CN115580648A - Data fusion system design method, system, electronic device and storage medium - Google Patents
Data fusion system design method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN115580648A CN115580648A CN202211211183.2A CN202211211183A CN115580648A CN 115580648 A CN115580648 A CN 115580648A CN 202211211183 A CN202211211183 A CN 202211211183A CN 115580648 A CN115580648 A CN 115580648A
- Authority
- CN
- China
- Prior art keywords
- target
- data
- service
- micro
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a design method and a system of a data fusion system, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: firstly, a target application scene and a target data source corresponding to the target application scene are determined, and a target micro service is determined from a plurality of candidate micro services according to the target data source and the target application scene. Then, a data distribution relation of each target micro service is established, and the execution sequence and the execution times of each target micro service are determined. And finally, generating a data fusion program pipeline consisting of the target microservices according to the execution sequence and the execution times. According to the method and the system, the target micro-services are organized and arranged in a way of integrated data fusion micro-service and through assembly line, and then data communication is carried out through data distribution service, so that a data fusion system with low coupling, high expandability and high maintainability is built, the maintenance is simple, and the operation and maintenance cost is low.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and a system for designing a data fusion system, an electronic device, and a storage medium.
Background
The data fusion is to organically synthesize the data of multiple modes by adopting a proper fusion algorithm, so that an evaluation result which is more accurate, more complete and more reliable than a single mode is obtained. The data fusion is widely applied to scenes such as multi-source image composition, robot and intelligent instrument systems, unmanned platforms, image analysis and understanding, target detection and tracking, automatic target identification and the like.
In the related art, a data fusion system designed by the existing method only considers a specific data fusion scene, and modularization is performed based on the specific scene. If the processing algorithm of a certain module needs to be modified, all modules need to be correspondingly adapted, and the whole process is restarted, so that the method is time-consuming, labor-consuming and low in flexibility.
Disclosure of Invention
Embodiments of the present invention provide a method and a system for designing a data fusion system, an electronic device, and a storage medium, which are used to solve or partially solve the problems in the background art.
In order to solve the technical problem, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a method for designing a data fusion system, where the method includes:
determining a target application scene and a target data source corresponding to the target application scene;
determining a target microservice from a plurality of candidate microservices according to a target data source and a target application scene, and/or generating the target microservice in real time according to the target data source;
establishing a data distribution relation of each target microservice;
determining the execution sequence and the execution times of each target micro service;
and generating a data fusion program pipeline consisting of the target micro-services according to the execution sequence and the execution times.
Optionally, the step of determining a target microservice from the plurality of candidate microservices according to the target data source and the target application scenario includes:
calculating a first score of the candidate micro-service according to the adaptation degree of the target data source and the candidate micro-service;
calculating a second score of the candidate micro-service according to the adaptation degree of the target application scene and the candidate micro-service;
calculating the total score of each candidate micro-service according to the first score and the second score of the candidate micro-service;
and determining the candidate micro-service with the total score larger than a preset threshold value as the target micro-service.
Optionally, the step of establishing a data distribution relationship for each target microservice includes:
establishing a data distribution relation between each target microservice and a target data source;
and establishing a data distribution relation between each target micro service.
Optionally, the step of determining a data distribution relationship between each target microservice and the target data source comprises:
the data distribution service issues a first data flow theme according to the target data source;
the target micro service subscribes to a second data flow theme to the data distribution service;
under the condition that the first data stream theme is matched with the second data stream theme, a first data channel between a target data source and a target micro service is established;
and sending the first real-time target data to the target micro-service through the first data channel.
Optionally, the step of determining a data distribution relationship between the target microservices comprises:
the first target micro service issues a third data flow theme to the data distribution service;
the second target microservice subscribes to a fourth data flow theme to the data distribution service;
under the condition that the third data stream theme is matched with the fourth data stream theme, a second data channel between the first target micro service and the second target micro service is established;
and the first target micro service sends second real-time target data to a second target micro service through a second data channel.
Optionally, the step of generating a data fusion program pipeline composed of target microservices according to the execution order and the execution times comprises:
determining the number of sub-data fusion program pipelines according to the execution times;
determining an API (application program interface) calling interface of each target microservice;
according to the execution sequence, sequencing the API call interfaces of each target micro service in order to generate a sub-data fusion program production line;
and combining the sub-data fusion program flow lines to generate a data fusion program flow line consisting of the target micro-services.
Optionally, the method further comprises:
determining a target micro service to be processed;
determining a control instruction corresponding to a target micro service to be processed, wherein the control instruction at least comprises: one of a stop instruction, an update instruction, and a replace instruction;
and processing the target micro service to be processed according to the control instruction.
A second aspect of the embodiments of the present invention provides a data fusion system, where the system includes:
the determining module is used for determining a target application scene and a target data source corresponding to the target application scene;
the selection module is used for determining a target microservice from a plurality of candidate microservices according to a target data source and a target application scene and/or generating the target microservice in real time according to the target data source;
the data relation determining module is used for establishing a data distribution relation of each target microservice;
the execution parameter determining module is used for determining the execution sequence and the execution times of each target micro service;
and the execution module is used for generating a data fusion program pipeline consisting of the target microservices according to the execution sequence and the execution times.
Optionally, the selection module comprises:
the first scoring submodule is used for calculating a first score of the candidate micro-service according to the adaptation degree of the target data source and the candidate micro-service;
the second scoring submodule is used for calculating a second score of the candidate micro-service according to the adaptation degree of the target application scene and the candidate micro-service;
the third scoring submodule is used for calculating the total score of each candidate micro service according to the first score and the second score of the candidate micro service;
and the screening submodule is used for determining the candidate micro-service with the total score larger than the preset threshold value as the target micro-service.
Optionally, the data relationship determining module includes:
the first data relation determining submodule is used for establishing a data distribution relation between each target microservice and a target data source;
and the second data relationship determining submodule is used for establishing a data distribution relationship between each target micro service.
Optionally, the first data relationship determination sub-module includes:
the issuing unit is used for issuing a first data stream theme by the data distribution service according to the target data source;
the subscription unit is used for subscribing the second data stream theme from the target micro service to the data distribution service;
the matching unit is used for establishing a first data channel between a target data source and a target micro service under the condition that a first data stream theme is matched with a second data stream theme;
and the transmission unit is used for transmitting the first real-time target data to the target micro service through the first data channel.
Optionally, the second data relationship determining sub-module includes:
the issuing unit is used for issuing a third data flow theme to the data distribution service by the first target micro service;
the subscription unit is used for subscribing a fourth data stream theme from the second target micro service to the data distribution service;
the matching unit is used for establishing a second data channel between the first target micro service and the second target micro service under the condition that the third data stream theme is matched with the fourth data stream theme;
and the transmission unit is used for sending second real-time target data to the second target micro service by the first target micro service through a second data channel.
Optionally, the execution module includes:
the quantity determining submodule is used for determining the quantity of the sub-data fusion program production lines according to the execution times;
the interface determining submodule is used for determining an API calling interface of each target microservice;
the sequencing submodule is used for sequencing the API call interface of each target microservice according to the execution sequence to generate a subdata fusion program pipeline;
and the combination submodule is used for combining the subdata fusion program production line to generate a data fusion program production line consisting of target microservices.
Optionally, the system further comprises a correction module, the correction module comprising:
the positioning sub-module is used for determining target micro-services to be processed;
the instruction determining submodule is used for determining a control instruction corresponding to the target micro service to be processed, wherein the control instruction at least comprises: one of a stop instruction, an update instruction, and a replace instruction;
and the execution submodule is used for processing the target micro-service to be processed according to the control instruction.
A third aspect of an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
the processor is configured to implement the method steps set forth in the first aspect of the embodiment of the present invention when executing the program stored in the memory.
A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the method as set forth in the first aspect of the embodiments of the present invention.
The embodiment of the invention has the following advantages: firstly, a target application scene and a target data source corresponding to the target application scene are determined, and a target micro service is determined from a plurality of candidate micro services according to the target data source and the target application scene. Then, a data distribution relation of each target micro service is established, and the execution sequence and the execution times of each target micro service are determined. And finally, generating a data fusion program pipeline consisting of the target microservices according to the execution sequence and the execution times. According to the application, the target micro-services are organized and arranged in a way of integrating and micro-services of the whole data, and then data communication is carried out through the data distribution service, so that a data integration system with low coupling, high expandability and high maintainability is built.
In some embodiments of the application, intermediate and final data of data fusion are distributed in a data distribution service mode, so that a large amount of data interaction among micro services is avoided, and the data transmission convenience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart illustrating steps of a method for designing a data fusion system according to an embodiment of the present invention;
FIG. 2 is a topology structure diagram of a data fusion system according to an embodiment of the present invention;
fig. 3 is a block diagram of a data fusion system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Interpretation of terms:
DDS: the Data Distribution Service is a new generation of distributed real-time communication middleware protocol, adopts a publish/subscribe system architecture, emphasizes taking Data as a center, provides rich QoS (quality of Service) strategies to ensure that the Data is distributed efficiently and flexibly in real time, and can meet various distributed real-time communication application requirements.
RPC (Remote Procedure Call), RPC (Remote Procedure Call) Remote Procedure Call protocol, a protocol that requests services from Remote computers over a network without knowledge of the underlying network technology. RPC it assumes the existence of certain protocols, such as TPC/UDP, etc., to carry information data between communication procedures.
Pipeline architecture, it can be understood that a thing that needs to be done repeatedly is cut into various phases, each phase being responsible for a separate unit.
In the design process of the traditional data fusion system, the design is generally targeted and comprises the steps of data preprocessing, data fusion, feature extraction, fusion attribute description and the like, and most of the steps are a streaming architecture or a modular architecture. Usually, precise customization is performed based on specific scenes and data sources, and if a certain module needs to be modified or upgraded, corresponding adaptation needs to be performed on other modules, and the operation of the whole system needs to be stopped and restarted, so that the modification can be effective. If the system is bulky, the expansibility and maintainability of the system are poor, which results in high time and labor costs for upgrading or modifying. Meanwhile, no matter the flow type structure or the modular structure is adopted, if the number of modules of the system is increased, the system becomes overstocked and redundant and is difficult to maintain.
As an example, when a conventional data fusion system is designed, it is usually based on a specific application scenario, for example, if the data fusion system is designed for depth of field recognition, it will make precise customization and module design according to a data source required for depth of field recognition, and finally form a modular structure, but if the data fusion system for depth of field recognition is used for image recognition or the data fusion system for depth of field recognition is upgraded, it is first necessary to modify some module or add a new module, then perform corresponding adaptation modification on other modules in the system, and at the same time, restart the system as a whole to enable the modification to be effective. Therefore, when the conventional data fusion system is designed, the conventional data fusion system is designed based on a single thread, and therefore, what steps are executed first, then, and finally, what steps are executed again need to be specified, so that the whole system is often stopped due to faults occurring in the system. And due to the adoption of a single-threaded mechanism, the timing between threads needs to be locked, and the throughput of the whole system is reduced.
Therefore, the method aims at the problems of high coupling, poor expandability and low maintainability of a data fusion system designed by the current traditional data fusion system design method. It will be appreciated that a data fusion system, which is known as a repetitive loop a-B-C-D, splits into sub-steps a-B-C-D, whereas the inventors contemplate that the process of fusing the multidimensional data is a repetitive loop. The inventor proposes the technical idea of the application: the method comprises the steps of firstly, integrating data fusion and micro-serving in an RPC mode, namely dividing an integrated data fusion system into a plurality of micro-service units which run independently, then organizing the connection relation of the specific micro-service units in a pipeline mode, and then carrying out data communication through a DDS, so that the data fusion system with low coupling, high expandability and high maintainability is obtained.
The data fusion system design method of the present application is described below with respect to establishing a data fusion system for performance testing of a civil unmanned aerial vehicle, and as shown in fig. 1, fig. 1 shows a flow diagram of the data fusion system design method of the present application.
S101: and determining a target application scene and a target data source corresponding to the target application scene.
In this embodiment, first, when designing a data fusion system, a user needs to determine what application scenario the data fusion system is specifically used in, and then what data needs to be acquired in the application scenario is what data, that is, what data acquired by a sensor needs to be used as a target data source.
As an example, if the data fusion system is designed for performance testing of a civil unmanned aerial vehicle, then the video data source collected by the camera of the civil unmanned aerial vehicle, the audio data collected by the audio collection device of the unmanned aerial vehicle, the power data of the unmanned aerial vehicle, the operating parameters of the unmanned aerial vehicle, etc. may be selected as the target data source. Therefore, the selection of the application scenario and the corresponding data source are adjusted according to the requirements of the user, which is not limited in the present application.
Since the formats of data collected by different sensors are different, when each data source is imported into the system, corresponding data preprocessing needs to be performed on the data source, so that the formats of different data sources input into the system are uniform.
In a feasible implementation, the target data source corresponding to the target application scenario may be real data acquired by an actual sensor, or may also be simulation data generated by a simulation engine, and the generation manner of the target data source is not limited in the present application.
S102: and determining the target micro service according to the target data source and the target application scene, and/or generating the target micro service in real time according to the target data source.
In the embodiment, common steps such as data preprocessing, data fusion, feature extraction, fusion attribute description and the like are performed in an RPC mode, different algorithm modules in the same step are generated into different micro services and are numbered, wherein the steps in the flow class can be numbered by using English letters, the steps in the algorithm class can be numbered by using numbers, and the steps can be adjusted according to the computing capacity of the system and the requirements of users in the process of generating the micro services.
For example, if the computing power of the system is strong, the steps of data preprocessing and feature extraction may be put together to jointly generate the microservice numbered a, and if the user needs to put together the data fusion and fusion attribute description, the steps of data fusion and fusion attribute may be put together to jointly generate the microservice numbered B.
In a possible implementation manner, in addition to selecting the best matching target microservice at the preset target microservice according to the target data source and the target application scenario, the target microservice may be established in real time according to the target data source.
As an example, if the preset target data source is the data source of the camera a, the corresponding micro service is B, and if the target data source is replaced by the data source of the camera C, the configuration parameters may be written according to the data source of the camera C, so as to generate the micro service D for preprocessing the image data of the data source of the camera C in real time.
And then selecting the required micro service, namely the target micro service, from the plurality of candidate micro services according to the target data source and the target application scene determined by the user. And the step of determining the target microservice from the candidate microservices may comprise:
s102-1: and calculating a first score of the candidate micro-service according to the adaptation degree of the target data source and the candidate micro-service.
In this embodiment, the degree of adaptation between the target data source and the candidate micro service may be determined according to the degree of adaptation between the data source commonly used by the candidate micro service and the target data source, and the higher the degree of adaptation is, the higher the probability that the candidate micro service is selected as the target micro service is.
As an example, the target data source includes four data sources, which are a, b, c, and d, respectively, and the common data sources of the candidate micro service with the number a are a, b, c, and e, then it may be determined that the degree of adaptation between the target data source and the candidate micro service with the number a is seventy-five percent, and the corresponding first score is 75. It should be noted that, the calculation manner of the adaptation degree of the target data source and the candidate microservice may be adjusted according to different requirements of the user, which is not limited in the present application.
S102-2: and calculating a second score of the candidate micro-service according to the adaptation degree of the target application scene and the candidate micro-service.
In this embodiment, the adaptation degree of the target application scenario and the candidate micro service may be determined according to a historical probability that the candidate micro service is selected as the target application scenario or a target micro service of an application scenario close to the target application scenario, where the higher the probability is, the higher the adaptation degree is, the higher the probability is that the candidate micro service is selected as the target micro service.
As an example, if the application scenario is a performance test of a civil drone, and the historical probability of the selection of the application scenario with the number a of the civil drone under the performance test is fifty percent, the corresponding second score is 50. It should be noted that the calculation manner of the adaptation degree between the target application scenario and the candidate microservice may be adjusted according to different requirements of the user, which is not limited in the present application.
S102-3: and calculating the total score of each candidate micro service according to the first score and the second score of the candidate micro service.
In this embodiment, after the first score and the second score of each candidate micro-service are obtained, the total score of each candidate micro-service is calculated according to the corresponding preset scale factor of each score, and the scale factor of each score can be set according to the requirement of the user.
As an example, if the first score of the user is 75 scores, the scale factor corresponding to the first score is 0.5, the second score is 50 scores, and the scale factor corresponding to the second score is 0.5, then each score is multiplied by the corresponding scale factor, and finally the multiplied results are accumulated, the total score of the candidate micro service with the number a can be calculated to be 62.5 scores. If the user considers that the adaptation degree of the target data source and the candidate micro-service is more important, the scale factor of the first score is higher than that of the second score, and if the adaptation degree of the user application scene and the candidate micro-service is more important, the scale factor of the first score is lower than that of the second score. It should be noted that, for setting the specific value of the scaling factor, the setting and the adjustment may be performed according to the requirement and the actual situation of the user, which is not limited in the present invention.
S102-4: and determining the candidate micro-service with the total score larger than a preset threshold value as the target micro-service.
In this embodiment, after calculating the total score of each candidate micro-service, comparing the total score with a preset score threshold, and determining the candidate micro-service larger than the preset threshold as the target micro-service, the determination of the corresponding score threshold may be determined according to the requirement of the user on the system, if the user wants the system to have high precision, the score threshold may be set to be very high, for example, 100 points, if the user wants the system to have compatibility and adaptability, the score threshold may be set to be higher, for example, 85 points.
As an example, if the data fusion system is designed for performance testing of a civil drone, and there are 20 candidate micro services, the candidate micro services with a total score greater than a preset threshold are the micro service number a, the micro service number 1, the micro service number B, the micro service number 2, the micro service number C, and the micro service number 3. The target microservice is determined to be microservice number a (for data pre-processing), with microservice number 1 (algorithm for data pre-processing), with microservice number B (for data fusion), with microservice number 2 (data fusion algorithm), with microservice number C (for feature extraction), with microservice number 3 (feature extraction algorithm).
S103: and establishing a data distribution relation of each target micro service.
In this embodiment, after determining the target microservice, it is further necessary to determine a data distribution relationship of the target microservice, that is, for any one target microservice, it is necessary to determine from which places it acquires data and to which places it sends data, and the data distribution relationship specifically includes two levels of data distribution relationships, and the specific steps include:
s103-1: and establishing a data distribution relation between each target micro service and the target data source.
In this embodiment, the first-level data distribution relationship refers to a data distribution relationship between each target microservice and an external target data source, that is, it needs to be determined which target microservices are input after the external data source is input, and the specific steps may include:
s103-1-1: the data distribution service publishes a first data stream topic according to a target data source.
In this embodiment, it can be understood that the data distribution service first needs to determine the data stream topic that is distributed by the data distribution service according to the connection condition of the target data source, and taking a television program as an example, if an operator owns video playback rights of multiple television stations such as a beijing television station and a shandong television station, the operator may distribute to represent a television channel owned by the operator and a television program corresponding to the television channel.
As an example, if the data distribution service receives four types of data sources, namely, a type a, b type c and d type d, where the type a data source represents image data and the type b data source represents audio data, the first data stream topic issued by the data distribution service includes an image data stream topic and an audio data stream topic to characterize that the data distribution service establishes an image data stream and an audio data stream.
S103-1-2: the target microservice subscribes to a second data stream topic to the data distribution service.
In this embodiment, the target microserver cannot acquire which specific data stream topics are in the data distribution service, and can only know what data is needed by the target microserver, so the target microserver subscribes to the second data stream topic according to the needed data, and the data corresponding to the second data stream topic is the data needed by the target microserver, and the description is continued with the television program.
For example, for the target microservice with the number a, one, zero, or multiple data that needs to be acquired from the target data source may be acquired, and if the required data is image data, the target microservice with the number a subscribes to the image data stream topic to the data distribution service.
In a possible implementation manner, when the target microservice subscribes to the data distribution service, the subscription to the second data stream topic can be performed not only according to the content, but also based on time.
S103-1-3: and under the condition that the first data stream theme is matched with the second data stream theme, establishing a first data channel of the target data source and the target micro service.
In this embodiment, the data distribution service checks whether each target microservice sends a subscription request, and if the target microservice sends a subscription request and the second data stream topic matches the first data stream topic published by the data distribution service, then a first data channel between the target data source and the target microservice is established.
S103-1-4: and sending first real-time target data to the target micro service through a first data channel.
In this embodiment, the first real-time target data refers to real-time data transmitted between a target data source and a target micro service, and when the target micro service is started, the data distribution service sends real-time target data generated by the target data source to the target micro service through the first data channel, and stores the real-time target data in a local storage space of the target micro service.
S103-2: and establishing a data distribution relation between each target microservice.
In this embodiment, in addition to the first-layer data distribution relationship between each target microservice and the external target data source, the data distribution relationship between any one target microservice and another target microservice is also to be determined, and the specific steps may be:
s103-2-1: the first target microservice publishes a third data stream topic to a data distribution service.
In this embodiment, any one of the target microservices may be a publisher or a subscriber, and when the subscriber wants to publish a message, the subscriber creates a publisher if the target microservice is to publish the message; if the program is to receive a message, a subscriber is created, and the subscriber and publisher are associated by topic. Thus, the first target microservice is a publisher.
As an example, for a target microservice with the number a, after processing real-time image data generated by a target data source, and after the processing is completed, first storing the processed real-time image data in a local storage of the target microservice with the number a, then issuing a third data stream topic to a data distribution service, where the third data stream topic is a data stream topic corresponding to data processed by the target microservice, that is, a data stream topic corresponding to the processed real-time image data, and the third data stream topic is: and (4) processing the image data stream theme after the target microservice with the number of A.
S103-2-2: the second target microservice subscribes to a fourth data stream topic to the data distribution service.
In this embodiment, the second target microservice is a subscriber, and sends a subscription request to the distribution service according to a specific type of data required by the second target microservice, that is, the data type corresponding to the fourth data stream topic is a data type required by the target microservice as the subscriber.
As an example, for the target microservice with number 1, which is used to further calculate and optimize the data generated by the target microservice with number a, the required data is the image data processed by the target microservice with number a, so the fourth data stream topic: and (4) processing the image data stream theme after the target microservice with the number of A.
S103-2-3: and under the condition that the third data stream theme is matched with the fourth data stream theme, establishing a second data channel between the first target micro service and the second target micro service.
In this embodiment, the data distribution service checks whether the third data stream topic and the fourth data stream topic received by the data distribution service are matched, if not, the subscriber does not receive the data, and if so, a second data channel between the first target micro service and the second target micro service is established, where the second data channel is used to transmit data between the first target micro service as the publisher and the second target micro service as the subscriber. The second data channel established between different target microservices is different.
As an example, since the third data stream topic published by the target microservice with the number a matches the fourth data stream topic subscribed by the target microservice with the number 1, a second data channel for transmitting data between the target microservice with the number a and the target microservice with the number 1 is established.
S103-2-4: and the first target micro service sends second real-time target data to a second target micro service through a second data channel.
In this embodiment, the second real-time target data refers to real-time data between different target micro services, and when the second target micro service is started, the first target micro service sends the second real-time target data to the second target micro service through the second data channel.
As an example, when the target microservice with the number of 1 is started, the target microservice with the number of a sends the image data processed by the target microservice with the number of a, that is, the second real-time target data, to the target microservice with the number of 1 through the second data channel. And then, the target micro service with the number of 1 performs corresponding calculation and processing on the image data processed by the target micro service with the number of A according to a built-in algorithm, so as to obtain the image data processed by the target micro service with the number of 1, stores the image data in a local storage of the target micro service with the number of 1, then releases the image data, and sends the image data to other target micro services subscribed with the target micro service with the number of 1.
S104: the execution order and execution times of each target microservice are determined.
In this embodiment, after the data distribution relationship of each target microservice is established, when the target data fusion system is established, the execution order and the execution times of each target microservice need to be determined. So that when the pipeline structure is built, the building can be rapidly completed.
For example, if the identified target microservices are the microservices numbered a, the microservices numbered 1, the microservices numbered B, the microservices numbered 2, the microservices numbered C, and the microservices numbered 3, the microservices numbered B, the microservices numbered 2, and the microservices numbered 3. According to the requirement of a user, the execution times of the micro service with the number A is twice, the execution order of the two times is the first bit and the third bit respectively, the micro service with the number 1 also needs to be executed twice, the execution order of the two times is the second bit and the fourth bit respectively, the execution times of the micro service with the number B is once, the execution order is the fifth bit, the execution times of the micro service with the number 2 is once, the execution order is the sixth bit, the execution times of the micro service with the number C is once, the execution order is the seventh bit, the execution times of the micro service with the number 3 is once, and the execution order is the eighth bit.
S105: and generating a data fusion program pipeline consisting of the target micro-services according to the execution sequence and the execution times.
In this embodiment, after the execution order and the execution times of each target micro-service are determined, different target micro-services are required to be added to the pipeline as different processes, so as to generate a data fusion program pipeline, thereby obtaining a data fusion system with low coupling, high expandability and high maintainability.
The method comprises the following specific steps:
s105-1: determining the number of sub-data fusion program pipelines according to the execution times;
s105-2: determining an API (application program interface) calling interface of each target micro service;
s105-3: according to the execution sequence, sequencing the API call interfaces of each target microservice in a bit order to generate a sub-data fusion program pipeline;
s105-4: and combining the sub-data fusion program flow lines to generate a data fusion program flow line consisting of the target micro-services.
In the embodiment of steps S105-1 to S105-4, different sub-pipelines can be obtained due to the combination of different target microservices, and the different sub-pipelines can be combined together to obtain the data fusion program pipeline.
As an example, if the execution times of the micro service with the number a is two, the execution orders of the two times are the first bit and the third bit respectively, and the micro service with the number 1 needs to be executed twice, the execution orders of the two times are the second bit and the fourth bit respectively, the execution times of the micro service with the number B is one, the execution order is the fifth bit, the execution times of the micro service with the number 2 is one, the execution order is the sixth bit, the execution times of the micro service with the number C is one, the execution order is the seventh bit, the execution times of the micro service with the number 3 is one, and the execution order is the eighth bit.
Because the maximum value of the execution times is two, it is determined that the number of the sub-data fusion program pipelines is two, and an API call Interface (Application Programming Interface Application program Interface) is provided for an external access point using the target micro-service, that is, the target micro-service can be called through the API call Interface. Since the number of the sub-data fusion program pipelines is two, the order of one sub-data fusion program pipeline may be: the order of microservice number a-microservice number 1-microservice number B-microservice number 2-microservice number C-microservice number 3, and another sub-data fusion program pipeline may be: microservice number a-microservice number 1. And then, distributing each API calling interface according to the sequence to generate two subdata fusion program pipelines, and then integrating the two subdata fusion program pipelines to obtain a total data fusion program pipeline. And then, when the total data fusion program pipeline is executed, calling the micro-service through the API according to the corresponding sequence, so as to distribute intermediate data and final data of data fusion in a data distribution service mode. Because different threads can be responsible for different sub-data fusion program pipelines, concurrency can be realized to a certain extent, and throughput is improved.
In one possible embodiment, the method further comprises:
determining a target micro service to be processed;
determining a control instruction corresponding to a target micro service to be processed, wherein the control instruction at least comprises: one of a stop instruction, an update instruction, and a replace instruction;
and processing the target micro service to be processed according to the control instruction.
In the embodiment, when a fault occurs in the system or an upgrade is needed, the conventional data fusion system needs to stop the system and make adaptive modification on the whole system, so that the system is very cumbersome and takes a long maintenance time, and the system cannot operate within the maintenance time. In the application, when a user needs to process a certain target microservice, the processing here may be to replace an algorithm in the target microservice and stop the target microservice, or to add a target microservice after the target microservice. Therefore, it is first necessary to determine which target microservice to be processed is, which can be determined according to the number of the target microservice to be processed, and then determine the control instruction to be made to the target microservice, if the target microservice needs to be processed: and in order to replace the algorithm in the target micro service, the corresponding control instruction is a replacement instruction, then the algorithm in the target micro service is replaced according to the replacement instruction, and the control instruction also comprises the processed data distribution relation of the target micro service.
In the application, the normal operation of the system cannot be influenced by processing the target micro-service, namely, the system cannot stop operating when one target micro-service is stopped. Therefore, the operation and maintenance cost and the operation and maintenance time of the system are greatly reduced, and the system has high expandability and high maintainability. The target micro service can be modified without influencing the target micro service, the whole system does not need to be restarted, and the target micro service can be modified only by simple replacement.
As shown in fig. 2, fig. 2 is a system architecture of a data fusion system generated based on the content of the first aspect of the embodiment of the present application, and the data fusion system architecture of the present application may be divided into three parts, namely, a data exchange module, a pipeline combination module, and a division module;
the data exchange module is used for issuing a first data stream theme according to the target data source; and a third data flow topic for target micro-service publishing and a fourth data flow topic for subscription;
the dividing module is used for dividing common steps and algorithms for data fusion into a plurality of micro services based on RPC;
and the pipeline combination module is used for generating a data fusion program pipeline consisting of the target micro services according to the execution sequence and the execution times.
An embodiment of the present invention further provides a data fusion system, and referring to fig. 3, a functional block diagram of a first aspect of an embodiment of the data fusion system according to the present invention is shown, where the system includes:
a determining module 301, configured to determine a target application scenario and a target data source corresponding to the target application scenario;
a selecting module 302, configured to determine a target microservice from multiple candidate microservices according to a target data source and a target application scenario, and/or generate a target microservice in real time according to the target data source;
a data relationship determining module 303, configured to establish a data distribution relationship of each target microservice;
an execution parameter determination module 304, configured to determine an execution order and an execution number of each target microservice;
and the execution module 305 is used for generating a data fusion program pipeline consisting of the target microservices according to the execution sequence and the execution times.
In one possible embodiment, the selection module comprises:
the first scoring submodule is used for calculating a first score of the candidate micro-service according to the adaptation degree of the target data source and the candidate micro-service;
the second scoring submodule is used for calculating a second score of the candidate micro-service according to the adaptation degree of the target application scene and the candidate micro-service;
the third scoring submodule is used for calculating the total score of each candidate micro-service according to the first score and the second score of the candidate micro-service;
and the screening submodule is used for determining the candidate micro-services with the total score larger than a preset threshold value as the target micro-services.
In one possible implementation, the data relationship determination module includes:
the first data relation determining submodule is used for establishing a data distribution relation between each target microservice and a target data source;
and the second data relation determining submodule is used for establishing a data distribution relation between each target microservice.
In one possible implementation, the first data relationship determination submodule includes:
the issuing unit is used for issuing a first data stream theme by the data distribution service according to the target data source;
the subscription unit is used for subscribing the second data flow theme from the target micro service to the data distribution service;
the matching unit is used for establishing a first data channel between a target data source and a target micro service under the condition that a first data stream theme is matched with a second data stream theme;
and the transmission unit is used for transmitting the first real-time target data to the target micro-service through the first data channel.
In one possible embodiment, the second data relationship determination submodule includes:
the issuing unit is used for issuing a third data flow theme to the data distribution service by the first target micro service;
the subscription unit is used for subscribing a fourth data stream theme from the second target micro service to the data distribution service;
the matching unit is used for establishing a second data channel between the first target micro service and the second target micro service under the condition that the third data stream theme is matched with the fourth data stream theme;
and the transmission unit is used for sending the second real-time target data to the second target microservice by the first target microservice through the second data channel.
In one possible implementation, the execution module includes:
the quantity determining submodule is used for determining the quantity of the sub-data fusion program production line according to the execution times;
the interface determining submodule is used for determining an API calling interface of each target microservice;
the sequencing submodule is used for sequencing the API call interfaces of each target microservice in order according to the execution sequence to generate a subdata fusion program pipeline;
and the combination submodule is used for combining the subdata fusion program production line to generate a data fusion program production line consisting of target microservices.
In one possible embodiment, the system further comprises a correction module, the correction module comprising:
the positioning submodule is used for determining target micro-service to be processed;
the instruction determining submodule is used for determining a control instruction corresponding to the target micro service to be processed, wherein the control instruction at least comprises: one of a stop instruction, an update instruction, and a replace instruction;
and the execution submodule is used for processing the target micro-service to be processed according to the control instruction.
Based on the same inventive concept, an embodiment of the present application further provides an electronic device, including:
at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data fusion system design method of the embodiment of the application.
In addition, to achieve the above object, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the data fusion system design method of the embodiment of the present application.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. "and/or" means that either or both of them can be selected. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrases "comprising one of \ 8230; \8230;" does not exclude the presence of additional like elements in a process, method, article, or terminal device that comprises the element.
The above detailed description is given to a method, a system, an electronic device and a storage medium for designing a data fusion system, which are provided by the present invention, and a specific example is applied in the present document to explain the principle and the implementation of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method for designing a data fusion system, the method comprising:
determining a target application scene and a target data source corresponding to the target application scene;
determining a target micro service from a plurality of candidate micro services according to the target data source and the target application scene, and/or generating the target micro service in real time according to the target data source;
establishing a data distribution relation of each target microservice;
determining the execution sequence and the execution times of each target microservice;
and generating a data fusion program pipeline consisting of the target microservices according to the execution sequence and the execution times.
2. The method of claim 1, wherein the step of determining a target microservice from a plurality of candidate microservices based on the target data source and the target application scenario comprises:
calculating a first score of the candidate micro-service according to the adaptation degree of the target data source and the candidate micro-service;
calculating a second score of the candidate micro-service according to the adaptation degree of the target application scene and the candidate micro-service;
calculating the total score of each candidate micro-service according to the first score and the second score of the candidate micro-service;
and determining the candidate micro-service with the total score larger than a preset threshold value as a target micro-service.
3. The method of claim 1, wherein the step of establishing a data distribution relationship for each of the target microservices comprises:
establishing a data distribution relation between each target micro service and the target data source;
and establishing a data distribution relation between each target microservice.
4. The data fusion system design method of claim 3, wherein the step of determining the data distribution relationship between each of the target microservices and the target data sources comprises:
the data distribution service issues a first data flow theme according to the target data source;
the target micro service subscribes to a second data stream topic from the data distribution service;
establishing a first data channel between the target data source and the target microservice if the first data stream topic and the second data stream topic match;
and sending first real-time target data to the target micro service through the first data channel.
5. The data fusion system design method of claim 3, wherein the step of determining the data distribution relationship between the target microservices comprises:
the first target micro service issues a third data flow theme to the data distribution service;
the second target micro service subscribes to a fourth data flow theme to the data distribution service;
establishing a second data channel between the first target microservice and the second target microservice if the third data stream topic and the fourth data stream topic match;
and the first target micro service sends second real-time target data to the second target micro service through the second data channel.
6. The method of designing a data fusion system according to claim 1, wherein the step of generating a data fusion program pipeline composed of the target microservices according to the execution order and the execution times comprises:
determining the number of sub-data fusion program pipelines according to the execution times;
determining an API (application program interface) calling interface of each target microservice;
according to the execution sequence, carrying out bit-order sequencing on an API call interface of each target micro service to generate the sub data fusion program pipeline;
and combining the sub-data fusion program assembly lines to generate a data fusion program assembly line consisting of the target micro-services.
7. The method of designing a data fusion system according to claim 1, further comprising:
determining a target micro service to be processed;
determining a control instruction corresponding to the target micro service to be processed, wherein the control instruction at least comprises: one of a stop instruction, an update instruction, and a replace instruction;
and processing the target micro service to be processed according to the control instruction.
8. A data fusion system, the system comprising:
the determining module is used for determining a target application scene and a target data source corresponding to the target application scene;
the selection module is used for determining a target micro service from a plurality of candidate micro services according to the target data source and the target application scene and/or generating the target micro service in real time according to the target data source;
the data relation determining module is used for establishing a data distribution relation of each target microservice;
the execution parameter determining module is used for determining the execution sequence and the execution times of each target micro service;
and the execution module is used for generating a data fusion program pipeline consisting of the target micro-services according to the execution sequence and the execution times.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method of designing a data fusion system according to any one of claims 1 to 7 when executing a program stored in a memory.
10. A storage medium storing computer instructions which, when executed by a processor, implement the data fusion system design method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211211183.2A CN115580648B (en) | 2022-09-30 | 2022-09-30 | Data fusion system design method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211211183.2A CN115580648B (en) | 2022-09-30 | 2022-09-30 | Data fusion system design method, system, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115580648A true CN115580648A (en) | 2023-01-06 |
| CN115580648B CN115580648B (en) | 2023-07-25 |
Family
ID=84583423
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211211183.2A Active CN115580648B (en) | 2022-09-30 | 2022-09-30 | Data fusion system design method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115580648B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190130004A1 (en) * | 2017-10-27 | 2019-05-02 | Streamsimple, Inc. | Streaming Microservices for Stream Processing Applications |
| CN111522535A (en) * | 2020-03-26 | 2020-08-11 | 杭州数跑科技有限公司 | Data source aggregation method and device, storage medium and computer equipment |
| CN113159091A (en) * | 2021-01-20 | 2021-07-23 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113691635A (en) * | 2021-09-01 | 2021-11-23 | 中电金信软件有限公司 | Method and device for calling micro service, electronic equipment and readable storage medium |
| CN114676179A (en) * | 2022-03-31 | 2022-06-28 | 北京科技大学 | Multi-source heterogeneous data interaction and fusion method and system for shield scene |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035436A (en) * | 2020-08-31 | 2020-12-04 | 国网天津市电力公司 | Energy big data application center system |
| CN114253531A (en) * | 2021-11-11 | 2022-03-29 | 广州云擎互动信息技术有限公司 | Micro-service publishing system |
| CN114895879B (en) * | 2022-07-14 | 2022-11-04 | 四川航天信息有限公司 | Management system design scheme determining method, device, equipment and storage medium |
-
2022
- 2022-09-30 CN CN202211211183.2A patent/CN115580648B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190130004A1 (en) * | 2017-10-27 | 2019-05-02 | Streamsimple, Inc. | Streaming Microservices for Stream Processing Applications |
| CN111522535A (en) * | 2020-03-26 | 2020-08-11 | 杭州数跑科技有限公司 | Data source aggregation method and device, storage medium and computer equipment |
| CN113159091A (en) * | 2021-01-20 | 2021-07-23 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113691635A (en) * | 2021-09-01 | 2021-11-23 | 中电金信软件有限公司 | Method and device for calling micro service, electronic equipment and readable storage medium |
| CN114676179A (en) * | 2022-03-31 | 2022-06-28 | 北京科技大学 | Multi-source heterogeneous data interaction and fusion method and system for shield scene |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115580648B (en) | 2023-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2452489B1 (en) | Systems and methods for managing video data | |
| CN101207699B (en) | Method for upgrading set-top box | |
| US20020032754A1 (en) | Method and apparatus for profiling in a distributed application environment | |
| CN101151901B (en) | Method of sending a command to a digital data flow server and apparatus used to implement said method | |
| CN103312684B (en) | For method and the information processing system of delivery of content in optimized network | |
| CN103534988A (en) | Publish and subscribe messaging method and apparatus | |
| CN110704749B (en) | Recommendation engine customization system, recommendation method, recommendation system and electronic equipment | |
| CN105828046A (en) | Data flow transmission method and device | |
| CN112073212B (en) | Parameter configuration method, device, terminal equipment and storage medium | |
| CN115580648A (en) | Data fusion system design method, system, electronic device and storage medium | |
| CN105812839A (en) | Video stream data acquisition method, page data transmission method, system and network server | |
| US20230104788A1 (en) | Method of distributing files through a content delivery network based also on artificial intelligence algorithms, telematic system and servers that allow to implement it | |
| CN111428128B (en) | Method for generating pluggable recommendation system, service recommendation method, device and equipment | |
| KR20220056836A (en) | Method and apparatus for determining voice response rate, electronic device, computer readable storage medium and computer program | |
| CN112738153B (en) | Gateway selection method, system, device, server and medium in service system | |
| EP3040891A1 (en) | Method for providing information to determine a graph associated with a data item | |
| US20080256178A1 (en) | Method and Apparatus for Providing Software by Functional Units in a Software Streaming System | |
| CN116980555B (en) | Input signal source acquisition method and system | |
| JP7181663B2 (en) | Communication device, communication program, and distributed processing method | |
| EP3767922B1 (en) | Method of channel mapping in an industrial process control system | |
| CN116192662A (en) | Service behavior prediction and deterministic network association model based and recommendation method | |
| CN115981805A (en) | Service control method, device, electronic equipment and medium based on scheduling system | |
| CN117471959A (en) | Multi-domain fusion data acquisition method, system and medium based on centralized architecture | |
| CN113065029A (en) | Data stream processing method and device | |
| CN114172873A (en) | Resolution adjustment method, resolution adjustment device, server and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |