CN111984648B - Data initialization method and system under micro-service architecture - Google Patents
Data initialization method and system under micro-service architecture Download PDFInfo
- Publication number
- CN111984648B CN111984648B CN202010839132.9A CN202010839132A CN111984648B CN 111984648 B CN111984648 B CN 111984648B CN 202010839132 A CN202010839132 A CN 202010839132A CN 111984648 B CN111984648 B CN 111984648B
- Authority
- CN
- China
- Prior art keywords
- data
- service
- initialization
- processing
- micro
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011423 initialization method Methods 0.000 title claims abstract description 7
- 238000012545 processing Methods 0.000 claims abstract description 48
- 238000000034 method Methods 0.000 claims abstract description 15
- 238000011161 development Methods 0.000 claims abstract description 10
- 238000007726 management method Methods 0.000 claims description 27
- 238000004891 communication Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 241001632422 Radiola linoides Species 0.000 claims 1
- 238000013515 script Methods 0.000 description 5
- 241001632427 Radiola Species 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/81—Indexing, e.g. XML tags; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data initialization method and system under a micro-service architecture, and relates to the field of software technology development. The system of the present invention includes a management service, an SDK, a service providing data or processing data; the management service is responsible for collecting data to be initialized with the SDK, analyzing the dependency relationship of the initialized data, issuing a data initialization instruction meeting the dependency to the SDK, and storing the instruction execution result; the SDK is responsible for collecting and reporting data to be initialized to a management service at service start-up under other micro-service systems that provide or process the data. The invention can finish the initialization of the initialization data with complex interdependence relationship under a micro-service system, can ensure that any service starting and dependency sequence can ensure the successful initialization of the final data, and ensures that most services meet the requirement of initialization data; the decoupling of service direct initialization data is satisfied.
Description
Technical Field
The invention belongs to the field of software technology development, and particularly relates to a data initialization system under a micro-service architecture and a data initialization method under the micro-service architecture.
Background
The invention is mainly based on the background of two aspects, namely the popularity of the software architecture in the form of micro services nowadays; secondly, most software architectures have the need for initializing the data in the database.
Currently, most software architectures have a requirement for initializing data in a database, and some preset data needs to be written into the database according to a pre-designed structure, such as default roles, default users, menu structures, and other service or system data. On the other hand, since the micro services are different from the conventional software architecture, a plurality of databases are used for storing data, and one of the advantages of the micro services is that the services are decoupled from each other, and the decoupling is hoped to be decoupled in terms of data initialization.
At present, no popular solution for initializing a database under a micro-service system exists, one of the common solutions is that each service writes an SQL script to complete the task when the SQL script is installed, and the disadvantage of the method is that the initialization of cross-service data dependence is very troublesome due to the uncertainty of SQL execution sequence in the deployment process of the micro-service; another way is that the entire system writes one or more unified SQL scripts to be executed sequentially at installation time, which has the disadvantage that this is not decoupled in the writing of database initialization scripts and that all service development teams maintain these scripts together, increasing communication and development costs. Therefore, it is important to provide a method and system for initializing data in micro-service architecture to solve the above problems.
Disclosure of Invention
The invention provides a data initialization method and a system under a micro-service architecture, which can efficiently finish initialization under the scenario of inter-dependence of cross-service data complexity, and written data can be distributed in respective services without writing and executing together. In addition, the scheme also provides a general initial data interface to be compatible with various data sources.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention relates to a data initialization system under a micro-service architecture, which comprises:
manage services, SDKs, provide data or process the service composition under other micro-service systems of the data;
the management service is the core of the system and is responsible for collecting data to be initialized with the SDK, analyzing the dependency relationship of the initialized data, issuing a data initialization instruction meeting the dependency to the SDK, and storing the instruction execution result;
the SDK is a bridge for communication between the management service and the service under the other micro-service system for providing data or processing data, is integrated by the service under the other micro-service system for providing data or processing data during development and has the access capability of a database or a data storage position, is responsible for collecting and reporting data needing to be initialized to the management service when the service under the other micro-service system for providing data or processing data is started, and executes and reports execution results when an initialization instruction of the management server arrives;
services under other micro-service architectures that provide data or process data need to integrate the SDK at development time and, depending on business needs, collect and convert data from any accessible data source into a format recognizable by the management service and report to the management service through the SDK.
The initialization method comprises initializing data providing service, managing service and initializing data processing service;
the initialization data providing service and the initialization data processing service are both services under other micro-service systems embedded with the provided data or the processed data of the SDK, and are configured to be only initial data providing service, or only initial data processing service or both initial data providing service and initial data processing service according to business requirements;
the initialization data adopts seed data to provide service, and comprises the following steps:
p1, manually added seed data is provided in the form of XML (Extensible Markup Language );
p2, automatically generating data such as interface authority data through a service self code;
p3, an extensible data providing form, an interface in the SDK is realized by the service, and return data is executed in the interface;
the management service is executed by monitoring the starting time of a new service instance, and comprises the following steps:
s1, after a seed data providing service instance is started, all seed data are pulled from the seed data providing service;
s2, splitting the seed data according to the source service, generating HASH check codes according to the content of the split data, using the source service, and using the HASH check codes as unique check, and if the check is inconsistent with the old version, storing the data in a database, wherein the table name for storing the data is as follows: the service_settings data has fields:
0) Data UUID id;
1) Source service_service;
2) MD5 code checksum;
3) Splitting the data data_json;
s3, splitting the split seed data according to records, and updating the split seed data into a data_external table, wherein the table structure is as follows:
0) Recording UUID id;
1) Associating the data ID data_id;
2) External reference ID external_id;
3) Target_service;
3) Actual data data_json;
4) The processing state process_status waits for dependence, processing is completed, and deletion is waited for;
s4, performing N times of data processing until the processed data is marked as 0, wherein the steps are as follows:
1) Inquiring unprocessed records, and grouping according to target services;
2) Processing each recorded reference, if the reference is satisfied, replacing the reference as real data, and if the reference is not satisfied, skipping first, wherein the replacement of each reference marks the processed data as +1;
3) Sending a request according to the group of the target service, enabling each service to process the data of the unprocessed record, writing back to an actual data field and maintaining unsatisfied references, if the references are satisfied, completing the processing, otherwise waiting for dependence;
the initializing data processing service includes the steps of:
providing a seed data processing interface, inserting data, updating data or deleting data and returning a processing result.
Compared with the prior art, the invention has the following beneficial effects:
the system can finish the initialization of the initialization data with complex interdependence relationship under a micro-service system, and can ensure that the final data is successfully initialized according to any service starting sequence and dependency sequence. And may show all data dependent and initialized execution status according to the management service record. The system provides a flexible SDK, which can ensure that the needs of most services on initialization data are met. The service data is collected and reported by each micro-service, and can meet the decoupling of the service direct initialization data.
Of course, it is not necessary for any one product to practice the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data initialization system under a micro-service architecture according to the present invention;
FIG. 2 is a flowchart showing steps of an initial data providing service according to the present invention;
FIG. 3 is a flowchart illustrating steps of a management service according to the present invention;
FIG. 4 is a flowchart illustrating steps of an initial data processing service according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a data initialization system under a micro service architecture of the present invention includes:
management services, SDKs (Software Development Kit, software development kits), services under other micro-services systems that provide data or process data (user services and order services are two examples in the figure);
the management service is the core of the system and is responsible for collecting data to be initialized with the SDK, analyzing the dependency relationship of the initialized data, issuing a data initialization instruction meeting the dependency to the SDK, and storing the instruction execution result;
the SDK is a bridge for communication between the management service and the service under the other micro-service system for providing data or processing data, is integrated by the service under the other micro-service system for providing data or processing data during development and has the access capability of a database or a data storage position, is responsible for collecting and reporting data needing to be initialized to the management service when the service under the other micro-service system for providing data or processing data is started, and executes and reports an execution result when an initialization instruction of the management server arrives;
services under other micro-service architectures that provide data or process data need to integrate the SDK at development time and collect and convert data from any accessible data source into a format recognizable by the management service and report to the management service through the SDK according to business needs.
As shown in fig. 2-4, a method for initializing data under a micro-service architecture includes initializing a data providing service, a management service, and an initializing data processing service;
the initialization data providing service and the initialization data processing service are both services under other micro-service systems embedded with the providing data or processing data of the SDK (SoftwareDevelopment Kit ), and are configured to be only initial data providing services, or only initial data processing services, or both initial data providing services and initial data processing services according to business requirements;
the initialization data adopts seed data to provide service, comprising the following steps:
p1, manually added seed data is provided in the form of XML (Extensible Markup Language );
p2, automatically generating data such as interface authority data through a service self code;
p3, an extensible data providing form, an interface in the SDK is realized by the service, and return data is executed in the interface;
the management service is executed by monitoring the starting time of a new service instance, and comprises the following steps:
s1, after a seed data providing service instance is started, all seed data are pulled from the seed data providing service;
s2, splitting the seed data according to the source service, generating HASH (HASH) check codes according to the content of the split data, using the HASH check codes as unique check for the source service, and storing the HASH check codes into a database if the check is inconsistent with the old version, wherein the table name for storing the data is as follows: the service_settings data has fields:
0) Data UUID id;
1) Source service_service;
2) MD5 code checksum;
3) Splitting the data data_json;
s3, splitting the split seed data according to records, and updating the split seed data into a data_external table, wherein the table structure is as follows:
0) Recording UUID id;
1) Associating the data ID data_id;
2) External reference ID external_id;
3) Target_service;
3) Actual data data_json;
4) The processing state process_status waits for dependence, processing is completed, and deletion is waited for;
s4, performing N times of data processing until the processed data is marked as 0, wherein the steps are as follows:
1) Inquiring unprocessed records, and grouping according to target services;
2) Processing each recorded reference, if the reference is satisfied, replacing the reference as real data, and if the reference is not satisfied, skipping first, wherein the replacement of each reference marks the processed data as +1;
3) Having each service process the data according to the packet transmission request of the target service, and writing back to the actual data field and holding unsatisfied references, if the references are already satisfied, completing the processing, otherwise waiting for the dependency;
the initializing data processing service comprises the steps of:
providing a seed data processing interface, inserting data, updating data or deleting data and returning a processing result.
The invention includes, but is not limited to, the following benefits:
the system can finish the initialization of the initialization data with complex mutual arrival relationship under a micro-service system, and can ensure that the final data is successfully initialized according to any service starting sequence and any dependency sequence. And may show all data dependent and initialized execution status according to the management service record. The system provides a flexible SDK, which can ensure that the needs of most services on initialization data are met. The service data is collected and reported by each micro-service, and can meet the decoupling of the service direct initialization data.
The preferred embodiments of the invention disclosed above are intended only to assist in the explanation of the invention. The preferred embodiments are not exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. The invention is limited only by the claims and the full scope and equivalents thereof.
Claims (1)
1. A method for initializing data in a micro-service architecture, comprising: manage services, SDKs, provide data or process the service composition under other micro-service systems of the data;
the management service is the core of the system and is responsible for collecting data to be initialized with the SDK, analyzing the dependency relationship of the initialized data, issuing a data initialization instruction meeting the dependency to the SDK, and storing the instruction execution result;
the SDK is a bridge for communication between the management service and the service under the other micro-service system for providing data or processing data, is integrated by the service under the other micro-service system for providing data or processing data during development and has the access capability of a database or a data storage position, is responsible for collecting and reporting data needing to be initialized to the management service when the service under the other micro-service system for providing data or processing data is started, and executes and reports execution results when an initialization instruction of the management server arrives; services under other micro-service systems that provide or process data need to integrate the SDK at development time and collect and convert data from any accessible data source into a format recognizable by the management service according to business requirements and report to the management service through the SDK;
the initialization method comprises initializing a data providing service, a management service and an initialization data processing service;
the initialization data providing service and the initialization data processing service are both services under other micro-service systems embedded with the provided data or the processed data of the SDK, and are configured to be only initial data providing service, or only initial data processing service or both initial data providing service and initial data processing service according to business requirements;
the initialization data adopts seed data to provide service, and comprises the following steps:
p1, manually added seed data is provided in the form of XML;
p2, automatically generating data through the service self code, including interface authority data;
p3, an extensible data providing form, an interface in the SDK is realized by the service, and return data is executed in the interface;
the management service is executed by monitoring the starting time of a new service instance, and comprises the following steps:
s1, after a seed data providing service instance is started, all seed data are pulled from the seed data providing service;
s2, splitting the seed data according to the source service, generating HASH check codes according to the content of the split data, using the source service, and using the HASH check codes as unique check, and if the check is inconsistent with the old version, storing the data in a database, wherein the table name for storing the data is as follows: the service_settings data has fields:
0) Data UUID id;
1) Source service_service;
2) MD5 code checksum;
3) Splitting the data data_json;
s3, splitting the split seed data according to records, and updating the split seed data into a data_external table, wherein the table structure is as follows:
0) Recording UUID id;
1) Associating the data ID data_id;
2) External reference ID external_id;
3) Target_service;
3) Actual data data_json;
4) The processing state process_status waits for dependence, processing is completed, and deletion is waited for;
s4, performing N times of data processing until the processed data is marked as 0, wherein the steps are as follows:
1) Inquiring unprocessed records, and grouping according to target services;
2) Processing each recorded reference, if the reference is satisfied, replacing the reference as real data, and if the reference is not satisfied, skipping first, wherein the replacement of each reference marks the processed data as +1;
3) Sending a request to each service according to the group of target services, enabling each service to process the data of the unprocessed records, writing back to an actual data field and maintaining unsatisfied references, if the references are satisfied, completing the processing, otherwise waiting for dependence;
the initializing data processing service includes the steps of:
providing a seed data processing interface, inserting data, updating data or deleting data and returning a processing result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010839132.9A CN111984648B (en) | 2020-08-19 | 2020-08-19 | Data initialization method and system under micro-service architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010839132.9A CN111984648B (en) | 2020-08-19 | 2020-08-19 | Data initialization method and system under micro-service architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111984648A CN111984648A (en) | 2020-11-24 |
| CN111984648B true CN111984648B (en) | 2023-10-31 |
Family
ID=73434807
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010839132.9A Active CN111984648B (en) | 2020-08-19 | 2020-08-19 | Data initialization method and system under micro-service architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111984648B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110287016A (en) * | 2019-07-01 | 2019-09-27 | 武汉兆格信息技术有限公司 | A kind of distribution flow chart Heterogeneous Computing dispatching method |
| CN110286884A (en) * | 2019-06-21 | 2019-09-27 | 北京百度网讯科技有限公司 | Micro services method of combination, device, equipment and computer readable storage medium |
| CN111198695A (en) * | 2019-12-25 | 2020-05-26 | 航天信息股份有限公司 | Automatic deployment method of micro-service administration management platform and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10824981B2 (en) * | 2017-04-24 | 2020-11-03 | Sap Se | Transaction orchestration for microservice |
-
2020
- 2020-08-19 CN CN202010839132.9A patent/CN111984648B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110286884A (en) * | 2019-06-21 | 2019-09-27 | 北京百度网讯科技有限公司 | Micro services method of combination, device, equipment and computer readable storage medium |
| CN110287016A (en) * | 2019-07-01 | 2019-09-27 | 武汉兆格信息技术有限公司 | A kind of distribution flow chart Heterogeneous Computing dispatching method |
| CN111198695A (en) * | 2019-12-25 | 2020-05-26 | 航天信息股份有限公司 | Automatic deployment method of micro-service administration management platform and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111984648A (en) | 2020-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9047392B2 (en) | System and method for conversion of JMS message data into database transactions for application to multiple heterogeneous databases | |
| US8341120B2 (en) | Apparatus and methods for transferring database objects into and out of database systems | |
| CN100570604C (en) | A kind of method of relation data base applications automatic upgrade | |
| CN101256650B (en) | Method and system for extracting enterprise data based on service entity | |
| US7895172B2 (en) | System and method for writing data dependent upon multiple reads in a distributed database | |
| US8635253B2 (en) | Decorated model architecture for efficient model-driven application development | |
| WO2012034440A1 (en) | Method and device for generating database upgrading script | |
| CN101196926A (en) | Database access platform and access method thereof | |
| CN115934855B (en) | Full-link field-level blood margin analysis method, system, equipment and storage medium | |
| CN111143451A (en) | SSM framework design method for hierarchical architecture design | |
| CN113032421A (en) | MongoDB-based distributed transaction processing system and method | |
| US20100153919A1 (en) | Systems and methods for tracking software stands in a software production landscape | |
| US7720872B1 (en) | Software interface mapping tool | |
| US20090063433A1 (en) | Apparatus, system, and method for command manager support for pluggable data formats | |
| US20070078840A1 (en) | Custom function library for inverse query evaluation of messages | |
| EP2718841A2 (en) | Code generation and implementation method, system, and storage medium for delivering bidirectional data aggregation and updates | |
| CN112487075B (en) | Method for integrating relational database data conversion operators and non-relational database data conversion operators | |
| CN111984648B (en) | Data initialization method and system under micro-service architecture | |
| CN115129740B (en) | Method and system for updating distributed micro-service database in cloud native environment | |
| CN101458628A (en) | Program edition management method | |
| CN110399386A (en) | A kind of SQL UPDATE method and control system based on Presto | |
| CN112487006A (en) | Implementation method for dynamically editing data structure and generating database table | |
| CN112685426A (en) | NiFi-based Kafka consumption NewSQL CDC stream data conversion method | |
| KR20170130911A (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| WO2024065188A1 (en) | Information model update method and apparatus, computing device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |