CN115098276A

CN115098276A - Service data management system and operation method, device and equipment thereof

Info

Publication number: CN115098276A
Application number: CN202210638859.XA
Authority: CN
Inventors: 马鑫; 刘闯
Original assignee: Neusoft Corp
Current assignee: Neusoft Corp
Priority date: 2022-06-07
Filing date: 2022-06-07
Publication date: 2022-09-23

Abstract

The application provides a service data management system and an operation method, device and equipment thereof. The system comprises: the system comprises a business management component and a big data warehouse ecology, wherein the business management component manages business data storage in each warehouse component of the big data warehouse ecology through a created data set, the data set is created according to a standardized storage format of each warehouse component configured in the business management component, and each warehouse component is provided with a corresponding warehouse adapter. The method and the system set a business management component as a middleware between business application and the large data number bin ecology, avoid business coupling between the business application and the large data number bin ecology, provide data support under various business scenes for the business application, shield complexity when various number bin components in the large data number bin ecology are mixed, reduce complexity and development difficulty of business data management, and reduce data source pressure of the large data number bin ecology.

Description

Service data management system and operation method, device and equipment thereof

Technical Field

The embodiment of the application relates to the technical field of data processing, in particular to a service data management system and an operation method, device and equipment thereof.

Background

With the continuous and deep application of big data technology, the amount of data accumulated in various service scenes is increased, and the data is diversified. In the case of a regional public health system, the data size is exponentially increased based on regional population, and the storage time is too long along with the life cycle of a user, so that the data query concurrency is too stressful.

At present, in order to deal with data analysis in a multi-service scenario, complex analysis of mass service data in different service scenarios is usually realized by means of distributed storage and distributed computation characteristics supported by a data warehouse technology in a big data ecology. At the moment, a plurality of ecologically complex warehouse counting components exist in the big data ecology, and different warehouse counting components are respectively suitable for different independent service scenes.

However, when the management and analysis of the service data are realized together by adopting a mixed manner of a plurality of warehouse components, the types of distributed storage, distributed computation and distributed file formats supported by each warehouse component are generally required to be docked, so that the complexity and development difficulty of service data management are greatly increased, and certain processing pressure is caused on a service data source.

Disclosure of Invention

The application provides a business data management system and an operation method, a device and equipment thereof, wherein a business management component is used as a middleware between a business application and a big data warehouse ecology, the decoupling of the business application from the business operation in a complex ecological environment is realized, the complexity when various warehouse components in the big data warehouse ecology are mixed is shielded, the complexity and the development difficulty of business data management are reduced, and the data source pressure of the big data warehouse ecology is reduced.

In a first aspect, an embodiment of the present application provides a system for managing service data, where the system includes: the system comprises a business management component and a big data warehouse ecology, wherein the business management component manages business data storage in each warehouse component of the big data warehouse ecology through a created data set, the data set is created according to a standardized storage format of each warehouse component configured in the business management component, and each warehouse component is provided with a corresponding warehouse adapter; wherein,

the service management component determines a corresponding request type and a target data set according to a current operation request of a service application, and generates a target operation statement under the request type through a target counting bin adapter corresponding to the target data set, so that the counting bin component corresponding to the target counting bin adapter is utilized to execute the target operation statement.

In a second aspect, an embodiment of the present application provides a method for operating service data, which is applied to a service management component in a management system for service data provided in the first aspect, and the method includes:

responding to a current operation request of a service application, and determining a corresponding request type and a target data set;

generating a target operation statement under the request type through a target data bin adapter corresponding to the target data set;

and forwarding the target operation statement to a silo counting component corresponding to the target silo counting adapter so as to execute the target operation statement.

In a third aspect, an embodiment of the present application provides an apparatus for operating service data, configured in a service management component in a management system for service data provided in the first aspect, where the apparatus includes:

the operation response module is used for responding to the current operation request of the service application and determining the corresponding request type and the target data set;

an operation statement generation module, configured to generate a target operation statement in the request type through a target data bin adapter corresponding to the target data set;

and the operation statement processing module is used for forwarding the target operation statement to the data bin assembly corresponding to the target data bin adapter so as to execute the target operation statement.

In a fourth aspect, an embodiment of the present application provides an electronic device, including:

a processor and a memory, the memory being used for storing a computer program, the processor being used for calling and executing the computer program stored in the memory to execute the operation method of the service data provided in the second aspect of the present application.

In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, where the computer program enables a computer to execute the operation method of the service data provided in the second aspect of the present application.

In a sixth aspect, the present application provides a computer program product, which includes computer programs/instructions, and is characterized in that when being executed by a processor, the computer programs/instructions implement the operation method of the service data as provided in the second aspect of the present application.

The business data management system, the operation method thereof, the device thereof and the equipment provided by the embodiment of the application set a business management component as a middleware between business application and big data warehouse ecology, so as to avoid business coupling between the business application and the big data warehouse ecology. The standardized storage format of each multi-bin component in the large data multi-bin ecology is pre-configured in the business management component, and the data set corresponding to the multi-bin component is created according to the standardized storage format of each multi-bin component, so that the business data storage in each multi-bin component is managed through the created data set, data support under multiple business scenes is provided for business application, and complexity when multiple multi-bin components in the large data multi-bin ecology are mixed is shielded. And moreover, a corresponding multi-bin adapter is set for each multi-bin component in the service management component, and then after a corresponding request type and a target data set are determined according to a current operation request of the service application, a target operation statement under the request type can be generated through the target multi-bin adapter corresponding to the target data set, and then the target operation statement is executed by using the multi-bin component corresponding to the target multi-bin adapter, so that service application is decoupled from service operation in a complex large data multi-bin ecology, the complexity and development difficulty of service data management are reduced, and the data source pressure of the large data multi-bin ecology is reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic diagram of a service data management system according to an embodiment of the present application;

fig. 2 is a data flow diagram of service data management according to an embodiment of the present application;

fig. 3 is a flowchart illustrating an operation method of service data according to an embodiment of the present application;

fig. 4 is a flowchart of a method for an incremental synchronization process of service data according to an embodiment of the present application;

fig. 5 is a schematic block diagram of an operation device for service data according to an embodiment of the present application;

fig. 6 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Considering that when management analysis of business data is realized jointly by adopting a mode of mixing a plurality of warehouse components in a big data warehouse ecology, the types of distributed storage, distributed computation and distributed file formats supported by each warehouse component are required to be docked, and the complexity and development difficulty of business data management are greatly increased. Therefore, the embodiment of the present application designs a service management component as a middleware between the service application and the big data warehouse ecology to avoid service coupling between the service application and the big data warehouse ecology. Meanwhile, a data set corresponding to each warehouse component is created according to the standardized storage format of each warehouse component configured in the service management component, so that service data storage in each warehouse component is managed through the created data set, data support under various service scenes can be provided for service application, and complexity of mixing various warehouse components in large-data warehouse ecology is shielded. And moreover, a corresponding multi-bin adapter is set for each multi-bin component in the service management component to process a target operation statement under the current operation request, so that service application is decoupled from service operation in a complex large-data multi-bin ecology, the complexity and development difficulty of service data management are reduced, and the data source pressure of the large-data multi-bin ecology is relieved.

Fig. 1 is a schematic diagram of a service data management system according to an embodiment of the present application. As shown in FIG. 1, the business data management system can include a business management component 110 and a big data silo ecosystem 120.

The business management component 110 manages business data storage in each warehouse component of the big data warehouse 120 through a created data set, the data set is created according to a standardized storage format of each warehouse component configured in the business management component 110, and each warehouse component is configured with a corresponding warehouse adapter.

Specifically, the service management component 110 determines a corresponding request type and a target data set according to the current operation request of the service application, and generates a target operation statement in the request type through a target counting bin adapter corresponding to the target data set, so as to execute the target operation statement by using the counting bin component corresponding to the target counting bin adapter.

In the present application, the big data number bin ecology 120 is a big data technology ecology circle supporting a bottom data storage and analysis technology, and is composed of a plurality of types of number bin components (for example, various databases such as an offline carbon data number bin, an approximate real-time number bin, an HBase daily increment table under XX enterprise ecology, an offline Impala number bin, an approximate real-time number bin, and an HBase daily increment table under CDH ecology) developed by different organizations. At this time, each bin component in the big data bin ecology 120 has the characteristics of corresponding distributed storage and distributed computation, and each bin component can handle various data operations in an independent business scenario. Therefore, the present application generally adopts a manner of mixing multiple warehouse components in the big data warehouse 120 to implement complex analysis of massive business data in a multi-business scenario.

However, considering that when the business application is directly coupled with the big data warehouse ecology 120, the requirement on the coupling structure between the business application and the big data warehouse ecology 120 is high, and the complexity and the development difficulty of business data management are greatly increased. Thus, the present application contemplates a business management component 110 as an intermediary between the business application and the big-data silo ecology 120 to avoid business coupling between the business application and the big-data silo ecology.

As an optional implementation scheme in the present application, since each warehouse component in the big data warehouse ecology 120 has a different distributed file format, in order to ensure the accuracy of uniform management of the business data in each warehouse component when the business data is operated in a manner of mixing multiple warehouse components in the big data warehouse ecology 120, the present application configures the standardized storage format of each warehouse component in advance in the business management component 110 for each warehouse component in the big data warehouse ecology 120. That is, the business management component 110 forms the best practices of (big data bin ecology- > bin component- > file format) for each bin component within the big data bin ecology 120 as the standardized storage format for each bin component.

Then, in order to achieve unified management of business data when a plurality of warehouse components are mixed and shield the complexity of the big-data warehouse ecology 120, the present application aggregates the multiple and complex business data units actually stored in the warehouse components into one data set according to the standardized storage format of each warehouse component in the big-data warehouse ecology 120.

Where the data set facing each bin component is the basic unit that supports structured data storage within that bin component, similar to a table of a relational database. In the present application, the service management component 110 may determine a warehouse component applicable in any service scenario according to specific management requirements of service data in different service scenarios, such as query cycle, high-concurrency query or multi-condition query. Then, according to the standardized storage format of the warehouse component configured in the service management component 110, various data sets suitable for the corresponding service scenarios can be created in the service management component 110.

It should be appreciated that to ensure the accuracy of the operation of the data set within the business management component 110, the present application may utilize distributed locks to maintain the meta information (meta information) and attribute information of the data set in a unified manner during the creation and modification of the data set.

The warehouse components developed by the same organization in the big data warehouse ecology 120 can be divided into near real-time warehouse components and offline warehouse components. Thus, the present application sets the Target attributes (i.e., Target parameters) of the data set to include both near real-time and offline types to apply the near real-time bin components and the offline bin components in the big data bin ecology 120 to support redundant storage of data of the same data set on the near real-time bin components and the offline bin components.

Moreover, in order to ensure the accuracy of data operations corresponding to different warehouse components, the present application sets a corresponding warehouse adapter for each warehouse component in the service management component 110, so as to form an ecological adaptation layer in the service management component 110. Furthermore, the business management component 110 may execute the business data management operation and the data set management operation associated with the data of the plurality of bins corresponding to any of the plurality of bin adapters by calling the plurality of bin adapters, so as to shield the underlying data difference between various bin components, thereby improving the stability of business application on business data and data set management.

As an optional implementation scheme in the present application, in order to ensure accurate and convenient calling of the business application, the business management component 110 in the present application sets a uniform external service interface for different business applications, so that each business application performs a corresponding data set management operation on each data warehouse component of the big data warehouse ecology 120, and performs a corresponding data management operation on the business data stored in each data warehouse component.

Specifically, different types of business applications may invoke the business management component 110 to perform corresponding data set management operations or business data management operations through an external service interface. For example, a certain business application may call the business management component 110 through an external service interface to perform management operations such as creation, deletion, modification, and query of data set definition on a data set, or call the business management component 110 to perform management operations such as writing, deletion, modification, and query on business data stored in a bin component pointed by the data set. That is, through the external service interface, each service application is supported to initiate a corresponding operation request to the service management component 110. Then, in response to each received operation request, the service management component 110 may perform a corresponding management operation on the target data set specified by the operation request or the service data in the target data set.

Moreover, the business application will periodically collect new business data and synchronize into the corresponding bin components of the big-data bin ecosystem 120. Considering that the amount of service data stored in each warehouse component of the big data warehouse ecology 120 is large, if a full-volume synchronization mode is adopted, the synchronization efficiency is greatly influenced. Therefore, the present application will set up a message queue based incremental sync channel within the service management component 110.

Specifically, according to the storage requirement of the newly acquired data to be synchronized in the big data warehouse 120, a corresponding data set can be created in the business management component 110, so as to enter each data to be synchronized. Meanwhile, the recorded data to be synchronized is cached continuously through a preset synchronization message queue. Then, after the service management component 110 receives the increment synchronization request of the service application, each data to be synchronized buffered in the synchronization message queue is consumed, so as to perform data set grouping on each data to be synchronized. And then, according to the grouped different data set attributes, corresponding synchronization operation is executed on the data to be synchronized through the data set corresponding to the multi-bin adapter, and the data to be synchronized are stored in the corresponding HBase day increment table. And subsequently, the data to be synchronized serving as the increment data is correspondingly synchronized into the warehouse counting assembly serving as the full-scale table according to the period of the increment table.

In addition, in order to ensure accurate execution of the business management operation, the application configures an additional function library supported by each bin component in the business management component 110 to compensate for different support degrees of the platform function set for business management among the bin components. Furthermore, the additional function library supports the service management component 110 to self-define the corresponding analysis function according to the data analysis characteristics of each array of the number bins.

In the present application, it is contemplated that the business management component 110 supports management operations in both data set management and business data management types. Therefore, after the service management component 110 receives the current operation request of the service application, it needs to determine the request type corresponding to the current operation request, that is, determine whether the current operation request is for data set management or service data management in a certain data set. And determining a target data set pointed by the current operation request, namely the data set to be operated at this time or the data set to which the service data to be operated at this time belongs. Then, according to the bin counting component pointed by the target data set, a corresponding target bin adapter is determined. And analyzing specific operation information in the current operation request through the target number bin adapter, so as to generate a target operation statement under the request type.

For example, if the request type is a Data set Definition operation, the target operation statement is a Data set operation statement in a Data Definition Language (DDL); if the request type is a management operation on business Data in any Data set, the target operation statement is a Structured Query Language (SQL) statement under a Data Management Language (DML).

Finally, the target warehouse adapter can forward the target operation statement to the corresponding warehouse data, and the warehouse component number executes the target operation statement to obtain the corresponding business operation result. And then, returning the service operation result to the service application initiating the current operation request.

As can be seen from the above, as shown in fig. 2, the service management component 110 in the present application can implement the following functions: the system comprises a data set management function, an ecological adaptation function, a data synchronization function and a data query function.

It should be noted that, in addition to the above functions, the service management component 110 can also implement basic database functions, such as a global write function, a data delete function, and a data modification function of each service data in the data set. The implementation of such underlying database functionality within the business management component 110 is similar to conventional implementations within a database in the present application and will not be described in great detail. The present application mainly explains a data synchronization function in which a data query function and incremental writing exist.

The above-described functions performed by the service management component 110 are explained next:

1) data set management function: for any warehouse component in the big data warehouse ecology 120, according to the standardized storage format configured in the service management component 110 by the warehouse component, the warehouse component is responsible for creating, deleting and modifying the data set corresponding to the warehouse component, inquiring the definition information of the data set, and the like.

As an exemplary scheme in the present application, as shown in fig. 2, for a collection-type business application as a data collection party, corresponding business data is collected continuously during the running process of the application, and the collected business data is stored into a corresponding warehouse component in the big-data warehouse ecology 120 through the business management component 110. At this time, the business management component 110 realizes the unified management of business data when a plurality of bin components are mixed by facing to the data sets which are created by each bin component. Therefore, the service application first initiates a corresponding data set management request to the service management component 110, where the data set management request may instruct the service management component 110 to perform some management operation of creating, deleting, modifying, and querying the data set definition information on the data set corresponding to the silo component according to the configured standardized storage format of the silo component.

Specifically, when receiving a data set management request initiated by a certain service application, the service management component 110 first determines a data set that needs to be managed this time, as a target data set. Then, in the process of creating or modifying the target data set, firstly, locking the target data set by using an externally provided distributed lock to prevent other processes from operating the target data set, and ensuring the accuracy of the data set management. Further, according to the specific management information carried in the data set management request, the meta information (meta information) configured for the locked target data set is obtained. And generating a corresponding DDL statement according to a specific operation in the creation, modification, deletion and data set definition information query executed for the time on the target data set and the meta information of the target data set, wherein the DDL statement is used for indicating a specific management operation on the meta information of the target data set. Subsequently, the DDL statement is correspondingly converted through an ecological adaptation function in the service management component 110, the converted DDL statement is forwarded to a several-bin component corresponding to a target data set in the big-data several-bin ecology 120 by using a target several-bin adapter corresponding to the target data set, and the several-bin component completes management of the meta information of the target data set by creating or modifying a data table therein.

And when the target attribute of the target data set managed this time is configured with the offline type, a custom synchronization cycle is also added to the target data set. At this time, the synchronization period added by each data set configured with the offline type is saved in the task scheduling engine in the service management component 110, so that a personalized synchronization task can be formed for the data to be synchronized under each data set configured with the offline type when the data synchronization function is subsequently executed.

2) Ecological adaptation function: and the statement conversion of the warehouse adapter suitable for the corresponding warehouse component is performed.

Specifically, the service management component 110 supports the SQL92 protocol and the platform function set based on the function library of Hive; a unified JDBC SQL interface for data insertion, update and query; the bin storage standardized configuration creates different bin adapters for different bin components within the big data bin ecosystem 120, as well as the optimal optimized parameter configuration and storage format (i.e., standardized storage format in the present application). Moreover, an additional function library supported by each multi-bin component is configured, wherein the additional function library comprises a function name mapping table, additional function libraries aiming at different multi-bins and the like, the different support degrees of various multi-bin components on the platform function set are made up, and the additional function library can be used for physical examination of self-defined functions. Meanwhile, each bin adapter is an independent Java Virtual Machine (JVM) process, and Context corresponding to JDBC interfaces of different bin components is built in the JVM process.

Therefore, for any business operation request of the business application to the target data set, a target operation statement suitable for the type of the request can be generated through the target data bin adapter corresponding to the target data set. For example, a data set operation statement in DDL language or an SQL statement in DML language, etc.

3) The data synchronization function: is responsible for incremental storage of business data periodically collected by the business application within the corresponding bin components of the big-data bin ecosystem 120.

In the present application, the business management component 110 will implement incremental synchronization of business applications for data to be synchronized within the big-data silo 120 through a preset incremental synchronization channel based on a message queue.

As shown in fig. 2, a synchronization message queue is first set up to buffer each service data newly collected by the service application. Then, each data to be synchronized in the synchronization message queue is continuously consumed through the incremental synchronization channel, and a data set to which each data to be synchronized belongs is judged. Furthermore, the data sets of each data to be synchronized are grouped, that is, the data to be synchronized belonging to the same data set are grouped into one group, so that the data to be synchronized in different grouped data sets can be obtained.

Then, by judging the target attribute of each packet data set to be a near real-time type and/or an offline type, the data to be synchronized in the packet data set can be distributed to a corresponding near real-time queue and/or offline queue. For example, if the target attribute of a certain packet data set is a near real-time type, the data to be synchronized in the packet data set is distributed to the corresponding near real-time queue. And if the target attribute of a certain packet data set is of an offline type, distributing the data to be synchronized in the packet data set to a corresponding offline queue. And if the target attribute of a certain packet data set is a near real-time type and an off-line type, respectively distributing the data to be synchronized under the packet data set to a corresponding near real-time queue and an off-line queue. And finally, for the data to be synchronized in the off-line columns, periodically performing synchronization processing according to the synchronization period of the corresponding packet data set in the task scheduling engine, so as to store the data to be synchronized in batches into the corresponding HBase day increment table. However, for the data to be synchronized in the near-real-time alignment column, near-real-time streaming synchronization processing may be directly performed to store the data to be synchronized in batches into the corresponding HBase day increment table. And subsequently, correspondingly synchronizing the data to be synchronized serving as the incremental data into an offline warehouse counting component and/or a near-real-time warehouse counting component serving as a full-scale table through a corresponding warehouse counting adapter under the ecological adaptation function according to the period of the incremental table. At this time, the application reduces the data source pressure of the big data warehouse ecology 120 by providing two storage modes of near real-time storage and offline storage for the same data to be synchronized.

In addition, after the data to be synchronized of the off-line columns are periodically synchronized, corresponding data arrival notification is set for the off-line synchronization condition of the data to be synchronized, so that the delay of the data analysis function is reduced.

4) Data query function: and the system is responsible for analyzing the actual query of the service application on the service data in each created data set, and supports the configuration of the target attribute of the additional query target data set and the survival time of the query operation.

Illustratively, upon receiving a query request of a business application for business data in any target data set, the business management component 110 generates an initial SQL statement in the DML language for representing data query operations in the target data set. Then, according to a Target attribute (Target) parameter in the query request, the initial SQL statement may be forwarded to the near-real time bin adapter and/or the off-line number bin adapter in the big data number bin ecology 120 corresponding to the Target data set. And then, the initial SQL statement is converted into a target SQL statement under the language supported by the bin assemblies corresponding to the near-real time bin adapter and/or the off-line bin adapter by combining the near-real time bin adapter and/or the off-line bin adapter with the corresponding additional function library, so as to shield the difference of different bin assemblies for SQL languages.

Then, the near-real-time bin adapter and/or the off-line bin adapter can maintain the context of the corresponding near-real-time bin assembly and/or off-line bin assembly in the data query process, and forward the target SQL statement to the near-real-time bin assembly and/or off-line bin assembly. And executing actual data query operation in the context of the near real-time bin counting component and/or the off-line bin counting component to obtain a corresponding query result.

At this time, in order to ensure the efficiency of the data query, the statement identifier (denoted as UUID) and the storage time (denoted as Keeptime) of the target SQL statement are set according to the number of query results. For example, if the number of query results is less than or equal to 1000, the query results are returned to the business application directly through the near real-time bin counting component and/or the offline bin counting component, and the context of the near real-time bin counting component and/or the offline bin counting component is closed. If the quantity of the query results is more than 1000, returning the first 1000 pieces of data in the query results to the business application through the near real-time bin assembly and/or the off-line bin assembly, and setting statement identification and storage time of the target SQL statement, wherein the near real-time bin adapter and/or the off-line bin adapter maintains the context of the corresponding near real-time bin assembly and/or off-line bin assembly in the survival time (keep time). Therefore, the business application can retrieve the queried data in batches within the time-to-live according to the statement mark, returning 1000 pieces each time. If no request for retrieving the next batch of query data in the query result is received after the survival time is exceeded, closing the Context of the near real-time data warehouse component and/or the off-line data warehouse component. However, if a request to retrieve the next batch of query data within the query results is received within the time-to-live, the context lease of the near real-time and/or offline data storage components is automatically renewed.

According to the technical scheme, the service management component is set to serve as the middleware between the service application and the big data warehouse ecology, and service coupling between the service application and the big data warehouse ecology is avoided. The standardized storage format of each multi-bin component in the large data multi-bin ecology is pre-configured in the business management component, and the data set corresponding to the multi-bin component is created according to the standardized storage format of each multi-bin component, so that the business data storage in each multi-bin component is managed through the created data set, data support under multiple business scenes is provided for business application, and complexity when multiple multi-bin components in the large data multi-bin ecology are mixed is shielded. And moreover, a corresponding multi-bin adapter is set for each multi-bin component in the service management component, and then after a corresponding request type and a target data set are determined according to a current operation request of the service application, a target operation statement under the request type can be generated through the target multi-bin adapter corresponding to the target data set, and then the target operation statement is executed by using the multi-bin component corresponding to the target multi-bin adapter, so that service application is decoupled from service operation in a complex large data multi-bin ecology, the complexity and development difficulty of service data management are reduced, and the data source pressure of the large data multi-bin ecology is reduced.

The specific operation steps of the service management component in the service data management system for performing various service data operations will be described in detail below.

Fig. 3 is a flowchart of an operation method of service data according to an embodiment of the present application, and the present embodiment is mainly applied to a service management component in a management system of service data provided in the foregoing embodiment.

Referring to fig. 3, the method may specifically include the following steps:

s310, responding to the current operation request of the business application, and determining the corresponding request type and the target data set.

It is contemplated that the business management components in the present application support management operations in both data set management and business data management types. Therefore, after receiving a current operation request of a service application, first, a request type corresponding to the current operation request needs to be determined, that is, it is determined whether the current operation request manages a data set or service data in a certain data set. And determining a target data set pointed by the current operation request, namely the data set to be operated at this time or the data set to which the service data to be operated at this time belongs.

It should be appreciated that to ensure accuracy of data set management, a lock is placed on a target data set if the request type of the current operation request defines an operation for the data set. In the process of creating or modifying the target data set, the target data set is locked by the externally provided distributed locks, so that other processes are prevented from operating the target data set, and the accuracy of the data set management is ensured. Then, after the target operation statement in the application is executed, the target data set is unlocked.

And S320, generating a target operation statement under the request type through the target data set corresponding target number bin adapter.

Optionally, for the current operation request, a corresponding initial operation statement is generated first. The initial operation statement can represent specific operation information which needs to be executed at this time. And, determining a corresponding target bin adapter according to the bin counting component pointed to by the target data set. The initial operation statement is then converted, by the target data set adapter in conjunction with the additional function library, into a function language appropriate for the data set pointed to by the target data set, thereby generating the target operation statement under the request type.

As an alternative implementation in the present application, if the request type is a data set definition operation, a data set operation statement in the DDL language may be generated by a target data set corresponding to the target data set as a target operation statement in the present application. However, if the request type is a management operation on business data in any data set, an SQL statement in the DML language may be generated by the target data bin adapter corresponding to the target data set as a target operation statement in the present application.

S330, the target operation statement is forwarded to the warehouse counting component corresponding to the target warehouse counting adapter so as to execute the target operation statement.

After the target operation statement of the current operation is obtained through the target counting bin adapter, the target operation statement is directly forwarded to the counting bin assembly corresponding to the target counting bin adapter. At this time, according to the fact that the target attribute of the target data set is the offline type and/or the near real-time type, it can be determined that the bin counting assembly is the corresponding offline bin counting assembly and/or the near real-time bin counting assembly. Then, the target operation statement is executed in the context of the data warehouse component, and a corresponding operation result is obtained.

Taking the query of the business data in the target data set as an example, the target operation statement in the present application is a data query statement in an SQL statement. At this time, the data query statement may be forwarded to the bin counting component corresponding to the target bin counting adapter, and the bin counting component executes a corresponding data query operation in the context thereof, so as to obtain a corresponding query result.

Then, in order to ensure the high efficiency of the data query, the statement identification and the survival time of the data query statement can be determined according to the number of query results. And locking the context lease of the warehouse component corresponding to the target warehouse adapter according to the survival time. And then, feeding back a corresponding query result to the service application batch according to the statement identifier in the context lease.

For example, if the number of query results is less than or equal to 1000, the query results are returned to the business application directly through the near real-time bin counting component and/or the offline bin counting component, and the context of the near real-time bin counting component and/or the offline bin counting component is closed. And if the quantity of the query results is more than 1000, returning the first 1000 data in the query results to the business application through the near real-time bin counting component and/or the off-line bin counting component, and setting statement identification and storage time of the target SQL statement, wherein the near real-time bin adapter and/or the off-line bin adapter maintains the context of the corresponding near real-time bin counting component and/or off-line bin counting component in the survival time (keep time).

Therefore, the business application can retrieve the queried data in batches within the time-to-live according to the statement mark, returning 1000 pieces each time. If no request for retrieving the next batch of query data in the query result is received after the survival time is exceeded, closing the Context of the near real-time data warehouse component and/or the off-line data warehouse component. However, if a request to retrieve the next batch of query data within the query results is received within the time-to-live, the context lease of the near real-time and/or offline data storage components is automatically renewed.

In addition, the present application also supports an operation of performing incremental synchronization on the service data in the target data set, as shown in fig. 4, the incremental synchronization operation of the service data includes the following steps:

s410, responding to the increment synchronous request of the service application, and carrying out data set grouping on the data to be synchronized in the synchronous message queue to obtain the data to be synchronized in different grouped data sets.

Aiming at the increment synchronization of the service data, a synchronization message queue is set for caching each newly acquired service data of the service application. Then, after receiving an incremental synchronization request of any business application, each data to be synchronized in the synchronization message queue is continuously consumed through an incremental synchronization channel, and a data set to which each data to be synchronized belongs is judged. Furthermore, the data sets of each data to be synchronized are grouped, that is, the data to be synchronized belonging to the same data set are grouped into one group, so that the data to be synchronized in different grouped data sets can be obtained.

And S420, forwarding the data to be synchronized in each packet data set to a corresponding near real-time queue and/or offline queue according to the target attribute of the packet data set.

By judging the target attribute of each packet data set to be a near real-time type and/or an off-line type, the data to be synchronized in the packet data set can be distributed to a corresponding near real-time queue and/or off-line queue. For example, if the target attribute of a certain packet data set is a near real-time type, the data to be synchronized in the packet data set is distributed to the corresponding near real-time queue. And if the target attribute of a certain packet data set is of an offline type, distributing the data to be synchronized in the packet data set to a corresponding offline queue. And if the target attribute of a certain packet data set is a near real-time type and an off-line type, respectively distributing the data to be synchronized under the packet data set to a corresponding near real-time queue and an off-line queue.

S430, writing the data to be synchronized in the near real-time queue into the corresponding near real-time data bin assembly through the near real-time data bin adapter corresponding to each grouped data set; and/or writing the data to be synchronized in the offline queue into the corresponding offline bin assembly according to the configured synchronization period of the grouped data sets through the offline bin adapter corresponding to each grouped data set.

Optionally, for the data to be synchronized in the off-line pair column, synchronization processing may be periodically performed according to a synchronization period of the corresponding packet data set in the task scheduling engine, so as to store the data to be synchronized in the corresponding HBase antenna increment table in batches. However, for the data to be synchronized in the near real-time alignment column, near real-time streaming synchronization processing can be directly performed, so as to store the data to be synchronized in batches into the corresponding HBase day increment table. And subsequently, correspondingly synchronizing the data to be synchronized serving as the incremental data into an offline warehouse counting component and/or a near-real-time warehouse counting component serving as a full-scale table through a corresponding warehouse counting adapter under the ecological adaptation function according to the period of the incremental table. At the moment, the method and the device have the advantages that two storage modes of near real-time storage and offline storage are provided for the same data to be synchronized, and the ecological data source pressure of the large data storage bin is reduced.

Fig. 5 is a schematic block diagram of an operation device for service data according to an embodiment of the present application, where the present embodiment is mainly configured in a service management component in a management system for service data provided in the foregoing embodiment. As shown in fig. 5, the apparatus 500 may include:

an operation response module 510, configured to determine, in response to a current operation request of a service application, a corresponding request type and a target data set;

an operation statement generating module 520, configured to generate a target operation statement in the request type through a target data bin adapter corresponding to the target data set;

an operation statement processing module 530, configured to forward the target operation statement to the bin counting component corresponding to the target bin counting adapter, so as to execute the target operation statement.

Further, the operation statement generating module 520 may be specifically configured to:

if the request type is data set definition operation, generating a data set operation statement under a data definition language through a target data bin adapter corresponding to a target data set;

and if the request type is the management operation on the business data in any data set, generating a Structured Query Language (SQL) statement under the data manipulation language through a target data bin adapter corresponding to the target data set.

Further, the operation statement processing module 530 may be specifically configured to:

if the target operation statement is a data query statement in the SQL statement, forwarding the data query statement to a data bin assembly corresponding to the target data bin adapter to obtain a corresponding query result;

determining statement identification and survival time of the data query statement according to the number of the query results;

locking the context lease of the warehouse component corresponding to the target warehouse adapter according to the survival time;

and feeding back a corresponding query result to the service application batch according to the statement identifier in the context lease.

Further, the operation device 500 for the service data may further include a data set locking module;

the data set locking module may be specifically configured to:

locking the target data set if the request type defines an operation for the data set;

and after the target operation statement is executed, unlocking the target data set.

Further, the operation apparatus 500 for service data may further include an incremental synchronization module;

the incremental synchronization module may be specifically configured to:

responding to the increment synchronous request of the service application, and performing data set grouping on the data to be synchronized in the synchronous message queue to obtain the data to be synchronized in different grouped data sets;

according to the target attribute of each grouped data set, the data to be synchronized in the grouped data sets are forwarded to the corresponding near real-time queue and/or off-line queue;

writing the data to be synchronized in the near real-time queue into the corresponding near real-time data bin assembly through the near real-time data bin adapter corresponding to each packet data set; and/or the presence of a gas in the gas,

and writing the data to be synchronized in the offline queue into a corresponding offline bin assembly according to the configured synchronization period of each grouped data set through the offline bin adapter corresponding to each grouped data set.

In the embodiment of the application, a service management component is set to serve as a middleware between the service application and the big data warehouse ecology, so that service coupling between the service application and the big data warehouse ecology is avoided. The standardized storage format of each warehouse component in the big data warehouse ecology is pre-configured in the business management component, and the data set corresponding to the warehouse component is created according to the standardized storage format of each warehouse component, so that the business data storage in each warehouse component is managed through the created data set, data support under multiple business scenes is provided for business application, and complexity when multiple warehouse components in the big data warehouse ecology are mixed is shielded. And moreover, a corresponding multi-bin adapter is set for each multi-bin assembly in the service management assembly, and then after a corresponding request type and a target data set are determined according to a current operation request of the service application, a target operation statement under the request type can be generated through the target multi-bin adapter corresponding to the target data set, and then the multi-bin assembly corresponding to the target multi-bin adapter is utilized to execute the target operation statement, so that service operation decoupling of the service application from a complex large data multi-bin ecology is realized, the complexity and the development difficulty of service data management are reduced, and the data source pressure of the large data multi-bin ecology is reduced.

It is to be understood that apparatus embodiments and method embodiments may correspond to one another and that similar descriptions may refer to method embodiments. To avoid repetition, further description is omitted here. Specifically, the apparatus 500 shown in fig. 5 may perform any method embodiment provided in the present application, and the foregoing and other operations and/or functions of each module in the apparatus 500 are respectively for implementing corresponding processes in each method of the embodiment of the present application, and are not described herein again for brevity.

The apparatus 500 of the embodiments of the present application is described above in connection with the figures from the perspective of functional modules. It should be understood that the functional modules may be implemented by hardware, by instructions in software, or by a combination of hardware and software modules. Specifically, the steps of the method embodiments in the present application may be implemented by integrated logic circuits of hardware in a processor and/or instructions in the form of software, and the steps of the method disclosed in conjunction with the embodiments in the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. Alternatively, the software modules may be located in random access memory, flash memory, read only memory, programmable read only memory, electrically erasable programmable memory, registers, and the like, as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps in the above method embodiments in combination with hardware thereof.

Fig. 6 is a schematic block diagram of an electronic device 600 provided in an embodiment of the present application.

As shown in fig. 6, the electronic device 600 may include:

a memory 610 and a processor 620, the memory 610 being adapted to store a computer program and to transfer the program code to the processor 620. In other words, the processor 620 may call and execute a computer program from the memory 610 to implement the method in the embodiment of the present application.

For example, the processor 620 may be configured to perform the above-described method embodiments according to instructions in the computer program.

In some embodiments of the present application, the processor 620 may include, but is not limited to:

general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like.

In some embodiments of the present application, the memory 610 includes, but is not limited to:

volatile memory and/or non-volatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example but not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic random access memory (DDR SDRAM), Enhanced Synchronous SDRAM (ESDRAM), SLDRAM, and Direct Rambus RAM (DR RAM).

In some embodiments of the present application, the computer program may be partitioned into one or more modules, which are stored in the memory 610 and executed by the processor 620 to perform the methods provided herein. The one or more modules may be a series of computer program instruction segments capable of performing certain functions, the instruction segments describing the execution of the computer program in the electronic device.

As shown in fig. 6, the electronic device may further include:

a transceiver 630, the transceiver 630 may be connected to the processor 620 or the memory 610.

The processor 620 may control the transceiver 630 to communicate with other devices, and in particular, may transmit information or data to the other devices or receive information or data transmitted by the other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

It should be understood that the various components in the electronic device are connected by a bus system that includes a power bus, a control bus, and a status signal bus in addition to a data bus.

Embodiments of the present application also provide a computer storage medium having a computer program stored thereon, where the computer program, when executed by a computer, enables the computer to execute the method of the above method embodiments. In other words, the present application also provides a computer program product containing instructions, which when executed by a computer, cause the computer to execute the method of the above method embodiments.

When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the present application are generated in whole or in part when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that is integrated into one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a Digital Video Disc (DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), among others.

Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, may be located in one place, or may be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. For example, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A system for managing service data, comprising: the system comprises a business management component and a big data warehouse ecology, wherein the business management component manages business data storage in each warehouse component of the big data warehouse ecology through a created data set, the data set is created according to a standardized storage format of each warehouse component configured in the business management component, and each warehouse component is provided with a corresponding warehouse adapter; wherein,

2. The system of claim 1, wherein the business management component is configured with a unified external service interface for different business applications to support each business application performing corresponding data set management operations on each data warehouse component of the big data warehouse ecosystem and corresponding data management operations on business data stored in each data warehouse component.

3. The system of claim 1, wherein the target attributes of the data set include a near real-time type and an offline type to support redundant storage of data of the data set on a near real-time bin assembly and an offline bin assembly.

4. The system of claim 1, wherein a message queue-based incremental synchronization channel is configured in the service management component to implement incremental synchronization of data under different data sets.

5. The system of claim 1, wherein an additional library of functions supported by each of the bin components is configured in the traffic management component to assist the target bin adapter in generating a target operation statement for the request type.

6. A method for operating service data, which is applied to a service management component in a service data management system according to any one of claims 1 to 5, and comprises:

responding to a current operation request of a business application, and determining a corresponding request type and a target data set;

7. The method of claim 6, wherein generating the target operation statement under the request type through the target data set corresponding target data bin adapter comprises:

and if the request type is the management operation on the business data in any data set, generating a Structured Query Language (SQL) statement under a data manipulation language through a target data set corresponding to the target data set.

8. The method of claim 7, wherein forwarding the target operation statement to a corresponding bin component of the target bin adapter to execute the target operation statement comprises:

if the target operation statement is a data query statement in the SQL statement, forwarding the data query statement to a warehouse component corresponding to the target warehouse adapter to obtain a corresponding query result;

9. The method of claim 7, further comprising:

locking the target dataset if the request type defines an operation for the dataset;

10. The method of claim 6, further comprising:

11. An apparatus for operating service data, configured in a service management component in a service data management system according to any one of claims 1 to 5, comprising:

and the operation statement processing module is used for forwarding the target operation statement to the warehouse counting assembly corresponding to the target warehouse counting adapter so as to execute the target operation statement.

12. An electronic device, comprising:

a processor and a memory, the memory for storing a computer program, the processor for calling and executing the computer program stored in the memory to perform the method of operation of the business data of any one of claims 6-10.

13. A computer-readable storage medium for storing a computer program which causes a computer to perform the method of operation of the service data according to any one of claims 6-10.

14. A computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the method of operation of the service data according to any of claims 6-10.