CN110019440B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN110019440B CN110019440B CN201710764963.2A CN201710764963A CN110019440B CN 110019440 B CN110019440 B CN 110019440B CN 201710764963 A CN201710764963 A CN 201710764963A CN 110019440 B CN110019440 B CN 110019440B
- Authority
- CN
- China
- Prior art keywords
- data
- configurable interface
- interface file
- module
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Abstract
The invention discloses a data processing method and device, relates to the technical field of information processing, and mainly aims to solve the problems of poor expandability and low maintainability of a traditional data warehouse. The technical scheme of the invention comprises the following steps: acquiring a data processing request, wherein the data processing request comprises a data name of data to be processed; calling the first configurable interface file, and acquiring data to be processed corresponding to the data name from the data storage module; calling the second configurable interface file, and acquiring a processing method corresponding to the to-be-processed data corresponding to the data name from the data processing module; processing the data to be processed according to the acquired processing method; and calling the third configurable interface file, determining a data application style corresponding to the data name from the data application module, and displaying the processed data to be processed based on the data application style.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a data processing method and apparatus.
Background
The traditional data warehouse uses a single data resource as a center to perform transaction Processing, and data and information of the data warehouse are extracted from different data sources, so that the data warehouse has the characteristic of Analysis Processing, is a structured data environment of a Decision Support System (DSS) and an On-Line Analysis Processing (OLAP), and can provide all types of data Support. As usage time increases, data warehouses need to support more data processing and more complex user queries, and therefore data warehouses need to have the ability to add new functional components while maintaining the capabilities of the original system. However, because the mechanism coupling degree in the conventional data warehouse system is high, the decision support system is tightly combined with the functional components of data mining, so that the expandability and maintainability of the data warehouse are poor, and the requirement of adding new functional components cannot be met.
In order to improve the expansibility and maintainability of the conventional data warehouse, the following two methods are mainly adopted in the prior art: the first method is to adopt a layering strategy to expand each layer in the system, increase components in each layer, and improve the expandability of the data warehouse by modularizing the components of the system; the second approach is to utilize high performance hardware, place components on different hardware, and improve the scalability of the data warehouse through scalable software parallelism and hardware parallelism.
In the process of implementing the invention, the inventor finds that in the prior art, if the first method is adopted, the single modules of the system are increased continuously, the internal coupling of each layer is improved continuously, and the maintainability of the system is reduced; if the second method is adopted, a high-performance platform needs to be established to run, high-performance software is parallel to hardware, and components with weaker performance in the system can reduce the overall performance level of the data warehouse, thereby increasing the development and maintenance difficulty of the data warehouse. Therefore, the prior art cannot realize the method, and the maintainability of the data warehouse is ensured, and the expandability of the traditional data warehouse is improved.
Disclosure of Invention
In view of this, the present invention provides a data processing method and apparatus, and mainly aims to solve the problems of poor expandability and reduced maintainability of a conventional data warehouse.
In order to solve the above problems, the present invention mainly provides the following technical solutions:
in a first aspect, the present invention provides a data repository comprising:
the data management layer is internally provided with a metadata control center, and the metadata control center performs data interaction with a plurality of different functional modules through a plurality of configurable interface files;
wherein each configurable interface file has a mapping relationship with one of the function modules, and the plurality of configurable interface files include: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module;
the data storage module is used for storing data;
the data processing module is used for recording a processing method corresponding to the data;
and the data application module is used for recording the application style of the processed data.
In a second aspect, the present invention provides a data processing method, which is applied to a data warehouse, where the data warehouse includes a data management layer, a plurality of configurable interface files, and a plurality of different functional modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and each configurable interface file records a data name of data contained in the corresponding functional module;
acquiring a data processing request, wherein the data processing request comprises a data name of data to be processed;
calling the first configurable interface file, and acquiring data to be processed corresponding to the data name from the data storage module;
calling the second configurable interface file, and acquiring a processing method corresponding to the to-be-processed data corresponding to the data name from the data processing module;
processing the data to be processed according to the acquired processing method;
and calling the third configurable interface file, determining a data application style corresponding to the data name from the data application module, and displaying the processed data to be processed based on the data application style.
Optionally, before acquiring the data to be processed from the data storage module by calling the first configurable interface file of the metadata control center, the method further includes:
respectively acquiring classified data components contained in the data storage module, the data processing module and the data application module, wherein the classified data components are used for recording classified data in different functional modules;
and recording the data names in the classified data components in the first configurable interface file, the second configurable interface file and the third configurable interface file, wherein the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
Optionally, the method further includes:
monitoring the classified data assembly in real time, and determining whether the classified data assembly is updated;
and if so, updating the configurable interface file based on the updated classification data component.
Optionally, based on the first configurable interface file of the metadata control center, acquiring the data to be processed from the data storage module includes:
acquiring a data name of data to be processed, and determining whether the data name exists in the first configurable interface file;
and if the data exists, acquiring the data to be processed corresponding to the data name from the data storage module.
In a third aspect, the present invention further provides a data processing apparatus, where the apparatus is applied to a data warehouse, where the data warehouse includes a data management layer, a plurality of configurable interface files, and a plurality of different functional modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and each configurable interface file records a data name of data contained in the corresponding functional module; the method comprises the following steps:
the device comprises a first acquisition unit, a second acquisition unit and a processing unit, wherein the first acquisition unit is used for acquiring a data processing request which comprises a data name of data to be processed;
the first calling unit is used for calling the first configurable interface file;
the second obtaining unit is used for obtaining the data to be processed corresponding to the data name from the data storage module after the first calling unit calls the first configurable interface file;
the second calling unit is used for calling the second configurable interface file;
a third obtaining unit, configured to obtain, from the data processing module, a processing method corresponding to the to-be-processed data corresponding to the data name after the second configurable interface file is called by the second calling unit;
the processing unit is used for processing the data to be processed according to the processing method acquired by the third acquisition unit;
the third calling unit is used for calling the third configurable interface file;
a first determining unit, configured to determine, after the third calling unit calls the third configurable interface file, a data application style corresponding to the data name from the data application module;
and the display unit is used for displaying the processed data to be processed based on the data application style after the first determination unit determines the data application style.
Optionally, the apparatus further comprises:
a fourth obtaining unit, configured to obtain, before the second obtaining unit obtains the data to be processed from the data storage module, classification data components included in the data storage module, the data processing module, and the data application module, respectively, where the classification data components are used to record classified data in different functional modules;
the recording unit is configured to record the data name in the classified data component acquired by the fourth acquiring unit in the first configurable interface file, the second configurable interface file, and the third configurable interface file, where the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
Optionally, the apparatus further comprises:
the monitoring unit is used for monitoring the classified data components in real time;
the second determining unit is used for determining whether the classified data component is updated or not after the monitoring unit monitors the classified data component;
and the updating unit is used for updating the configurable interface file based on the updated classification data component when the classification data component is determined to be updated.
Optionally, the second obtaining unit includes:
the first acquisition module is used for acquiring the data name of the data to be processed;
a determining module, configured to determine whether the data name exists in the first configurable interface file after the name of the to-be-processed data acquired by the first acquiring module;
and the second acquisition module is used for acquiring the data to be processed corresponding to the data name from the data storage module when the determination module determines that the data name exists in the first configurable interface file.
In a fourth aspect, the present invention further provides a storage medium including a stored program, where the apparatus on which the storage medium is located is controlled to execute the data processing method according to any one of the second aspects when the program runs.
In a fifth aspect, the present invention further provides a processor, configured to execute a program, where the program executes the method for processing data according to any one of the second aspects.
By the technical scheme, the technical scheme provided by the invention at least has the following advantages:
after a data processing request is obtained, the data to be processed corresponding to the data name is obtained from the data storage module by calling the first configurable interface file of the metadata control center, the processing method corresponding to the data to be processed corresponding to the data name is obtained from the data processing module by calling the second configurable interface file, the data application style corresponding to the data name is determined from the data application module by calling the third configurable interface file, and the processed data to be processed is displayed, compared with the prior art, the data processing method and the device add the corresponding data interface to the metadata control center according to the data layer specification in a data warehouse, control the expansion and maintenance of each component in each data layer through the data interface, and realize the data exchange between the metadata control center and the data layer, and the expandability and the maintainability of the data warehouse are further improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 illustrates a block diagram of the components of a raw data warehouse, as known in the art;
FIG. 2 is a schematic diagram of an architecture of a data warehouse provided by an embodiment of the present invention;
FIG. 3 is a flow chart of a method for processing data according to an embodiment of the present invention;
fig. 4 is a flowchart illustrating a second data processing method according to an embodiment of the present invention;
FIG. 5 illustrates a block diagram of components that may be used to provide a data warehouse in accordance with an embodiment of the present invention;
FIG. 6 is a block diagram illustrating an apparatus for processing data according to an embodiment of the present invention;
fig. 7 is a block diagram illustrating a second data processing apparatus according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In the prior art, in order to enable a data warehouse system to meet more user requirements and support more data processing and query processing, a new functional module needs to be added while an original functional module in the system is maintained, and a hierarchical policy method or a method of adding a high-performance component is generally adopted. As shown in fig. 1, an original data warehouse in the prior art includes a data management layer, a data layer a, a data layer B, and a data layer C, and each data layer includes several components. When a user needs a new application function I, a new functional component needs to be added into the original data warehouse, and the layering strategy is that a component a4 is added at a data A layer, a component B4 is added at a data B layer, and a component C4 is added at a data C layer, so as to realize corresponding operations which need to be executed on the implementation of the new function I by each data layer, and the expandability of the data warehouse is improved by modularizing the components of the system; the method for increasing the high-performance components is that a data D layer is added in the system, the components D1, D2 and D3 which can realize the application function I are added into the data D layer, the data D layer is placed in a new hardware device to run, and the expandability of the data warehouse is improved through the extensible software parallelism and the hardware parallelism.
However, when more application functions are required to be added subsequently, for example, application function ii, application function iii, application function iv, and the like, if the method of the hierarchical strategy is continuously adopted, the number of components in the data layer a, the data layer B, and the data layer C will be increased continuously, the coupling inside each layer is improved continuously, and the maintainability of the system is reduced; if the method of increasing high-performance components is continuously adopted, a platform with high performance needs to be established, the data layer A, the data layer B and the data layer C with lower performance of the original data warehouse can generate a barrier effect on the performance of the whole data warehouse, and in addition, the development and maintenance difficulty of the data warehouse is increased aiming at a plurality of hardware development software.
In order to solve the above problems, the embodiment of the present invention modifies a structure system of a conventional data warehouse to solve the problems of poor expandability and reduced maintainability of the conventional data warehouse. As shown in fig. 2, an architecture diagram of a data warehouse provided in an embodiment of the present invention is shown, where the data warehouse according to the embodiment of the present invention includes:
the data management layer is internally provided with a metadata control center, and the metadata control center performs data interaction with a plurality of different functional modules through a plurality of configurable interface files;
wherein each configurable interface file has a mapping relationship with one of the function modules, and the plurality of configurable interface files include: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module;
the data storage module is used for storing data;
the data processing module is used for recording a processing method corresponding to the data;
and the data application module is used for recording the application style of the processed data.
It should be noted that, in the embodiment of the present invention, a data warehouse includes three modules (a data storage module, a data processing module, and a data application module) as an example, but it should be understood that this description is not intended to limit that the metadata control center can only interact with the three modules, in practical application, one or at least two functional modules may be added, and at the same time, the third configurable interface file and the fourth configurable file … are added to establish an interaction relationship between the newly added functional module and the metadata control center through configurable files, so that compared with the prior art, data storage and processing are performed through the data warehouse structure shown in fig. 2, so that the data warehouse is easier to expand and maintain.
When the data module is expanded, different modules can be expanded, and the data storage capacity in the same module can also be expanded. For different modules, a data interface is added in the metadata control center while one module is added, the data interface performs data interaction with the newly added module, and specific interaction content is recorded in a configurable interface file (assumed as a fourth class configuration interface file) corresponding to the newly added data interface. For the same module to be expanded, only the data name of newly added data needs to be updated in the configurable interface file, the hardware storage address is modified, and the data can be synchronously updated in the configurable interface file. Therefore, the data warehouse provided by the embodiment of the invention is easy to expand and maintain.
In the above embodiments, reference is made to a first configurable interface file, a second configurable interface file, a third configurable interface file, and a fourth configurable interface file, where the first, second, and third … are merely used to distinguish data types of the metadata control center when interacting with the data module, and the configurable interface files all play the same role in the embodiments of the present invention, that is, record data names, data storage addresses, data volume sizes, modification times, and the like of data contained in different function modules.
An embodiment of the present invention provides a data processing method, where the method is applied to a data warehouse shown in fig. 2, and as shown in fig. 3, the method includes:
101. and acquiring a data processing request, wherein the data processing request comprises a data name of the data to be processed.
The data processing request acquired by the data warehouse may be, but is not limited to, a request for querying data, a request for analyzing data, or a request for updating data from a front end, and the receiving manner may be real-time receiving, so long as the front end sends a data processing request, the data processing request is immediately processed; or receiving in batches, the front end sending data processing requests, receiving the data processing requests in a batch at fixed time or fixed quantity, and the number of the data processing requests received in each batch is not limited.
For example, when a user queries, processes, and updates data based on a data warehouse, the user sends a data processing request to the data warehouse through a search engine, where the data request includes a data name of data to be processed, for example, when the user demands 1+1 data calculation processing, the data to be processed is 1, and an addition operation is performed.
102. And calling the first configurable interface file, and acquiring the data to be processed corresponding to the data name from the data storage module.
Referring to fig. 2, the data warehouse according to the embodiment of the present invention includes a data management layer, a plurality of configurable interface files, and a plurality of different function modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and a data name of data contained in the corresponding functional module is recorded in each configurable interface file.
And after receiving the data processing request, acquiring the data to be processed from the data storage module by calling a first configurable interface file of the metadata control center.
The type of the metadata control center can be technical metadata, and the metadata is used by a data engineer developing and managing a data warehouse and describes data related to development, management and maintenance of the data warehouse, wherein the data comprises data source information, data conversion description, a data warehouse model, data cleaning and updating rules, data mapping, access rights and the like; it may also be business metadata that serves management and business analysts, describes data from a business perspective, including business terms, what data is in the data warehouse, the location of the data, and the availability of the data, etc., to help business personnel better understand which data is available in the data warehouse and how it is used. In addition, the metadata control center comprises at least three configurable interface files, at least one configurable interface file which can be exchanged with the data warehouse, and at least one configurable interface file which can be exchanged with the data processing module, wherein the configurable interface files are uniquely associated with the function modules, the contents of the configurable interface files comprise, but are not limited to, packet names of classes, class names of interfaces, method names in the classes, parameters, return values and the like, and are used for guiding the exchange of data, and the mode for calling the interface files can be used for realizing the interface methods implicitly or realizing the interface methods explicitly.
103. And calling the second configurable interface file, and acquiring a processing method corresponding to the to-be-processed data corresponding to the data name from the data processing module.
The second configurable interface file can be called by adopting an implicit interface implementation method or an explicit interface implementation method, and the second interface file standardizes the operation of the data processing interface, guides the data interface to acquire corresponding data according to the performance of each component in the data processing module, or outputs the corresponding data to the meta-control center.
The data processing method included in the data processing module includes but is not limited to one or two or more of the following:
(1) the aggregation of data refers to simple aggregation based on specific needs (aggregation based on multidimensional data is present in a multidimensional data model), and may be summarized data such as total Pageviews, Visits, Unique Visitors and the like of a website, or average data such as avg.
(2) The method comprises the steps of constructing a multi-dimensional data model, and carrying out multi-angle and multi-level analysis application, wherein the specific processing method can be a selling star model and a snowflake model constructed based on time dimension, region dimension and the like, can also be cross query of each time dimension and region dimension, and can also be subdivision based on the time dimension and the region dimension.
(3) The business model is constructed based on certain data analysis and decision support, and can be one or more than two of a user evaluation model, an association recommendation model, an RFM analysis model and the like, or one or more than two of a linear programming model, an inventory model and the like supported by decision.
104. And processing the data to be processed according to the processing method.
The data processing method in step 103 is adopted to perform data processing on the data to be processed acquired in step 102, and specific data types include but are not limited to: bit, int, smallnt, tinyint, numeric, decimal, money, smallmoney, float, datetime, smalldatetime, curror, timetag, uniquedentifier, char, text, binary, image, and the like.
The data to be processed may be preprocessed before data processing, and the method includes but is not limited to: and checking the validity of the data and checking the integrity of the data. The specific treatment mode can be one or two or more of the following:
(1) the cleaning of the repeated data can be to directly delete the repeated data from the data storage table or the storage chain, or to reserve the deleted repeated data to another data table or the data chain while deleting the repeated data, so as to facilitate the subsequent searching of the repeated data;
(2) the cleaning of useless fields can comprise the following steps: the method is realized by adopting an ALTER TABLE field or directly by using an UPDATE statement, and a specific cleaning method or the adopted statement is not limited;
(3) the conversion of data may include, but is not limited to: conversion of data type, conversion of object name, conversion of data encoding, conversion of table structure, and the like.
In addition, the specific sequence executed by the data processing module may be that each data processing component simultaneously and respectively executes the corresponding processing task, or that each component executes the corresponding task once according to the execution steps of the processing task.
105. And calling the third configurable interface file, determining a data application style corresponding to the data name from the data application module, and displaying the processed data to be processed based on the data application style.
The third configurable interface file may be called by using an implicit implementation interface method or an explicit implementation interface method, and the third interface file standardizes the operation of the data application interface and instructs the data interface how to obtain corresponding data according to the performance of each component in the data application module.
The components of the data application style provided in the data module include, but are not limited to: query tool components, reporting tool components, OLAP components, data mining tool components, and the like. The form of the display result obtained specifically may be, but is not limited to, any one or more of the following:
(1) high performance MAPPs include, but are not limited to: full/incremental data loading, heterogeneous data source support, distributed parallel computing and the like;
(2) visual modeling includes, but is not limited to: drag-and-drop modeling, cross-data warehouse table association, data dictionaries, calculation fields, model filters, format conversion and the like;
(3) self-service analysis includes but is not limited to: intelligent question answering, perspective analysis, chart linkage, dynamic reference line, What-If analysis, personalized theme, measurement prediction and the like;
(4) data reporting includes, but is not limited to: offline data collection, approval and release, form customization and the like.
After a data processing request is obtained, acquiring data to be processed corresponding to a data name from a data storage module by calling a first configurable interface file of a metadata control center, acquiring a processing method corresponding to the data to be processed corresponding to the data name from a data processing module by calling a second configurable interface file, determining a data application style corresponding to the data name from a data application module by calling a third configurable interface file, and displaying the processed data to be processed, compared with the prior art, the data processing method provided by the embodiment of the invention adds a corresponding data interface to the metadata control center according to the data layer specification in a data warehouse, controls the expansion and maintenance of each component in each data layer through the data interface, and realizes the data exchange between the metadata control center and the data layer, and the expandability and the maintainability of the data warehouse are further improved.
An embodiment of the present invention further provides another data processing method, as shown in fig. 4, the method includes:
201. and adding a data management layer in the data warehouse, and constructing a metadata control center in the data management layer.
The data warehouse can be suitable for computer environments of different hosts, operating systems, heterogeneous networks, heterogeneous data warehouses and the like in computer application systems of enterprises, and the data management layer is used as the core of the whole system and comprises functions of acquiring, managing, storing and the like of metadata and various service interfaces for controlling the metadata. The system completes the brain functions of receiving, distributing and managing tasks in the whole process of the system, can realize transaction management, namely, the capability of providing correct and simultaneous access to a data warehouse for a plurality of users, can also realize access control, namely, the capability of limiting unauthorized users to access data, the capability of detecting the validity of the data and the like without being limited to other functions of the functions, and the data management layer can be composed of the following three parts: data Description Language (DDL), Data Manipulation Language (DML), and Data warehouse routine, but the present invention is not limited to the above configuration.
The metadata control center may be built using the OIM standard including, but not limited to, MDC and the OMG CWM standard. The meta-control center can store and maintain the meta-data in the meta-data warehouse, and can also realize the message transmission among data warehouse modeling tools, data acquisition tools, front-end tools and the like, coordinate the work among all modules and tools and the like.
202. And adding a corresponding data interface for the metadata control center according to the data layer specification, and establishing communication connection between the metadata control center and a data storage module, a data processing module and a data application module respectively based on the data interface.
After the metadata control center is formed through various ways, in order to enable the metadata control center to establish a connection with the data storage module, the data processing module and the data application module, so that the metadata control center can perform operations related to content change such as content addition, deletion and update on content in the modules and conventional metadata operations such as metadata content retrieval, query, placement and organization, data interfaces corresponding to the data storage module, the data processing module and the data application module are added to the metadata control center respectively. Wherein the setting needs to be done according to a layer specification describing the functionality of the corresponding layer, which is greater than the sum of the functionalities of each classified data component in the layer.
After the metadata control center establishes a connection relationship with the module, the module may adopt any one of the following management modes:
(1) the system management mode is data warehouse-oriented and is completed by professional personnel of a data warehouse management system, a data user has a use right but does not have an operation right of metadata, the metadata of a data set newly generated in a data application project is also transmitted to a data warehouse manager by an application system, and then the data warehouse manager manages the metadata in a unified way;
(2) the user management mode is oriented to application projects, namely, the change information of the metadata of the data application of some data users is allowed to be directly fed back to the metadata warehouse, so that the dynamic update of the metadata and the timely capture and writing of the metadata of the newly generated data set into a metadata file can be ensured;
(3) a combination of a system management mode and a user management mode.
203. And respectively acquiring classified data components contained in the data storage module, the data processing module and the data application module, wherein the classified data components are used for recording classified data in different functional modules.
The data storage module may include, but is not limited to, any one or more than two of a source data component, a comprehensive data component, a historical data component, a current data component, and the like. The data processing module may include, but is not limited to, any one or more of an algorithm component, a tool component, a modeling component, and an analysis component. The data application module may include, but is not limited to, any one or more than two of a presentation application component, a query application component, a report application component, and the like. The specific acquisition mode is not specified, and may be a single acquisition in sequence, or a plurality of acquisitions in a time-division manner.
In addition, after the classification data component is obtained, the classification data component can be monitored in real time to determine whether the classification data component is updated. The update form of the specific occurrence of the classified data component may be, but is not limited to: deleting an existing taxonomy data component, adding a new taxonomy data component, or modifying an existing taxonomy data component. The specific way of updating the data components may be immediate updating, or on-time updating according to a preset time, or batch updating according to the classification data components that need to be updated.
If the classified data component is updated, the configurable interface file is updated based on the updated classified data component, the configurable file is updated, and the freshness and the accuracy of data storage and data processing in the data warehouse system can be ensured. Specifically, the configurable file may be updated by adding a trigger during data source design, making an effective update in the process of data addition, modification or deletion, or adding some audit tables, recording and tracking the operation details of data in the process of data addition, deletion and modification, or adding a self-increment column, a time and date stamp, etc.
204. And recording the data names in the classified data components in the first configurable interface file, the second configurable interface file and the third configurable interface file, wherein the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
First, before the name of the data to be processed is obtained, data names corresponding to the data storage module, the data processing module and the data application function module are recorded in the first configurable interface file, the second configurable interface file and the third configurable interface file. The phase of the recorded data may be during the design and modeling process, during the data transformation, cleaning and filtering process, during the data migration process, and metadata may be obtained from data storage software and front-end presentation tools. Wherein the data name is a unique identification of the data.
And after the name of the data to be processed is obtained, determining whether the name of the data exists in the first configurable interface file, wherein the determination mode can be a mode of traversing a data table or a data chain, and a specific mode is not required. And when the data name exists in the first configurable interface file, acquiring the data to be processed corresponding to the data name from the data storage module.
The data processing method provided by the embodiment of the invention acquires the data to be processed corresponding to the data name from the data storage module by calling the first configurable interface file of the metadata control center after acquiring the data processing request, acquires the processing method corresponding to the data to be processed corresponding to the data name from the data processing module by calling the second configurable interface file, determines the data application style corresponding to the data name from the data application module by calling the third configurable interface file, and displays the processed data to be processed, compared with the prior art, the data processing method provided by the embodiment of the invention adds the corresponding data interface to the metadata control center according to the data layer specification in a data warehouse, controls the expansion and maintenance of each component in each data layer through the data interface, and realizes the data exchange between the metadata control center and the data layer, and the expandability and the maintainability of the data warehouse are further improved, and the data transmission and exchange among the data function modules are facilitated.
Further, as an implementation of the method shown in fig. 3 and fig. 4, another embodiment of the present invention further provides a data processing apparatus. The embodiment of the apparatus corresponds to the embodiment of the method, and for convenience of reading, details in the embodiment of the apparatus are not repeated one by one, but it should be clear that the apparatus in the embodiment can correspondingly implement all the contents in the embodiment of the method.
Further, an embodiment of the present invention further provides a data warehouse, as shown in fig. 5, including:
a data management layer 31, in which a metadata control center is disposed, and the metadata control center performs data interaction with a plurality of different functional modules through a plurality of configurable interface files;
wherein each configurable interface file has a mapping relationship with one of the function modules, and the plurality of configurable interface files include: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: data storage module, data processing module and data application module
The data storage module 33 is used for storing data;
the data processing module 34 is configured to record a processing method corresponding to the data;
the data application module 35 is configured to record an application style of the processed data.
An embodiment of the present invention provides a data processing apparatus, as shown in fig. 6, where the apparatus is applied to a data warehouse, where the data warehouse includes a data management layer, a plurality of configurable interface files, and a plurality of different functional modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and each configurable interface file records a data name of data contained in the corresponding functional module; the method comprises the following steps:
a first obtaining unit 401, configured to obtain a data processing request, where the data processing request includes a data name of data to be processed;
a first calling unit 402, configured to call the first configurable interface file;
a second obtaining unit 403, configured to obtain, after the first calling unit 402 calls the first configurable interface file, to-be-processed data corresponding to the data name from the data storage module;
a second calling unit 404, configured to call the second configurable interface file;
a third obtaining unit 405, configured to obtain, from the data processing module, a processing method corresponding to the to-be-processed data corresponding to the data name after the second configurable interface file is called by the second calling unit;
a processing unit 406, configured to process the to-be-processed data according to the processing method obtained by the third obtaining unit 405;
a third calling unit 407, configured to call the third configurable interface file;
a first determining unit 408, configured to determine, after the third calling unit 407 calls the third configurable interface file, a data application style corresponding to the data name from the data application module;
a presentation unit 409, configured to, after the first determination unit 408 determines the data application style, present the processed data to be processed based on the data application style.
Further, as shown in fig. 7, the apparatus further includes:
a fourth obtaining unit 410, configured to obtain, before the second obtaining unit 403 obtains the data to be processed from the data storage module, classified data components included in the data storage module, the data processing module, and the data application module, respectively, where the classified data components are used to record the classified data in different functional modules;
a recording unit 411, configured to record the data name in the classified data component acquired by the fourth acquiring unit 410 in the first configurable interface file, the second configurable interface file, and the third configurable interface file, where the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
Further, as shown in fig. 7, the apparatus further includes:
a monitoring unit 412, configured to perform real-time monitoring on the classified data components;
a second determining unit 413, configured to determine whether there is an update in the classified data component after the monitoring unit monitors the classified data component;
an updating unit 414, configured to update the configurable interface file based on the updated classification data component when it is determined that the classification data component has an update.
Further, as shown in fig. 7, the second obtaining unit 403 includes:
a first obtaining module 4031, configured to obtain a data name of data to be processed;
a determining module 4032, configured to determine, after the name of the to-be-processed data acquired by the first acquiring module, whether the data name exists in the first configurable interface file;
a second obtaining module 4033, when the determining module determines that the data name exists in the first configurable interface file, the user obtains the to-be-processed data corresponding to the data name from the data storage module.
The data processing device comprises a processor and a memory, the first acquiring unit 401, the first calling unit 402, the second acquiring unit 403, the second calling unit 404, the third acquiring unit 405, the processing unit 406, the third calling unit 407, the first determining unit 408, the displaying unit 409 and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. One or more kernels can be set, and the problems of poor expandability and low maintainability of the traditional data warehouse are solved by adjusting kernel parameters.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a storage medium on which a program is stored, the program implementing a processing method of the data when executed by a processor.
The embodiment of the invention provides a processor, which is used for running a program, wherein the program executes a processing method of data when running.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein the processor executes the program and realizes the following steps: the data warehouse comprises a data management layer, a plurality of configurable interface files and a plurality of different functional modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and each configurable interface file records a data name of data contained in the corresponding functional module;
acquiring a data processing request, wherein the data processing request comprises a data name of data to be processed;
calling the first configurable interface file, and acquiring data to be processed corresponding to the data name from the data storage module;
calling the second configurable interface file, and acquiring a processing method corresponding to the to-be-processed data corresponding to the data name from the data processing module;
processing the data to be processed according to the acquired processing method;
and calling the third configurable interface file, determining a data application style corresponding to the data name from the data application module, and displaying the processed data to be processed based on the data application style.
Optionally, before acquiring the data to be processed from the data storage module by calling the first configurable interface file of the metadata control center, the method further includes:
respectively acquiring classified data components contained in the data storage module, the data processing module and the data application module, wherein the classified data components are used for recording classified data in different functional modules;
and recording the data names in the classified data components in the first configurable interface file, the second configurable interface file and the third configurable interface file, wherein the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
Optionally, the method further includes:
monitoring the classified data assembly in real time, and determining whether the classified data assembly is updated;
and if so, updating the configurable interface file based on the updated classification data component.
Optionally, based on the first configurable interface file of the metadata control center, acquiring the data to be processed from the data storage module includes:
acquiring a data name of data to be processed, and determining whether the data name exists in the first configurable interface file;
and if the data exists, acquiring the data to be processed corresponding to the data name from the data storage module.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform program code for initializing the following method steps when executed on a data processing device: acquiring a data processing request, wherein the data processing request comprises a data name of data to be processed;
calling the first configurable interface file, and acquiring data to be processed corresponding to the data name from the data storage module;
calling the second configurable interface file, and acquiring a processing method corresponding to the to-be-processed data corresponding to the data name from the data processing module;
processing the data to be processed according to the acquired processing method;
and calling the third configurable interface file, determining a data application style corresponding to the data name from the data application module, and displaying the processed data to be processed based on the data application style.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (11)
1. A data warehouse, comprising:
the data management layer is internally provided with a metadata control center, and the metadata control center performs data interaction with a plurality of different functional modules through a plurality of configurable interface files;
wherein each configurable interface file has a mapping relationship with one of the function modules, and the plurality of configurable interface files include: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module;
the data storage module is used for storing data;
the data processing module is used for recording a processing method corresponding to the data;
and the data application module is used for recording the application style of the processed data.
2. The data processing method is applied to a data warehouse, and is characterized in that the data warehouse comprises a data management layer, a plurality of configurable interface files and a plurality of different functional modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and each configurable interface file records a data name of data contained in the corresponding functional module;
acquiring a data processing request, wherein the data processing request comprises a data name of data to be processed;
calling the first configurable interface file, and acquiring data to be processed corresponding to the data name from the data storage module;
calling the second configurable interface file, and acquiring a processing method corresponding to the to-be-processed data corresponding to the data name from the data processing module;
processing the data to be processed according to the acquired processing method;
and calling the third configurable interface file, determining a data application style corresponding to the data name from the data application module, and displaying the processed data to be processed based on the data application style.
3. The method of claim 2, wherein prior to retrieving the data to be processed from the data storage module by calling the first configurable interface file of the metadata control center, the method further comprises:
respectively acquiring classified data components contained in the data storage module, the data processing module and the data application module, wherein the classified data components are used for recording classified data in different functional modules;
and recording the data names in the classified data components in the first configurable interface file, the second configurable interface file and the third configurable interface file, wherein the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
4. The method of claim 3, further comprising:
monitoring the classified data assembly in real time, and determining whether the classified data assembly is updated;
and if so, updating the configurable interface file based on the updated classification data component.
5. The method of claim 2, wherein obtaining the data to be processed from the data storage module based on the first configurable interface file of the metadata control center comprises:
acquiring a data name of data to be processed, and determining whether the data name exists in the first configurable interface file;
and if the data exists, acquiring the data to be processed corresponding to the data name from the data storage module.
6. The data processing device is applied to a data warehouse, and is characterized in that the data warehouse comprises a data management layer, a plurality of configurable interface files and a plurality of different functional modules; a metadata control center is arranged in the data management layer, and the plurality of configurable interface files comprise: the interface file comprises a first configurable interface file, a second configurable interface file and a third configurable interface file; the plurality of different functional modules includes: the system comprises a data storage module, a data processing module and a data application module; the metadata control center performs data interaction with a plurality of different functional modules in the data warehouse through a plurality of configurable interface files, each configurable interface file has a mapping relation with one functional module in the data warehouse, and each configurable interface file records a data name of data contained in the corresponding functional module; the device comprises:
the device comprises a first acquisition unit, a second acquisition unit and a processing unit, wherein the first acquisition unit is used for acquiring a data processing request which comprises a data name of data to be processed;
the first calling unit is used for calling the first configurable interface file;
the second obtaining unit is used for obtaining the data to be processed corresponding to the data name from the data storage module after the first calling unit calls the first configurable interface file;
the second calling unit is used for calling the second configurable interface file;
a third obtaining unit, configured to obtain, from the data processing module, a processing method corresponding to the to-be-processed data corresponding to the data name after the second configurable interface file is called by the second calling unit;
the processing unit is used for processing the data to be processed according to the processing method acquired by the third acquisition unit;
the third calling unit is used for calling the third configurable interface file;
a first determining unit, configured to determine, after the third calling unit calls the third configurable interface file, a data application style corresponding to the data name from the data application module;
and the display unit is used for displaying the processed data to be processed based on the data application style after the first determination unit determines the data application style.
7. The apparatus of claim 6, further comprising:
a fourth obtaining unit, configured to obtain, before the second obtaining unit obtains the data to be processed from the data storage module, classification data components included in the data storage module, the data processing module, and the data application module, respectively, where the classification data components are used to record classified data in different functional modules;
the recording unit is configured to record the data name in the classified data component acquired by the fourth acquiring unit in the first configurable interface file, the second configurable interface file, and the third configurable interface file, where the data storage module corresponds to the first configurable interface file, the data processing module corresponds to the second configurable interface file, and the data application module corresponds to the third configurable interface file.
8. The apparatus of claim 7, further comprising:
the monitoring unit is used for monitoring the classified data components in real time;
the second determining unit is used for determining whether the classified data component is updated or not after the monitoring unit monitors the classified data component;
and the updating unit is used for updating the configurable interface file based on the updated classification data component when the classification data component is determined to be updated.
9. The apparatus of claim 6, wherein the second obtaining unit comprises:
the first acquisition module is used for acquiring the data name of the data to be processed;
a determining module, configured to determine whether the data name exists in the first configurable interface file after the name of the to-be-processed data acquired by the first acquiring module;
and the second acquisition module is used for acquiring the data to be processed corresponding to the data name from the data storage module when the determination module determines that the data name exists in the first configurable interface file.
10. A storage medium, characterized in that the storage medium comprises a stored program, wherein when the program runs, a device where the storage medium is located is controlled to execute the data processing method of any one of claims 2 to 5.
11. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the data processing method according to any one of claims 2 to 5 when the program is run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710764963.2A CN110019440B (en) | 2017-08-30 | 2017-08-30 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710764963.2A CN110019440B (en) | 2017-08-30 | 2017-08-30 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019440A CN110019440A (en) | 2019-07-16 |
| CN110019440B true CN110019440B (en) | 2021-06-08 |
Family
ID=67186162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710764963.2A Active CN110019440B (en) | 2017-08-30 | 2017-08-30 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019440B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143453A (en) * | 2019-12-24 | 2020-05-12 | 济宁昊瀛天成大数据信息技术有限公司 | Parallel processing method for industrial big data heterogeneous data |
| CN111858467B (en) * | 2020-07-22 | 2024-05-03 | 平安证券股份有限公司 | File data processing method, device, equipment and medium based on artificial intelligence |
| CN115167891B (en) * | 2022-07-07 | 2023-09-05 | 北京蓝天航空科技股份有限公司 | Data updating method, device and equipment of interface control file and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572456A (en) * | 2014-12-30 | 2015-04-29 | 北京锐安科技有限公司 | Data collection method and device |
| CN105893063A (en) * | 2016-06-13 | 2016-08-24 | 积成电子股份有限公司 | Application program interface based on XML format configuration file and construction method of application program interface |
| CN106202361A (en) * | 2016-07-07 | 2016-12-07 | 上海携程商务有限公司 | Data query method and system |
| CN106649734A (en) * | 2016-12-23 | 2017-05-10 | 中国科学院南京地理与湖泊研究所 | Water quality target management method and system with configurable model |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007047349A2 (en) * | 2005-10-14 | 2007-04-26 | Symantec Operating Corporation | Technique for improving scalability and portability of a storage management system |
| IN2013CH05996A (en) * | 2013-12-20 | 2015-06-26 | Infosys Ltd | |
| CN103902671B (en) * | 2014-03-19 | 2018-04-13 | 北京科技大学 | A kind of dynamic integrity method and system of isomerous multi-source data |
| US10372685B2 (en) * | 2014-03-31 | 2019-08-06 | Amazon Technologies, Inc. | Scalable file storage service |
| CN107533570B (en) * | 2015-10-23 | 2020-11-03 | 甲骨文国际公司 | System and method for automatically inferring cube schema from tabular data |
| CN105867932B (en) * | 2016-04-07 | 2018-12-18 | 浙江工业大学 | Game cloud design method based on metadata reflex chain |
-
2017
- 2017-08-30 CN CN201710764963.2A patent/CN110019440B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572456A (en) * | 2014-12-30 | 2015-04-29 | 北京锐安科技有限公司 | Data collection method and device |
| CN105893063A (en) * | 2016-06-13 | 2016-08-24 | 积成电子股份有限公司 | Application program interface based on XML format configuration file and construction method of application program interface |
| CN106202361A (en) * | 2016-07-07 | 2016-12-07 | 上海携程商务有限公司 | Data query method and system |
| CN106649734A (en) * | 2016-12-23 | 2017-05-10 | 中国科学院南京地理与湖泊研究所 | Water quality target management method and system with configurable model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019440A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220350819A1 (en) | System and method for improved performance in a multidimensional database environment | |
| US20230084389A1 (en) | System and method for providing bottom-up aggregation in a multidimensional database environment | |
| US8555018B1 (en) | Techniques for storing data | |
| US10235430B2 (en) | Systems, methods, and apparatuses for detecting activity patterns | |
| CA2824319C (en) | Column smart mechanism for column based database | |
| US20220253433A1 (en) | Maintaining data stream history for generating materialized views | |
| JP4222947B2 (en) | Method, program, and system for representing multimedia content management objects | |
| CN110431545A (en) | Inquiry is executed for structural data and unstructured data | |
| US10909114B1 (en) | Predicting partitions of a database table for processing a database query | |
| CN112580914A (en) | Method and device for realizing enterprise-level data middling platform system for collecting multi-source data | |
| US11429636B2 (en) | Smart elastic scaling based on application scenarios | |
| CN110297869B (en) | AI data warehouse platform and operation method | |
| CN110019440B (en) | Data processing method and device | |
| US10157213B1 (en) | Data processing with streaming data | |
| US11645247B2 (en) | Ingestion of master data from multiple applications | |
| EP3958128B1 (en) | Customized processing of sensor data | |
| US20220058069A1 (en) | Interface for processing sensor data with hyperscale services | |
| US20230418824A1 (en) | Workload-aware column inprints | |
| CN114730312A (en) | Managed materialized views created from heterogeneous data sources | |
| US10095738B1 (en) | Dynamic assignment of logical partitions according to query predicate evaluations | |
| Soldatos et al. | A reference architecture model for big data systems in the finance sector | |
| US20220138343A1 (en) | Method of determining data set membership and delivery | |
| US11615061B1 (en) | Evaluating workload for database migration recommendations | |
| Mayer et al. | Variety management for big data | |
| US11868349B2 (en) | Row secure table plan generation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |