CN116450715A - Information integration data processing method, system, electronic equipment and storage medium - Google Patents
Information integration data processing method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116450715A CN116450715A CN202211341734.7A CN202211341734A CN116450715A CN 116450715 A CN116450715 A CN 116450715A CN 202211341734 A CN202211341734 A CN 202211341734A CN 116450715 A CN116450715 A CN 116450715A
- Authority
- CN
- China
- Prior art keywords
- data
- resources
- resource
- resource pool
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000010354 integration Effects 0.000 title claims abstract description 28
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000013499 data model Methods 0.000 claims abstract description 49
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000004891 communication Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 16
- 238000006243 chemical reaction Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 claims description 11
- 238000013500 data storage Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000007726 management method Methods 0.000 abstract description 11
- 238000005065 mining Methods 0.000 abstract description 6
- 230000009286 beneficial effect Effects 0.000 abstract description 4
- 238000010276 construction Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an information integration data processing method, an information integration data processing system, electronic equipment and a storage medium, wherein the method comprises the following steps: storing data resources of at least one service information system into a data resource pool; generating a data catalog of the data resource pool according to a preset data model; determining a data access interface according to the data directory and the data access request; accessing a target data resource in the data resource pool based on the data access interface. The embodiment of the invention realizes the integration of the data resources of different service information systems, shares the data resources of different service information systems, is beneficial to realizing the mining of the data resources, can improve the utilization efficiency of the data resources and improves the scientific management degree of the service system information.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to an information integration data processing method, system, electronic device, and storage medium.
Background
The information system is an important foundation stone for the development and utilization of business information resources, and the data resources are organized, operated and managed through the information system, so that the integration of all elements can be realized. The data resource is used as a foundation for information system construction, the information resource is required to be integrated and applied, and the integration of the data resource gradually becomes an important determining factor for informatization construction. Currently, integrated processing of data resources within various information systems generally suffers from the following disadvantages:
1. the information systems are independently built and are not interconnected with the outside, so that most of the information systems cannot be mutually compatible, the information systems lack uniform platform standards, data sharing among the information systems cannot be realized, standardization and scientization of informatization management are greatly influenced, and coordinated development of informatization construction is severely restricted.
2. The sharing degree of the data resources of the information systems is not high, the data resources are deposited in the information systems, and the data resources are lack of effective excavation and utilization, so that the utilization rate of the data resources is low, and scientific data resource management is lacked in the information systems.
Disclosure of Invention
The invention provides an information integration data processing method, an information integration data processing system, electronic equipment and a storage medium, which are used for realizing the integration of data resources of different information systems, realizing the sharing of the data resources among different systems, being beneficial to realizing the mining of the data resources, improving the utilization efficiency of the data resources and improving the scientific management degree of information.
According to an aspect of the present invention, there is provided an information integration data processing method, wherein the method includes:
storing data resources of at least one service information system into a data resource pool;
generating a data catalog of the data resource pool according to a preset data model;
determining a data access interface according to the data directory and the data access request;
accessing a target data resource in the data resource pool based on the data access interface.
According to another aspect of the present invention, there is provided an information-integration data processing system, wherein the system includes:
the data storage module is used for storing the data resources of at least one business information system into a data resource pool;
the catalog generation module is used for generating a data catalog of the data resource pool according to a preset data model;
the interface determining module is used for determining a data access interface according to the data catalogue and the data access request;
and the resource access module is used for accessing the target data resources in the data resource pool based on the data access interface.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the information-integration data-processing method of any one of the embodiments of the invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute the information-integration data processing method according to any one of the embodiments of the present invention.
According to the technical scheme, the data resources of different business information systems are stored in the data resource pool, the data directory is generated for the data resource pool according to the preset data model, the data access interface is determined according to the data target and the data access request, the access of the target data resources in the data resource pool is realized according to the call of the data access interface, the data resource integration of the different business information systems is realized, the data resource sharing among the different systems is realized, the data resource mining is facilitated, the utilization efficiency of the data resources can be improved, and the scientific management degree of the system information is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an information integration data processing method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a method for processing information integration data according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of an information-integration data processing system according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing an information integration data processing method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of an information integration data processing method according to an embodiment of the present invention, where the method may be implemented by an information integration data processing system, and the information integration data processing system may be implemented in hardware and/or software, and the information integration data processing system may be configured in a server or a server cluster. As shown in fig. 1, the method includes:
step 110, data resources of at least one service information system are stored in a data resource pool.
The service information system can be an informatization system in each professional field, the development mode and the software and hardware architecture of the service information system can be different, and the service information system can respectively belong to different units. In some embodiments, the service information system may include a conventional relational database system, a file system, a network messaging system, a distributed database, a distributed file system, etc., and it is understood that the development voice and the software and hardware architecture of the service information system may be different. The data resources may be valuable data generated by the service information system during operation, the data resources may include automated data and non-automated data, the data resources may include different data types according to different functions implemented by the service information system, and the data resources may include structured data and unstructured data.
In the embodiment of the invention, communication connection can be established with different service information systems, data resources of the different service information systems can be acquired in the communication connection, and the acquired data resources can be stored in a data resource pool. The data resource pool can be a shared set of data resources, the data resource pool can be a data warehouse or a data lake, the data resource pool can be built by adopting a distributed architecture, different business information systems can be connected, the data islands of the different business information systems are eliminated, and the data resource pool can support structured data, semi-structured data and unstructured data. In some embodiments, a data resource pool can be built through a Hadoop distributed cluster, and data resources of each service information system can be stored into the built data resource pool after being carded.
And 120, generating a data catalog of the data resource pool according to the preset data model.
The preset data model can be generated based on data feature abstraction of professional field information, can comprise a set of concepts and definitions of data, and can be composed of a data structure, data operation and data constraint. It can be appreciated that the number of the preset data models can be one or more, and the preset data models can be respectively generated in an abstract mode according to the data characteristics of different data in different service information systems. The data directory may be information for managing the data resource pool, and may include one or more directory entries therein, where each directory entry may correspond to a type of data resource, and the data directory may be used for managing, analyzing, and navigating the data resource. The data directory may include one or more of a local directory, a global directory, and a conditional shared directory, which may be divided according to storage locations.
In an embodiment of the present invention, one or more preset data models configured in advance may be read, and the preset data models may be generated by an administrator based on data features of different service information systems. The data resources in the data resource pool can be combed according to the preset data model, and the data resources are stored as directory entries into a data directory according to different preset data models, and it can be understood that the data directory can be updated in a timing or quantitative manner after being generated. In one embodiment, when a new data resource is stored in the data resource pool, a directory entry for the data resource may be registered in the data directory. In another embodiment, the preset data model may be defined according to service attribution, and data resources corresponding to different data assets may be queried in the data resource pool according to the preset data model, and the data assets may be organized into a data directory.
Step 130, determining a data access interface according to the data directory and the data access request.
The data access request may be a request of a user to access a data resource, the data access request may be from different service information systems, the data access request may be generated according to an access format specified by a data resource pool or may be generated according to an access format of a service information system, and in some embodiments, the data access request acquired by the data resource pool may be unified in format so as to be compatible with communication modes of different service information systems. The data access interface can be a function interface for accessing the data in the data resource pool, the data access interface can be configured with access rules of the data resources, the data access interface can be set according to the types of the data resources in the data resource pool, and it can be understood that the data access interfaces of different data resources can be different.
In the embodiment of the invention, the data access request piece can be monitored, after the data access request is acquired, the data resource to be accessed can be queried in the data catalog according to the data access request, and the matched data access interface can be acquired according to the data resource. It can be understood that the number of the data access requests can be one or more, and the data resources corresponding to each data access request can be queried in the data directory at the same time, so as to improve the data access efficiency. In some embodiments, the resource name of the data resource may be stored in the data directory, and the resource name in the data access request may be matched with the data directory, so as to query the data resource to be accessed. In some embodiments, if the data resource to be accessed is not queried in the data directory, the data query error request can be fed back to the data source of the data access request.
Step 140, accessing the target data resource in the data resource pool based on the data access interface.
The target data resource may be a data resource to be queried by a data access request, and the data access request may be different from a data source of the target data resource, for example, the data access request is from a first service information system, the target data resource is from a second service information system, and the data resource pool may implement data sharing between different service information systems.
In the implementation of the present invention, the data access interface may be invoked, so that the target data resource in the data resource pool is accessed through the data access rule configured in the data access interface, and it may be understood that when the target data resource is read, the target data may be processed according to the content of the data access request, where the processing includes, but is not limited to, data update, data deletion, and data addition. In some embodiments, the data resource access rules of different data structures are configured in different data access interfaces, and the access of the data resources of different data structures can be realized by calling the data access interfaces. In other embodiments, the data access interface is configured with data resource access rules of different file systems, and the data resource pool is formed by different data storage systems, so that access to traditional relational database data, file system data, network message system data, distributed message queue data, distributed database data and distributed file system data can be realized by calling the data access interface, thereby realizing integrated management of data resources and improving compatibility of different service information systems.
According to the embodiment of the invention, the data resources of different service information systems are stored in the data resource pool, the data directory is generated for the data resource pool according to the preset data model, the data access interface is determined according to the data target and the data access request, the access of the target data resources in the data resource pool is realized according to the call of the data access interface, the data resource integration of the different service information systems is realized, the data resource sharing among the different systems is realized, the data resource mining is facilitated, the utilization efficiency of the data resources can be improved, and the scientific management degree of the system information is improved.
Example two
Fig. 2 is a flowchart of an information integration data processing method according to a second embodiment of the present invention, where the embodiment of the present invention is embodied on the basis of the above embodiment of the present invention, and referring to fig. 2, the embodiment of the present invention specifically includes the following steps:
step 210, reading data configuration information of the service information system.
The data configuration information may be preset parameter information for accessing different service information systems, the data configuration information may include authority authentication information, communication protocols and the like of the different service information systems, and the data configuration information may exist in the form of an encrypted file.
In the embodiment of the invention, the data configuration information of different service information systems can be read, and the data configuration information of different service information systems can be positioned at the same storage position or different storage positions. In some embodiments, the data configuration files of different service information systems may be stored in the same configuration file, and different data configuration information may be read in one configuration file. In other embodiments, the data configuration information may be stored in a hardware key of a different user, and the data configuration information of the service information system may be read after the hardware key is accessed.
Step 220, determining a data access channel in a preset configuration file according to the data configuration information.
The preset configuration file may be a data file including access rules of different service information systems, and may be configured in a server or a server cluster for executing the method of the present invention, where the preset configuration file includes one or more configuration items, and each configuration item may correspond to an access rule of a service information system, where it may be understood that the access rule may include a data communication protocol. The data access channel can be a data transmission channel constructed with the service information system, and the data access channel can specifically comprise a nested word data transmission channel, a hypertext transmission protocol channel, a streaming media transmission protocol channel and the like.
In the embodiment of the invention, the preset configuration file can be loaded into the memory, one or more construction rules of the data access channel can be searched in the preset configuration file according to the read data configuration information, and the data access channel can be constructed according to the construction rules of the data access channel and the data configuration information and service information system. It may be understood that the manner of determining the data access channel according to the data configuration information in the preset configuration file may include searching for an identifier of the service information system corresponding to the data configuration information, and searching for the data access channel stored in association with the identifier. In some embodiments, the data access channel is already pre-established, and the access right of the data access channel can be acquired according to the acquired data configuration information.
Step 230, the data resources of the business information system are read in the data access channel, and the data resources are stored in the data resource pool.
In the embodiment of the invention, the data resource can be acquired in the service information system through the data access channel, and it can be understood that the data resource can be actively pushed by the service information system through the data access channel or pulled by the execution device of the method of the invention through the data access channel in the service information system, and can be stored in the data resource pool after the data resource is acquired. In some embodiments, the access authority of the data resource can be set in the service information system, so that the control of the data resource sharing is realized, and the data resource can be acquired through the data access channel only after the sharing authority is set in the service information system, so that the safety of the data resource is improved on the premise of ensuring the data sharing utilization.
Step 240, reading a data model definition of a preset data model, wherein the data model definition at least comprises a data structure.
Wherein the data model definition may include definition information describing different data resources, the data model definition may be composed of at least a data structure, the data structure may be used to describe static characteristics of the data resources, may include types, contents, properties of the data resources, and associations between data, etc., and the associations may include hierarchical structures and relational structures between different data resources, etc. The data model definition may further include information such as data operations and data constraints, where the data operations may describe dynamic characteristics of the data resources, may include insertion, modification, deletion, and query of data, and the data constraints may include integrity rules of the data resources, may be constraint and storage rules of the data resources and data resource association, for example, the value of the data resources is not negative, or the data type of the data resources is an integer type.
In the embodiment of the invention, corresponding data model definitions can be extracted for each preset data model, the data model definitions can be generated by an administrator according to the data characteristic abstraction of the service information system, and each data model definition can be composed of elements such as a data structure, data operation, data constraint and the like. It will be appreciated that the data model definitions may exist in the form of configuration files.
Step 250, searching the data resource pool for the data resources corresponding to the data model definitions.
In particular, the data resources may be categorized within the data resource pool according to data model definitions, each of which may have a respective corresponding one or more data resources, and the data resources within each category may satisfy element information of the corresponding data model definition, including, but not limited to, data structures, data operations, and data constraints. In one embodiment, data structures defined by each data model may be extracted, and corresponding data resources are queried within the data resource pool according to the extracted data structures, respectively.
Step 260, determining at least one association hierarchical relationship of the data resources according to the data structure for each data resource.
The association hierarchical relationship may be information reflecting an interrelation relationship between different data resources, and the association hierarchical relationship may reflect whether an association exists between at least two data resources.
In the embodiment of the invention, the data structure in the data model definition of the data resources can be extracted aiming at the data resources belonging to different data model definitions, and the association hierarchical relation with at least one other data resource can be determined according to the static characteristics in the data structure. For example, when a data resource may be associated with data resource a according to a data structure, identification information may be created for the data resource and data resource a to represent an association hierarchy.
Step 270, generating a data catalog corresponding to each data resource according to each association hierarchy.
In the embodiment of the invention, the hierarchical architecture among different data resources can be determined according to the determined association hierarchical relationship, the data resources can be converted into the directory entry according to the hierarchical architecture, and the directory entry is stored as a data directory according to the hierarchical architecture.
Step 280, extracting a data characteristic identifier in the data access request.
The data characteristic identifier may be information reflecting the characteristics of the data resource, and may include information such as a data name, a data type, and a service attribution.
In the embodiment of the invention, the data characteristic identifiers such as the data name, the data type, the service attribution and the like can be extracted from each data access request and used for matching the data resources.
Step 290, searching the data catalogue for the data resource matching the data characteristic identifier.
In the implementation of the invention, the data characteristic identifier and the directory entry can be matched in the data directory, and the data resource corresponding to the matched directory entry can be used as the data resource matched with the data characteristic identifier. It will be appreciated that the matching process may be achieved by matching the directory entry with a string of data characteristic identifiers or by regular matching.
Step 2100, extracting a data access interface matched with the data resource from the preset interface file.
In the embodiment of the invention, the preset interface file may be a configuration file for storing the data access interfaces, one or more data access interfaces may be included in the preset interface file, and the matched data access interfaces may be searched according to the resource name or the resource type of the data resource.
Step 2110, accessing a target data resource in the data resource pool based on the data access interface.
According to the embodiment of the invention, the access channel is determined in the preset configuration file according to the data configuration information by reading the data configuration information of different service information systems, the data resources of the service information systems are acquired in the data access channel, the data resources are stored in the data resource pool, the data model definitions comprising the data structures are read, the data model definitions are searched in the data resource pool, the data resources respectively matched with the data model definitions are searched, the data resources are processed into the data catalogue according to the data structure in the corresponding data model definitions according to the association hierarchical relation, the data characteristic identification is extracted in the data access request, the data resources matched with the data characteristic identification in the data catalogue are searched, the data access interface matched with the data resources is acquired, the access to the target data resources in the data resource pool is realized by calling the access interface, the data resources of different service information systems are integrated, the data resources of different service information systems are shared, the data resource mining of the data systems is realized, the utilization efficiency of the data resources can be improved, and the scientific management degree of the system information is improved.
In some embodiments, before storing the data resources in the data resource pool, further comprising:
extracting a data structure of the data resource, and determining a preset format conversion rule according to the data structure; and converting the data format of the data resource according to the preset conversion rule.
The preset format conversion rule may be a rule for converting a format of the data resource, and the preset format conversion rule may include a source data format and a target data format.
In the embodiment of the invention, the data formats of the data resources stored in the data resource pool can be unified, the data resources aiming at different data structures can be extracted to obtain the corresponding preset conversion rules, and the data resources can be converted from the source data format to the target data format according to the preset conversion rules.
In other embodiments, the generating the data directory corresponding to each data resource according to each association hierarchical relationship includes:
generating a minimum communication diagram by the data resource and the association hierarchical relationship; and respectively taking each data resource as a catalog entry of the data catalog according to the minimum communication diagram.
The minimum communication graph can be a graph formed by taking data resources as nodes and taking management hierarchical relations as edges, and can reflect the simplest association relation among the data resources.
In the embodiment of the invention, the data resource is taken as a node, the association hierarchy relationship is taken as an edge, the minimum communication graph is constructed, the construction process can be generated by a Prim algorithm and a Kruskal algorithm, the data resource according to each node in the minimum communication graph can be taken as a catalog item, each catalog item is organized into a data catalog according to the hierarchy structure of the minimum communication graph, and it can be understood that the hierarchy relationship of different catalog items in the data catalog is the same as the hierarchy relationship of the nodes of each catalog item in the minimum communication graph.
In some embodiments, further comprising: and generating a data quality analysis result of the data resource according to a preset data granularity, wherein the preset data granularity at least comprises a null value, a repetition value, a format, a reference value and a fluctuation.
In the embodiment of the invention, the data quality analysis can be performed on each data resource, preset data granularity such as null value, repeated value, format, reference value, volatility and the like in the data resource can be analyzed and counted in the data quality analysis process, different preset data granularity can have different weight values, and the sum of the weight values of each data resource can be used as a data quality analysis result.
Example III
Fig. 3 is a schematic structural diagram of an information-integration data processing system according to a third embodiment of the present invention. As shown in fig. 3, the system includes: a data storage module 310, a catalog generation module 320, an interface determination module 330, and a resource access module 340.
A data storage module 310, configured to store data resources of at least one service information system into a data resource pool.
The catalog generation module 320 is configured to generate a data catalog of the data resource pool according to a preset data model.
The interface determining module 330 is configured to determine a data access interface according to the data directory and the data access request.
A resource access module 340, configured to access a target data resource in the data resource pool based on the data access interface.
According to the embodiment of the invention, the data storage module is used for storing the data resources of different service information systems in the data resource pool, the catalog generation module is used for generating the data catalog for the data resource pool according to the preset data model, the interface determination module is used for determining the data access interface according to the data target and the data access request, and the resource access module is used for realizing the access of the target data resources in the data resource pool according to the call of the data access interface, realizing the data resource integration of different service information systems, realizing the data resource sharing among different systems, being beneficial to realizing the data resource mining, improving the utilization efficiency of the data resources and improving the scientific management degree of the system information.
Further, on the basis of the embodiment of the present invention, the data storage module 310 includes:
and the information reading unit is used for reading the data configuration information of the service information system.
And the channel determining unit is used for determining a data access channel in a preset configuration file according to the data configuration information.
And the resource storage unit is used for reading the data resources of the service information system in the data access channel and storing the data resources into the data resource pool.
Further, on the basis of the above embodiment of the present invention, the system further includes: the format conversion module is used for extracting the data structure of the data resource and determining a preset format conversion rule according to the data structure; and converting the data format of the data resource according to the preset conversion rule.
Further, on the basis of the above embodiment of the present invention, the catalog generating module 320 includes:
and the definition reading unit is used for reading the data model definition of the preset data model, wherein the data model definition at least comprises a data structure.
And the resource classification unit is used for searching the data resources corresponding to the data model definitions in the data resource pool.
And the hierarchical relationship unit is used for determining at least one association hierarchical relationship of the data resources according to the data structure for each data resource.
And the catalog generation unit is used for generating the data catalog corresponding to each data resource according to each association hierarchical relationship.
Further, on the basis of the embodiment of the present invention, the catalog generating unit is specifically configured to: generating a minimum communication diagram by the data resource and the association hierarchical relationship; and respectively taking each data resource as a catalog entry of the data catalog according to the minimum communication diagram.
Further, on the basis of the above embodiment of the present invention, the interface determining module 330 includes:
and the identifier extraction unit is used for extracting the data characteristic identifier in the data access request.
And the catalog searching unit is used for searching the data resources matched with the data characteristic identifiers in the data catalog.
And the interface extraction unit is used for extracting the data access interface matched with the data resource in a preset interface file.
Further, on the basis of the above embodiment of the present invention, the system further includes: and the data quality module is used for generating a data quality analysis result of the data resource according to a preset data granularity, wherein the preset data granularity at least comprises a null value, a repetition value, a format, a reference value and volatility.
The information integration data processing system provided by the embodiment of the invention can execute the information integration data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example IV
Fig. 4 is a schematic structural diagram of an electronic device implementing an information integration data processing method according to an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the respective methods and processes described above, such as an information integration data processing method.
In some embodiments, the information-integration data-processing method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the information-integration data-processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the information-integration data-processing method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. An information integration data processing method, characterized by comprising:
storing data resources of at least one service information system into a data resource pool;
generating a data catalog of the data resource pool according to a preset data model;
determining a data access interface according to the data directory and the data access request;
accessing a target data resource in the data resource pool based on the data access interface.
2. The method of claim 1, wherein storing the data resources of the at least one business information system to the data resource pool comprises:
reading data configuration information of the service information system;
determining a data access channel in a preset configuration file according to the data configuration information;
and reading the data resources of the service information system in the data access channel, and storing the data resources into the data resource pool.
3. The method of claim 2, wherein prior to storing the data resources in the data resource pool, further comprising:
extracting a data structure of the data resource, and determining a preset format conversion rule according to the data structure;
and converting the data format of the data resource according to the preset conversion rule.
4. The method of claim 1, wherein the generating the data catalog of the data resource pool according to a preset data model comprises:
reading a data model definition of the preset data model, wherein the data model definition at least comprises a data structure;
searching the data resources corresponding to each data model definition in the data resource pool;
determining at least one association hierarchical relationship of the data resources according to the data structure for each data resource;
and generating the data catalogue corresponding to each data resource according to each association hierarchical relationship.
5. The method of claim 4, wherein the generating the data directory corresponding to each data resource according to each association hierarchy relationship comprises:
generating a minimum communication diagram by the data resource and the association hierarchical relationship;
and respectively taking each data resource as a catalog entry of the data catalog according to the minimum communication diagram.
6. The method of claim 1, wherein said determining a data access interface based on said data directory and data access request comprises:
extracting a data characteristic identifier in the data access request;
searching the data resource matched with the data characteristic identifier in the data catalog;
and extracting the data access interface matched with the data resource from a preset interface file.
7. The method according to any one of claims 1-6, further comprising:
and generating a data quality analysis result of the data resource according to a preset data granularity, wherein the preset data granularity at least comprises a null value, a repetition value, a format, a reference value and a fluctuation.
8. An information-integration data processing system, comprising:
the data storage module is used for storing the data resources of at least one business information system into a data resource pool;
the catalog generation module is used for generating a data catalog of the data resource pool according to a preset data model;
the interface determining module is used for determining a data access interface according to the data catalogue and the data access request;
and the resource access module is used for accessing the target data resources in the data resource pool based on the data access interface.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the information-integration data-processing method of any one of claims 1-7.
10. A computer readable storage medium, characterized in that the computer readable storage medium stores computer instructions for causing a processor to implement the information-integration data processing method of any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211341734.7A CN116450715A (en) | 2022-10-27 | 2022-10-27 | Information integration data processing method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211341734.7A CN116450715A (en) | 2022-10-27 | 2022-10-27 | Information integration data processing method, system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116450715A true CN116450715A (en) | 2023-07-18 |
Family
ID=87134369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211341734.7A Pending CN116450715A (en) | 2022-10-27 | 2022-10-27 | Information integration data processing method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116450715A (en) |
-
2022
- 2022-10-27 CN CN202211341734.7A patent/CN116450715A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034988B (en) | Accounting entry generation method and device | |
| CN112162965B (en) | Log data processing method, device, computer equipment and storage medium | |
| CN112559271A (en) | Method, device, equipment and storage medium for monitoring interface performance of distributed application | |
| CN111221785A (en) | Semantic data lake construction method of multi-source heterogeneous data | |
| CN113568938A (en) | Data stream processing method and device, electronic equipment and storage medium | |
| CN110109983B (en) | Method and device for operating Redis database | |
| CN112528067A (en) | Graph database storage method, graph database reading method, graph database storage device, graph database reading device and graph database reading equipment | |
| CN112559717A (en) | Search matching method and device, electronic equipment and storage medium | |
| CN116611411A (en) | Business system report generation method, device, equipment and storage medium | |
| CN116719794A (en) | Data processing method, device, electronic equipment, medium and program product | |
| WO2022111148A1 (en) | Metadata indexing for information management | |
| CN112783887A (en) | Data processing method and device based on data warehouse | |
| CN113220710B (en) | Data query method, device, electronic equipment and storage medium | |
| CN113239054B (en) | Information generation method and related device | |
| CN111061763A (en) | Method and device for generating rule execution plan of rule engine | |
| CN113722600A (en) | Data query method, device, equipment and product applied to big data | |
| CN116955856A (en) | Information display method, device, electronic equipment and storage medium | |
| CN108768742B (en) | Network construction method and device, electronic equipment and storage medium | |
| US10885157B2 (en) | Determining a database signature | |
| CN116450715A (en) | Information integration data processing method, system, electronic equipment and storage medium | |
| CN115408547A (en) | Dictionary tree construction method, device, equipment and storage medium | |
| CN114547477A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114281586A (en) | Fault determination method and device, electronic equipment and computer readable storage medium | |
| US20230086429A1 (en) | Method of recognizing address, electronic device and storage medium | |
| US12093315B2 (en) | Asserted relationships matching in an identity graph data structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |