CN113268500B - Service processing method and device and electronic equipment - Google Patents
Service processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN113268500B CN113268500B CN202110638853.8A CN202110638853A CN113268500B CN 113268500 B CN113268500 B CN 113268500B CN 202110638853 A CN202110638853 A CN 202110638853A CN 113268500 B CN113268500 B CN 113268500B
- Authority
- CN
- China
- Prior art keywords
- metadata information
- data
- service
- information set
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides a service processing method, a device and electronic equipment, wherein the method comprises the steps of receiving a data query request corresponding to a service to be processed, wherein the data query request comprises a service identifier, acquiring service source data corresponding to the service identifier from a first type database and a second type database according to the data query request, wherein the service source data comprises basic service data and a task scheduling relationship, and processing the basic service data according to the task scheduling relationship, so that the service to be processed is realized, the efficiency of data query and calling is improved, the waiting time of a user is further reduced, and the use experience of the user is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a business processing method, a business processing device and electronic equipment.
Background
With the development of the Internet, the number of users and the number of information source nodes are increased explosively, so that the data form is diversified. Although relational databases have extremely strong management capabilities, information sharing and data exchange are severely impacted due to differences between various database systems, as well as the heterogeneous nature of the operating systems relied upon.
In the prior art, when related business is realized, a plurality of reports are often needed, and data personnel can extract various data and task scheduling information for processing the data, and the extracted data and task scheduling information are uniformly stored in a database.
However, because the extracted data and task scheduling information are various, more time is required to be spent in the subsequent query or call process to acquire the corresponding data from the database, the efficiency of data query and call is reduced, the waiting time of the user is further increased, and the use experience of the user is reduced.
Disclosure of Invention
The embodiment of the invention provides a service processing method, a service processing device and electronic equipment, which are used for improving the efficiency of data query and call.
In a first aspect, an embodiment of the present invention provides a service processing method, including:
receiving a data query request corresponding to a service to be processed, wherein the data query request comprises a service identifier;
acquiring service source data corresponding to the service identifier from a first type database and a second type database according to the data query request, wherein the service source data comprises basic service data and task scheduling relation;
and processing the basic service data according to the task scheduling relation, thereby realizing the service to be processed.
Optionally, the first type database is a search type database, the second type database is a graph type database, and service source data corresponding to the service identifier is obtained from the first type database and the second type database according to the data query request, where the service source data includes basic service data and a task scheduling relationship, and the method includes:
acquiring service source data corresponding to the service identifier from the search class database and the graphic class database according to the data query request;
the business source data comprises basic business data and task scheduling relations, wherein the basic business data comprises basic business data of graphic types and basic business data of text types.
Optionally, before the receiving the data query request corresponding to the service to be processed, the method further includes:
metadata information stored in a report system, a scheduling system and different types of data storage systems is acquired, and a metadata information set is obtained;
integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration;
storing the integrated first sub-metadata information set into the first type database, and storing the integrated second sub-metadata information set into the second type database.
Optionally, the integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration framework to obtain a first sub-metadata information set and a second sub-metadata information set after integration, including:
acquiring target task dependency chains corresponding to different types of metadata information in the metadata information set;
acquiring all task variables from the target task dependency chain;
judging whether the task variables are task variables of a preset type or not according to each task variable to obtain a judging result;
and if the judging result is that the task variable is a task variable of a preset type, integrating and classifying the metadata information corresponding to the task variable in the metadata information set according to a preset data integration frame to obtain an integrated first sub-metadata information set and second sub-metadata information set.
Optionally, the method further comprises:
and if the judging result is that the task variable is not a task variable of a preset type, deleting the metadata information corresponding to the task variable in the metadata information set.
Optionally, the metadata information includes at least one data table, and each data table includes metadata, a column number, an annotation of the table itself, and a column annotation.
In a second aspect, an embodiment of the present invention provides a service processing apparatus, including:
the receiving module is used for receiving a data query request corresponding to a service to be processed, wherein the data query request comprises a service identifier;
the processing module is used for acquiring service source data corresponding to the service identifier from a first type database and a second type database according to the data query request, wherein the service source data comprises basic service data and task scheduling relation;
and the processing module is also used for processing the basic service data according to the task scheduling relationship so as to realize the service to be processed.
In a third aspect, an embodiment of the present invention provides an electronic device, including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing computer-executable instructions stored in the memory causes the at least one processor to perform the business processing method of any of the first aspects.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium, where computer executable instructions are stored, and when executed by a processor, implement the service processing method according to any one of the first aspects.
In a fifth aspect, embodiments of the present invention provide a computer program product comprising a computer program which, when executed by a processor, implements the business processing method according to the first aspect and the various possible designs of the first aspect.
The embodiment of the invention provides a service processing method, a device and electronic equipment, which can firstly receive a data query request containing a service identifier corresponding to a service to be processed, and then acquire service source data corresponding to the service identifier from a first type database and a second type database according to the data query request, wherein the service source data contains basic service data and task scheduling relation, and then process the basic service data according to the task scheduling relation, thereby realizing the service to be processed.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
Fig. 1 is a schematic diagram of an architecture of an application system of a service processing method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a service processing method according to an embodiment of the present invention;
fig. 3 is an application schematic diagram of an parsing process according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a metadata structure according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a service processing device according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be capable of including other sequential examples in addition to those illustrated or described. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In the prior art, when related business is realized, a plurality of reports are often needed, and data personnel can extract various data and task scheduling information for processing the data, and the extracted data and task scheduling information are uniformly stored in a database. However, because of the wide variety of extracted data and task scheduling information, it takes a long time to acquire the corresponding data from the database during the subsequent query or call. The extracted data and task scheduling information comprise two types of type a and type B, the data and task scheduling information corresponding to the type a and the type B are uniformly stored in one database, and then the data storage mode corresponding to the database is probably not suitable for storing the type a or the type B data, so that in the subsequent query or call process, more time is required to be consumed to acquire the corresponding data from the database, the data query and call efficiency is reduced, the waiting time of a user is further increased, and the use experience of the user is reduced.
Based on the problems, the method and the device realize that the service source data of different types adopt different query and call modes by adopting the mode of storing the service source data of different types by adopting the databases of the first type and the second type, thereby improving the efficiency of data query and call, reducing the waiting time of a user and improving the technical effect of using experience of the user.
Fig. 1 is a schematic architecture diagram of an application system of a service processing method according to an embodiment of the present invention, where, as shown in fig. 1, the application system may include: search server 101, metadata server 102, front-end server 103, data source 104, and server 105. The search server 101 and the metadata server 102 may correspond to different types of databases, and are used to implement storage of different types of basic service data and task scheduling relationships. The front-end server 103 is responsible for the Web UI that interacts with the user, and may be, for example, a flash-based Web application, where the presentation layer is constructed using React with reduce, bootstrap, weback, and Babel. The data source 104 may be a reporting system, a scheduling system, and various types of data storage systems, and may be Oracle, hive, mySQL, for example. The server 105 may acquire corresponding data from the search server 101 and the metadata server 102 according to the received data query request corresponding to the service to be processed, and then further process the data according to the acquired data, so as to implement the service to be processed.
The search server 101 and the metadata server 102 may be deployed separately or may be deployed in a unified manner in the server 105. In this embodiment, the search server 101 and the metadata server 102 are deployed separately.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Fig. 2 is a flow chart of a service processing method provided by an embodiment of the present invention, where the method of the embodiment may be executed by a server, as shown in fig. 2, and the method of the embodiment may include:
s201: and receiving a data query request corresponding to the service to be processed, wherein the data query request comprises a service identifier.
In this embodiment, a plurality of to-be-processed services may be displayed in the front-end server, and the plurality of to-be-processed services may be displayed in a permutation manner according to a preset display style, and when a user wants to implement any to-be-processed service, the to-be-processed service may be triggered by a touch operation manner.
Further, the touch operation may be a single click operation, a double click operation, a long press operation, a heavy press operation, or the like through a mouse.
The server may receive a data query request corresponding to a service to be processed triggered by the touch operation, where the data query request may include a service identifier, and is used to uniquely identify the service to be processed.
The type of the service identifier may be set by user definition, which is not defined in detail herein.
S202: and acquiring service source data corresponding to the service identifier from the first type database and the second type database according to the data query request, wherein the service source data comprises basic service data and task scheduling relation.
In this embodiment, different types of service source data may be stored in different types of databases, so that the efficiency of subsequent service source data query and scheduling may be improved.
Further, the first type database is a search type database, the second type database is a graph type database, and service source data corresponding to the service identifier is obtained from the first type database and the second type database according to a data query request, where the service source data includes basic service data and a task scheduling relationship, and the method specifically may include:
and acquiring service source data corresponding to the service identifier from the search class database and the graphic class database according to the data query request.
The business source data comprises basic business data and task scheduling relations, wherein the basic business data comprises basic business data of graphic types and basic business data of text types.
Specifically, the service source data is stored in a mode of combining the search class database and the graph class database, so that most types of data can be covered, and the data query and scheduling efficiency is also remarkably improved. Illustratively, the search class database may be an elastiscearch, the graph class database may be a Neo4J graph database, and the structured data may be stored in a network.
In addition, the more database types are stored, the more the pressure of server call can be increased, and most of data query and call conditions can be met by storing the database in two types of databases, and the database can be stored in at least three types of databases under special conditions.
S203: and processing the basic service data according to the task scheduling relationship, thereby realizing the service to be processed.
In this embodiment, the service source data may include basic service data and task scheduling relationship, where the basic service data may be various types of data required for implementing the service to be processed, and examples may be text, or may be an image, etc. The task scheduling relationship is an overall process for realizing the service to be processed, and the service to be processed comprises three sub-service modules, and the task scheduling relationship is a scheduling relationship among the three sub-service modules. Correspondingly, after the task scheduling relation is determined, the basic service data can be processed according to the task scheduling relation, so that the service to be processed is realized.
After the scheme is adopted, the data query request containing the service identifier corresponding to the service to be processed can be received, then the service source data corresponding to the service identifier is obtained from the first type database and the second type database according to the data query request, wherein the service source data contains the basic service data and the task scheduling relation, the basic service data is processed according to the task scheduling relation, the service to be processed is further realized, and the mode that the two different types of databases of the first type database and the second type database are adopted for storing the different types of service source data is adopted, so that the different types of service source data adopt different query and calling modes, the efficiency of data query and calling is improved, the waiting time of a user is further reduced, and the use experience of the user is improved.
The examples of the present specification also provide some specific embodiments of the method based on the method of fig. 2, which is described below.
Further, in another embodiment, before S201, it may further include:
and acquiring metadata information stored in the reporting system, the scheduling system and the data storage systems of different types to obtain a metadata information set.
And integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration.
Storing the integrated first sub-metadata information set into a first type database, and storing the integrated second sub-metadata information set into the second type database.
In this embodiment, the metadata information set may include a reporting system, a scheduling system, and different types of storage systems, where the different types of storage systems may be Oracle, hive, mySQL, etc.
Further, integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration, which specifically may include:
and acquiring target task dependency chains corresponding to different types of metadata information in the metadata information set.
All task variables are obtained from the target task dependency chain.
And judging whether the task variable is a task variable of a preset type or not according to each task variable to obtain a judging result.
And if the judging result is that the task variable is a task variable of a preset type, integrating and classifying the metadata information corresponding to the task variable in the metadata information set according to a preset data integration frame to obtain an integrated first sub-metadata information set and second sub-metadata information set.
Furthermore, the method may further comprise:
and if the judging result is that the task variable is not a task variable of a preset type, deleting the metadata information corresponding to the task variable in the metadata information set.
Specifically, different types of metadata information in the metadata information set may correspond to different target task dependency chains, and specific corresponding rules may be determined in an existing corresponding manner, which is not discussed in detail herein. The target task dependency chain may be a DAG in airflow, and may be a directed acyclic graph composed of multiple task dependencies.
The metadata service interacts with the Neo4j graph database using the Python language interface of Neo4j and provides metadata for the front-end service. In the metadata model, the table is the core of the whole metadata model, and first the table is stored in some type of database, which may be Oracle, imala, etc. Because the data finds that the metadata to be searched is stored in different types of databases, each database is named for distinction. The database itself also includes similar schemas to distinguish between database tables. In summary, we use database types, database names and schemas to uniquely distinguish tables, and can locate a specific unique and unique table by using the database types, database names and schemas.
When the generation of the table is determined, it is also necessary to determine where the data in the table comes from. Scheduling tasks related to table generation may be found by parsing the DAG file of the airflow. Fig. 3 is an application schematic diagram of an parsing process provided in an embodiment of the present invention, as shown in fig. 3, in this embodiment, all task variables are obtained from a DAG file of Airflow, and an exemplary task variable may be an Operator, and for each Operator, whether the Operator is a task variable of a preset type is determined, and for an exemplary task variable, whether the Operator is an SQL Operator or a DataX Operator may be determined. If the metadata information is not SQL Operator or DataX Operator, the Operator is not processed any more, and the metadata information corresponding to the Operator is deleted. Otherwise, if the SQL Operator is the SQL Operator, the SQL statement is extracted from the Operator, the SQL statement is analyzed by a Parse method, and the TABLE name behind the INTO/OVERWRETE/TABLE is extracted from the SQL statement. If the metadata Operator is the data Operator, the Json is analyzed, and then the table name of the metadata information pair is extracted according to the agreed path, so that the metadata Operator is convenient to call when the service to be processed is realized.
In addition, the data integration framework may employ an existing framework responsible for extracting metadata from data sources and importing the metadata into the first type of database and the second type of database, and illustratively into Neo4J and the elastic search database.
Furthermore, in another embodiment, the metadata information includes at least one data table, each of the data tables including metadata, a column number, an annotation of the table itself, and a column annotation.
In this embodiment, the metadata information may be composed of different data tables, and each data table may include, in addition to its own metadata, a column number, an annotation of the table itself, and a column annotation, so that the table is conveniently described from multiple dimensions for subsequent recall.
Fig. 4 is a schematic diagram of a metadata structure provided in an embodiment of the present invention, as shown in fig. 4, in this embodiment, data related to a table may be extracted from a preset metadata base, so as to determine which billboard the table may be applied to. By way of example, a Superset metadata base may be provided. In addition, meta information of the table may be contained in the table, such as which columns are in the table, comments of the table itself, column comments, and the like.
Based on the same idea, the embodiment of the present disclosure further provides a device corresponding to the method, and fig. 5 is a schematic structural diagram of a service processing device provided by the embodiment of the present disclosure, as shown in fig. 5, may include:
the receiving module 501 is configured to receive a data query request corresponding to a service to be processed, where the data query request includes a service identifier.
And the processing module 502 is configured to obtain, according to the data query request, service source data corresponding to the service identifier from a first type database and a second type database, where the service source data includes basic service data and a task scheduling relationship.
In this embodiment, the first type database is a search class database, the second type database is a graph class database, and the processing module 502 is further configured to:
and acquiring service source data corresponding to the service identifier from the search class database and the graphic class database according to the data query request.
The business source data comprises basic business data and task scheduling relations, wherein the basic business data comprises basic business data of graphic types and basic business data of text types.
The processing module 502 is further configured to process the basic service data according to the task scheduling relationship, thereby implementing the service to be processed.
Furthermore, in another embodiment, the processing module 502 is further configured to:
and acquiring metadata information stored in the reporting system, the scheduling system and the data storage systems of different types to obtain a metadata information set.
And integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration.
Storing the integrated first sub-metadata information set into the first type database, and storing the integrated second sub-metadata information set into the second type database.
Furthermore, in another embodiment, the processing module 502 is further configured to:
and acquiring target task dependency chains corresponding to different types of metadata information in the metadata information set.
And acquiring all task variables from the target task dependency chain.
And judging whether the task variable is a task variable of a preset type or not according to each task variable to obtain a judging result.
And if the judging result is that the task variable is a task variable of a preset type, integrating and classifying the metadata information corresponding to the task variable in the metadata information set according to a preset data integration frame to obtain an integrated first sub-metadata information set and second sub-metadata information set.
In addition, the processing module 502 is further configured to:
and if the judging result is that the task variable is not a task variable of a preset type, deleting the metadata information corresponding to the task variable in the metadata information set.
The metadata information comprises at least one data table, and each data table comprises metadata information, a column number, comments of the table and column comments.
The device provided by the embodiment of the present invention can implement the method of the embodiment shown in fig. 2, and its implementation principle and technical effects are similar, and will not be described herein.
Fig. 6 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention, as shown in fig. 6, an apparatus 600 provided in this embodiment includes: at least one processor 601 and a memory 602. The processor 601 and the memory 602 are connected by a bus 603.
In a specific implementation, at least one processor 601 executes computer-executable instructions stored in the memory 602, so that the at least one processor 601 performs the method in the above method embodiments.
The specific implementation process of the processor 601 may refer to the above-mentioned method embodiment, and its implementation principle and technical effects are similar, and this embodiment will not be described herein again.
In the embodiment shown in fig. 6, it should be understood that the processor may be a central processing unit (english: central Processing Unit, abbreviated as CPU), or may be other general purpose processors, digital signal processors (english: digital Signal Processor, abbreviated as DSP), application specific integrated circuits (english: application Specific Integrated Circuit, abbreviated as ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
The memory may comprise high speed RAM memory or may further comprise non-volatile storage NVM, such as at least one disk memory.
The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (Peripheral Component Interconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or one type of bus.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the service processing method of the method embodiment is realized.
The embodiment of the invention also provides a computer program product, comprising a computer program which realizes the service processing method when being executed by a processor.
The computer readable storage medium described above may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. A readable storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.
An exemplary readable storage medium is coupled to the processor such the processor can read information from, and write information to, the readable storage medium. In the alternative, the readable storage medium may be integral to the processor. The processor and the readable storage medium may reside in an application specific integrated circuit (Application Specific Integrated Circuits, ASIC for short). The processor and the readable storage medium may reside as discrete components in a device.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (7)
1. A method for processing a service, comprising:
receiving a data query request corresponding to a service to be processed, wherein the data query request comprises a service identifier;
acquiring service source data corresponding to the service identifier from a first type database and a second type database according to the data query request, wherein the service source data comprises basic service data and task scheduling relation;
processing the basic service data according to the task scheduling relationship, thereby realizing the service to be processed;
before receiving the data query request corresponding to the service to be processed, the method further comprises the following steps:
metadata information stored in a report system, a scheduling system and different types of data storage systems is acquired, and a metadata information set is obtained;
integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration;
storing the integrated first sub-metadata information set into the first type database, and storing the integrated second sub-metadata information set into the second type database;
the integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration, including:
acquiring target task dependency chains corresponding to different types of metadata information in the metadata information set;
acquiring all task variables from the target task dependency chain;
judging whether the task variables are task variables of a preset type or not according to each task variable to obtain a judging result;
and if the judging result is that the task variable is a task variable of a preset type, integrating and classifying the metadata information corresponding to the task variable in the metadata information set according to a preset data integration frame to obtain an integrated first sub-metadata information set and second sub-metadata information set.
2. The method according to claim 1, wherein the first type database is a search type database, the second type database is a graph type database, the service source data corresponding to the service identifier is obtained from the first type database and the second type database according to the data query request, wherein the service source data includes basic service data and task scheduling relation, and the method includes:
acquiring service source data corresponding to the service identifier from the search class database and the graphic class database according to the data query request;
the business source data comprises basic business data and task scheduling relations, wherein the basic business data comprises basic business data of graphic types and basic business data of text types.
3. The method according to claim 1, wherein the method further comprises:
and if the judging result is that the task variable is not a task variable of a preset type, deleting the metadata information corresponding to the task variable in the metadata information set.
4. The method of claim 1, wherein the metadata information comprises at least one data table, each of the data tables comprising metadata, a column number, an annotation of the table itself, and a column annotation.
5. A service processing apparatus, comprising:
the receiving module is used for receiving a data query request corresponding to a service to be processed, wherein the data query request comprises a service identifier;
the processing module is used for acquiring service source data corresponding to the service identifier from a first type database and a second type database according to the data query request, wherein the service source data comprises basic service data and task scheduling relation;
the processing module is further used for processing the basic service data according to the task scheduling relationship, so as to realize the service to be processed;
the processing module is further configured to:
metadata information stored in a report system, a scheduling system and different types of data storage systems is acquired, and a metadata information set is obtained;
integrating and classifying the metadata information of different types in the metadata information set according to a preset data integration frame to obtain a first sub-metadata information set and a second sub-metadata information set after integration;
storing the integrated first sub-metadata information set into the first type database, and storing the integrated second sub-metadata information set into the second type database;
the processing module is further configured to:
acquiring target task dependency chains corresponding to different types of metadata information in the metadata information set;
acquiring all task variables from the target task dependency chain;
judging whether the task variables are task variables of a preset type or not according to each task variable to obtain a judging result;
and if the judging result is that the task variable is a task variable of a preset type, integrating and classifying the metadata information corresponding to the task variable in the metadata information set according to a preset data integration frame to obtain an integrated first sub-metadata information set and second sub-metadata information set.
6. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing computer-executable instructions stored in the memory causes the at least one processor to perform the business processing method of any of claims 1 to 4.
7. A computer readable storage medium having stored therein computer executable instructions which, when executed by a processor, implement the business processing method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110638853.8A CN113268500B (en) | 2021-06-08 | 2021-06-08 | Service processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110638853.8A CN113268500B (en) | 2021-06-08 | 2021-06-08 | Service processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113268500A CN113268500A (en) | 2021-08-17 |
| CN113268500B true CN113268500B (en) | 2023-06-02 |
Family
ID=77234546
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110638853.8A Active CN113268500B (en) | 2021-06-08 | 2021-06-08 | Service processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113268500B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115145992B (en) * | 2022-09-05 | 2022-12-27 | 深圳市明源云科技有限公司 | Multi-source data processing method and device, electronic equipment and readable storage medium |
| CN115269561B (en) * | 2022-09-21 | 2023-01-24 | 国网智能电网研究院有限公司 | Hybrid database management method and device, hybrid database and electronic equipment |
| CN115375380B (en) * | 2022-10-26 | 2023-02-03 | 南昌惠联网络技术有限公司 | Service data processing method and processing device based on attribute classification |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970899A (en) * | 2014-05-27 | 2014-08-06 | 重庆大学 | Service-oriented metadata relevance extraction management method and management system |
| CN106815270A (en) * | 2015-12-02 | 2017-06-09 | 大唐移动通信设备有限公司 | A kind of method and device for servicing dynamic management |
| CN108701325A (en) * | 2015-12-21 | 2018-10-23 | 星茶瓦公司 | The transaction system that is self-regulated and the method for the system |
| CN109857756A (en) * | 2019-01-14 | 2019-06-07 | 中国联合网络通信集团有限公司 | Generation method, device, equipment and the readable storage medium storing program for executing of Database Report |
| US10673716B1 (en) * | 2017-10-31 | 2020-06-02 | Amazon Technologies, Inc. | Graph-based generation of dependency-adherent execution plans for data center migrations |
| CN111708794A (en) * | 2020-06-22 | 2020-09-25 | 中国平安财产保险股份有限公司 | Data comparison method and device based on big data platform and computer equipment |
| CN112507010A (en) * | 2020-12-14 | 2021-03-16 | 深圳佑驾创新科技有限公司 | Service data processing method and device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011034283A1 (en) * | 2009-09-20 | 2011-03-24 | Lg Electronics Inc. | Method of processing epg metadata in network device and the network device for controlling the same |
-
2021
- 2021-06-08 CN CN202110638853.8A patent/CN113268500B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970899A (en) * | 2014-05-27 | 2014-08-06 | 重庆大学 | Service-oriented metadata relevance extraction management method and management system |
| CN106815270A (en) * | 2015-12-02 | 2017-06-09 | 大唐移动通信设备有限公司 | A kind of method and device for servicing dynamic management |
| CN108701325A (en) * | 2015-12-21 | 2018-10-23 | 星茶瓦公司 | The transaction system that is self-regulated and the method for the system |
| US10673716B1 (en) * | 2017-10-31 | 2020-06-02 | Amazon Technologies, Inc. | Graph-based generation of dependency-adherent execution plans for data center migrations |
| CN109857756A (en) * | 2019-01-14 | 2019-06-07 | 中国联合网络通信集团有限公司 | Generation method, device, equipment and the readable storage medium storing program for executing of Database Report |
| CN111708794A (en) * | 2020-06-22 | 2020-09-25 | 中国平安财产保险股份有限公司 | Data comparison method and device based on big data platform and computer equipment |
| CN112507010A (en) * | 2020-12-14 | 2021-03-16 | 深圳佑驾创新科技有限公司 | Service data processing method and device, computer equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| Detection of spam-posting accounts on Twitter;Isa Inuwa-Dutse et al.;《Neurocomputing》;496-511 * |
| 基于继电保护故障及信息系统的定值自动校核;苏俊妮;《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》;C042-574 * |
| 数据网格中目录服务的研究与应用;程珅;《中国优秀博硕士学位论文全文数据库 (硕士) 信息科技辑》;I139-159 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113268500A (en) | 2021-08-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113268500B (en) | Service processing method and device and electronic equipment | |
| CN110795455B (en) | Dependency analysis method, electronic device, computer apparatus, and readable storage medium | |
| AU2009238294B2 (en) | Data transformation based on a technical design document | |
| CN108171528B (en) | Attribution method and attribution system | |
| CN110781183B (en) | Processing method and device for incremental data in Hive database and computer equipment | |
| CN112115152B (en) | Data increment updating and inquiring method and device, electronic equipment and storage medium | |
| CN110427188B (en) | Configuration method, device, equipment and storage medium of single-test assertion program | |
| CN113672781A (en) | Data query method and device, electronic equipment and storage medium | |
| CN110795697A (en) | Logic expression obtaining method and device, storage medium and electronic device | |
| CN116450890A (en) | Graph data processing method, device and system, electronic equipment and storage medium | |
| CN113987337A (en) | Search method, system, equipment and storage medium based on componentized dynamic arrangement | |
| CN114003818A (en) | Page recommendation method, device, server and storage medium | |
| CN113342472A (en) | Micro-service cluster creating method and device, electronic equipment and readable storage medium | |
| CN115295109A (en) | Medical data tracing method and device, storage medium and terminal equipment | |
| CN117009397A (en) | Data query method, data query device, electronic equipment and storage medium | |
| CN113609128A (en) | Method and device for generating database entity class, terminal equipment and storage medium | |
| CN113687827A (en) | Data list generation method, device and equipment based on widget and storage medium | |
| CN113448985A (en) | API (application program interface) interface generation method, calling method and device and electronic equipment | |
| US8321844B2 (en) | Providing registration of a communication | |
| US11960560B1 (en) | Methods for analyzing recurring accessibility issues with dynamic web site behavior and devices thereof | |
| CN114489886A (en) | Resource page generation method, device, equipment and medium | |
| CN111079391B (en) | Report generation method and device | |
| US20240104297A1 (en) | Analysis of spreadsheet table in response to user input | |
| CN117742779A (en) | Method, device, equipment and storage medium for checking resource configuration information | |
| US20150081380A1 (en) | Complement self service business intelligence with cleansed and enriched customer data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |