CN110689245A - Method and system for analyzing call relation of big data workflow - Google Patents
Method and system for analyzing call relation of big data workflow Download PDFInfo
- Publication number
- CN110689245A CN110689245A CN201910877666.8A CN201910877666A CN110689245A CN 110689245 A CN110689245 A CN 110689245A CN 201910877666 A CN201910877666 A CN 201910877666A CN 110689245 A CN110689245 A CN 110689245A
- Authority
- CN
- China
- Prior art keywords
- workflow
- file
- name
- work
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0633—Workflow analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
A method for analyzing call relations of big data workflows comprises the following steps: storing metadata of a workflow scheduling engine, and obtaining the name of a workflow and the storage position of a description file; reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow; reading the first file, and obtaining the operation type and the execution file; writing the name of the workflow and the execution file into a first table, wherein the first table records the workflow attribution of the current operation; and when the job type is hive, continuously reading the execution file, and obtaining a target table and a source table in a data table. By the method for analyzing the call relation of the big data workflow, the relation between the big data workflow scheduling engine workflow and the operation and the relation between the operation and the operation are analyzed. A global invocation relationship chain may be established to assist in querying and analyzing dependencies between them.
Description
Technical Field
The application relates to the field of databases, in particular to a method and a system for analyzing call relation of big data workflow.
Background
The existing big data workflow scheduling engines mainly comprise Oozie, Azkaban, AirFlow and the like, a DAG graph is used for providing a group of operations and flows which are run in a workflow in a specific sequence, and the supported operation types comprise java, map-reduce, hive, shell, python and the like. However, none of the existing workflow scheduling engines provides a method for querying which workflow a certain scheduling job belongs to, and the existing workflow scheduling engines need to manually traverse all workflows one by one to search. The invention provides a method for analyzing the scheduling relation between jobs and workflows, which helps to quickly find the workflow to which the jobs belong.
The disadvantages are as follows:
1. the existing big data workflow scheduling engine does not provide a method for inquiring the workflow to which the job belongs
2. The existing big data workflow scheduling engine does not provide the dependency relationship between internal tables of hive jobs
Disclosure of Invention
The main purpose of the present application is to provide a method for analyzing a big data workflow invocation relationship, which includes:
storing metadata of a workflow scheduling engine, and obtaining the name of a workflow and the storage position of a description file;
reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow;
reading the first file, and obtaining the operation type and the execution file;
writing the name of the workflow and the execution file into a first table, wherein the first table records the workflow attribution of the current operation;
and when the job type is hive, continuously reading the execution file, and obtaining a target table and a source table in a data table.
Optionally, the workflow attribution of the job execution file is queried by executing the file table, and the workflow attribution of the data table generated by querying the data warehouse is queried by executing the file table and the word _ table join.
Optionally, the metadata of ozie is stored through mysql, and WF _ JOBS table is searched through sql below, so as to obtain the name app _ name of the workflow and the saving position app _ path of the description file in hdfs.
Optionally, the first file is a workflow file that reads the lower side of the folder according to the location of the app _ path.
Optionally, the job type work _ type is obtained through a name attribute of an action tag, the execution file work _ script is obtained through a value of a script, and the app _ name and the work _ script are written into an app _ work table of mysql.
Optionally, when the work _ type is live, continuing to read the work _ script file, where the work _ script file records sql that needs to be executed by the job, and analyzing the sql by the following regular expressions to obtain a target table and a source table in a data table:
target table name dst _ table: insert [ \\ n \ t ] + overlay [ \\ n \ t ] + table [ \\ n \ t ] + ([ ^ n \ n ^ t ] +) source table name src _ table: [ \\ n \ t ] from [ \ n \ t ] + ([ ^ n \ n ^ t ] +) or join [ \ n \ t ] + ([ ^ n \ n ^ t ] +)
And writing the work _ script, the dst _ table and the src _ table into a work _ table of mysql, wherein the work _ table records the dependency relationship of the hive operation table.
The application also provides a system for analyzing the call relation of the big data workflow, which comprises the following steps:
the storage module is used for storing the metadata of the workflow scheduling engine and acquiring the name of the workflow and the storage position of the description file;
the reading module is used for reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow, and the job types and the execution files are obtained;
the writing module is used for writing the name of the workflow and the execution file into a first table, and the first table records the workflow attribution of the current operation;
and the obtaining module is used for continuously reading the execution file when the operation type is hive, and obtaining a target table and a source table in a data table.
The application also discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of being executed by the processor, wherein the processor realizes the method of any one of the above items when executing the computer program.
The application also discloses a computer-readable storage medium, a non-volatile readable storage medium, having stored therein a computer program which, when executed by a processor, implements the method of any of the above.
The present application also discloses a computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of the above.
By the method for analyzing the call relation of the big data workflow, the relation between the big data workflow scheduling engine workflow and the operation and the relation between the operation and the operation are analyzed. A global invocation relationship chain may be established to assist in querying and analyzing dependencies between them.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, serve to provide a further understanding of the application and to enable other features, objects, and advantages of the application to be more apparent. The drawings and their description illustrate the embodiments of the invention and do not limit it. In the drawings:
FIG. 1 is a schematic flow diagram of a method of analyzing big data workflow invocation relationships, according to one embodiment of the present application;
FIG. 2 is a flowchart illustrating a method for analyzing call relations of a big data workflow according to an embodiment of the present application
FIG. 3 is a schematic diagram of a computer device according to one embodiment of the present application; and
FIG. 4 is a schematic diagram of a computer-readable storage medium according to one embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Furthermore, the terms "mounted," "disposed," "provided," "connected," and "sleeved" are to be construed broadly. For example, it may be a fixed connection, a removable connection, or a unitary construction; can be a mechanical connection, or an electrical connection; may be directly connected, or indirectly connected through intervening media, or may be in internal communication between two devices, elements or components. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1, an embodiment of the present application provides a method for analyzing a call relationship of a big data workflow, including:
s2: storing metadata of a workflow scheduling engine, and obtaining the name of a workflow and the storage position of a description file;
s4: reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow;
s6: reading the first file, and obtaining the operation type and the execution file;
s8: writing the name of the workflow and the execution file into a first table, wherein the first table records the workflow attribution of the current operation;
s10: and when the job type is hive, continuously reading the execution file, and obtaining a target table and a source table in a data table.
In an embodiment of the application, the workflow attribution of the job execution file is queried through the execution file table, and the workflow attribution of the data table generated by the data warehouse is queried through the execution file table and the word _ table join.
In an embodiment of the present application, the mysql is used to store metadata of Oozie, and the WF _ JOBS table is searched by the following sql to obtain the name app _ name of the workflow and the storage location app _ path of the description file in hdfs.
In an embodiment of the present application, the first file is a workflow.xml file that reads the bottom of the folder according to the location of the app _ path.
In an embodiment of the application, the job type work _ type is obtained through a name attribute of an action tag, the execution file work _ script is obtained through a value of a script, and the app _ name and the work _ script are written into an app _ work table of mysql.
In an embodiment of the application, when work _ type is live, continuing to read the work _ script file, where the work _ script file records sql to be executed for a job change, and analyzing the sql through the following regular expressions to obtain a target table and a source table in a data table:
target table name dst _ table: insert [ \\ n \ t ] + overlay [ \\ n \ t ] + table [ \\ n \ t ] + ([ ^ n \ n ^ t ] +) source table name src _ table: [ \\ n \ t ] from [ \ n \ t ] + ([ ^ n \ n ^ t ] +) or join [ \ n \ t ] + ([ ^ n \ n ^ t ] +)
And writing the work _ script, the dst _ table and the src _ table into a work _ table of mysql, wherein the work _ table records the dependency relationship of the hive operation table.
The application also provides a system for analyzing the call relation of the big data workflow, which comprises the following steps:
the storage module is used for storing the metadata of the workflow scheduling engine and acquiring the name of the workflow and the storage position of the description file;
the reading module is used for reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow, and the job types and the execution files are obtained;
the writing module is used for writing the name of the workflow and the execution file into a first table, and the first table records the workflow attribution of the current operation;
and the obtaining module is used for continuously reading the execution file when the operation type is hive, and obtaining a target table and a source table in a data table.
Referring to fig. 3, the present application further provides a computer device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method of any one of the above methods when executing the computer program.
Referring to fig. 4, the present application also provides a computer-readable storage medium, a non-volatile readable storage medium, having stored therein a computer program, which when executed by a processor implements the method of any of the above.
The present application also provides a computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of the above.
Referring to fig. 2, in an embodiment of the present application, a process of analyzing a call relationship of a big data workflow scheduling engine (Oozie, Azkaban, AirFlow, etc.) is shown in the above diagram, and specific analysis steps are described in detail below with Oozie as a column:
1. the Oozie metadata is stored by using mysql, and the WF _ JOBS table is searched by using the following sql, so that the names app _ name of all workflows and the position app _ path of the workflow description file stored in hdfs can be obtained.
SELECT max(id)id,app_name,max(app_path)app_path FROM WF_JOBS GROUP BYapp_name
2. Reading a workflow file below the folder according to the position of the app _ path, wherein the workflow file describes all job types and job execution sequences in the current workflow
3. Xml file is read, the job type work _ type can be obtained by the name attribute of < action > tag, and the job execution file work _ script can be obtained by the value of < script >. And writing the app _ name and the work _ script in the step 1 into an app _ work table of mysql, wherein the table records which workflow the job belongs to.
4. If work _ type is live, continuing to read a work _ script file, recording the sql which needs to be executed by the job, and analyzing the sql by using the following regular expression to find out a target table and a source table in the data table:
target table name dst _ table: insert [ \\ n \ t ] + overlay [ \\ n \ t ] + table [ \ n \ t ] + ([ ^ n \ n ^ t ] +)
Source table name src _ table: [ \\ n \ t ] from [ \ n \ t ] + ([ ^ n \ n ^ t ] +) or join [ \ n \ t ] + ([ ^ n \ n ^ t ] +)
Writing work _ script, dst _ table and src _ table into mysql work _ table, wherein the table records the dependency relationship of the table in hive operation
5. The workflow to which the job execution file belongs can be inquired by using the work _ script table, and the workflow in which the data table generated by the data warehouse is generated can be inquired by performing join on the work _ script table and the work _ table.
The basic concepts that may be used in this application are as follows:
big data:
https://wiki.mbalib.com/wiki/％E5％A4％A7％E6％95％B0％E6％8D％AE
Hadoop：https://hadoop.apache.org/
big data scheduling engine:
https://www.cnblogs.com/barneywill/p/10109820.html
Oozie:https://oozie.apache.org/
Hive：https://www.yiibai.com/hive/
and (3) data warehouse:
library https:// baike.baidu.com/% E6% 95% B0% E6% 8D% AE% E4% BB% 93% E5% BA% 93
mysql: is an open source code relational database management system (RDBMS) using the most common databases
Management language-Structured Query Language (SQL) for database management
The regular expression is as follows: also known as regular expressions, is a concept of computer science. Regular expressions are typically used to retrieve, replace, text that conforms to a certain pattern (rule).
A DAG: directed acyclic graphs, which are graphs in which any one side has a direction and no loop exists
Oozie: is a workflow scheduler for managing Apache Hadoop jobs. Oozie combines multiple jobs sequentially into one logical unit of work as a Directed Acyclic Graph (DAG) of operations
hive, a data warehouse tool based on Hadoop can map the structured data file into a database table, provide a simple sql query function and convert the sql statement into a MapReduce task to run
hdfs: HDFS Hadoop distributtered FILE SYSTEM is short for, and is a DISTRIBUTED file system
sql: is a standard database language
By the method for analyzing the call relation of the big data workflow, the relation between the big data workflow scheduling engine workflow and the operation and the relation between the operation and the operation are analyzed. A global invocation relationship chain may be established to assist in querying and analyzing dependencies between them.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for analyzing call relation of big data workflow is characterized by comprising the following steps:
storing metadata of a workflow scheduling engine, and obtaining the name of a workflow and the storage position of a description file;
reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow;
reading the first file, and obtaining the operation type and the execution file;
writing the name of the workflow and the execution file into a first table, wherein the first table records the workflow attribution of the current operation;
and when the job type is hive, continuously reading the execution file, and obtaining a target table and a source table in a data table.
2. The method for analyzing call relation of big data workflow according to claim 1, wherein workflow attribution of job execution file is inquired by executing file table, and workflow attribution of data table generated by data warehouse is inquired by executing file table and word _ table join.
3. The method for analyzing the call relation of the big data workflow as claimed in claim 2, wherein the name app _ name of the workflow and the saving location app _ path of the description file in hdfs are obtained by storing the metadata of Oozie through mysql and looking up the WF _ JOBS table through the following sql.
4. The method for analyzing big data workflow invocation relationship according to claim 3, wherein said first file is a workflow.xml file under the folder read according to the location of said app _ path.
5. The method for analyzing call relation of big data workflow as claimed in claim 4, wherein the job type work _ type is obtained by name attribute of action tag, the execution file work _ script is obtained by value of script, and the app _ name and the work _ script are written into app _ work table of mysql.
6. The method for analyzing call relation of big data workflow as claimed in claim 5, wherein when work _ type is live, continuing to read the work _ script file, the work _ script file records sql that needs to be executed for job change, and the target table and the source table in the data table are obtained by analyzing sql through the following regular expressions:
target table name dst _ table: insert [ \\ n \ t ] + overlay [ \\ n \ t ] + table [ \ n \ t ] + ([ ^ n \ n ^ t ] +)
Source table name src _ table: [ \\ n \ t ] from [ \ n \ t ] + ([ ^ n \ n ^ t ] +) or join [ \ n \ t ] + ([ ^ n \ n ^ t ] +)
And writing the work _ script, the dst _ table and the src _ table into a work _ table of mysql, wherein the work _ table records the dependency relationship of the hive operation table.
7. A system for analyzing call relations of a big data workflow, comprising:
the storage module is used for storing the metadata of the workflow scheduling engine and acquiring the name of the workflow and the storage position of the description file;
the reading module is used for reading a first file according to the storage position, wherein the first file describes all job types and execution sequences of the current workflow, and the job types and the execution files are obtained;
the writing module is used for writing the name of the workflow and the execution file into a first table, and the first table records the workflow attribution of the current operation;
and the obtaining module is used for continuously reading the execution file when the operation type is hive, and obtaining a target table and a source table in a data table.
8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable by the processor, the processor implementing the method of any one of claims 1-6 when executing the computer program.
9. A computer-readable storage medium, a non-transitory readable storage medium, having stored therein a computer program, which when executed by a processor, implements the method of any one of claims 1-6.
10. A computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910877666.8A CN110689245B (en) | 2019-09-17 | 2019-09-17 | Method and system for analyzing call relation of big data workflow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910877666.8A CN110689245B (en) | 2019-09-17 | 2019-09-17 | Method and system for analyzing call relation of big data workflow |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110689245A true CN110689245A (en) | 2020-01-14 |
| CN110689245B CN110689245B (en) | 2022-07-12 |
Family
ID=69109305
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910877666.8A Active CN110689245B (en) | 2019-09-17 | 2019-09-17 | Method and system for analyzing call relation of big data workflow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110689245B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651460A (en) * | 2020-06-11 | 2020-09-11 | 上海德易车信息科技有限公司 | Data management method and device, electronic equipment and readable storage medium |
| CN112506957A (en) * | 2020-12-18 | 2021-03-16 | 杭州数梦工场科技有限公司 | Method and device for determining workflow dependency relationship |
| CN115525680A (en) * | 2022-09-21 | 2022-12-27 | 京信数据科技有限公司 | Data processing job scheduling method and device, computer equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101645180A (en) * | 2009-05-31 | 2010-02-10 | 北京龙德时代科技发展有限公司 | Intelligent data acquisition unit |
| CN106020955A (en) * | 2016-05-12 | 2016-10-12 | 深圳市傲天科技股份有限公司 | Infinite big data workflow dispatching platform |
| US20170193069A1 (en) * | 2015-12-31 | 2017-07-06 | Dropbox, Inc. | Propagating Computable Dependencies Within Synchronized Content Items Between First and Third-Party Applications |
| CN107105009A (en) * | 2017-03-22 | 2017-08-29 | 北京荣之联科技股份有限公司 | Job scheduling method and device based on Kubernetes system docking workflow engines |
| CN107689982A (en) * | 2017-06-25 | 2018-02-13 | 平安科技（深圳）有限公司 | Multi-data source method of data synchronization, application server and computer-readable recording medium |
| CN109670780A (en) * | 2018-12-03 | 2019-04-23 | 中国建设银行股份有限公司 | Workflow processing method, equipment and storage medium under complex scene |
| US20190361718A1 (en) * | 2018-05-25 | 2019-11-28 | Vmware, Inc. | 3D API Redirection for Virtual Desktop Infrastructure |
-
2019
- 2019-09-17 CN CN201910877666.8A patent/CN110689245B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101645180A (en) * | 2009-05-31 | 2010-02-10 | 北京龙德时代科技发展有限公司 | Intelligent data acquisition unit |
| US20170193069A1 (en) * | 2015-12-31 | 2017-07-06 | Dropbox, Inc. | Propagating Computable Dependencies Within Synchronized Content Items Between First and Third-Party Applications |
| CN106020955A (en) * | 2016-05-12 | 2016-10-12 | 深圳市傲天科技股份有限公司 | Infinite big data workflow dispatching platform |
| CN107105009A (en) * | 2017-03-22 | 2017-08-29 | 北京荣之联科技股份有限公司 | Job scheduling method and device based on Kubernetes system docking workflow engines |
| CN107689982A (en) * | 2017-06-25 | 2018-02-13 | 平安科技（深圳）有限公司 | Multi-data source method of data synchronization, application server and computer-readable recording medium |
| US20190361718A1 (en) * | 2018-05-25 | 2019-11-28 | Vmware, Inc. | 3D API Redirection for Virtual Desktop Infrastructure |
| CN109670780A (en) * | 2018-12-03 | 2019-04-23 | 中国建设银行股份有限公司 | Workflow processing method, equipment and storage medium under complex scene |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651460A (en) * | 2020-06-11 | 2020-09-11 | 上海德易车信息科技有限公司 | Data management method and device, electronic equipment and readable storage medium |
| CN112506957A (en) * | 2020-12-18 | 2021-03-16 | 杭州数梦工场科技有限公司 | Method and device for determining workflow dependency relationship |
| CN115525680A (en) * | 2022-09-21 | 2022-12-27 | 京信数据科技有限公司 | Data processing job scheduling method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110689245B (en) | 2022-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110689245B (en) | Method and system for analyzing call relation of big data workflow | |
| CN107463632B (en) | Distributed NewSQL database system and data query method | |
| Hagedorn et al. | The STARK framework for spatio-temporal data analytics on spark | |
| US11423082B2 (en) | Methods and apparatus for subgraph matching in big data analysis | |
| US8898181B2 (en) | Subscription for integrating external data from external system | |
| JP6964384B2 (en) | Methods, programs, and systems for the automatic discovery of relationships between fields in a mixed heterogeneous data source environment. | |
| CN105630881A (en) | Data storage method and query method for RDF (Resource Description Framework) | |
| US9406018B2 (en) | Systems and methods for semantic data integration | |
| Santos et al. | Data models in NoSQL databases for big data contexts | |
| CN110555035A (en) | Method and device for optimizing query statement | |
| CN104156385B (en) | A kind of method and apparatus of processing time sequence data | |
| US20200159722A1 (en) | Presenting updated data using persisting views | |
| US10095737B2 (en) | Information storage system | |
| US9773003B2 (en) | Computer implemented system and method for investigative data analytics | |
| CN114564482A (en) | Multi-entity-oriented label system and processing method | |
| JPWO2017170459A6 (en) | Method, program, and system for automatic discovery of relationships between fields in a heterogeneous data source mixed environment | |
| RU2015112157A (en) | SYSTEM AND METHOD OF DATA SEARCH IN THE DATABASE OF GRAPHS | |
| US8825596B2 (en) | Systems and methods for robust data source access | |
| CN109471904B (en) | Method and system for organizing labels | |
| JP2020160494A (en) | Information processing apparatus, document management system and program | |
| US10324927B2 (en) | Data-driven union pruning in a database semantic layer | |
| JP2004192657A (en) | Information retrieval system, and recording medium recording information retrieval method and program for information retrieval | |
| JP5998835B2 (en) | Information processing apparatus and program | |
| US11126604B2 (en) | Aggregation apparatus, aggregation method, and storage medium | |
| GB2596769A (en) | Extensible data skipping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 200125, Room 325, 3rd Floor, Unit 2, No. 231, Shibocun Road, China (Shanghai) Pilot Free Trade Zone, Pudong New Area, Shanghai Patentee after: Wheel interconnection technology (Shanghai) Co.,Ltd. Address before: 200125 Room 501, 5 / F, building 3, 3601 Dongfang Road, Pudong New Area, Shanghai Patentee before: SHANGHAI YIDIAN SPACE NETWORK Co.,Ltd. |