WO2018058707A1 - 任务处理方法和分布式计算框架 - Google Patents
任务处理方法和分布式计算框架 Download PDFInfo
- Publication number
- WO2018058707A1 WO2018058707A1 PCT/CN2016/102124 CN2016102124W WO2018058707A1 WO 2018058707 A1 WO2018058707 A1 WO 2018058707A1 CN 2016102124 W CN2016102124 W CN 2016102124W WO 2018058707 A1 WO2018058707 A1 WO 2018058707A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- operator
- data set
- distributed
- task
- distributed computing
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2452—Query translation
- G06F16/24524—Access plan code generation and invalidation; Reuse of access plans
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9027—Trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9032—Query formulation
- G06F16/90332—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
Definitions
- the present application relates to the field of computers, and in particular to the field of distributed computing, and in particular to a task processing method and a distributed computing framework.
- logical planning is required to describe distributed computing tasks.
- the usual way of constructing a logical plan is to describe the upstream and downstream relationship of the operator.
- the present application provides a task processing method and a distributed computing framework for solving the technical problems existing in the above background art.
- the application provides a task processing method, which includes: parsing an expression corresponding to a distributed computing task, and constructing task description information corresponding to the distributed computing task, where the task description information is used to describe the operator and the distributed data set.
- a task processing method which includes: parsing an expression corresponding to a distributed computing task, and constructing task description information corresponding to the distributed computing task, where the task description information is used to describe the operator and the distributed data set.
- the operator acts on a distributed data set and/or distributes the distributed data set Data set; based on the task description information, determine the distributed data set used by the operator; use the operator to perform distributed computing on the distributed data set used by the operator.
- the application provides a distributed computing framework, where the distributed computing framework includes: a building unit configured to parse an expression corresponding to the distributed computing task, and construct a task description information corresponding to the distributed computing task, and the task description information For describing a correspondence between an operator and a distributed data set, wherein the operator acts on the distributed data set and/or the distributed data set obtained after grouping the distributed data set; the determining unit is configured to be based on the task Descriptive information, a distributed data set for which the operator is used; a computing unit configured to perform distributed computing on the distributed data set for the operator using the operator.
- the task processing method and the distributed computing framework provided by the present application construct task description information corresponding to the distributed computing task by parsing the expression corresponding to the distributed computing task, and the task description information is used to describe the correspondence between the operator and the distributed data set. a relationship in which an operator acts on a distributed data set and/or a distributed data set obtained by grouping distributed data sets; based on task description information, determining a distributed data set for which the operator is used; using an operator Distributed computing is performed on the distributed data sets used by the operators.
- the scope and nesting relationship of the topology description operator are constructed, and the same operator can be applied to the distributed data set and the distributed data set obtained after grouping the distributed data set to describe Distributed computing tasks with different computing needs.
- FIG. 1 shows a flow chart of one embodiment of a task processing method according to the present application
- FIG. 2 shows an exemplary effect diagram of describing a distributed computing task using a DAG diagram
- FIG. 3 shows an exemplary effect diagram of describing a distributed computing task in a tree structure
- FIG. 4 shows an exemplary effect diagram of describing a distributed computing task using a tree structure and a DAG diagram
- Figure 5 shows a structural representation of one embodiment of a distributed computing framework in accordance with the present application. intention
- FIG. 6 is a block diagram of a computer system suitable for implementing a distributed computing framework of an embodiment of the present application.
- FIG. 1 illustrates a flow 100 of one embodiment of a task processing method in accordance with the present application.
- the method includes the following steps:
- Step 101 Analyze an expression corresponding to the distributed computing task, and construct task description information corresponding to the distributed computing task.
- the distributed computing task can be represented by an expression.
- Task description information can be used to describe distributed computing tasks, and task description information can be referred to as logical plans.
- logical plans When dealing with distributed computing tasks, you can parse the expressions of distributed computing tasks and build logical plans for distributed computing tasks.
- the logical plan of the distributed computing task can include the operator and the domain in which the operator is located.
- the operator can indicate the operation of the data. Taking the student's grade data of the school as an example, the operation of taking the top three scores of the student's grade data can be referred to as the operator who takes the top 3 scores of the student's grade data.
- a domain can be used to represent the grouping of data.
- the domain in which the operator is located can determine the distributed data set to which the operator acts. For example, if the domain of the top 3 operators of the student's grade data is in the grade field, then the grade field represents the distributed data set of the grade data of all the students including the entire school, according to the grade, for the students.
- the distributed data set with the scores of the top 3 operators of the score data is a distributed data set of the score data of the students containing one grade obtained after grouping.
- the logical plan can be defined in the following form:
- Child node null
- the semantics of the entry domain and the leave domain can be defined.
- Enter a domain Start grouping distributed data sets. To enter a domain, you can first access an operator that reads the keywords used in the expression to group the distributed data sets. When leaving a domain, distributed data sets are no longer grouped. All distributed data sets generated by the leaving domain are merged together and flow to the next node.
- the logical plan can be summarized as follows: First, there is a global domain, and the scope of the global domain is the score of all students. There is an input on the global domain, the grade information can be read from the input, and the grade field can be generated. The first 3 operators can be placed in the grade field to take the top 3 scores for the grades of each grade. . At the same time, in the grade field, the class information can be read again to generate the class domain, and the top 3 operators can be placed in the class domain, which is used to take the top 3 students in each class. Finally, the results of the first two top two can be output through the output operator located on the global domain.
- the task description information is a topology
- the topology includes: an operator, a domain, and a domain is used to indicate a range corresponding to the distributed data set.
- the task description information may be a topology.
- the topology can include operators and domains.
- a field is used to represent a grouping of data.
- the parent node of the domain is empty or domain, and the child nodes of the domain are domains or operators.
- the expression includes: a group operator keyword, a grouping keyword, and an operator operator keyword.
- the expression corresponding to the distributed computing task includes: a group operator keyword, a grouping keyword, and an operation operator keyword.
- parsing the expression corresponding to the distributed computing task, and constructing the task description information corresponding to the distributed computing task includes: creating a domain corresponding to the grouping keyword; determining, corresponding to the operation operator keyword An operation operator; constructing a topology, wherein the child nodes of the domain in the topology include: a group operator corresponding to the group operator keyword, and an operation operator.
- a distributed data set for the score data of all students including the school may be determined according to the grouping keyword according to the grade level. Grouping.
- the grouping keyword in the expression contains an operator operator keyword corresponding to the operator of the top 3 of the student's grade data
- the top 3 of the student's grade data may be determined according to the operator operator keyword.
- the distributed data set of the operator action is a distributed data set of the score data of the students containing one grade after grouping.
- a distributed data set for the grade data of the student including one grade can be determined according to the group keyword according to the class keyword. Grouping.
- the grouping keyword in the expression contains the operator operator keyword corresponding to the operator of the top 3 of the student's grade data, the operator operator keyword can be used to determine the top 3 of the student's grade data.
- the distributed data set of the operator function is a distributed data set containing the grade data of the students of one class.
- the distributed data set of the top three operators of the student's performance data is determined as the score data of the students of each grade and the grades of the students of each class. After the data, it is possible to construct a distributed computing task for describing the student's grade data in the first three grades and the top three in the shift. Topology.
- a field that represents the student's grade data is the domain of the student's grade data for all students in the school, ie, the global domain.
- the child nodes of the domain may include an input node, an output node, and a domain representing the grade data of the student as a grade, that is, a grade domain.
- a distributed data set containing grade data for all students in the school can be entered from the input node. The results obtained by taking the top 3 scores of the grades of the students of one grade and the scores of the top 3 of the grades of the grades of the grades of one grade can be summarized and output at the output node.
- the child nodes of the domain representing the grade data of the student are the grouping operator for reading the grade information, the operator operator of the top 3 for the student's grade data, and the grade of the student.
- the range of data is the domain of the class.
- the grouping operator is used to read the grade information from the input node, that is, group the distributed datasets of the grade data of all the students input by the school input by the input node according to the grade, and obtain the grade data of the students including each grade. Distributed data set.
- the top 3 operation operators for the student's grade data are used to take the top 3 scores for each grade student's grade data.
- the child nodes representing the student's grade data in the domain of the class include a grouping operator for reading class information, and an operator operator for taking the top three scores of the student's grade data.
- the grouping operator is used to read the class information, that is, group the distributed data sets of the grade data of the students including one grade according to the class, and obtain a distributed data set containing the grade data of the students of each class.
- the top 3 operation operators for the student's grade data are used to take the top 3 scores for the grade data of each class.
- FIG. 2 shows an exemplary rendering of a distributed computing task using a DAG diagram.
- an input node, a reading grade information node, a reading class information node, a top three node of a score, and an output node are shown.
- the operators represented by the above nodes can form an upstream and downstream relationship to form a DAG graph (Directed Acyclic Graph).
- the input node is an input operator for receiving a distributed data set containing the score data for all students of the school.
- the reading grade information node is a grouping operator for reading the grade information.
- the class information node is read as a grouping operator for reading class information.
- the top 3 nodes in the score are the top 3 operation operators for the grade data of the grade students. And the scores of the students in the class take the top 3 operation operators.
- the output node is an output operator for outputting the score data of the student of the grade level and the result obtained by taking the top three scores of the grade data of the student of the class.
- FIG. 3 illustrates an exemplary rendering of a distributed computing task using a tree structure.
- FIG. 3 an input node, a read grade information node, a read class information node, a top three nodes of an achievement, an output node, a global domain node, a grade domain node, and a class domain node are shown.
- the child nodes of the global domain node contain input nodes, output nodes, and grade domain nodes.
- the input node may be an input operator that receives the input distributed data set containing the student's grade data.
- the output node may be an output operator that outputs a calculation result obtained by distributed calculation of the distributed data set using an operator.
- the child nodes of the grade domain contain the reading of the grade information node and the top 3 nodes of the grade.
- the reading grade information node may be a grouping operator for reading the grade information
- the first three nodes of the score may be the top three operation operators for the student's grade data.
- the child nodes of the class domain contain the reading of the class information node and the top 3 nodes of the score.
- the reading class information node may be a grouping operator that reads the class information
- the first three nodes of the score may be the top three operation operators for the student's performance data.
- FIG. 4 illustrates an exemplary rendering of a distributed computing task using a tree structure and a DAG diagram.
- an input node represented by a solid line, a reading grade information node, a read class information node, a top three nodes of an achievement, an output node, a global domain node represented by a broken line, a grade domain node, Class domain node.
- the input node may be an input operator that receives the input distributed data set containing the student's grade data.
- the output node may be an output operator that outputs a calculation result obtained by distributed calculation of the distributed data set using an operator.
- the reading grade information node may be a grouping operator for reading the grade information, and the first three nodes of the score may be the top three operation operators for the student's grade data.
- the reading class information node may be a grouping operator that reads the class information, and the first three nodes of the score may be the top three operation operators for the student's performance data.
- connection between operators can be used to describe the upstream and downstream relationship of the operator, and the nesting between nodes can be used to describe the parent-child relationship.
- Step 102 based on the task description information, determine a distributed data set for the operator to use.
- the distributed data that the operator acts on may be determined based on the task description information. set.
- the task description information of the distributed computing task constructed in step 101 includes the root node.
- the nodes in the topology structure can be traversed. During the traversal process, it is determined that the top three operation operators for the student's performance data are respectively located in the grade level domain and the class domain, and then the top 3 scores of the student's grade data are determined.
- the distributed data set of the operator functions is the grade data of the students of each grade and the grade data of the students of each class.
- Step 103 Perform an distributed calculation on the distributed data set used by the operator by using an operator.
- the operator after determining the distributed data set for the operator based on the task description information by step 102, the operator can perform distributed computing on the distributed data set for the operator.
- a distributed data set is determined which takes the top 3 operator functions for the student's grade data.
- the top three can be divided into the scores of the grades of the students of each grade and the grades of the grades of the students of each class.
- the result can then be output by the output node in the global domain.
- the present application provides an embodiment of a distributed computing framework, which corresponds to the method embodiment shown in FIG. 2.
- the distributed computing framework 500 of this embodiment includes: a building unit 501, Determination unit 502, calculation unit 503.
- the constructing unit 501 is configured to parse an expression corresponding to the distributed computing task, and construct task description information corresponding to the distributed computing task, where the task description information is used to describe a correspondence between the operator and the distributed data set, where a distributed data set obtained after the distributed data set and/or grouping the distributed data set;
- the determining unit 502 is configured to determine a distributed data set for the operator to use based on the task description information;
- the 503 is configured to perform distributed computing on the distributed data sets used by the operators using the operators.
- the expression includes: a group operator keyword, a grouping keyword, and an operator operator keyword.
- the task description information is a topology
- the topology includes: an operator, a domain, and a domain is used to indicate a range corresponding to the distributed data set.
- the building unit 501 includes: a creating subunit (not shown) configured to create a domain corresponding to the grouping keyword; an operator determining subunit (not shown), Configuring an operation operator for determining an operation operator keyword; a topology construction subunit (not shown) configured to construct a topology, wherein the child nodes of the domain in the topology include: a group operator keyword Corresponding grouping operator and operation operator.
- the child node of the root node of the topology includes: output for outputting a calculation result obtained by using the operator to perform distributed calculation on the distributed data set used by the operator operator.
- FIG. 6 is a block diagram showing the structure of a computer system suitable for implementing the distributed computing framework of the embodiments of the present application.
- computer system 600 includes a central processing unit (CPU) 601 that can be loaded into a program in random access memory (RAM) 603 according to a program stored in read only memory (ROM) 602 or from storage portion 608. And perform various appropriate actions and processes.
- RAM random access memory
- ROM read only memory
- RAM 703 various programs and data required for the operation of the system 600 are also stored.
- the CPU 701, the ROM 602, and the RAM 603 are connected to each other through a bus 604.
- An input/output (I/O) interface 605 is also coupled to bus 604.
- the following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a storage portion 608 including a hard disk or the like. And include such as LAN cards, A communication portion 609 of a network interface card such as a modem. The communication section 609 performs communication processing via a network such as the Internet.
- Driver 610 is also coupled to I/O interface 605 as needed.
- a removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like, is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage portion 608 as needed.
- an embodiment of the present disclosure includes a computer program product comprising a computer program tangibly embodied on a machine readable medium, the computer program comprising program code for executing the method illustrated in the flowchart.
- the computer program can be downloaded and installed from the network via communication portion 609, and/or installed from removable media 611.
- each block of the flowchart or block diagrams can represent a module, a program segment, or a portion of code that includes one or more logic for implementing the specified.
- Functional executable instructions can also occur in a different order than that illustrated in the drawings. For example, two successively represented blocks may in fact be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved.
- each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts can be implemented in a dedicated hardware-based system that performs the specified function or operation. Or it can be implemented by a combination of dedicated hardware and computer instructions.
- the present application further provides a non-volatile computer storage medium, which may be a non-volatile computer storage medium included in the device described in the above embodiments; It may be a non-volatile computer storage medium that exists alone and is not assembled into the terminal.
- the non-volatile computer storage medium stores one or more programs, when the one or more programs are executed by a device, causing the device to: parse an expression corresponding to a distributed computing task, and construct a distributed computing Task description information corresponding to the task, the task description information is used to describe a correspondence between the operator and the distributed data set, wherein the operator distributed data set and/or the distributed data a distributed data set obtained after grouping; determining, based on the task description information, a distributed data set for which the operator is used; and using the operator to make a distributed data set for the operator Perform distributed computing.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (12)
- 一种任务处理方法,其特征在于,所述方法包括:解析分布式计算任务对应的表达式,以及构建分布式计算任务对应的任务描述信息,所述任务描述信息用于描述算子与分布式数据集的对应关系,其中,所述算子作用于分布式数据集和/或对所述分布式数据集进行分组之后得到的分布式数据集;基于所述任务描述信息,确定所述算子所作用于的分布式数据集;利用所述算子对所述算子所作用于的分布式数据集进行分布式计算。
- 根据权利要求1所述的方法,其特征在于,所述表达式包括:分组算子关键字、分组关键字、操作算子关键字。
- 根据权利要求2所述的方法,其特征在于,任务描述信息为拓扑结构,所述拓扑结构包括:算子、域,所述域用于指示分布式数据集对应的范围。
- 根据权利要求3所述的方法,其特征在于,所述解析分布式计算任务对应的表达式,以及构建分布式计算任务对应的任务描述信息包括:创建所述分组关键字对应的域;确定所述操作算子关键字对应的操作算子;构建所述拓扑结构,其中,所述域在所述拓扑结构中的子节点包括:所述分组算子关键字对应的分组算子、所述操作算子。
- 根据权利要求4所述的方法,其特征在于,所述拓扑结构的根节点的子节点包括:用于输出利用所述算子对所述算子所作用于的分布式数据集进行分布式计算得到的计算结果的输出算子。
- 一种分布式计算框架,其特征在于,所述分布式计算框架包括:构建单元,配置用于解析分布式计算任务对应的表达式,以及构建分布式计算任务对应的任务描述信息,所述任务描述信息用于描述算子与分布式数据集的对应关系,其中,所述算子分布式数据集和/或对所述分布式数据集进行分组之后得到的分布式数据集;确定单元,配置用于基于所述任务描述信息,确定所述算子所作用于的分布式数据集;计算单元,配置用于利用所述算子对所述算子所作用于的分布式数据集进行分布式计算。
- 根据权利要求6所述的分布式计算框架,其特征在于,所述表达式包括:分组算子关键字、分组关键字、操作算子关键字。
- 根据权利要求7所述的分布式计算框架,其特征在于,任务描述信息为拓扑结构,所述拓扑结构包括:算子、域,所述域用于指示分布式数据集对应的范围。
- 根据权利要求8所述的分布式计算框架,其特征在于,所述构建单元包括:创建子单元,配置用于创建所述分组关键字对应的域;操作算子确定子单元,配置用于确定所述操作算子关键字对应的操作算子;拓扑结构构建子单元,配置用于构建所述拓扑结构,其中,所述域在所述拓扑结构中的子节点包括:所述分组算子关键字对应的分组算子、所述操作算子。
- 根据权利要求9所述的分布式计算框架,其特征在于,所述拓扑结构的根节点的子节点包括:用于输出利用所述算子对所述算子所作用于的分布式数据集进行分布式计算得到的计算结果的输出算子。
- 一种设备,包括:处理器;和存储器,所述存储器中存储有能够被所述处理器执行的计算机可读指令,在所述计算机可读指令被执行时,所述处理器执行任务处理方法,所述方法包括:解析分布式计算任务对应的表达式,以及构建分布式计算任务对应的任务描述信息,所述任务描述信息用于描述算子与分布式数据集的对应关系,其中,所述算子作用于分布式数据集和/或对所述分布式数据集进行分组之后得到的分布式数据集;基于所述任务描述信息,确定所述算子所作用于的分布式数据集;利用所述算子对所述算子所作用于的分布式数据集进行分布式计算。
- 一种非易失性计算机存储介质,所述计算机存储介质存储有能够被处理器执行的计算机可读指令,当所述计算机可读指令被处理器执行时,所述处理器执行任务处理方法,所述方法包括:解析分布式计算任务对应的表达式,以及构建分布式计算任务对应的任务描述信息,所述任务描述信息用于描述算子与分布式数据集的对应关系,其中,所述算子作用于分布式数据集和/或对所述分布式数据集进行分组之后得到的分布式数据集;基于所述任务描述信息,确定所述算子所作用于的分布式数据集;利用所述算子对所述算子所作用于的分布式数据集进行分布式计算。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2019505219A JP6781819B2 (ja) | 2016-09-30 | 2016-10-14 | タスク処理方法及び分散コンピューティングフレームワークシステム |
KR1020197002250A KR102161545B1 (ko) | 2016-09-30 | 2016-10-14 | 태스크 프로세싱 방법 및 분산 컴퓨팅 프레임워크 |
EP16917455.4A EP3474139A4 (en) | 2016-09-30 | 2016-10-14 | TASK PROCESSING AND DISTRIBUTED COMPUTER FRAMEWORK |
US16/352,678 US11709894B2 (en) | 2016-09-30 | 2019-03-13 | Task processing method and distributed computing framework |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610873646.X | 2016-09-30 | ||
CN201610873646.XA CN106383738B (zh) | 2016-09-30 | 2016-09-30 | 任务处理方法和分布式计算框架 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/352,678 Continuation US11709894B2 (en) | 2016-09-30 | 2019-03-13 | Task processing method and distributed computing framework |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018058707A1 true WO2018058707A1 (zh) | 2018-04-05 |
Family
ID=57937180
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2016/102124 WO2018058707A1 (zh) | 2016-09-30 | 2016-10-14 | 任务处理方法和分布式计算框架 |
Country Status (6)
Country | Link |
---|---|
US (1) | US11709894B2 (zh) |
EP (1) | EP3474139A4 (zh) |
JP (1) | JP6781819B2 (zh) |
KR (1) | KR102161545B1 (zh) |
CN (1) | CN106383738B (zh) |
WO (1) | WO2018058707A1 (zh) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107193579B (zh) * | 2017-03-29 | 2021-04-27 | 武汉斗鱼网络科技有限公司 | 计划任务的实现方法及装置 |
CN113553286A (zh) * | 2020-04-24 | 2021-10-26 | 中科寒武纪科技股份有限公司 | 基于多处理节点来构建通信拓扑结构的方法和设备 |
KR20220097631A (ko) | 2020-12-30 | 2022-07-08 | 주식회사 프리딕션 | 텍스트 기반의 문서에 대하여 관련 문서를 추천하는 관련 문서 추천 시스템 및 방법, 컴퓨터 프로그램 및 컴퓨터 판독가능 기록 매체 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101820444A (zh) * | 2010-03-24 | 2010-09-01 | 北京航空航天大学 | 一种基于描述信息匹配相似度的资源服务匹配与搜索方法 |
CN101853179A (zh) * | 2010-05-10 | 2010-10-06 | 深圳市极限网络科技有限公司 | 基于插件执行任务分解的通用分布式动态运算技术 |
CN105760511A (zh) * | 2016-02-24 | 2016-07-13 | 南京信息职业技术学院 | 一种基于storm的大数据自适应拓扑处理方法 |
CN105824957A (zh) * | 2016-03-30 | 2016-08-03 | 电子科技大学 | 分布式内存列式数据库的查询引擎系统及查询方法 |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4846736B2 (ja) * | 2005-12-22 | 2011-12-28 | 独立行政法人海洋研究開発機構 | 並列処理支援装置 |
JP4073033B1 (ja) | 2007-04-27 | 2008-04-09 | 透 降矢 | 結合演算の処理機能の向上を考慮した合成関係演算を利用したマルチオペレーション・プロセッシングを用いたデータベースのクエリー処理システム |
US8195648B2 (en) * | 2009-10-21 | 2012-06-05 | Microsoft Corporation | Partitioned query execution in event processing systems |
CN102479217B (zh) * | 2010-11-23 | 2015-07-15 | 腾讯科技(深圳)有限公司 | 一种分布式数据仓库中实现计算均衡的方法及装置 |
DE102011103305A1 (de) * | 2011-06-03 | 2012-12-06 | Eads Deutschland Gmbh | Verfahren der Datenfusion und Bereitstellung von Information in einem Computersystem |
CN102508640B (zh) * | 2011-10-27 | 2015-04-29 | 西北工业大学 | 基于任务分解的分布式rfid复杂事件检测方法 |
US20130339934A1 (en) * | 2012-06-13 | 2013-12-19 | Josef Troch | Updating virtualized services |
US9910894B2 (en) * | 2012-07-16 | 2018-03-06 | Microsoft Technology Licensing, Llc | Data scope origination within aggregation operations |
US9542400B2 (en) * | 2012-09-07 | 2017-01-10 | Oracle International Corporation | Service archive support |
JP6020014B2 (ja) * | 2012-10-02 | 2016-11-02 | 日本電気株式会社 | 分散データストア管理装置、分散並列処理実行装置、分散並列処理システム、分散データストア管理方法、分散並列処理実行方法、および、コンピュータ・プログラム |
US9367806B1 (en) * | 2013-08-08 | 2016-06-14 | Jasmin Cosic | Systems and methods of using an artificially intelligent database management system and interfaces for mobile, embedded, and other computing devices |
US9646003B2 (en) * | 2013-11-20 | 2017-05-09 | Wolfram Research, Inc. | Cloud storage methods and systems |
US10776325B2 (en) * | 2013-11-26 | 2020-09-15 | Ab Initio Technology Llc | Parallel access to data in a distributed file system |
CN104809242B (zh) * | 2015-05-15 | 2018-03-02 | 成都睿峰科技有限公司 | 一种基于分布式结构的大数据聚类方法和装置 |
CN105808746A (zh) * | 2016-03-14 | 2016-07-27 | 中国科学院计算技术研究所 | 一种基于Hadoop体系的关系型大数据无缝接入方法及系统 |
US10453074B2 (en) * | 2016-07-08 | 2019-10-22 | Asapp, Inc. | Automatically suggesting resources for responding to a request |
-
2016
- 2016-09-30 CN CN201610873646.XA patent/CN106383738B/zh active Active
- 2016-10-14 JP JP2019505219A patent/JP6781819B2/ja active Active
- 2016-10-14 KR KR1020197002250A patent/KR102161545B1/ko active IP Right Grant
- 2016-10-14 WO PCT/CN2016/102124 patent/WO2018058707A1/zh unknown
- 2016-10-14 EP EP16917455.4A patent/EP3474139A4/en not_active Ceased
-
2019
- 2019-03-13 US US16/352,678 patent/US11709894B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101820444A (zh) * | 2010-03-24 | 2010-09-01 | 北京航空航天大学 | 一种基于描述信息匹配相似度的资源服务匹配与搜索方法 |
CN101853179A (zh) * | 2010-05-10 | 2010-10-06 | 深圳市极限网络科技有限公司 | 基于插件执行任务分解的通用分布式动态运算技术 |
CN105760511A (zh) * | 2016-02-24 | 2016-07-13 | 南京信息职业技术学院 | 一种基于storm的大数据自适应拓扑处理方法 |
CN105824957A (zh) * | 2016-03-30 | 2016-08-03 | 电子科技大学 | 分布式内存列式数据库的查询引擎系统及查询方法 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3474139A4 * |
Also Published As
Publication number | Publication date |
---|---|
CN106383738A (zh) | 2017-02-08 |
JP2019528522A (ja) | 2019-10-10 |
KR20190020800A (ko) | 2019-03-04 |
CN106383738B (zh) | 2019-10-11 |
US20190213217A1 (en) | 2019-07-11 |
KR102161545B1 (ko) | 2020-10-05 |
JP6781819B2 (ja) | 2020-11-04 |
EP3474139A4 (en) | 2019-06-26 |
US11709894B2 (en) | 2023-07-25 |
EP3474139A1 (en) | 2019-04-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
AU2019213302B2 (en) | Filtering data lineage diagrams | |
US20240126596A1 (en) | Scheduling operations on a computation graph | |
CN109960810B (zh) | 一种实体对齐方法及装置 | |
CN112106056A (zh) | 构造虚构的话语树来提高回答聚敛性问题的能力 | |
US20140330871A1 (en) | Identifying common data objects representing solutions to a problem in different disciplines | |
US9466041B2 (en) | User selected flow graph modification | |
JP5494999B1 (ja) | テキストマイニングシステム、テキストマイニング方法及びプログラム | |
WO2022218186A1 (zh) | 个性化知识图谱的生成方法、装置及计算机设备 | |
CN111159220B (zh) | 用于输出结构化查询语句的方法和装置 | |
US20150007084A1 (en) | Chaining applications | |
US20160232229A1 (en) | Filtering data lineage diagrams | |
WO2020005769A1 (en) | Visualization of user intent in virtual agent interaction | |
JP2019133622A (ja) | Apiパラメータのマッピング | |
US11709894B2 (en) | Task processing method and distributed computing framework | |
CN106445645B (zh) | 用于执行分布式计算任务的方法和装置 | |
JP2022166215A (ja) | 文字位置決めモデルのトレーニング方法及び文字位置決め方法 | |
CN106445913A (zh) | 基于MapReduce的语义推理方法及系统 | |
US10983997B2 (en) | Path query evaluation in graph databases | |
Cheng et al. | Optimal alignments between large event logs and process models over distributed systems: An approach based on Petri nets | |
US20180341645A1 (en) | Methods and systems for translating natural language requirements to a semantic modeling language statement | |
KR101870658B1 (ko) | 언어지능 모듈 실시간 분산처리 최적화 시스템 및 방법 | |
CN115358397A (zh) | 一种基于数据采样的并行图规则挖掘方法及装置 | |
CN115168673B (zh) | 一种数据的图形化处理方法、装置、设备及存储介质 | |
Bernardelli | Econometric modeling of panel data using parallel computing with Apache Spark | |
Kemper et al. | 1 „Advancing Analytical Database Systems “von Andreas Michael Kipf, TU München, Januar 2020 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16917455 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20197002250 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2016917455 Country of ref document: EP Effective date: 20190115 |
|
ENP | Entry into the national phase |
Ref document number: 2019505219 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |