SG11201908969WA - Systems and methods for performing data processing operations using variable level parallelism - Google Patents

Abstract

FIGBEGIN ACCESS INFORMATION SPECIFYINGA DATAFLOW GRAPI- h-202 —204 OBTAIN LAYOUT(S) FOR INPUT NODES OBTAIN LAYOUT(SIFOR OUTPLT NODE;S) L-206 1 ,-- 208 ASSIGN LAYOUTS \"0 INTERMEDIATE NODES OF \"FE DA - WOW GRAPH ____\"..-208a PROPAGATE LAYOUTS FRCM OUTPUT NODE(S) TO INPUT NODE(S 210 ADJACENT NODES NO HAVE MISIXCHED LAYOUTS? YES INSERT REPARTIONING OPERATION r 212 C END W O 20 18/ 18367 6 Al PROPAGATE LAYOUTS FROM INPUT NODE1S) TO OUTPUT NODES (12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) (19) World Intellectual Property Organization International Bureau (43) International Publication Date 04 October 2018 (04.10.2018) W I PO I PCT ill Hu omits °nolo HE mil moo mil iflo oimIE (10) International Publication Number WO 2018/183676 Al (51) International Patent Classification: GOOF 9 / 50 (2006.01) (21) International Application Number: PCT/US2018/025144 (22) International Filing Date: 29 March 2018 (29.03.2018) (25) Filing Language: English (26) Publication Language: English (30) Priority Data: 62/478,390 29 March 2017 (29.03.2017) US (71) Applicant: AB INITIO TECHNOLOGY LLC [US/US]; 201 Spring Street, Lexington, MA 02421 (US). (72) Inventor: DICKIE, Garth, Allen; 161 Parker Road, Fram- ingham, MA 01702 (US). (74) Agent: RUDOY, Daniel, G.; Wo1f, Greenfield & Sacks, P.C., 600 Atlanitc Avenue, Boston, MA 02210-2206 (US). (81) Designated States (unless otherwise indicated, for every kind of national protection available): AE, AG, AL, AM, AO, AT, AU, AZ, BA, BB, BG, BH, BN, BR, BW, BY, BZ, CA, CH, CL, CN, CO, CR, CU, CZ, DE, DJ, DK, DM, DO, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN, HR, HU, ID, IL, IN, IR, IS, JO, JP, KE, KG, KH, KN, KP, KR, KW, KZ, LA, LC, LK, LR, LS, LU, LY, MA, MD, ME, MG, MK, MN, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM, PA, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SA, SC, SD, SE, SG, SK, SL, SM, ST, SV, SY, TH, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW. (84) Designated States (unless otherwise indicated, for every kind of regional protection available): ARIPO (BW, GH, GM, KE, LR, LS, MW, MZ, NA, RW, SD, SL, ST, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU, TJ, TM), European (AL, AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU, LV, MC, MK, MT, NL, NO, PL, PT, RO, RS, SE, SI, SK, SM, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, KM, ML, MR, NE, SN, TD, TG). Published: with international search report (Art. 21(3)) (54) Title: SYSTEMS AND METHODS FOR PERFORMING DATA PROCESSING OPERATIONS USING VARIABLE LEVEL PARALLELISM (57) : Techniques for determining processing layouts to nodes of a dataflow graph. The techniques include: obtaining information specifying a 200 dataflow graph, the dataflow graph comprising a plurality of nodes and a plural- ity of edges connecting the plurality nodes, the plurality of edges representing flows of data among nodes in the plurality of nodes, the plurality of nodes com- prising: a first set of one or more nodes; and a second set of one or more nodes disjoint from the first set of nodes; obtaining a first set of one or more processing layouts for the first set of nodes; and determining a processing layout for each node in the second set of nodes based on the first set of processing layouts and one or more layout determination rules, the one or more layout determination rules including at least one rule for selecting among processing layouts having different degrees of parallelism.