CN116248757A

CN116248757A - Method for setting format of executable file, method and device for migrating data stream program

Info

Publication number: CN116248757A
Application number: CN202211531189.8A
Authority: CN
Inventors: 罗秋明
Original assignee: Shenzhen Atu Technology Co ltd
Current assignee: Shenzhen Atu Technology Co ltd
Priority date: 2022-12-01
Filing date: 2022-12-01
Publication date: 2023-06-09

Abstract

The invention discloses a format setting method of an executable file, a data stream program migration method and a device, wherein the format setting method of the executable file is used for creating a node program topological graph formed based on a starting address list and a topology information list of each data stream node in the executable file, and when an operating system participates in data scheduling, data layout and data optimization in a data stream execution mode, the operating system is beneficial to quickly obtaining the topology information and the starting address of each data stream node, aligning at a page boundary based on the starting address of each data stream node, and is beneficial to the dynamic migration of subsequent nodes by taking the node as a unit, thereby improving the running speed of the system. The data flow program migration method is based on a node program topological graph-DAG graph, so that mapping and demapping of virtual pages and real pages can be performed in a basis, and the system running speed can be remarkably improved.

Description

Method for setting format of executable file, method and device for migrating data stream program

Technical Field

The present invention relates to the field of executable file format setting, and in particular, to a method for setting a format of an executable file, and a method and apparatus for migrating a data stream program.

Background

Executable files refer to files which can be loaded and executed by an operating system, and the executable files are presented in different modes under different operating system environments. In the windows operating system, the executable file may be an exe file, while in the Linux operating system environment, the executable file is an ELF file (Executable Linkable Format).

The data flow program is executed in a data driving mode, data to be processed is distributed to each core, calculation and communication of the data are separated, and potential parallelism in the flow program is fully mined by utilizing the parallelism characteristic of software pipelining through task scheduling and distribution, so that loads among the cores are balanced.

Because most of the current software and hardware platforms are designed based on control flow programs, the ELF file format is also matched with the mode of the control flow programs, and the ELF file format in the mode cannot be directly applied to the ELF files of the data flow programs. Therefore, in order to coordinate with the operating system of the data flow program, the ELF file format of the data flow program also needs to be adapted and modified accordingly.

On current mainstream computing systems, for example: under the environment of Linux operating systems running on hardware platforms such as x86, ARM, MIPS and the like, ELF files mainly record the content of codes and data, and relevant information of ELF file formats is recorded on application programs, so that when the ELF files based on data stream programs execute running tasks, the data stream executing modes of the ELF files are completely and automatically responsible by the application programs, or the ELF files are supported by the application programs and a run time library of user states together, the operating systems cannot participate in data scheduling, data layout and data optimization in the data stream executing modes, and finally, the data stream migration speed is slow, and the system running is influenced.

Disclosure of Invention

Therefore, the technical problem to be solved by the invention is to overcome the problems that an operating system in the prior art cannot participate in data scheduling, data layout and data optimization in a data stream execution mode, and finally the data stream migration speed is slow and the system operation is affected, so as to provide a format setting method of an executable file, a data stream program migration method and a device.

According to a first aspect, the embodiment of the invention also discloses a method for setting an executable file format based on a data stream program, which comprises the following steps:

acquiring each data stream node of the function code;

creating a starting address list and a topology information list of each data stream node, wherein the topology information list comprises: the total number of the data stream nodes and/or the topological relation among the data stream nodes and/or the sequence of execution of the data stream nodes and/or the current scheduling execution state of the data stream nodes during dynamic migration;

creating a node program topological graph formed based on the starting address list and the topological information list of each data stream node;

arranging the starting address list and the topology information list to a first program section, and arranging each data stream node to a second program section;

And forming a target executable file according to the contents of the first program segment and the second program segment, and adding the position identification information of the node program topological graph on the target executable file.

By executing the embodiment, the node program topological graph formed based on the starting address list and the topology information list of each data stream node is created in the executable file, when an operating system participates in data scheduling, data layout and data optimization in a data stream execution mode, the operating system is facilitated to quickly acquire the topology information and the starting address of each data stream node, the starting addresses of the data stream nodes are aligned at the page boundaries based on the starting addresses of the data stream nodes, and the node program topological graph is facilitated to dynamically migrate subsequent nodes by taking the nodes as units, so that the running speed of the system is improved.

With reference to the first aspect, in an implementation manner of the first aspect, when linking the data stream nodes, linking the data stream nodes into PIC position independent codes, and independently segmenting the PIC position independent codes; when linking the data stream nodes, aligning the start points of the data stream nodes at page boundaries based on the start addresses of the data stream nodes.

By executing the embodiment, when compiling each data stream node, each data stream node is linked into PIC position independent codes, and the PIC position independent codes are independently segmented. When each data stream node is called, the data stream node can normally operate in any stack space, the data stream node can normally operate in any position, address information of each data stream node is not required to be modified again, and finally the calling speed of each data stream node is improved. And meanwhile, the starting points of the data stream nodes are aligned at page boundaries, so that the subsequent migration of the executable file is facilitated, the executable file is allowed to take the nodes as migration units, the migration possibility and flexibility of the data stream nodes are improved, and the data stream operation speed is further improved.

With reference to any implementation manner of the first aspect, in another implementation manner of the first aspect, when linking the data flow nodes, address information of input data of the data flow nodes is relocated to the target stack area, and address information of output data of the data flow nodes is not relocated temporarily.

By implementing the embodiment, the migration in the process space can be simply carried out without complex dynamic linking and repositioning again, and only the initial address of the output data is needed to be simply repositioned. When each data stream node is linked, the address information of the input data of each data stream node is loaded in the stack area, which is beneficial to improving the running speed of the data stream when each data stream node is dynamically migrated.

According to a second aspect, an embodiment of the present invention provides a migration method of an executable file data stream program, including the following steps:

determining a data stream node to be migrated in a target executable file;

repositioning the starting address of the data flow node to be migrated to the target stack area based on the node program topological graph in the setting method of the executable file format based on the data flow program in the first aspect or any implementation mode of the first aspect, and updating the current starting address of the target stack area in the node program topological graph;

acquiring the current scheduling running state of the data flow node to be migrated according to the node program topological graph;

and determining whether the data stream node to be migrated of the virtual storage area is mapped on a physical storage area according to the current scheduling running state of the data stream node to be migrated.

With reference to the second aspect, in any implementation manner of the second aspect, determining whether to map the to-be-migrated data stream node of the virtual storage area on the physical storage area according to the current scheduled running state of the to-be-migrated data stream node includes:

If the current scheduling running state of the data stream node to be migrated is an executed state, the mapping corresponding relation of the data stream node to be migrated of the virtual storage area on a physical storage area is released;

if the current scheduling running state of the data stream node to be migrated is an executing state, locking the data stream node to be migrated of the virtual storage area on a physical storage area;

and if the current scheduling running state of the data stream node to be migrated is the state to be executed, mapping the data stream node to be migrated of the virtual storage area on a physical storage area is not needed.

By executing the implementation manner of the second aspect, since the node program topological graph-DAG graph in the implementation manner of the first aspect is used, mapping and demapping of virtual and real pages are performed in a basis instead of blindly using the LRU algorithm as the page changing algorithm in the embodiment of the present invention, so that the page missing rate can be significantly reduced, and the running speed of the system can be further improved.

With reference to any implementation manner of the second aspect, in another implementation manner of the second aspect, the migration method of the executable file data stream program further includes: and if the storage space of the physical storage area is smaller than or equal to a preset threshold value, mapping the data stream nodes to be migrated of the virtual storage area on the physical storage area according to the priority execution sequence of the node program topological graph.

By executing the embodiment, the address mapping is preferentially established for the nodes which are positioned in front of the virtual page, and the mapping relation between the virtual page and the real page can be flexibly established by combining the actual situation of the physical storage area, so that the migration efficiency of the data flow node program is improved, and the running speed of the system is improved.

With reference to any implementation manner of the second aspect, in another implementation manner of the second aspect, when the to-be-migrated data stream node is migrated, repositioning is performed on an input address of the to-be-migrated data stream node to the target stack area, repositioning is not required on an output address of the to-be-migrated data stream node, the output data of the to-be-migrated data stream node is written into the stack area where the subsequent node is located, repositioning is not required on the output address of the to-be-migrated data stream node, and the output data of the to-be-migrated data stream node is written into the stack area where the subsequent node is located.

With reference to any implementation manner of the second aspect, in another implementation manner of the second aspect, based on the node program topology diagram, when the to-be-migrated data flow node is migrated, a stack frame start address of the to-be-migrated data flow node executed on a same processor is the same, and when the to-be-migrated data flow node is executed, a page table address mechanism is used to limit a data memory address of the to-be-migrated data flow node to be in a memory space of a node.

According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer instructions are stored, where the computer instructions are configured to cause the computer to perform the method for setting an executable file format based on a data stream program according to the first aspect or any implementation manner of the first aspect; or, the migration method of the executable file data stream program in the second aspect or any implementation manner of the second aspect.

According to a fourth aspect, an embodiment of the present invention provides a computer apparatus, comprising: the device comprises a memory and a processor, wherein the memory and the processor are in communication connection, the memory stores computer instructions, and the processor executes the computer instructions so as to execute the method for setting the executable file format based on the data flow program in the first aspect or any implementation mode of the first aspect; or, the migration method of the executable file data stream program in the second aspect or any implementation manner of the second aspect.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of a specific example of a method for setting an executable file format based on a data stream program according to an embodiment of the present invention;

FIG. 2 is a topology of a node procedure in an embodiment of the invention;

FIG. 3 is a schematic diagram illustrating a configuration of a data structure in an executable file according to an embodiment of the present invention;

FIG. 4A is a diagram illustrating loading of a conventional ELF executable file;

FIG. 4B is a diagram illustrating the loading of an ELF executable file in an embodiment of the present invention;

FIG. 5 is a schematic diagram of writing data to a relay node b for a predecessor node a according to an embodiment of the present invention;

FIG. 6 is a flowchart of a specific example of a migration method of an executable file data stream program according to an embodiment of the present invention;

FIG. 7 is a schematic diagram illustrating an operation mode of an executable file according to an embodiment of the present invention;

FIG. 8 is a schematic diagram illustrating migration of a node program b of a data stream to be migrated in an executable file according to an embodiment of the present invention;

fig. 9 is a schematic diagram of a current scheduling execution state of each data flow node in the embodiment of the present invention;

FIG. 10 is a schematic diagram illustrating migration of a data stream node program of an executable file according to an embodiment of the present invention;

FIG. 11 is a schematic diagram illustrating the configuration and migration of executable file formats based on a data stream program according to an embodiment of the present invention;

Fig. 12 is a schematic structural diagram of a setting device of an executable file format based on a data stream program according to an embodiment of the present invention;

FIG. 13 is a block diagram illustrating an exemplary embodiment of a migration apparatus for executable file data stream programs;

fig. 14 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, or can be communicated inside the two components, or can be connected wirelessly or in a wired way. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.

In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.

The embodiment of the invention discloses a setting method of an executable file format based on a data stream program, which is shown in fig. 1 and comprises the following steps:

step S11: each data stream node of the function code is acquired.

Each data flow node of the function code is executed in a data driving mode, data to be processed is distributed to each core, calculation and communication of the data are separated, and potential parallelism in a flow program is fully mined by utilizing the parallelism characteristic of software flow through task scheduling and distribution, so that loads among the cores are balanced. The connection operation of the input and output is well defined. Therefore, each data flow node does not adopt a command mode, and relevant operations are immediately executed whenever data is ready and input is valid, so that the data flow programming is parallel in nature and can well run in a multi-core processor and a large-scale distributed system. For example: each data flow node for acquiring the function code is a, b, c, d, e, f, g.

Step S12: creating a starting address list and a topology information list of each data stream node, wherein the topology information list comprises: the total number of the data stream nodes and/or the topological relation among the data stream nodes and/or the sequence of execution of the data stream nodes and/or the current scheduling execution state of the data stream nodes during dynamic migration.

For example: the initial address list is used for recording the initial position of each data stream node, and the initial address list is equivalent to the initial address index list of each data stream node; for example: the start address list records the start position of each data stream node a, b, c, d, e, f, g when each data stream node is a, b, c, d, e, f, g.

For example: a, b, c, d, e, f, g for each data stream node, the total number of each data stream node in the topology information list is 7; for example: the topology relation among the data stream nodes in the topology information list is the input-to-output relation of any data stream node, namely the predecessor-successor relation of any data stream node; for example: for each data flow node, a, b, c, d, if a is executed first, b is executed then c and d are executed, the sequence of execution in the topology information list is sequentially as follows; a- & gt, b- & gt, c and d, wherein a is a predecessor node of b, and c and d are successor nodes of b; for example: for the current scheduling execution state of each data flow node a, b and c, a is an executed state, b is a positive execution state, c is a non-executed state, and the current scheduling execution state of each data flow node during dynamic migration comprises: an executed state, a executing state, and a non-executing state;

In a specific embodiment, the topology information list of each data stream node further includes: the data size of the topological relation among the data stream nodes and/or the entry function address of each data stream node and/or the stack frame size of the stack area to which each data stream node belongs.

For example: as shown in fig. 2, the topology input item of the node e is a node b, and the topology output item is a node g; the input term of the node c is a and b, and the topology output term is f. The data size of the topological relation between the data stream nodes is 2 bytes, and the data size of the data stream nodes is 1 byte; the entry function address of each data stream node and the stack frame size of the stack area to which each data stream node belongs are typically fixed values, for example: the entry function address of each data stream node and the stack frame size of the stack area to which each data stream node belongs are 256 bytes.

Step S13: a node program topology map formed based on the starting address list and the topology information list of each data stream node is created.

For example: the static examples of the data stream nodes are described as a directed graph DAG according to the starting address list of each node and the topology information list of each node, namely the node program topology graph in the above. In fig. 2, nodes in the diagram represent calculation units, edges represent data transmission paths, data is transmitted between adjacent nodes through the edges, the nodes consume the data to perform calculation, and the generated data is output to an input-output sequence as an input of a next calculation unit. The topological relation among the data stream nodes is described through the node program topological graph, so that the input and output among the data stream nodes can be expressed more vividly.

Step S14: arranging a starting address list and a topology information list to the first program section, and arranging each data stream node to the second program section.

For example: as shown in fig. 3, the first program segment records a starting address list and a topology information list; the second program segment records the data flow function, namely each of the data flow nodes df1-dfn described above.

In a specific embodiment, the content data size of the first program segment is set to a preset byte value. For example: the data size of the initial address list is M bytes, the data size of the topology information list is N bytes, therefore, the content data size of the first program segment pair is m+n bytes, the preset byte value is m+n bytes, and certainly, the preset byte value is not limited by this and can be set in combination with practical situations.

In a specific embodiment, the offset of the first program segment in the file is stored on the program header of the target executable file.

Step S15: and forming a target executable file according to the contents of the first program segment and the second program segment, and adding the position identification information of the node program topological graph on the target executable file.

In the executable file there is a single segment, i.e. the content of the first program segment, for recording the node topology information of the data stream, and the above-mentioned location identification information is used for indicating the location of the node program topology map in the executable file, which location identification information indicates the initial byte offset of the node program topology map in the executable file, which location identification is advantageous for fast retrieval of the node program topology map, i.e. the DAG map in fig. 2.

In a specific embodiment, when linking each data stream node, linking each data stream node into PIC position independent codes, and independently forming the PIC position independent codes into sections; when linking the data stream nodes, the start points of the data stream nodes are aligned at page boundaries to form segments.

When linking each data stream node, each data stream node is linked into PIC position independent codes, and the PIC position independent codes are independently segmented. When each data stream node is called, the data stream node can normally operate in any stack space, the data stream node can normally operate in any position, address information of each data stream node is not required to be modified again, and finally the calling speed of each data stream node is improved.

In a specific embodiment, when linking the data stream nodes, the start points of the data stream nodes are aligned at page boundaries based on the start addresses of the data stream nodes to form segments. The starting points of the data stream nodes are aligned at the page boundaries, so that the subsequent migration of the executable files is facilitated, the executable files are allowed to take the nodes as migration units, the possibility and flexibility of the data stream node migration are improved, and the data stream operation speed is further improved.

FIG. 4A is a schematic diagram of loading a conventional ELF executable file, and the memory is not loaded in units of nodes, and FIG. 4B is a schematic diagram of loading an ELF executable file in an embodiment of the present invention, wherein the start points of the nodes of each data stream are aligned at page boundaries.

In a specific embodiment, when linking the data stream nodes, the address information of the input data of each data stream node is limited in the stack area, and the address information of the output data of each data stream node is not relocated.

For example: when linking the functions of each data stream node, the input data of each data stream node is in its associated stack, and its output data address has not been relocated (within the stack frame of the subsequent data stream node). Similar to the dynamic linking process of the dynamic library, the difference is that the function codes of these data stream nodes are defined not to access global variables, so that migration in the process space can be simplified without complex dynamic linking and relocation again, and only simple relocation of the start address of the output data is required. Therefore, when the data stream nodes are linked, the address information of the input data of each data stream node is limited in the stack area, which is beneficial to improving the running speed of the data stream when the following data stream nodes dynamically migrate.

For example: as shown in fig. 5, a data stream node a outputs data to a data stream node b, which is a schematic diagram of a predecessor node a writing data to a successor node b. The dataflow node a gets bound to thread 1 with its input data in the stack of thread 1, but the output data needs to be written to the stack of dataflow node program b tasks, and must be bound to a given thread (here thread 2) and learn of its stack before it can determine its output data address.

After the node of the function of each data flow node is initially loaded, the function code of each data flow node can be migrated from the original NUMA node to another NUMA node along with the calculation promotion of the node program topological graph-DAG graph. After the migration, only the starting point position of the output data is needed to be relocated. For example: in FIG. 5, after the data flow node b task has migrated to thread 3, which is bound to another NUMA node, then the data written by a is to be diverted to a new location (from right-most in FIG. 5 to left-most in FIG. 5).

In a specific implementation manner, the method for setting an executable file format based on a data flow program in the embodiment of the present invention is based on a node program topology diagram, and when each data flow node is migrated, global variables of each data flow node are not accessed when each data flow node is accessed.

The executable file is allowed to take the node as a migration unit, when the global variable of each data stream node is not accessed, only the starting address information of each data stream node is needed to be repositioned, the global variable of each data stream node is not needed to be accessed, and the running speed of the data stream can be further improved.

The following specific example is used to illustrate a method for setting an executable file format based on a data flow program in an embodiment of the present invention, wherein a node of a data flow node program diagram, i.e. a DAG diagram, is used as an illustration, and a topological relation (a preceding-going-following relation) is used to record, and meanwhile, a node function pointer and a required stack frame size are recorded, for example:

in the above program code function, wherein pre_count is used to record the number of predecessor nodes and pres [ ] is used to record pointers of the pre_count predecessor nodes. The succ_count and succ [ ] are then used to record the successor node relationships; node-id is used to record the identified node.

In FIG. 2, there are seven nodes, and if recorded with the variable mynodes, there should be struct node mynodes [7]. The mynodes are formed individually into a section of an ELF executable file that individually forms an ELF executable segment, function (void) for recording a list of starting addresses for each data stream node.

For all output data, addressing is done in a "base + offset" manner. For example: one node transmits 10 bytes of data to the successor node. Then its base address is the bottom of the stack of successor nodes on the corresponding processor and its base address of the input data of successor nodes is addr1. And then writing the data into the address range of [ addr 1-addr1+9 ] to finish data transmission.

According to the setting method of the executable file format based on the data flow program, the step S11-step S15 is executed, and the node program topological graph formed based on the initial address list and the topology information list of each data flow node is created in the executable file, so that when an operating system participates in data scheduling, data layout and data optimization in a data flow execution mode, the operating system is facilitated to quickly acquire the topology information and the initial address of each data flow node, and the initial address of each data flow node is aligned at a page boundary based on the initial address of each data flow node, and the dynamic migration of the subsequent node is facilitated by taking the node as a unit, and the running speed of the system is improved.

Based on the same conception, the embodiment of the invention also discloses a migration method of the executable file data stream program, as shown in fig. 6, comprising the following steps:

Step S61: and determining the data flow node to be migrated in the target executable file.

For example: at each data flow node a, b, c, d, e, f, g included in the target executable file, the data flow node to be migrated is a.

Step S62: based on the node program topology diagram in the setting method of the executable file format of the data flow program in the above embodiment, the starting address of the data flow node to be migrated is relocated to the target stack area, and the current starting address of the target stack area is updated in the node program topology diagram. By implementing step S62, it is ensured that the scheduler can migrate a task and can correctly find the entry address of the task.

In fig. 5, a is a data stream node to be migrated, a is also a predecessor node of b, the predecessor node a writes data to a successor node b, and the data stream node a outputs data to the data stream node b. The dataflow node a gets bound to thread 1 to run with its input data in the thread 1 target stack area, but the output data needs to be written to the stack of dataflow node b tasks, and must be bound to a given thread (here thread 2) and learn of its stack before it can determine its output data address.

Therefore, the node program topological graph is generated based on the starting address list of each data stream node, the starting address of the data stream node a to be migrated is relocated to the target stack area based on the node program topological graph-DAG graph, the starting points of the data stream nodes to be migrated are aligned at page boundaries, and the data stream node b to be migrated can be rapidly located without relocating the output address of the data stream node a to be migrated.

Fig. 7 is a schematic diagram illustrating an operation manner of an executable file in an embodiment of the present invention. After loading each data stream node in the above embodiment into the memory, the migration of tasks is allowed, and based on the setting method of the executable file format in the embodiment, we only use stack variables, so that the function code can be migrated to where to run normally. Since the DAG map information recorded in the process PCB (process control block) has function node information, the new address after migration is updated to the function start address information on task b in the DAG map information in the PCB. Fig. 8 is a schematic migration diagram of a data flow node b to be migrated in an executable file according to an embodiment of the present invention.

Step S63: and acquiring the current scheduling running state of the data flow node to be migrated according to the node program topological graph.

Because the topology information list of the node program topology chart records the current scheduling running state of each data stream node in the dynamic migration process, the current scheduling running state comprises the following steps: an executed state, a executing state, and a non-executing state. As shown in fig. 9, the current scheduling operation state of each data flow node in the embodiment of the present invention is shown.

Step S64: and determining whether to map the data stream nodes to be migrated of the virtual storage area on the physical storage area according to the current scheduling running state of the data stream nodes to be migrated.

In a preferred embodiment, in fig. 6, the step S64, according to the current scheduled running state of the data flow node to be migrated, of determining whether to map the data flow node to be migrated of the virtual storage area on the physical storage area, includes:

step S641: and if the current scheduling running state of the data stream node to be migrated is the executed state, the mapping corresponding relation of the data stream node to be migrated of the virtual storage area on the physical storage area is removed.

Step S642: and if the current scheduling running state of the data stream node to be migrated is the executing state, locking the data stream node to be migrated of the virtual storage area on the physical storage area.

Step S643: if the current scheduling running state of the data stream node to be migrated is the state to be executed, mapping of the data stream node to be migrated of the virtual storage area on the physical storage area is not needed.

The tasks are executed in no order from step S641 to step S643.

The space of the node corresponding to the executed data stream node function code can be released, and the mapped physical memory can be recovered. The unexecuted data flow node function space may be temporarily unassigned with physical memory without creating a map.

Fig. 10 is a schematic diagram illustrating migration of a data stream node program of an executable file according to an embodiment of the present invention. The physical memory is the virtual memory region, i.e., the programming space in fig. 10, and the physical memory is the physical memory in fig. 10. Because the node program topological graph-DAG graph in the embodiment is used, the mapping and the demapping of virtual and real pages are carried out in a basis instead of blindly using the LRU algorithm as a page changing algorithm in the embodiment of the invention, so that the page missing rate can be obviously reduced.

For example: in fig. 9, data stream node a has been running to end, its memory can be reclaimed, d is not running to build no map, and b, c is executing, has a physical page frame map and is not swapped out. Instead of performing mapping and demapping according to the scheduling process, the migration page replacement of the executable file is performed without using an LRU page replacement algorithm on the conventional operating system, and instead of using a common memory page prefetching algorithm, prefetching content is determined according to the DAG map, so that the page shortage rate can be significantly reduced.

As shown in fig. 11, the executable file format setting and migration scheme based on the data stream program are shown. In fig. 11, by the format setting method of the executable file based on the data flow program in the above embodiment, the DAG map is loaded into the memory, and the Node functions Df1 to Dfn of the data flow map are loaded into the memory (initial loading), and the physical mapping is on Node 0. If Df2 needs to be allocated to NUMA node 1 for scheduling reasons to run, then Df2 code can be remapped to create a new mapping of virtual page VM to real page PM. After Df2 migration, if Df1 needs to transfer data to Df2, then the base address in the data address "base+offset" used by the output code in Df1 needs to be redefined according to where Df2 is located—the data stream task stack location used by the new processor core.

In another preferred embodiment, the migration method of the executable file data stream program further includes: and if the storage space of the physical storage area is smaller than or equal to a preset threshold value, mapping the data stream nodes to be migrated of the virtual storage area on the physical storage area based on the sequence of executing the node program topological graph.

Because the priority execution sequence is recorded in the node topology information list in the node program topology, if the physical memory is insufficient, the address mapping is preferentially established for the node in front according to the depth ordering of the node program topology-DAG graph, namely the priority execution sequence. Therefore, the migration method of the executable file data stream program in the embodiment of the invention can flexibly establish the mapping relation between the virtual page and the real page by combining the actual condition of the physical storage area, thereby improving the migration efficiency of the data stream node program and the running speed of the system.

In a specific implementation manner, according to the migration method of the executable file data stream program in the embodiment of the invention, when the data stream node to be migrated is migrated, the input address of the data stream node to be migrated is relocated to the target stack area, the output address of the data stream node to be migrated is not relocated, and the output data of the data stream node to be migrated is written into the stack area where the subsequent node is located based on the node program topological graph.

Repositioning is carried out based on the input address of the data stream node to be migrated, and the executable file is allowed to take the node as a migration unit, so that the possibility and flexibility of the data stream node migration are improved, and the data stream operation speed is further improved.

In a specific implementation manner, according to the method for setting the executable file format based on the data flow program in the embodiment of the present invention, when migrating the data flow nodes to be migrated, the data flow nodes to be migrated sequentially executed on the same processor share the same stack based on the node program topology diagram.

In a specific implementation manner, according to the migration method of the executable file data stream program in the embodiment of the present invention, based on a node program topology diagram, when a data stream node to be migrated is migrated, a stack frame start address of the data stream node to be migrated executed on the same processor is the same, and when the data stream node to be migrated is executed, a page table address mechanism is used to limit an access range of a data memory address of the data stream node to be migrated in a memory space of the node.

Based on the data stream node to be migrated, the stack frame starting addresses of the target nodes executed on the same processor are the same, and the executable file is allowed to take the node as a migration unit, so that the possibility and flexibility of data stream node migration are improved, and the data stream running speed is further improved. Because the same stack space is repeatedly used by the data flow task functions on the same processor, the code data of the data flow task functions can be locked, the page fault rate is reduced, and the calculation performance is improved.

According to the migration method of the executable file data stream program, through executing the steps, the node program topological diagram in the setting method of the executable file format based on the data stream program is relocated based on the initial address of the data stream node to be migrated, virtual-real page mapping and demapping are carried out on a basis, instead of blindly using the LRU algorithm as a page changing algorithm, the page missing rate can be obviously reduced, and the running speed of a system is further improved. Or, the address mapping is established for the nodes which are positioned in front of the virtual page, and the mapping relation between the virtual page and the real page can be flexibly established by combining the actual situation of the physical storage area, so that the migration efficiency of the data flow node program is improved, and the running speed of the system is improved.

Based on the same conception, the embodiment of the invention also discloses a setting device of the executable file format based on the data stream program, as shown in fig. 12, comprising the following modules:

the data stream obtaining module 121 is configured to obtain each data stream node of the function code.

The list creation module 122 is configured to create a starting address list and a topology information list of each data stream node, where the topology information list includes: the total number of the data stream nodes and/or the topological relation among the data stream nodes and/or the sequence of execution of the data stream nodes and/or the current scheduling execution state of the data stream nodes during dynamic migration.

The topology map creation module 123 is configured to create a node program topology map formed based on the starting address list and the topology information list of each data stream node.

The program segment arrangement module 124 is configured to arrange the start address list, the topology information list to the first program segment, and each data stream node to the second program segment.

And the executable file forming module is used for forming a target executable file according to the contents of the first program segment and the second program segment, and adding the position identification information of the node program topological graph on the target executable file.

In a specific embodiment, when linking each data stream node, linking each data stream node into PIC position independent codes, and independently forming the PIC position independent codes into sections; when linking the data stream nodes, the start points of the data stream nodes are aligned at page boundaries based on the start addresses of the data stream nodes to constitute segments.

In a specific embodiment, when linking the data flow nodes, the address information of the input data of each data flow node is relocated to the target stack area, while the address information of the output data of each data flow node is not relocated.

Based on the same conception, the embodiment of the invention also discloses a migration device of the executable file data stream program, as shown in fig. 13, comprising the following modules:

a data flow determining module 131, configured to determine a data flow node to be migrated in the target executable file;

the data stream updating module 132 is configured to reposition a start address of a data stream node to be migrated based on a node program topology map in a setting method of an executable file format of a data stream program, and update a current initial address of the data stream node to be migrated to a target stack area in the node program topology map;

The data stream state obtaining module 133 is configured to obtain a current scheduling execution state of a data stream node to be migrated according to the node program topology map;

the mapping relation executing module 134 is configured to determine whether to map the data stream node to be migrated of the virtual storage area on the physical storage area according to the current scheduling execution status of the data stream node to be migrated.

In a specific embodiment, the mapping relation execution module includes:

the mapping relation canceling submodule is used for canceling the mapping relation of the data flow node to be migrated in the physical storage area if the current scheduling execution state of the data flow node to be migrated is the executed state;

the mapping relation locking sub-module is used for locking the data stream node to be migrated in the virtual storage area on the physical storage area if the current scheduling execution state of the data stream node to be migrated is the positive execution state;

and the mapping relation to-be-executed sub-module is used for mapping the to-be-migrated data stream node of the virtual storage area on the physical storage area if the current scheduling execution state of the to-be-migrated data stream node is the to-be-executed state.

In a specific implementation manner, the migration device of the executable file data stream program in the embodiment of the present invention further includes: a mapping relation determining module for

And if the storage space of the physical storage area is smaller than or equal to the preset threshold value, mapping the data stream nodes to be migrated of the virtual storage area on the physical storage area according to the sequence of executing the node program topological graph.

In a specific embodiment, based on the node program topological graph, when the data stream node to be migrated is migrated, the input address of the data stream node to be migrated is relocated to the target stack area, the output address of the data stream node to be migrated is not required to be relocated, and the output data of the data stream node to be migrated is written into the stack area where the subsequent node is located.

In a specific embodiment, based on a node program topological graph, when the data stream node to be migrated is migrated, the stack frame starting addresses of the data stream node to be migrated executed on the same processor are the same, and when the data stream node to be migrated is executed, a page table address mechanism is used for limiting the access range of the data memory address of the data stream node to be migrated in the memory space of the node.

The embodiment of the present invention further provides a computer device, as shown in fig. 14, where the computer device may include a processor 141 and a memory 142, where the processor 141 and the memory 142 may be connected by a bus or other manners, and in fig. 14, the connection is exemplified by a bus.

Processor 141 may be a central processing unit (Central Processing Unit, CPU). Processor 141 may also be a chip such as other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination thereof.

Memory 142, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 141 executes various functional applications of the processor and data processing by executing non-transitory software programs, instructions, and modules stored in the memory 142, that is, implements the migration method of the executable file data stream program in the above-described embodiment, or the migration method of the executable file data stream program.

Memory 142 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created by the processor 141, etc. In addition, memory 142 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 142 optionally includes memory remotely located relative to processor 141, which may be connected to processor 141 via a network. Examples of such networks include, but are not limited to, the power grid, the internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The one or more modules are stored in memory 142 that, when executed by processor 141, perform the method of setting an executable file format based on a data stream program or the method of migrating an executable file data stream program in the embodiment shown in the figures.

The details of the computer device may be understood with reference to the corresponding related descriptions and effects of the embodiments shown in the drawings, which are not repeated herein.

It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.

It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.

Claims

1. A method for setting an executable file format based on a data stream program, comprising the steps of:

acquiring each data stream node of the function code;

2. The method for setting an executable file format based on a data stream program according to claim 1, wherein when linking the data stream nodes, linking the data stream nodes into PIC position independent codes, and independently segmenting the PIC position independent codes; when linking the data stream nodes, aligning the start points of the data stream nodes at page boundaries based on the start addresses of the data stream nodes to form segments.

3. The method according to claim 1, wherein when linking the data stream nodes, the address information of the input data of each data stream node is relocated to the target stack area, and the address information of the output data of each data stream node is not relocated.

4. A method for migrating an executable file data stream program, comprising the steps of:

determining a data stream node to be migrated in a target executable file;

repositioning a start address of the data flow node to be migrated into the target stack area based on a node program topology in the data flow program-based executable file format setting method as claimed in any one of claims 1 to 3, and updating a current start address of the target stack area in the node program topology;

5. The migration method of executable file data stream program according to claim 4, wherein determining whether to map the data stream node to be migrated of a virtual storage area on a physical storage area according to a current scheduled running state of the data stream node to be migrated comprises:

6. The method of migrating an executable file data stream program according to claim 4, further comprising: and if the storage space of the physical storage area is smaller than or equal to a preset threshold value, mapping the data stream nodes to be migrated of the virtual storage area on the physical storage area according to the sequence of executing the node program topological graph.

7. The migration method of executable file data stream program according to claims 4 to 6, wherein, based on the node program topology, when migrating the data stream node to be migrated, the input address of the data stream node to be migrated is relocated to the target stack area, and the output address of the data stream node to be migrated is not required to be relocated, and the output data of the data stream node to be migrated is written into the stack area where the subsequent node is located.

8. The migration method of executable file data stream programs according to claims 4 to 6, wherein, based on the node program topology, when migrating the data stream nodes to be migrated, stack frame start addresses of the data stream nodes to be migrated executed on the same processor are the same, and when executing the data stream nodes to be migrated, a page table address mechanism is used to limit a data memory address access range of the data stream nodes to be migrated in a memory space of the node.

9. A computer-readable storage medium storing computer instructions for causing the computer to execute the method of setting an executable file format based on a data stream program according to any one of claims 1 to 3; or, a migration method of an executable file data stream program according to any one of claims 4 to 8.

10. A computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the method of setting a dataflow program based executable file format according to any one of claims 1 to 3 by executing the computer instructions; or, a migration method of an executable file data stream program according to any one of claims 4 to 8.