CN111694789A - Embedded reconfigurable heterogeneous determination method, system, storage medium and processor - Google Patents
Embedded reconfigurable heterogeneous determination method, system, storage medium and processor Download PDFInfo
- Publication number
- CN111694789A CN111694789A CN202010323642.0A CN202010323642A CN111694789A CN 111694789 A CN111694789 A CN 111694789A CN 202010323642 A CN202010323642 A CN 202010323642A CN 111694789 A CN111694789 A CN 111694789A
- Authority
- CN
- China
- Prior art keywords
- task
- computing
- node
- heterogeneous
- resources
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/76—Architectures of general purpose stored program computers
- G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
- G06F15/7807—System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2273—Test methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2289—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing by configuration test
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2294—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing by remote test
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/76—Architectures of general purpose stored program computers
- G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
- G06F15/7867—Architectures of general purpose stored program computers comprising a single central processing unit with reconfigurable architecture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention belongs to the technical field of reconfigurable computing, and discloses an embedded reconfigurable heterogeneous measurement method, a system, a storage medium and a processor, wherein the method is used for centralizing the computing power of a plurality of distributed embedded computing boards bearing various heterogeneous computing resources and constructing a task-driven reconfigurable heterogeneous computing platform; the tasks and heterogeneous computing resources of the cluster are uniformly managed by using a dynamic cluster construction mode, and a reconfigurable virtual computing environment is constructed by using a virtualization technology. The invention constructs a heterogeneous computing platform with self-organizing cooperation and unified management capability of resources and reconfigurable computing environment. A Web visualization module in a user interface layer provides an interactive interface for a user, wherein a safety mechanism performs multiple division on user levels to provide guarantee for access and authorization of various users; and the management of tasks, resources, users and the like is finished by using a graphical interface, so that the use complexity of the users is reduced, and the task load balancing capability is provided.
Description
Technical Field
The invention belongs to the technical field of reconfigurable computing, and particularly relates to an embedded reconfigurable heterogeneous measurement method, a system, a storage medium and a processor.
Background
At present, in order to meet the demand of computational diversification, a traditional general computing system based on a CPU cannot meet the demand of artificial intelligence and the like on high computing capacity, more and more scenes begin to introduce hardware such as a GPU, an FPGA and the like for acceleration, and heterogeneous computing is carried out at the same time. Heterogeneous Computing (Heterogeneous Computing) refers to Computing systems composed of Computing units using different types of instruction sets and architectures. Heterogeneous computing is a technology with balanced performance, cost and power consumption, and on the basis of computing task parallelism, tasks are allocated to computing resources which are most suitable for executing the tasks to be executed, so that the total execution time of the computing tasks is minimized, and the performance and the cost are optimized. Due to low integration level, long design period, weak computing power and poor flexibility of the special processor chip, the development requirement of new technology cannot be met. The reconfigurable computing can upgrade or modify the computer hardware structure, thereby better meeting the flexible and changeable task requirements. The reconfigurable computing related technology develops rapidly, the development trends and the technical requirements of framework co-fusion, resource sharing and cooperative processing are presented, but the effective load of a single computing node is limited, the computing capacity is limited, the task processing is easy to reach the load performance limit, and the task processing efficiency is influenced, so that independent loads are necessarily interconnected in an effective mode to form a computing platform with larger scale and higher computing capacity, and the problem of load computing bottleneck is solved well.
As the reconfigurable computing concept is provided for the hardware level, the defects of long development period and poor flexibility of the application specific integrated circuit are overcome, more researches are still focused on improving the performance of the reconfigurable logic device and the system hardware, and the improvement of the system performance by the platform level architecture, the system management software and the scheduling strategy is less considered, so that the improvement of the system performance by the management scheduling at the system level and the reconfigurable computing at the software level is ignored. The reconfigurable computing needs to exert the efficiency advantages, the reconfigurable hardware is the foundation, the platform architecture is the key, the self-organization coordination and dynamic self-evolution theory and technology of resources, communication and architecture are the core, the rapid information storage and retrieval are effective ways for improving the platform efficiency, and the related contents need to be deeply researched. In addition, for the task-driven model, in the process from task access to task execution completion, problems such as automatic detection of faults, automatic switching of computing nodes, automatic recovery of tasks, and the like are involved. To ensure that tasks are performed continuously and efficiently, high availability of computing platforms is essential. Therefore, a highly available reconfigurable heterogeneous computing platform with task-driven, self-organizing, cooperative and unified management of heterogeneous resources is urgently needed.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) at present, the effective load of a single computing node is limited, the computing capacity is limited, and the task processing is easy to reach the load performance limit, so that the task processing efficiency is influenced.
(2) At present, much research on reconfigurable computing still focuses on the performance improvement of system hardware, and the improvement of system performance by platform-level architecture, system management software and scheduling policy is less considered.
The difficulty in solving the above problems and defects is:
an embedded computing platform constructed by embedded computing nodes has the characteristics of distribution, heterogeneity and the like physically, and the realization of resource self-organization cooperation and unified management by utilizing the communication technology among the nodes is the basis of cluster construction; the computing platform does not depend on the main control node strongly any more, and how to ensure the high availability of the platform when the main control node fails is the key for constructing the platform; task execution on a single computing node has a risk of single point failure, and how to ensure high availability of task execution at a platform level is a key point of research.
The significance of solving the problems and the defects is as follows:
the problems and the defects are solved, a plurality of embedded computing nodes can be cascaded together by means of a communication technology, and self-organization cooperation and unified management of heterogeneous resources are achieved; the cluster does not strongly depend on the main control node any more, meanwhile, automatic migration of tasks on fault nodes is guaranteed, a centerless, high-performance and high-availability embedded computing platform is constructed, and a task-driven and reconfigurable computing environment is provided.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides an embedded reconfigurable heterogeneous determination method, a system, a storage medium and a processor.
The invention is realized in such a way, and provides an embedded reconfigurable heterogeneous measurement method, which integrates the computing power of a plurality of distributed embedded computing boards bearing various heterogeneous computing resources and constructs a task-driven reconfigurable heterogeneous computing platform; the tasks and heterogeneous computing resources of the cluster are uniformly managed by using a dynamic cluster construction mode, and a reconfigurable virtual computing environment is constructed by using a virtualization technology.
Further, the embedded reconfigurable heterogeneous measurement method receives task input of a user, provides a web visual interaction page, divides nodes into a plurality of independent clusters, builds an agent on the upper layer of the clusters and provides load balance;
in the proxy node, a plurality of nodes share one virtual IP based on a VRRP protocol, when a Master of a node bound by the virtual IP fails, the virtual IP drifts to a Slave node, and the Slave node rises to be a new Master, so that the functions of task access and load balancing are continuously provided.
Further, the embedded reconfigurable heterogeneous measurement method constructs a cluster through a dynamic cluster construction module, constructs a task pool and a resource pool of the cluster through a task pool construction module and a resource pool construction module, and matches tasks and resources through a virtual computing environment construction module to generate a virtual computing environment for task execution.
Further, the dynamic cluster construction of the embedded reconfigurable heterogeneous measurement method comprises heartbeat detection, database consistency and a dynamic center election strategy;
the heartbeat detection is used for carrying out survival detection on each node in a master-slave service mode with equal node positions, adding a new computing node into a cluster and deleting a fault node; when the computing node is deleted, the heartbeat detection of the system can automatically detect the fault information of the deleted development board; after the deletion is finished, synchronizing the content of the system resources in the database through a database consistency strategy;
the database consistency strategy synchronizes data in the database, so that each computing node knows tasks and resource configuration information of a cluster, when a main node fails, a dynamic center election strategy is utilized to elect a new main node, and the data in the database is synchronized;
the dynamic central election realizes the dynamic selection of the main control node in the cluster, and the election strategy is used for dynamically selecting the main node on the basis of the database consistency module to realize non-centralization; the selected main node is responsible for issuing tasks, and distributing specific tasks and configuration information related to the tasks to a certain computing node according to a certain distribution strategy; when a certain computing node fails, the main node is responsible for fault migration and re-issues tasks on the failed node.
Further, when a task X is coming, the main node distributes tasks according to the resource state of the computing nodes and sends the tasks and configuration information thereof to the computing nodes i meeting the requirements; if the resources are insufficient, the task is queued for waiting; after the resources are successfully distributed, changing the database of the main node, and synchronizing the operation to other databases; when a computing node i fails, heartbeat detection finds the failure and clears resource information of the node i in a database, after the database is synchronized through a database consistency strategy, a main node obtains the failure information of the node i, then the task X which is being executed on the node i is judged to have failed, the main node issues the task X again according to information such as resource states of nodes which survive in a cluster, and the task X is issued to a node j for re-execution.
Further, when the embedded computing board card is started, a resource pool construction module of the embedded reconfigurable heterogeneous measurement method finishes scanning of resource configuration, acquires registered equipment information, performs health detection on the equipment, stores available resource information into a resource list of a database, realizes resource discovery and availability detection, and constructs a cluster resource pool by means of a database consistency strategy;
the virtual computing environment construction matches the tasks with resources required by the tasks, the main node constructs related configuration information of the task execution environment according to the requirements of the tasks and issues the related configuration information to the computing nodes along with the tasks, and the computing nodes integrate the tasks and the configuration environment to construct the virtual computing environment. Packaging the application and the operating environment into a docker mirror image, uploading the docker mirror image to a docker warehouse, and constructing a virtual computing environment in a docker mode, wherein the virtual computing environment is quick to start and belongs to the second level; and the computing node pulls the mirror image from the docker warehouse according to the configuration information issued by the main node, and constructs a virtual computing environment for executing the task, so that the installation of the application and the configuration of the environment can be automatically completed.
Furthermore, the hardware resources of the embedded reconfigurable heterogeneous measurement method adopt a bus-component architecture, and controller boards with heterogeneous computing resources are subjected to networking interconnection communication through standard interfaces defined by a message bus; heterogeneous resources on the embedded computing board card are uniformly accessed, and the boards are networked and interconnected, so that the standard extensible high-speed system bus and heterogeneous resource uniform component packaging and accessing are realized.
It is another object of the present invention to provide a program storage medium for receiving user input, the stored computer program causing an electronic device to perform the steps comprising: centralizing the computing power of a plurality of distributed embedded computing board cards bearing various heterogeneous computing resources, and constructing a task-driven reconfigurable heterogeneous computing platform; the tasks and heterogeneous computing resources of the cluster are uniformly managed by using a dynamic cluster construction mode, and a reconfigurable virtual computing environment is constructed by using a virtualization technology.
Another object of the present invention is to provide an embedded reconfigurable heterogeneous assay system for implementing the embedded reconfigurable heterogeneous assay method, the embedded reconfigurable heterogeneous assay system including:
the user interface layer is used for providing a Web visual task access mode and providing a load balancing function;
the system middleware layer is used for constructing a cluster and managing heterogeneous computing resources in a unified manner;
the hardware layer is used for uniform access of heterogeneous resources on the embedded computing board card and networking interconnection among boards;
the user interface layer includes:
the Web visualization module is used for providing a Web visualization interface to manage tasks, resources and users and providing guarantee for access and authorization;
the task access module is used for providing a uniform access address, ensuring the high availability of uniform access by using an agent and distributing tasks among clusters according to a load balancing strategy;
the system middleware layer comprises a dynamic cluster building module, a task pool building module, a resource pool building module and a virtual computing environment building module;
the dynamic cluster building module comprises heartbeat detection, database consistency and a dynamic center election strategy and is used for building and managing a cluster and realizing unified scheduling of tasks and resources;
the task pool construction module is used for realizing the construction of a cluster task pool;
the resource pool building module is used for realizing discovery and health check of heterogeneous resources;
another object of the present invention is to provide a processor having the embedded reconfigurable heterogeneous measurement system mounted thereon.
By combining all the technical schemes, the invention has the advantages and positive effects that: the invention provides a task-driven reconfigurable heterogeneous computing platform on an embedded platform, which realizes self-organization cooperation and unified management of heterogeneous resources and constructs a reconfigurable task virtual computing environment, and ensures high availability of task access, task issuing and task execution. The invention integrates the computing power of distributed embedded nodes, and constructs a heterogeneous computing platform with the resource self-organization cooperation and unified management capability and a reconfigurable computing environment. A Web visualization module in a user interface layer provides an interactive interface for a user, wherein a safety mechanism performs multiple division on user levels to provide guarantee for access and authorization of various users; and the management of tasks, resources, users and the like is finished by using a graphical interface, so that the use complexity of the users is reduced, and the task load balancing capability is provided.
The invention realizes the continuous high-availability task access, establishes the agent cluster on the upper layer of the cluster on the user interface layer, realizes that the nodes in the agent cluster share one Virtual IP (VIP) based on the VRRP protocol, and when the node bound by the VIP fails, the VIP automatically drifts to a new agent node, thereby providing a uniform and high-availability access interface for the user and ensuring the high availability of the task access.
The invention realizes the continuous high-availability task issuing, and adopts the centerless main control node dynamic election technology to ensure the high availability of the platform on the task issuing. When the current main node in the cluster fails, the main node is removed from the cluster through heartbeat detection and a database consistency strategy, a new main node is elected through a dynamic central election strategy after a database is synchronized, and resource allocation and task issuing are continuously carried out, so that decentralization of the cluster is realized, namely, a static main control node is not needed, the breakdown of the whole cluster caused by the breakdown of the main node is avoided, the high availability of the main node is ensured, and the high availability of task issuing is ensured.
The invention realizes the execution of the continuous high-availability tasks, when a certain computing node fails, the computing node is removed from the cluster through heartbeat detection and a database consistency strategy, all the tasks which are being executed on the node fail and are distributed and issued again by the main node, thereby realizing the task migration and ensuring the high availability of the task execution.
The invention realizes a dynamic reconfigurable task execution environment, provides an independent virtual computing environment for each task on the basis of a virtualization technology, and realizes a task customized task execution environment, namely the dynamic reconfigurable task execution environment, wherein computing resources in the virtual computing environment dynamically change according to task requirements.
The invention realizes the self-organization coordination and the unified management of heterogeneous resources, detects the survival state of nodes through heartbeat detection when a new computing node is added into a cluster or one node is in fault (is separated from the cluster), dynamically adds and deletes the heterogeneous resources through the resource discovery and the availability detection of a resource pool building module, synchronizes the cluster resource state through the database consistency strategy, realizes the hot plug of the computing node and the self-adaption access and deletion of the heterogeneous resources, and then is uniformly managed by the cluster, namely the self-organization coordination and the unified management of the heterogeneous resources are realized.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained from the drawings without creative efforts.
Fig. 1 is a flowchart of an embedded reconfigurable heterogeneous determination method according to an embodiment of the present invention.
FIG. 2 is a schematic structural diagram of an embedded reconfigurable heterogeneous assay system provided by an embodiment of the present invention;
in the figure: 1. a user interface layer; 2. a system middleware layer; 3. and a hardware layer.
Fig. 3 is a schematic architecture diagram of an embedded reconfigurable heterogeneous measurement system according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a user interface layer proxy structure provided in the embodiment of the present invention.
Fig. 5 is a schematic diagram of implementing a dynamic center according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of task issuing and failover provided in the embodiment of the present invention.
FIG. 7 is a schematic diagram of a virtual computing environment according to an embodiment of the present invention.
Fig. 8 is a schematic diagram of interconnection of boards of heterogeneous computing resources according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides an embedded reconfigurable heterogeneous assay method, system, storage medium, and processor, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the embedded reconfigurable heterogeneous assay method provided by the present invention includes the following steps:
s101: centralizing the computing power of a plurality of distributed embedded computing board cards bearing various heterogeneous computing resources (such as CPU, GPU, FPGA, DSP and the like) and constructing a task-driven reconfigurable heterogeneous computing platform;
s102: the tasks and heterogeneous computing resources of the cluster are uniformly managed by using a dynamic cluster construction mode, and a reconfigurable virtual computing environment is constructed by using a virtualization technology.
As shown in fig. 2, the embedded reconfigurable heterogeneous measurement system provided by the present invention includes:
and the user interface layer 1 is used for providing a task access mode for Web visualization and providing a load balancing function.
And the system middleware layer 2 is used for constructing a cluster and uniformly managing heterogeneous computing resources.
And the hardware layer 3 is used for uniform access of heterogeneous resources on the embedded computing board card and networking interconnection among boards.
In the present invention, the user interface layer 1 includes:
and the Web visualization module is used for providing a Web visualization interface for the user to manage tasks, resources, users and the like and providing guarantee for the access and authorization of various users.
And the task access module is used for providing a uniform access address for the user, ensuring the high availability of uniform access of the user by using the agent and distributing the task among the clusters according to a certain load balancing strategy.
And the Web visualization module is used for providing a visual interactive interface for the user to manage, realizing multiple division of the user identity based on a security authentication and authorization mechanism and providing corresponding management service for different users.
The task access module is used for receiving a task request of a user, the task access module is an agent cluster constructed by a plurality of computing nodes, a Virtual IP (VIP) is shared by the nodes in the agent cluster based on a Virtual Router Redundancy Protocol (VRRP), and when a node bound by the VIP fails, the VIP is automatically migrated to a new agent node, so that a uniform and high-availability access interface is provided for the user, and the high availability of task access is ensured. Meanwhile, the agent cluster distributes the tasks to the nodes of the plurality of clusters by using a load balancing strategy, so that the concurrency and the throughput of the tasks are improved, and the utilization rate of resources is improved.
In the invention, the system middleware layer 2 comprises a dynamic cluster building module, a task pool building module, a resource pool building module and a virtual computing environment building module.
And the dynamic cluster building module comprises heartbeat detection, database consistency and a dynamic center election strategy and is used for building and managing a cluster and realizing unified scheduling of tasks and resources.
The heartbeat detection realizes the detection of the health state of the nodes in the cluster, the survival detection is carried out on each node in a master-slave service mode of equal node status, a new computing node is added into the cluster, and a fault node is deleted. When the computing node is deleted, synchronizing the content of the system resource in the database through a database consistency strategy, thereby ensuring that the task is not issued to the computing node which has failed;
the database consistency realizes the data in the synchronous database, so that each computing node knows the task and resource configuration information of the cluster, when the main node fails, a dynamic center election strategy is utilized to elect a new main node, and the data in the database is synchronized, thereby solving the problem of failure transfer. The consistency of the database ensures that each node in the cluster can become a new main node when the main control node fails, thereby ensuring the high availability of the system;
the dynamic center election realizes the dynamic selection of the main control node in the cluster, and the election strategy is used for dynamically selecting the main node on the basis of the consistency of the database, thereby realizing the dynamic construction of the cluster and ensuring the high availability of the cluster. The elected main node is responsible for issuing tasks, namely distributing specific tasks and configuration information related to the tasks to a certain computing node according to a certain distribution strategy. When a certain computing node fails, the main node is responsible for fault migration and re-issues tasks on the failed node.
And the task pool building module is used for building the cluster task pool.
And the resource pool building module comprises resource discovery and availability detection and realizes discovery and health check of heterogeneous resources. And (3) resource discovery, namely completing the scanning of resource configuration when the embedded computing board card is started, acquiring the registered equipment information and acquiring heterogeneous computing resource information. And (3) availability detection, namely performing health detection on the found heterogeneous resources and storing the available resource information into a resource list of the database.
And the hardware layer 3 is used for uniform access of heterogeneous resources on the embedded computing board card and networked interconnection among boards, and realizes the encapsulation and access of a standard extensible high-speed system bus and a uniform heterogeneous resource component, so that the heterogeneous resources can be managed and communicated according to a uniform calling interface and a uniform protocol, and hardware support is provided for the high-expansibility of the system and the componentization service of the heterogeneous resources.
The technical solution of the present invention is further described below with reference to the accompanying drawings.
The embedded reconfigurable heterogeneous measuring system provided by the invention comprises:
the user interface layer 1, as shown in fig. 4, is responsible for accepting task input from a user and providing a web visualization interactive page. Meanwhile, the nodes are divided into a plurality of independent clusters, and an agent is built on the upper layer of the clusters to provide a load balancing function. This ensures relative independence of the database between clusters, similar to the role of cluster sharding. For a single cluster, the task concurrency is premised on the resource allocation of the master nodes, and for a plurality of clusters, the task concurrency is allocated to a plurality of master nodes, so that the task concurrency and the task throughput are improved. In addition, the utilization rate of resources is improved.
In the agent node, a plurality of nodes share a Virtual IP (VIP) based on a VRRP protocol, when a Master of a VIP bound node fails, the VIP can drift to a Slave node, and the Slave node rises to be a new Master, so that the functions of task access and load balancing are continuously provided. Therefore, for the user, only one uniform access interface is provided, so that the high availability of the agent node is ensured, and the high availability of task access is further ensured.
And the system middleware layer 2 constructs a cluster through the dynamic cluster tool module, constructs a task pool and a resource pool of the cluster through the task pool constructing module and the resource pool constructing module, and generates a virtual computing environment for executing the task by matching the task and the resource through the virtual computing environment constructing module.
The dynamic cluster building module consists of heartbeat detection, database consistency and a dynamic center election strategy.
The heartbeat detection is responsible for dynamic adding and deleting of nodes in the cluster, and is used for carrying out survival detection on each node in a master-slave service mode with equal node positions, adding a new computing node into the cluster and deleting a fault node, so that the method is the basis for realizing resource self-organization cooperation and unified management. When the computing node is deleted, the heartbeat detection of the system can automatically detect the fault information of the deleted development board. After the deletion is completed, the content of the system resources in the database is synchronized through the database consistency strategy, so that the task is not issued to the failed computing node.
The database consistency realizes the data in the synchronous database, so that each computing node knows the task and resource configuration information of the cluster, when the main node fails, a new main node is elected by utilizing the following dynamic center election strategy, and the data in the database is synchronized, thereby solving the problem of fault transfer. The consistency of the database ensures that each node in the cluster can become a new main node when the main control node fails, thereby ensuring the high availability of the system.
The dynamic central election module dynamically selects the main control node in the cluster, as shown in fig. 5, on the basis of the consistency of the database, the main node is dynamically selected by using an election strategy, so that decentralization is realized, and at the moment, if the main node fails, a new main node is elected to continue to maintain the service of the cluster, so that the dependence of the cluster on the static main nodes is solved, the high availability of the cluster is ensured, and the high availability of task issuing is further ensured. The elected main node is responsible for issuing tasks, namely distributing specific tasks and configuration information related to the tasks to a certain computing node according to a certain distribution strategy. When a certain computing node fails, the main node is responsible for fault migration and re-issues tasks on the failed node.
As shown in fig. 6, when a task X comes, the master node performs task allocation according to the resource state of the computing node, and sends the task and the configuration information thereof to the computing node i that meets the requirement. If the resources are not sufficient, the task is queued up. After the resources are successfully allocated, the database of the main node is changed, and the operation is synchronized to other databases. As shown in fig. 7, when a failure occurs in a computing node i, heartbeat detection will find the failure and clear resource information and the like of the node i in a database, and after synchronizing the database through a database consistency policy, a master node obtains the failure information of the node i, and then determines that a task X being executed on the node i has failed, the master node issues the task X again according to information such as resource states of surviving nodes in a cluster, and the task X is issued on a node j for re-execution.
The task pool construction module constructs a task pool of the cluster.
When the embedded computing board card is started, the resource pool building module completes scanning of resource configuration, acquires registered equipment information, performs health detection on the equipment, stores available resource information into a resource list of a database, achieves resource discovery and availability detection, and builds a cluster resource pool by means of a database consistency strategy.
The virtual computing environment construction module matches the task with the resources required by the task, the main node constructs a virtual computing environment executed by the task according to the requirement of the task, namely the relevant configuration information of the task execution environment is constructed and is issued to the computing node along with the task, and the computing node integrates the task and the configuration environment to construct the virtual computing environment. On the task-driven computing platform, the application and the operating environment are packaged into a docker mirror image and uploaded to a docker warehouse, and a virtual computing environment is established in a docker mode, is started quickly and belongs to the second level. As shown in fig. 7, the computing node pulls the mirror image from the docker warehouse according to the configuration information issued by the master node, and constructs a virtual computing environment for executing the task, so that the installation of the application and the configuration of the environment can be automatically completed.
In the virtual computing environment construction, the task execution environment is dynamically generated by the task and the task configuration information, namely the task execution environment is driven by the task and is dynamically reconfigurable.
The hardware layer 3 is configured as shown in fig. 8, and the hardware resource module adopts a bus-component architecture design to perform networking interconnection communication on the controller boards loaded with heterogeneous computing resources through a standard interface defined by a message bus. When a new computing board is clamped, discovery and positioning of new resources can be realized only by carrying out corresponding configuration on a corresponding board IP in the system, and the same method is adopted when the board is pulled out. The hardware layer is responsible for uniform access of heterogeneous resources on the embedded computing board card and networked interconnection among boards, and realizes encapsulation and access of standard extensible high-speed system buses and heterogeneous resource uniform components, so that heterogeneous resources can be managed and communicated according to uniform calling interfaces and protocols, and hardware support is provided for system high expansibility and heterogeneous resource componentization services.
The granularity of resources occupied by the tasks is refined based on the containerized task allocation and deployment mode, the granularity is not a slice level any more, but the granularity is the size of the resources required by the tasks, namely, the tasks respectively occupy a part of the resources, so that the utilization rate of the resources is greatly improved.
In order to test the difference between the computing performance and the resource occupation situation of a CPU when a task is executed in a Docker and a task is executed in a Host (Host), the computing performance of the CPU is tested by using a Linpack testing tool on a Xilinx UltraScale + MPSoC ZCU102 platform and taking a floating point computing peak value as an evaluation index, and the Nginx service is subjected to pressure test by using Apache benchmark and the resource occupation situation is monitored by using Nmon.
(1) Linpack test
Assuming that the number of floating-point operations of the CPU per clock cycle is 1, the theoretical CPU floating-point operation peak value is 1.2GHz × 1 × 4 cores — 4.8 gfops.
The Linkpack test result is shown in table 1, where Max represents the actual CPU floating point calculation peak obtained by the test, and the maximum efficiency is defined as the ratio of Max to the theoretical CPU floating point calculation peak.
TABLE 1 comparison of maximum efficiency of CPU floating point calculation peaks in Host and Docker
Max(Gflops) | Maximum efficiency (Max/theoretical peak) | |
Host | 4.065e-01 | 8.47% |
Docker(Alpine) | 5.6567e-01 | 11.78% |
As can be seen in table 1, there is no significant loss of performance in docker compared to Host.
(2) Apache benchmark test
Remote access pressure tests are respectively carried out on the Nginx services in Host and Docker, and the monitoring results are shown in Table 2. Wherein, CPU (usr% + sys%) represents the occupation ratio of the user space to the kernel space CPU, and CPU-WAvg represents the weighted average occupation ratio of the CPU.
Table 2 comparison of CPU resource occupation in Host and Docker
CPU(usr%+sys%) | CPU-WAvg | |
Host | 25%+70% | 17.6% |
Docker | 20%+50% | 11.0% |
As can be seen in table 2, compared to Host, the weighted average occupancy of the CPU in Docker is reduced by 6.6%, i.e., the resource occupancy in Docker is less.
By combining the two tests, compared with the Host, the method has the advantages that the calculation performance is not obviously reduced in the Docker environment, and the resource occupation is effectively reduced. In addition, the containerization technology is utilized to improve the deployment efficiency of tasks and reduce the complexity of operation and maintenance.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided on a carrier medium such as a disk, CD-or DVD-ROM, programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier, for example. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. An embedded reconfigurable heterogeneous measurement method is characterized in that the embedded reconfigurable heterogeneous measurement method integrates the computing power of a plurality of distributed embedded computing boards bearing various heterogeneous computing resources and constructs a task-driven reconfigurable heterogeneous computing platform; the tasks and heterogeneous computing resources of the cluster are uniformly managed by using a dynamic cluster construction mode, and a reconfigurable virtual computing environment is constructed by using a virtualization technology.
2. The embedded reconfigurable heterogeneous measurement method according to claim 1, wherein the embedded reconfigurable heterogeneous measurement method accepts task input of a user, provides a web visual interactive page, divides nodes into a plurality of independent clusters, builds an agent on an upper layer of the clusters, and provides load balancing;
in the proxy node, a plurality of nodes share one virtual IP based on a VRRP protocol, when a Master of a node bound by the virtual IP fails, the virtual IP drifts to a Slave node, and the Slave node rises to be a new Master, so that the functions of task access and load balancing are continuously provided.
3. The embedded reconfigurable heterogeneous measurement method according to claim 1, wherein the embedded reconfigurable heterogeneous measurement method creates a cluster through a dynamic cluster creation module, creates a task pool and a resource pool of the cluster through a task pool creation module and a resource pool creation module, and generates a virtual computing environment for task execution by matching tasks and resources through a virtual computing environment creation module.
4. The embedded reconfigurable heterogeneous assay of claim 3, wherein the dynamic cluster construction of the embedded reconfigurable heterogeneous assay comprises heartbeat detection, database consistency, and dynamic centric election strategies;
the heartbeat detection is carried out on each node in a master-slave service mode of equal node status, new computing nodes are added into the cluster, and fault nodes are deleted; when the computing node is deleted, the heartbeat detection of the system can automatically detect the fault information of the deleted development board; after the deletion is finished, synchronizing the content of the system resources in the database through a database consistency strategy;
the database consistency realizes the synchronization of the data in the database, so that each computing node knows the task and resource configuration information of the cluster, when the main node fails, a new main node is elected by using a dynamic central election strategy, and the data in the database is synchronized;
the dynamic central election realizes the dynamic selection of the main control node in the cluster, and the election strategy is used for dynamically selecting the main node on the basis of the database consistency module to realize non-centralization; the selected main node is responsible for issuing tasks, and distributing specific tasks and configuration information related to the tasks to a certain computing node according to a certain distribution strategy; when a certain computing node fails, the main node is responsible for fault migration and re-issues tasks on the failed node.
5. The embedded reconfigurable heterogeneous measurement method according to claim 3, wherein when a task X comes, the master node performs task allocation according to the resource state of the computing node, and sends the task and configuration information thereof to a computing node i meeting the requirements; if the resources are insufficient, the task is queued for waiting; after the resources are successfully distributed, changing the database of the main node, and synchronizing the operation to other databases; when a computing node i fails, heartbeat detection finds the failure and clears resource information of the node i in a database, after the database is synchronized through a database consistency strategy, a main node obtains the failure information of the node i, then the task X which is being executed on the node i is judged to have failed, the main node issues the task X again according to information such as resource states of nodes which survive in a cluster, and the task X is issued to a node j for re-execution.
6. The embedded reconfigurable heterogeneous measurement method according to claim 3, wherein the resource pool of the embedded reconfigurable heterogeneous measurement method is constructed by completing scanning of resource configuration when an embedded computing board is started, acquiring registered device information and performing health detection on devices, storing available resource information into a resource list of a database to realize resource discovery and availability detection, and constructing a clustered resource pool by means of a database consistency strategy;
the virtual computing environment construction matches the task with the resources required by the task, the main node constructs a virtual computing environment for executing the task according to the requirement of the task, constructs related configuration information of the task execution environment, and sends the configuration information to the computing nodes along with the task, and the computing nodes integrate the task and the configuration environment to construct the virtual computing environment. Packaging the application and the operating environment into a docker mirror image, uploading the docker mirror image to a docker warehouse, and constructing a virtual computing environment in a docker mode, wherein the virtual computing environment is quick to start and belongs to the second level; and the computing node pulls the mirror image from the docker warehouse according to the configuration information issued by the main node, and constructs a virtual computing environment for executing the task, so that the installation of the application and the configuration of the environment are automatically completed.
7. The embedded reconfigurable heterogeneous measurement method according to claim 1, wherein hardware resources of the embedded reconfigurable heterogeneous measurement method adopt a bus-component architecture, and controller boards loaded with heterogeneous computing resources are networked and communicated with each other through a standard interface defined by a message bus; heterogeneous resources on the embedded computing board card are uniformly accessed, and the boards are networked and interconnected, so that the standard extensible high-speed system bus and heterogeneous resource uniform component packaging and accessing are realized.
8. A program storage medium for receiving user input, the stored computer program causing an electronic device to perform the steps comprising: centralizing the computing power of a plurality of distributed embedded computing board cards bearing various heterogeneous computing resources, and constructing a task-driven reconfigurable heterogeneous computing platform; the tasks and heterogeneous computing resources of the cluster are uniformly managed by using a dynamic cluster construction mode, and a reconfigurable virtual computing environment is constructed by using a virtualization technology.
9. An embedded reconfigurable heterogeneous assay system for implementing the embedded reconfigurable heterogeneous assay method according to any one of claims 1 to 7, the embedded reconfigurable heterogeneous assay system comprising:
the user interface layer is used for providing a Web visual task access mode and providing a load balancing function;
the system middleware layer is used for constructing a cluster and managing heterogeneous computing resources in a unified manner;
the hardware layer is used for uniform access of heterogeneous resources on the embedded computing board card and networking interconnection among boards;
the user interface layer includes:
the Web visualization module is used for providing a Web visualization interface to manage tasks, resources and users and providing guarantee for access and authorization;
the task access module is used for providing a uniform access address, ensuring the high availability of uniform access by using an agent and distributing tasks among clusters according to a load balancing strategy;
the system middleware layer comprises a dynamic cluster building module, a task pool building module, a resource pool building module and a virtual computing environment building module;
the dynamic cluster building module comprises heartbeat detection, database consistency and a dynamic center election strategy and is used for building and managing a cluster and realizing unified scheduling of tasks and resources;
the task pool construction module is used for realizing the construction of a cluster task pool;
and the resource pool building module is used for realizing the discovery and health check of the heterogeneous resources.
10. A processor carrying the embedded reconfigurable heterogeneous assay system of claim 9.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010323642.0A CN111694789A (en) | 2020-04-22 | 2020-04-22 | Embedded reconfigurable heterogeneous determination method, system, storage medium and processor |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010323642.0A CN111694789A (en) | 2020-04-22 | 2020-04-22 | Embedded reconfigurable heterogeneous determination method, system, storage medium and processor |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111694789A true CN111694789A (en) | 2020-09-22 |
Family
ID=72476528
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010323642.0A Pending CN111694789A (en) | 2020-04-22 | 2020-04-22 | Embedded reconfigurable heterogeneous determination method, system, storage medium and processor |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111694789A (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112328396A (en) * | 2020-11-09 | 2021-02-05 | 西安电子科技大学 | Dynamic self-adaptive SOPC fault-tolerant method based on task level |
CN112884633A (en) * | 2021-02-26 | 2021-06-01 | 上海联影医疗科技股份有限公司 | Image reconstruction apparatus, image reconstruction system, image reconstruction method, computer device, and medium |
CN112948118A (en) * | 2021-03-12 | 2021-06-11 | 上海哔哩哔哩科技有限公司 | Edge calculation method, platform, computer device and readable storage medium |
CN113111016A (en) * | 2021-04-19 | 2021-07-13 | 南京翼辉信息技术有限公司 | Single channel device communication middleware |
CN114124389A (en) * | 2021-11-09 | 2022-03-01 | 国网山东省电力公司电力科学研究院 | Reconfigurable computing-based national cryptographic algorithm FPGA deployment method and system |
CN114598591A (en) * | 2022-03-07 | 2022-06-07 | 中国电子科技集团公司第十四研究所 | Embedded platform node fault recovery system and method |
CN115277346A (en) * | 2022-09-28 | 2022-11-01 | 中国电子科技集团公司第十研究所 | Cloud native container network system facing embedded heterogeneous bus |
CN115695136A (en) * | 2022-12-30 | 2023-02-03 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-source data distributed embedded processing device and on-demand configuration method thereof |
WO2023226601A1 (en) * | 2022-05-23 | 2023-11-30 | 中兴通讯股份有限公司 | Anomaly processing method and apparatus for heterogeneous acceleration resource, and storage medium and electronic apparatus |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103259832A (en) * | 2012-12-24 | 2013-08-21 | 中国科学院沈阳自动化研究所 | Cluster resource control method for achieving dynamic load balance, fault diagnosis and failover |
CN105607954A (en) * | 2015-12-21 | 2016-05-25 | 华南师范大学 | Stateful container online migration method and apparatus |
CN107087019A (en) * | 2017-03-14 | 2017-08-22 | 西安电子科技大学 | A kind of end cloud cooperated computing framework and task scheduling apparatus and method |
CN108234191A (en) * | 2017-05-31 | 2018-06-29 | 深圳市创梦天地科技有限公司 | The management method and device of cloud computing platform |
CN110413389A (en) * | 2019-07-24 | 2019-11-05 | 浙江工业大学 | A kind of task schedule optimization method under the unbalanced Spark environment of resource |
CN110580198A (en) * | 2019-08-29 | 2019-12-17 | 上海仪电(集团)有限公司中央研究院 | Method and device for adaptively switching OpenStack computing node into control node |
CN110912780A (en) * | 2019-12-13 | 2020-03-24 | 无锡华云数据技术服务有限公司 | High-availability cluster detection method, system and controlled terminal |
-
2020
- 2020-04-22 CN CN202010323642.0A patent/CN111694789A/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103259832A (en) * | 2012-12-24 | 2013-08-21 | 中国科学院沈阳自动化研究所 | Cluster resource control method for achieving dynamic load balance, fault diagnosis and failover |
CN105607954A (en) * | 2015-12-21 | 2016-05-25 | 华南师范大学 | Stateful container online migration method and apparatus |
CN107087019A (en) * | 2017-03-14 | 2017-08-22 | 西安电子科技大学 | A kind of end cloud cooperated computing framework and task scheduling apparatus and method |
CN108234191A (en) * | 2017-05-31 | 2018-06-29 | 深圳市创梦天地科技有限公司 | The management method and device of cloud computing platform |
CN110413389A (en) * | 2019-07-24 | 2019-11-05 | 浙江工业大学 | A kind of task schedule optimization method under the unbalanced Spark environment of resource |
CN110580198A (en) * | 2019-08-29 | 2019-12-17 | 上海仪电(集团)有限公司中央研究院 | Method and device for adaptively switching OpenStack computing node into control node |
CN110912780A (en) * | 2019-12-13 | 2020-03-24 | 无锡华云数据技术服务有限公司 | High-availability cluster detection method, system and controlled terminal |
Cited By (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112328396B (en) * | 2020-11-09 | 2022-10-21 | 西安电子科技大学 | Dynamic self-adaptive SOPC fault-tolerant method based on task level |
CN112328396A (en) * | 2020-11-09 | 2021-02-05 | 西安电子科技大学 | Dynamic self-adaptive SOPC fault-tolerant method based on task level |
CN112884633B (en) * | 2021-02-26 | 2022-12-23 | 上海联影医疗科技股份有限公司 | Image reconstruction apparatus, image reconstruction system, image reconstruction method, computer device, and medium |
WO2022179631A1 (en) * | 2021-02-26 | 2022-09-01 | Shanghai United Imaging Healthcare Co., Ltd. | Systems and methods for image reconstruction |
CN112884633A (en) * | 2021-02-26 | 2021-06-01 | 上海联影医疗科技股份有限公司 | Image reconstruction apparatus, image reconstruction system, image reconstruction method, computer device, and medium |
EP4278319A4 (en) * | 2021-02-26 | 2024-07-31 | Shanghai United Imaging Healthcare Co Ltd | Systems and methods for image reconstruction |
CN112948118A (en) * | 2021-03-12 | 2021-06-11 | 上海哔哩哔哩科技有限公司 | Edge calculation method, platform, computer device and readable storage medium |
CN112948118B (en) * | 2021-03-12 | 2024-01-16 | 上海哔哩哔哩科技有限公司 | Edge computing method, platform, computer device and readable storage medium |
CN113111016A (en) * | 2021-04-19 | 2021-07-13 | 南京翼辉信息技术有限公司 | Single channel device communication middleware |
CN113111016B (en) * | 2021-04-19 | 2024-06-11 | 南京翼辉信息技术有限公司 | Single channel equipment communication middleware |
CN114124389A (en) * | 2021-11-09 | 2022-03-01 | 国网山东省电力公司电力科学研究院 | Reconfigurable computing-based national cryptographic algorithm FPGA deployment method and system |
CN114124389B (en) * | 2021-11-09 | 2023-08-11 | 国网山东省电力公司电力科学研究院 | Reconfigurable computing-based FPGA (field programmable Gate array) deployment method and system |
CN114598591A (en) * | 2022-03-07 | 2022-06-07 | 中国电子科技集团公司第十四研究所 | Embedded platform node fault recovery system and method |
CN114598591B (en) * | 2022-03-07 | 2024-02-02 | 中国电子科技集团公司第十四研究所 | Embedded platform node fault recovery system and method |
WO2023226601A1 (en) * | 2022-05-23 | 2023-11-30 | 中兴通讯股份有限公司 | Anomaly processing method and apparatus for heterogeneous acceleration resource, and storage medium and electronic apparatus |
CN115277346A (en) * | 2022-09-28 | 2022-11-01 | 中国电子科技集团公司第十研究所 | Cloud native container network system facing embedded heterogeneous bus |
CN115277346B (en) * | 2022-09-28 | 2023-01-20 | 中国电子科技集团公司第十研究所 | Cloud native container network system facing embedded heterogeneous bus |
CN115695136B (en) * | 2022-12-30 | 2023-03-10 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-source data distributed embedded processing device and on-demand configuration method thereof |
CN115695136A (en) * | 2022-12-30 | 2023-02-03 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-source data distributed embedded processing device and on-demand configuration method thereof |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111694789A (en) | Embedded reconfigurable heterogeneous determination method, system, storage medium and processor | |
US11768695B2 (en) | Methods and apparatus to deploy a hybrid workload domain | |
CN102325192B (en) | Cloud computing implementation method and system | |
CN107066319B (en) | Multi-dimensional scheduling system for heterogeneous resources | |
KR101823505B1 (en) | A computer cluster arragement for processing a computation task and method for operation thereof | |
US11740921B2 (en) | Coordinated container scheduling for improved resource allocation in virtual computing environment | |
CN112948063B (en) | Cloud platform creation method and device, cloud platform and cloud platform implementation system | |
CN110888743A (en) | GPU resource using method, device and storage medium | |
CN111897654A (en) | Method and device for migrating application to cloud platform, electronic equipment and storage medium | |
CN110784539A (en) | Data management system and method based on cloud computing | |
US11886898B2 (en) | GPU-remoting latency aware virtual machine migration | |
US9755986B1 (en) | Techniques for tightly-integrating an enterprise storage array into a distributed virtualized computing environment | |
CN115080436B (en) | Test index determining method and device, electronic equipment and storage medium | |
US11531572B2 (en) | Cross-cluster host reassignment | |
CN113626150A (en) | Elastic container example implementation method | |
CN107528871A (en) | Data analysis in storage system | |
CN117170858A (en) | Dynamic scalable mimicry computing method and system | |
US10157066B2 (en) | Method for optimizing performance of computationally intensive applications | |
CN109558214B (en) | Host machine resource management method and device in heterogeneous environment and storage medium | |
CN106844021B (en) | Computing environment resource management system and management method thereof | |
CN118210634B (en) | Server system, resource scheduling method of server system, chip and core particle | |
CN115328651B (en) | Light-weight micro cloud system based on domestic VPX server | |
Xu et al. | Network-aware task assignment for MapReduce applications in shared clusters | |
US20240160487A1 (en) | Flexible gpu resource scheduling method in large-scale container operation environment | |
US20220138008A1 (en) | Methods and apparatus to manage resources in a hybrid workload domain |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200922 |
|
RJ01 | Rejection of invention patent application after publication |