CN109697153A - Monitoring method, monitoring system and computer readable storage medium - Google Patents
Monitoring method, monitoring system and computer readable storage medium Download PDFInfo
- Publication number
- CN109697153A CN109697153A CN201811616164.1A CN201811616164A CN109697153A CN 109697153 A CN109697153 A CN 109697153A CN 201811616164 A CN201811616164 A CN 201811616164A CN 109697153 A CN109697153 A CN 109697153A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring method
- container
- resource data
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/328—Computer systems status display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Present disclose provides a kind of monitoring method, monitoring system and computer readable storage medium, which includes: the first obtaining step, wherein obtains information associated with each working node to host node by the first container group of deployment；Second obtaining step, wherein obtain the achievement data of resource data and resource data according to the agent container that the information is disposed on each working node by the first container group of deployment；Calculate step, wherein polymerization calculating is carried out to the resource data and achievement data of acquisition by the first container group of deployment；And display step, wherein the data that polymerization calculates are shown by display unit.
Description
Technical field
This disclosure relates to a kind of monitoring method, monitoring system and computer readable storage medium, particularly for
Kubernetes container cluster management system.
Background technique
Kubernetes (typically, since letter is longer, also being write as " K8S ") is most to start to be designed and developed by google company
And contribute to the open source container cluster management project of Cloud Native Computing Foundation.Its design object
Be between mainframe cluster provide one can automatically dispose, expansion, O&M Open Source Platform.In general, Kubernetes is tied
The work of Docker container instrument is closed, and integrates the mainframe cluster of multiple operation Docker containers.
Under traditional O&M mode, the corresponding server of manual creation and resource are needed, and manually build the dependence of project
Environment manually estimates memory and CPU that service needs to occupy.Moreover, when break down such as server accident delay machine when, need
Artificial treatment.It can be seen that traditional mode is not only time-consuming and laborious, but also it cannot rationally utilize resource.
Since K8S carries out Container Management using Docker, so inherently having all characteristics of Docker, it is only necessary to make
It can operation service with the Docker mirror image of respective environment.Compared to traditional O&M mode, due to Docker container it is independent every
From the characteristics of, reduce it is various rely on environment conflicts, reduce O&M cost, also facilitate the migration of integrity service.
The entire life mid-term monitored in operation and maintenance system or even product is all important a link, for different applications
Scene, monitoring scheme can also be very different.One complete monitoring system includes: acquisition data, analysis storing data, exhibition
Registration evidence, alarm and automatic processing and the security mechanism of monitoring tools itself etc..
Currently, the monitor mode of opposite mainstream is carried out by Open-Source Tools, such as works such as zabbix, Prometheus
Tool carries out data acquisition, and the showing interfaces open source softwares such as grafna, kibana is cooperated to show monitoring data.But on the one hand,
Current Open-Source Tools are suitble to monitor relatively simple achievement data, it is difficult to meet the varigrained performance indicator of overall monitor
Demand；On the other hand, the use of Open-Source Tools, including deployment, query configuration it is comparatively laborious, in operation expanding need carry out compared with
More configuration adjustment.Especially in the case where number of containers reaches even hundreds of common situations, monitoring is significantly increased
Difficulty and cost.
Summary of the invention
The disclosure is intended to provide what one kind can especially monitor the progress of K8S container cluster management system comprehensively and efficiently
Method and system.
According to one aspect of the disclosure, a kind of monitoring method is provided, characterized by comprising: the first obtaining step,
Wherein, information associated with each working node is obtained to host node by the first container group of deployment；Second obtaining step,
In, by the first container group of deployment according to the agent container that the information is disposed on each working node obtain resource data with
And the achievement data of resource data；Calculate step, wherein by the first container group of deployment to the resource data and index of acquisition
Data carry out polymerization calculating；And display step, wherein the data that polymerization calculates are shown by display unit.
A kind of monitoring system another aspect of the present disclosure provides, characterized by comprising: first obtains list
Member is configured as obtaining information associated with each working node to host node by the first container group of deployment；Second obtains
Unit is configured as being obtained by the first container group of deployment according to the agent container that the information is disposed on each working node
The achievement data of resource data and resource data；Computing unit is configured as the first container group by deployment to acquisition
Resource data and achievement data carry out polymerization calculating；And display unit, it is configured as showing that polymerization calculates by display unit
Data.
A kind of monitoring system another aspect of the present disclosure provides, the monitoring system include: one or more
Processor and one or more memories, are stored with computer program above, when the computer program is held by processor
One or more processor is made to implement monitoring method as described above when row.
A kind of computer readable storage medium another aspect of the present disclosure provides, is stored with computer above
Program implements one or more processor
Monitoring method as described above.
According to the aspects above of the disclosure, the every of such as K8S is completely obtained by modes such as agency, system interfaces and is provided
The information such as source data, achievement data, and stored, analyzed and shown, to meet the monitoring demand of O&M, improve production O&M
Efficiency.
According to being described below referring to attached drawing, the other feature and advantage of the disclosure be will become apparent.
Detailed description of the invention
Be incorporated in specification and the attached drawing for constituting part of specification show embodiment of the disclosure, and with retouch
It states together for illustrating the principle of the disclosure.
Fig. 1 is to show the schematic diagram for the platform for implementing the monitoring method according to an embodiment of the present disclosure.
Fig. 2 is to show the block diagram of the basic configuration of the monitoring system according to an embodiment of the present disclosure.
Fig. 3 is to show the flow chart of the monitoring method according to an embodiment of the present disclosure.
Fig. 4 A-4B is to show the schematic diagram of the acquisition operation according to an embodiment of the present disclosure.
Fig. 5 is to show the schematic diagram of the converging operation according to an embodiment of the present disclosure.
Fig. 6 A-6B shows the Snipping Tool of the overall state of two sets of K8S platforms.
Fig. 7 A-7D shows the Snipping Tool of part K8S resource.
Fig. 8 shows the Snipping Tool of the resource information of fine granulation.
Fig. 9 is to show the exemplary configuration figure that computer equipment according to an embodiment of the present disclosure may be implemented.
Specific embodiment
For clarity and conciseness, all features of embodiment are not described in the description.It should be understood, however, that
Much settings specific to embodiment must be made, during implementing to embodiment to realize the tool of developer
Body target, for example, meeting restrictive condition those of related to equipment and business, and these restrictive conditions may be with implementation
Example difference and change.
Herein, it should be noted that in order to avoid having obscured the disclosure because of unnecessary details, only show in the accompanying drawings
The processing step and/or device structure closely related with the scheme according at least to the disclosure, and be omitted and disclosure relationship
Little other details.
For clear explanation, some related notions are introduced first.
Cluster (cluster): cluster is one group of computer that is mutually independent, being interconnected by high speed network, they are constituted
One group, and managed with the mode of triangular web.When one client and cluster interact, cluster seems an independence
Server.Cluster is for improving availability and scalability.
Container (container): container is that software package independently can be performed in a lightweight, comprising running one needed for it
It cuts: code, when operation, system tool, system library, and setting.The lightweight of container allows it to immediately begin to execute simultaneously
Use less CPU and content；Container is based on open standard, it is run in any architecture；And container
Application program and its ambient enviroment can be kept apart, therefore container improves security of system.
Container group (Pod): Pod is that the minimum that can be created and dispose in K8S is also most simple unit.One Pod is represented
The process run in cluster.Encapsulate one or more containers of application in Pod.Pod operate in one we be referred to as
In environment for node Node, Node is also possible to a virtual machine of cloud either physical machine.Usually on one node
Run several hundred a Pod.
Deployment (Deploy): Deploy be container instance template, container instance be created that according to Deploy come.?
The mirror image of container, the version of container, the information such as the quantity to be disposed of container can be write in Deploy object exactly.
Service (Service): Service service is the core of distributed type assemblies framework, a Service object possess as
Lower feature: 1. possess the name (such as mysql-server) uniquely specified；2. possess a virtual IP address (ClusterIP,
ServiceIP or VIP) and port numbers；3. being capable of providing remote service ability；4. being mapped to this service ability of offer
One group of container is using upper.
NameSpace (Namespace): being another important concept in K8S system, by by pair of internal system
As disparity items, group or the user group that " distribution " is into different Namespace, and formation is grouped in logic, convenient for different
It can also be managed respectively while being grouped in the shared resource for using entire cluster.
Interface (Interface): the api interface usually said is exactly the interface that application program or system provide.So below
What the K8S interface mentioned referred to is exactly the api interface that this system of K8S provides.Api interface can be used in embodiment of the disclosure
Obtain data.
Resource data: can be understood as the data of the resource object of such as K8S, for example, POD, Deploy these have it is specific right
As the data of example, just such as people, article.The parameter of resource data can be understood as the detailed content of resource data, such as
Pod just has title, affiliated namespace, the available cpu of distribution and the upper limit, lower limit, the container for including of memory of Pod etc.
Deng.The parameter of resource data can be understood as the static of resource data and limit.
Achievement data: can be understood as the attribute of example, such as cpu, the indicator memory of Pod, each resource type have
Different indexs.For example the index of people can be height, weight, the index of article can be volume or other.Resource data
Achievement data can be understood as occupied real resource when operation.Using the parameter and achievement data it can be concluded that the money
The loading condition of source data, as hereinafter referring to described by Fig. 5.
Each embodiment of the disclosure is described in detail with reference to the accompanying drawing.
Fig. 1 is to show the schematic diagram for the platform for implementing the monitoring method according to an embodiment of the present disclosure.
As shown in Figure 1, in the platform of implementing monitoring method, including multiple node 101-104 and monitoring system 100.This
A little node 101-104 include host node 101 and working node 102-103.Host node and working node are in such as K8S structure
Known node, host node be responsible for maintenance system database, maintenance cluster state and responsible scheduling of resource etc., working node
Be responsible for the period of maintenance container, service discovery and load balancing are provided and DNS service is provided etc., details are not described herein again.
Actual business function is disposed on K8S, business function can be realized by multiple business modules, can also be by one
Business module is realized.For example the crm system of telecommunications is deployed in K8S environmentally, this system contains many business moulds
Block, such as the push of order reception, order, enquiry module etc..Business module division is that business development project team voluntarily divides
, each business module is then generated by Pod in a manner of Deploy according to actual business requirement and is deployed in K8S, root is facilitated
Adjust the Pod quantity of each module (Deploy) operation in real time according to load.Business module is configured to the disclosure after having disposed
Above-mentioned platform in, facilitate platform to carry out subsequent monitoring.
Monitoring system 100 be responsible for monitoring K8S O&M state, such as acquisition data, analysis storing data, display data,
Alarm and automatic processing and the security mechanism of monitoring tools itself etc..It will be seen from figure 1 that monitoring system 100 passes through
The agency being deployed on node 101-104 from node 101-104 obtain data, storing data, data are carried out calculating analysis,
And display data, to realize above-mentioned monitoring purpose.The detail of these operations is described further below.
Fig. 2 shows the block diagrams according to the basic configuration of the monitoring system of an embodiment of the present disclosure.As described above, monitoring
System can be used for obtaining data, storing data in platform O&M, data be carried out with calculating analysis and display data, therefore
Including corresponding operation unit.As shown in Fig. 2, monitoring system may include such as first acquisition unit 201, second acquisition unit
202, computing unit 203 and display unit 204.These units 201-204 can be to be configured in system initialization process.
First acquisition unit 201 can be configured to obtain and each work by the first container group of deployment to host node 101
The associated information of node 102-104.
Second acquisition unit 202 can be configured to through the first container group of deployment according to the information to each working node
The agent container disposed on 102-104 obtains the achievement data of resource data and resource data.
Computing unit 203 can be configured to by the first container group of deployment to the resource data of acquisition and achievement data into
Row polymerization calculates.
Display unit 204 can be configured to show the data that polymerization calculates by display unit.
In addition, each unit, which can according to need, is combined or divided into subassembly.Although for example, first acquisition unit
201 and second acquisition unit 202 be described as isolated component, but first acquisition unit 201 and second acquisition unit 202 can be with
It is merged into a subassembly, such as is referred to as acquiring unit.Further, although computing unit 203 is described as obtaining list with first
Member 201 and second acquisition unit 202 separate, but computing unit 203 can be obtained with first acquisition unit 201 and/or second
Unit 202 is merged into a subassembly, such as acquisition computing unit.
Monitoring is described in detail according to the flow chart of the monitoring method of an embodiment of the present disclosure below with reference to shown in Fig. 3
The operation of system 100 and its each component units.Monitoring method in Fig. 3 system 100 that can be monitored executes at runtime.
Step S301 is the first obtaining step, wherein is obtained and each work by the first container group of deployment to host node 101
Make the associated information of node 102-104.Step S301 can be held by the first acquisition unit 201 in monitoring system 100
Row.
The meaning of container group is as mentioned before.The first container group can be by disposing to realize the main of monitoring system 100
The object of operation.For example, the first container group obtains required data from host node and working node, and the data of acquisition are gathered
It is total to calculate, as described in reference computing unit 303 hereinafter.Therefore, it is similar to previously described acquisition computing unit.
The first container group can be deployed on host node 101, or on one of being deployed in working node 102-104, only
It wants to obtain required data and execute polymerization to calculate.
Information associated with each working node 102-104, such as each working node are stored on host node 101
The identifier of 102-104 can uniquely identify corresponding working node by identifier.The information is without being limited thereto, can be with
The data such as file system capacity, CPU, memory for example including each working node 102-104.Therefore, by obtaining these letters
Breath, monitoring system 100 can preferably access each working node, and understand the ability of working node.
The first container group can be used various ways and obtain data to host node, such as, but not limited to:
Command line mode: it is connected to by way of ssh on target machine such as host node 101 and executes order to obtain
Access evidence.Ssh can simply be understood as remotely connecting.Such as be first coupled on host node 101 on the remote machine, then execute
Order is to obtain data.The data of acquisition can store in the storage medium such as database or file in monitoring system 100.
The active way of output: through application program during self-operating, actively will according to pre-set instruction
Target data is output in specified storage medium (such as database or file).For example, host node 101 can at runtime actively
It outputs data in database or file, then monitoring system 100 obtains data from the database or file.
Interface obtains: obtaining specified data by the api interface for calling application program open.As previously mentioned, this
Interface described in text can be the api interface of K8S system.For example, monitoring system 100 is obtained by the api interface from host node 101
Take required data.
Fig. 3 is returned below to continue to describe.Step S302 is the second obtaining step, wherein passes through the first container group of deployment
The finger of resource data and resource data is obtained according to the agent container that the information is disposed on each working node 102-104
Mark data.Step S302 can be executed by the second acquisition unit 202 in monitoring system 100.The data got can deposit
Storage is in the database of monitoring system 100.
As previously described, resource data can be understood as the data of the resource object of such as K8S, such as Pod,
These have the data of specific object instance to Deploy, just such as people, article.The parameter of resource data can be understood as resource data
Detailed content, such as Pod, just have the title of Pod, affiliated namespace, the available cpu of distribution and memory the upper limit, under
It limits, the container for including etc..
Achievement data can be understood as the attribute of example, such as cpu, the indicator memory of Pod, each resource type have not
Same index.For example the index of people can be height and weight, the index of article can be volume or other.
In existing monitoring platform, as described in the background section, using the adviser tool or interface tool of open source, if
Need to resource data to this fine granulation and its achievement data monitoring, then need to carry out individually relatively complicated order configuration
And deployment especially reaches even hundreds of common feelings in number of containers therefore, it is difficult to carry out comprehensive and efficient monitoring
Under condition.
Therefore, in the disclosure, other than the first container group of deployment, further in each working node 102-104
Upper deployment agent container, agent container can for example be created by Docker mirror image, wherein considering enterprising in working node
The individual cultivation of row data acquisition.When there are multiple working nodes, creation can be directly repeated for these working nodes
Agent container.
Referring to the detailed process of the flow chart citing description step S302 of Fig. 4 A.In step S401, agent container
The resource data for example read on each working node 102-104 can be directly acquired according to predetermined configurations.
In step S402, agent container can directly acquire according to predetermined configurations and for example read each working node
The achievement data of resource data on 102-104, so as to which these data are returned to the first container group.
For example, predetermined configurations may include at least one in following: the acquisition time of data, data type, data format.
Therefore, agent container can periodically obtain the data for meeting data format for specific resource data such as Pod.Herein,
Data format can be the reference format in system, be also possible to customized data format.
In step S403, the first container group can access agent container by API.
In step S404, the first container group obtains required data from agent container.
In addition, agent container can also combine one of previously described command line mode, the active way of output, interface mode
The data of acquisition to interact with the first container group.For example, although describing the first container group in preceding step S403 passes through API
Come the data for accessing agent container to obtain required.But the first container group can also obtain as follows data: on the remote machine
It is first coupled on one of working node 102-104, it is then corresponding to pass through using order line execution order on the remote machine
Agent container obtains data.In the latter case, step S403 can be executed before step S401 and step S402, thus
Flow chart shown in Fig. 4 B is formed, remaining duplicate part A referring to fig. 4 is repeated no more.
In the case where the data on working node can acquire otherwise, the first container group can also directly lead to
API is crossed to obtain these data.
In addition, though in the embodiment of Fig. 1 and Fig. 3, it is not shown agent container on host node 101, but can also be with
Agent container similarly is disposed on host node 101 with working node 102-104, obtains operation to realize.
Fig. 3 is returned below to continue to describe.Step S303 is to calculate step, wherein the first container group by deployment is to obtaining
The resource data and achievement data taken carries out polymerization calculating.
Before the process that description polymerization calculates, after illustrating the relationship between resource in order to understand by taking K8S as an example first
The description in face.
The Pod mentioned before is the minimum unit as K8S scheduling, may include one or more in an actually Pod
A container.Then Pod is generated by Deploy.Namespace can simply be understood as customized module and divide (at this
In disclosed practical business, all business do not refine point all at a namespace), Deploy belongs to some
namespace。
So relationship described above be K8S cluster-namespace-Deploy-Pod-container, from top to bottom for
Inclusion relation, to form tree structure.
So polymerization calculates, extended counter calculating is carried out both for relation above or tree structure.Because of agency
Container is all the achievement datas such as cpu, the memory of the bottom i.e. container acquired when acquiring data, but actual conditions
Under do not need the operating condition of concern too deep layer, it is only necessary to see some Pod or business module or even want to see entire K8S cluster
Operating condition, that just need the achievement data by collected each container summarized by relation above or tree structure or
Polymerization calculates.
That is, after the achievement datas such as cpu, the memory of the container for collecting the bottom, first belonging to one
The achievement datas such as cpu, the memory of each container of Pod are added, and the result of acquisition is exactly the achievement data of the Pod.Then, belonging to
The achievement datas such as cpu, the memory of each Pod of one Deploy are added, and the result of acquisition is exactly the achievement data of the Deploy.
Then, as soon as the achievement datas such as cpu, the memory of each Pod for belonging to a namespace can also be added, the result of acquisition
It is the achievement data of the namespace.And so on, the achievement data at all levels of entire tree structure can be formed, with
Just it stores in database and shows.
Next, illustrating the operation that polymerization calculates using the example of current business and referring to the flow chart of Fig. 5.One
Business module is deployed in K8S in a manner of Deploy, is explained according to hierarchical relationship before, with telecom client relation management
In crm system for one of order reception module.Under the Deploy of this order reception module may comprising one or
Multiple Pod, this Pod quantity are adjusted according to business load situation, and busy can increase, can reduce when idle；Include in each Pod
Two containers, the quantity of container is determined according to business function situation by exploitation in Pod.
In step S501, after step S301 and step S302 acquisition data, the computing unit 203 of monitoring system 100
It is capable of forming the following hierarchical relationship of resource data first:
The container in Pod-Pod under the Deploy-of the order reception module module.
Then, in step S502, according to the achievement data of the step S302 resource data obtained, i.e. container in Pod
Cpu, internal storage data, the information there are also container, so that it may according to the relationship of Pod and container, by the cpu of the container under the Pod, interior
Achievement data addition is deposited, is exactly the achievement data of the Pod.
Then, in step S503, if the business module heavy traffic, therefore multiple Pod are contained, similarly by Pod
Achievement data addition just obtain the business module or the achievement data of Deploy.Alternatively, if the business module only one
Pod, then directly using the achievement data of the Pod as the achievement data of the business module.
Above-mentioned cpu, internal storage data be all specifically to be worth, for example cpu has used how many a cores, and EMS memory occupation is more
Few G byte, but this can not reaction load the case where, so in the case where needing computational load, need to calculate cpu and interior
Deposit the percentage of occupancy.
Optionally, in step S504, for the resource data computational load in hierarchical relationship.It obtains in step s 302
Resource data parameter in can get corresponding resource allocation, such as the distribution mentioned in the description of S302 is available
The upper limit of cpu and memory, therefore cpu can be calculated according to the upper limit and according to the achievement data obtained in step S302
With the occupied percentage of memory.For example, according to the corresponding resource allocation upper limit obtained in step S302 and referring to for Pod
Data are marked, the load of the Pod can be obtained.
What the computing function of this part can be carried out in computing unit 203, or held in other modules of monitoring system 100
Row.
In another example, it includes being calculated according to the division of business module that polymerization, which calculates,.Such as in the description of Fig. 1
In refer to, business module division is that business development project team voluntarily divides, then according to actual business requirement by each industry
Business module is generated Pod in a manner of Deploy and is deployed in K8S, is facilitated and is adjusted each module (Deploy) fortune in real time according to load
Capable Pod quantity.Business module is configured in the above-mentioned platform of the disclosure after having disposed, platform is facilitated to carry out subsequent prison
Control.
In order to realize the monitoring to the multiple business modules so divided, similarly may be used with the example of the crm module in Fig. 5
To carry out polymerization calculating to other business modules, to realize the monitoring to multiple business modules.
Fig. 3 is returned below to continue to describe.Step S304 is display step, wherein shows that polymerization calculates by display unit
Data.These data are as described above, include resource data and its achievement data.
It is corresponding with the data that polymerization calculates, to these data show can also according to the hierarchical relationship of resource data into
Row.For example, being shown according to hierarchical relationship K8S cluster-namespace-Deploy-Pod-container.Appoint in addition, clicking
One level can also further display the data of next level.
Similarly, the display of these data can also be carried out according to the division of business module.Such as it is previously mentioned
Crm module.
Display operation can be shown in various types of displays in monitoring system 100 by known display technology
On (not shown).
The display example of the data of the polymerization calculating of the disclosure is provided below with reference to accompanying drawings.
Fig. 6 A-6B shows the real-time interface (Snipping Tool) of the periodic refreshing of the overall state of two sets of K8S platforms, wherein
Mainly show:
The cpu of cluster entirety, memory, storage service condition；
The resource quantity (NODE quantity, Deploy quantity, Pod quantity, Service quantity) of-K8S cluster；
(this is the configuration database of K8S to-K8S etcd, is the included configuration data for being used to save K8S itself of K8S
Database) operating status；
The operating status of-Rs, Rc, Deploy (expection whether the Pod quantity of Deploy operation such as configures is consistent)；
- Pod matrix shows the operation conditions of Pod, and different colors can be shown according to load, intuitively observes load
The excessively high or abnormal Pod of operation, indicates an abnormal Pod with arrow in fig. 6.
At least two dimensions may be selected to be presented in Pod matrix:
Fig. 6 A is presented according to the dimension of service center, and service center belongs to a upper level for business module, because
Practical business module number is too many, so being presented with the dimension of service center.It can be with intuitive monitoring according to business module from Fig. 6 A
The resource data for the fine granulation that (service center) divides and the state of resource data, such as kube-system this business
There are CPU or the memory usage of a Pod excessively high in center.Specific business module can be shown by clicking failure Pod.
Fig. 6 B is presented with the dimension of device label, and device label is special marking on each node node, mainly
For providing which platform node can only be used to which business disposed, the Pod of same business function is prevented excessively to be distributed to cluster
In each node, facilitate management.The resource data for the fine granulation that can also be divided from Fig. 6 B with intuitive monitoring according to business module
And the state of resource data, the layout of the histogram in Fig. 6 B is identical as Fig. 6 A, while not having in Pod matrix faulty
Pod, therefore not over arrows.
Fig. 7 A-7D shows part K8S resource, that is, resource data information Snipping Tool, these information are to pass through
The API of K8S is called to obtain.Fig. 7 A shows the information of node Node, and Fig. 7 B shows the information of service Service, and Fig. 7 C is shown
The information of container group Pod, Fig. 7 D show the information of business module.
The Snipping Tool of the resource information of Fig. 8 real-time display more fine granulation, the i.e. basic information and performance number of container
According to, while can check the process of receptacle and carry out container register, as shown on the right.
It can be seen that the disclosure completely obtains every resource of such as K8S by modes such as the first container group, agent containers
The information such as data, achievement data store it, are analyzed, and various dimensions, be shown to a variety of styles, avoid
The complex configurations and deployment to be executed in the case where being monitored using Open-Source Tools, thus meet the monitoring demand of O&M,
Improve production O&M efficiency.
Although front describes monitoring method and monitoring system in conjunction with the embodiments, monitoring system and monitoring method are unlimited
In above-described embodiment, other operations can also be performed or there are other functions.As it was noted above, monitoring method or monitoring system
100 also execute general following monitoring function:
Alarm function, mainly timing judge the alarm regulation configured in monitoring system 100, if abnormal
Display alarm and short message alarm in monitoring system；
The relevant alarm of-K8S, for example the cpu or memory of Pod have been more than threshold value；The Pod quantity of Deploy operation and pre-
Phase is not inconsistent, and Pod state is not running etc..These rules can be by configuring regulation；
It is automatic to execute in Pod in preprepared cleaning after automatic processing, such as the excessively high alarm of Pod memory
The script deposited clears up memory, achievees the purpose that automatic processing.Similarly, there are also the function of some other automatic processings
Can, it is realized according to user demand.
Inherently safe mechanism refers to that monitoring function itself (such as agent, calculate node) runs whether normal, mesh
Preceding agent and computing unit 203 are deployed in the K8S environment of monitoring in a manner of container, so we are to pass through tune
Information of container is obtained with the api interface of K8S, so that it may which whether normal, reach inherently safe monitoring if viewing monitoring associated vessel
Purpose.
The foregoing embodiments of the disclosure can not only can be also used for using the monitoring with K8S platform to single or multiple
Docker container is monitored.
Fig. 9 shows the exemplary configuration that computer equipment 2000 according to an embodiment of the present disclosure may be implemented.It calculates
Machine equipment 2000 is can be using the example of the hardware device of the above-mentioned aspect of the disclosure.Computer equipment 2000, which can be, is matched
It is set to any machine for executing processing and/or calculating.Computer equipment 2000 can be but be not limited to work station, server,
Desktop computer, laptop computer, tablet computer, personal data assistants (PDA), smart phone, car-mounted computer or with
Upper combination.The aforementioned monitoring system of the disclosure can be wholly or at least partially by above-mentioned computer equipment 2000 or similar to its
Equipment or system realize.
As shown in Fig. 9 figure, computer equipment 2000 may include that may connect via one or more Interface & Bus 2002
The one or more elements for connecing or communicating.For example, computer equipment 2000 may include bus 2002, one or more processors
2004, one or more input equipments 2006 and one or more output equipments 2008.Bus 2002 may include but unlimited
In Industry Standard Architecture (Industry Standard Architecture, ISA) bus, Micro Channel Architecture (Micro
Channel Architecture, MCA) bus, enhancing ISA (EISA) bus, Video Electronics Standards Association (VESA) part be total
Line and peripheral component interconnection (PCI) bus etc..One or more processing equipments 2004 can be any kind of processor,
And it can include but is not limited to one or more general processors or application specific processor (such as dedicated processes chip).Input is set
Standby 2006 can be and can input any kind of input equipment of information to computer equipment, and can include but is not limited to
Mouse, keyboard, touch screen, microphone and/or remote controllers.Output equipment 2008 can be any class that information can be presented
The equipment of type, and can include but is not limited to display, loudspeaker, video/audio outlet terminal, vibrator and/or printing
Machine.Computer equipment 2000 can also include or be connected to non-transient storage equipment 2010, the non-transient storage equipment 2010
It can be storage equipment any non-transient and that data storage may be implemented, and can include but is not limited to dish driving
Device, light storage device, solid-state memory, floppy disk, flexible disk, hard disk, tape or any other magnetic medium, compact disk or any
Other optical mediums, ROM (read-only memory), RAM (random access memory), buffer memory and/or any other storage
Chip or node, and/or computer can be from other any media for wherein reading data, instruction and/or code.It is non-transient to deposit
Storing up equipment 2010 can be detachably connected with any interface.Non-transient storage equipment 2010 can have it is being stored thereon,
For realizing aforementioned monitoring method and/or computer program/data of step.Computer equipment 2000 can also include that communication is set
Standby 2012, the communication equipment 2012 can be can enable any kind of equipment with external device (ED) and/or network communication or
System, and can include but is not limited to modem, network card, infrared communication equipment, wireless telecom equipment and/or core
Piece collection (such as bluetoothTMEquipment, 1302.11 equipment, WiFi equipment, WiMax equipment, cellular communication facility etc.).
Computer equipment 2000 can also include working storage 2014.The working storage 2014 can be and can store
Any kind of working storage of the instruction and/or data useful for processor 2004, and can include but is not limited to
Random access memory (RAM) and read-only memory (ROM).
Software element in above-mentioned working storage 2014 can include but is not limited to operating system 2016, one or
Multiple application programs 2018, driver and/or other data and code.Said one or multiple application programs 2018 may include
For executing the instruction of monitoring method as described above and each step.It can be by reading and executing one or more application program
2018 processor realizes device/cells/elements of monitoring system 100 above-mentioned.More specifically, for example, aforementioned monitoring system
First acquisition unit 201 in 100 can executed by processor 2004 with the instruction for executing the step S301 of Fig. 3
It is realized when application program 2018.In addition, for example, the second acquisition unit 202 in aforementioned monitoring system 100 can be by processor
2004 realizations when executing the application program 2018 with the instruction for executing step S302.In addition, for example, aforementioned monitoring is
Computing unit 203 in system 100 can execute the application journey having for executing the instruction of step S303 by processor 2004
It is realized when sequence 2018.In addition, for example, the display unit 204 in aforementioned monitoring system 100 can execute tool by processor 2004
Realization when having the application program 2018 of the instruction for executing step S304.It is unshowned other each in aforementioned monitoring system 100
A unit can also be realized with similar mode.The executable code or source code of the instruction of software element can store non-temporary
In state computer readable storage medium (such as storage equipment 2010 as described above), and can be by compiling and/or installing
It reads in working storage 2014.The executable code or source code of the instruction of software element can also be downloaded from remote location.
It should be appreciated that modification can be carried out according to particular requirement.It is, for example, possible to use the hardware of customization and/or specific members
Part can be realized in a manner of hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof.In addition,
It can be using the connection with other computer equipments (such as network inputs/output equipment).For example, the computer equipment of the disclosure
Some or all of can according to the disclosure by using assembler language programming hardware (e.g., including field-programmable gate array
Arrange the programmable logic circuit of (FPGA) and/or programmable logic array (PLA)) or logic and algorithm hardware program language
(such as VERILOG, VHDL, C++) Lai Shixian.
It will be further understood that the element of computer equipment 2000 can be distributed over the entire network.For example, can make
While executing some processing with a processor, other processing are executed using other teleprocessing units.Computer system 2000
Other elements can also similarly be distributed.Therefore, computer equipment 2000 is construed as executing processing in multiple places
Distributed computing system.
Disclosed method and equipment can be implemented in many ways.For example, can by software, hardware, firmware,
Or any combination thereof implement the disclosure.The order of above-mentioned method and step is merely illustrative, and disclosed method step is not
It is limited to order described in detail above, clearly states unless otherwise.In addition, in some embodiments, the disclosure may be used also
To be implemented as recording program in the recording medium comprising for realizing according to the machine readable finger of disclosed method
It enables.Thus, the disclosure covers storage also for realizing according to the recording medium of the program of disclosed method.
Although passed through example illustrates some embodiments of the present disclosure in detail, those skilled in the art should be managed
Solution, above-mentioned example are intended merely to be illustrative without limiting the scope of the present disclosure.It will be understood by those skilled in the art that above-mentioned implementation
Example can be modified in the case where not departing from the scope and essence of the disclosure.The scope of the present disclosure is wanted by appended right
Ask restriction.
Claims (15)
1. a kind of monitoring method, characterized by comprising:
First obtaining step, wherein letter associated with each working node is obtained to host node by the first container group of deployment
Breath；
Second obtaining step, wherein the generation disposed according to the information on each working node by the first container group of deployment
Manage the achievement data that container obtains resource data and resource data；
Calculate step, wherein polymerization calculating is carried out to the resource data and achievement data of acquisition by the first container group of deployment；
And
Show step, wherein the data that polymerization calculates are shown by display unit.
2. monitoring method according to claim 1, wherein the information includes at least the identifier of each working node.
3. monitoring method according to claim 1, wherein resource data includes at least one following object: container group
Pod services Service, disposes Deploy, mirror image Image, copy controller Rc, copy set Rs, NameSpace Namespace,
And configuration item ConfigMap.
4. monitoring method according to claim 3, wherein resource data further includes the parameter of the object.
5. monitoring method according to any one of claim 1 to 4, wherein in the second obtaining step, the first container group
It is obtained by K8S application programming interfaces to agent container.
6. monitoring method according to any one of claim 1 to 4, wherein achievement data include it is following at least one: hold
Device, cpu, memory and other memories.
7. monitoring method according to any one of claim 1 to 4, wherein the agency disposed on each working node holds
Device can directly acquire resource data and achievement data in relevant work node according to predetermined configurations.
8. monitoring method according to claim 7, wherein the predetermined configurations include at least at least one of the following: obtaining
Take time, data type, data format.
9. monitoring method according to any one of claim 1 to 4, wherein it includes according to resource data that polymerization, which calculates,
Hierarchical relationship is calculated.
10. monitoring method according to any one of claim 1 to 4, wherein it includes according to business module that polymerization, which calculates,
It divides to be calculated.
11. monitoring method according to any one of claim 1 to 4, wherein display step includes being shown with different dimensions
It polymerize the resource data and achievement data calculated.
12. monitoring method according to any one of claim 1 to 4, wherein the first container group can be deployed in main section
On point, or on one of being deployed in working node.
13. a kind of monitoring system, characterized by comprising:
First acquisition unit is configured as obtaining by the first container group of deployment to host node associated with each working node
Information；
Second acquisition unit is configured as being disposed according to the information on each working node by the first container group of deployment
Agent container obtains the achievement data of resource data and resource data；
Computing unit is configured as carrying out polymerization meter to the resource data and achievement data of acquisition by the first container group of deployment
It calculates；And
Display unit is configured as showing the data that polymerization calculates by display unit.
14. a kind of monitoring system, characterized by comprising:
One or more processors, and
One or more memories, are stored with computer program above, make when the computer program is executed by processor
Obtain monitoring method of one or more processor implementation as described in one of claim 1-12.
15. a kind of computer readable storage medium, is stored with computer program above, when the computer program by one or more
Multiple processors make one or more processor implement the monitoring side as described in one of claim 1-12 when executing
Method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811616164.1A CN109697153A (en) | 2018-12-28 | 2018-12-28 | Monitoring method, monitoring system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811616164.1A CN109697153A (en) | 2018-12-28 | 2018-12-28 | Monitoring method, monitoring system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109697153A true CN109697153A (en) | 2019-04-30 |
Family
ID=66232153
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811616164.1A Pending CN109697153A (en) | 2018-12-28 | 2018-12-28 | Monitoring method, monitoring system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109697153A (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110221950A (en) * | 2019-06-18 | 2019-09-10 | 杭州朗和科技有限公司 | Processing method, medium, device and the calculating equipment of monitoring data |
| CN110247810A (en) * | 2019-07-09 | 2019-09-17 | 浪潮云信息技术有限公司 | A kind of system and method for collection vessel service monitoring data |
| CN110262944A (en) * | 2019-06-21 | 2019-09-20 | 四川长虹电器股份有限公司 | The method that a kind of pair of K8s cluster container resource is monitored and is alerted |
| CN110515696A (en) * | 2019-08-09 | 2019-11-29 | 济南浪潮数据技术有限公司 | Information monitoring method and device |
| CN110942017A (en) * | 2019-11-25 | 2020-03-31 | 重庆紫光华山智安科技有限公司 | Multi-algorithm index comparison method and system based on automation |
| CN111506480A (en) * | 2020-04-23 | 2020-08-07 | 上海达梦数据库有限公司 | State detection method, device and system for components in cluster |
| CN111510351A (en) * | 2020-04-10 | 2020-08-07 | 星辰天合（北京）数据科技有限公司 | Anomaly detection method and device based on Promissuris monitoring system |
| CN111625418A (en) * | 2020-05-12 | 2020-09-04 | 深圳前海微众银行股份有限公司 | Process monitoring method and device |
| CN111666189A (en) * | 2020-06-12 | 2020-09-15 | 中信银行股份有限公司 | Method and system for declaratively visually configuring Prometheus monitoring alarm |
| CN111682976A (en) * | 2020-04-26 | 2020-09-18 | 合肥中科类脑智能技术有限公司 | Method for ensuring distributed multi-machine communication monitoring |
| CN111708611A (en) * | 2020-07-02 | 2020-09-25 | 浪潮云信息技术股份公司 | Lightweight Kubernetes monitoring system and method |
| CN111897551A (en) * | 2020-08-03 | 2020-11-06 | 上海嗨酷强供应链信息技术有限公司 | Platform and method for rapidly cloning software environment in cloud environment |
| CN112162817A (en) * | 2020-09-09 | 2021-01-01 | 新浪网技术(中国)有限公司 | Processing method and device for deploying service resources of container cluster and storage medium |
| CN112231128A (en) * | 2020-09-11 | 2021-01-15 | 中科可控信息产业有限公司 | Memory error processing method and device, computer equipment and storage medium |
| CN112559142A (en) * | 2019-09-26 | 2021-03-26 | 贵州白山云科技股份有限公司 | Container control method, device, edge calculation system, medium and equipment |
| CN112671602A (en) * | 2020-12-14 | 2021-04-16 | 北京金山云网络技术有限公司 | Data processing method, device, system, equipment and storage medium of edge node |
| WO2021212655A1 (en) * | 2020-04-24 | 2021-10-28 | 平安科技（深圳）有限公司 | Device monitoring method and apparatus, and computer device and storage medium |
| CN113704067A (en) * | 2021-09-09 | 2021-11-26 | 合肥新青罗数字技术有限公司 | Monitoring method for intangible asset management system |
| CN114281478A (en) * | 2021-12-24 | 2022-04-05 | 北京百度网讯科技有限公司 | Container operation method and device, electronic equipment and computer storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050005200A1 (en) * | 2003-03-12 | 2005-01-06 | Vladimir Matena | Method and apparatus for executing applications on a distributed computer system |
| CN103944769A (en) * | 2014-05-05 | 2014-07-23 | 江苏物联网研究发展中心 | RPC (Remote Procedure Call) protocol based cluster resource unified management system |
| CN107579931A (en) * | 2017-09-08 | 2018-01-12 | 杭州云象网络技术有限公司 | A kind of block chain based on Kubernetes is Service Source adaptation method |
| CN108829509A (en) * | 2018-05-03 | 2018-11-16 | 山东汇贸电子口岸有限公司 | Distributed container cluster framework resources management method based on domestic CPU and operating system |
-
2018
- 2018-12-28 CN CN201811616164.1A patent/CN109697153A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050005200A1 (en) * | 2003-03-12 | 2005-01-06 | Vladimir Matena | Method and apparatus for executing applications on a distributed computer system |
| CN103944769A (en) * | 2014-05-05 | 2014-07-23 | 江苏物联网研究发展中心 | RPC (Remote Procedure Call) protocol based cluster resource unified management system |
| CN107579931A (en) * | 2017-09-08 | 2018-01-12 | 杭州云象网络技术有限公司 | A kind of block chain based on Kubernetes is Service Source adaptation method |
| CN108829509A (en) * | 2018-05-03 | 2018-11-16 | 山东汇贸电子口岸有限公司 | Distributed container cluster framework resources management method based on domestic CPU and operating system |
Cited By (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110221950A (en) * | 2019-06-18 | 2019-09-10 | 杭州朗和科技有限公司 | Processing method, medium, device and the calculating equipment of monitoring data |
| CN110262944A (en) * | 2019-06-21 | 2019-09-20 | 四川长虹电器股份有限公司 | The method that a kind of pair of K8s cluster container resource is monitored and is alerted |
| CN110247810A (en) * | 2019-07-09 | 2019-09-17 | 浪潮云信息技术有限公司 | A kind of system and method for collection vessel service monitoring data |
| CN110247810B (en) * | 2019-07-09 | 2023-03-28 | 浪潮云信息技术股份公司 | System and method for collecting container service monitoring data |
| CN110515696A (en) * | 2019-08-09 | 2019-11-29 | 济南浪潮数据技术有限公司 | Information monitoring method and device |
| CN112559142A (en) * | 2019-09-26 | 2021-03-26 | 贵州白山云科技股份有限公司 | Container control method, device, edge calculation system, medium and equipment |
| CN112559142B (en) * | 2019-09-26 | 2023-12-19 | 贵州白山云科技股份有限公司 | Container control method, device, edge computing system, medium and equipment |
| CN110942017B (en) * | 2019-11-25 | 2022-12-02 | 重庆紫光华山智安科技有限公司 | Multi-algorithm index comparison method and system based on automation |
| CN110942017A (en) * | 2019-11-25 | 2020-03-31 | 重庆紫光华山智安科技有限公司 | Multi-algorithm index comparison method and system based on automation |
| CN111510351B (en) * | 2020-04-10 | 2021-09-14 | 星辰天合（北京）数据科技有限公司 | Anomaly detection method and device based on Promissuris monitoring system |
| CN111510351A (en) * | 2020-04-10 | 2020-08-07 | 星辰天合（北京）数据科技有限公司 | Anomaly detection method and device based on Promissuris monitoring system |
| CN111506480B (en) * | 2020-04-23 | 2024-03-08 | 上海达梦数据库有限公司 | Method, device and system for detecting states of components in cluster |
| CN111506480A (en) * | 2020-04-23 | 2020-08-07 | 上海达梦数据库有限公司 | State detection method, device and system for components in cluster |
| WO2021212655A1 (en) * | 2020-04-24 | 2021-10-28 | 平安科技（深圳）有限公司 | Device monitoring method and apparatus, and computer device and storage medium |
| CN111682976A (en) * | 2020-04-26 | 2020-09-18 | 合肥中科类脑智能技术有限公司 | Method for ensuring distributed multi-machine communication monitoring |
| CN111625418A (en) * | 2020-05-12 | 2020-09-04 | 深圳前海微众银行股份有限公司 | Process monitoring method and device |
| CN111666189B (en) * | 2020-06-12 | 2023-03-31 | 中信银行股份有限公司 | Method and system for declaratively visually configuring Prometheus monitoring alarm |
| CN111666189A (en) * | 2020-06-12 | 2020-09-15 | 中信银行股份有限公司 | Method and system for declaratively visually configuring Prometheus monitoring alarm |
| CN111708611A (en) * | 2020-07-02 | 2020-09-25 | 浪潮云信息技术股份公司 | Lightweight Kubernetes monitoring system and method |
| CN111897551A (en) * | 2020-08-03 | 2020-11-06 | 上海嗨酷强供应链信息技术有限公司 | Platform and method for rapidly cloning software environment in cloud environment |
| CN112162817B (en) * | 2020-09-09 | 2023-09-26 | 新浪技术（中国）有限公司 | Processing method and device for deployment service resources of container cluster and storage medium |
| CN112162817A (en) * | 2020-09-09 | 2021-01-01 | 新浪网技术(中国)有限公司 | Processing method and device for deploying service resources of container cluster and storage medium |
| CN112231128A (en) * | 2020-09-11 | 2021-01-15 | 中科可控信息产业有限公司 | Memory error processing method and device, computer equipment and storage medium |
| CN112671602A (en) * | 2020-12-14 | 2021-04-16 | 北京金山云网络技术有限公司 | Data processing method, device, system, equipment and storage medium of edge node |
| CN112671602B (en) * | 2020-12-14 | 2023-07-04 | 北京金山云网络技术有限公司 | Data processing method, device, system, equipment and storage medium of edge node |
| CN113704067A (en) * | 2021-09-09 | 2021-11-26 | 合肥新青罗数字技术有限公司 | Monitoring method for intangible asset management system |
| CN113704067B (en) * | 2021-09-09 | 2023-10-24 | 合肥新青罗数字技术有限公司 | Intangible asset management system monitoring method |
| CN114281478A (en) * | 2021-12-24 | 2022-04-05 | 北京百度网讯科技有限公司 | Container operation method and device, electronic equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109697153A (en) | Monitoring method, monitoring system and computer readable storage medium | |
| CN106067080B (en) | Configurable workflow capabilities are provided | |
| WO2021017301A1 (en) | Management method and apparatus based on kubernetes cluster, and computer-readable storage medium | |
| CN109074377B (en) | Managed function execution for real-time processing of data streams | |
| CN103380423B (en) | For the system and method for private cloud computing | |
| CN109684054A (en) | Information processing method and device, electronic equipment and memory | |
| CN106533804A (en) | Network operation support system | |
| EP3032442B1 (en) | Modeling and simulation of infrastructure architecture for big data | |
| US20180210808A1 (en) | System and methods for application activity capture, error identification, and error correction | |
| CN102880709B (en) | Data warehouse management system and data warehouse management method | |
| CN102722381B (en) | The technology of optimization and upgrading task | |
| EP3881183A1 (en) | Mesh-based event broker for distributed computing | |
| CN105653425A (en) | Complicated event processing engine based monitoring system | |
| CN105229591A (en) | Overall situation polymerization NameSpace is created in order to storage administration | |
| CN108431796A (en) | Distributed resource management system and method | |
| US20190205342A1 (en) | Identifying and structuring related data | |
| CN109634718A (en) | The method and system of cloud platform creation mirror image | |
| CN110365762A (en) | Service processing method, device, equipment and storage medium | |
| US9210098B2 (en) | Enhanced command selection in a networked computing environment | |
| CN109992956A (en) | The processing method and relevant apparatus of the security strategy of container | |
| CN101495978B (en) | Reduction of message flow between bus-connected consumers and producers | |
| Al-Ali et al. | Role of IoT technologies in big data management systems: A review and Smart Grid case study | |
| CN116136801B (en) | Cloud platform data processing method and device, electronic equipment and storage medium | |
| CN116383223A (en) | Asset data processing method, related device and storage medium | |
| US11392424B2 (en) | Method and device for aiding decision-making for the allocation of computing means on a high performance computing infrastructure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190430 |
|
| RJ01 | Rejection of invention patent application after publication |