CN110278218B - Container-based data receiving and analyzing method - Google Patents
Container-based data receiving and analyzing method Download PDFInfo
- Publication number
- CN110278218B CN110278218B CN201810207667.7A CN201810207667A CN110278218B CN 110278218 B CN110278218 B CN 110278218B CN 201810207667 A CN201810207667 A CN 201810207667A CN 110278218 B CN110278218 B CN 110278218B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- container
- containers
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0273—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/29—Flow control; Congestion control using a combination of thresholds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
Abstract
The invention discloses a data receiving and analyzing method based on a container, which comprises the following steps: providing a standard container mirror image, wherein basic components required by the operation of data receiving and analyzing services are packaged in the standard container mirror image; the automatic deployment platform automatically makes the user code into a container mirror image based on Webhook and pushes the container mirror image to the central library; automatically deploying a container mirror image through an automated deployment platform; monitoring the flow and the connection number of each data receiving and analyzing service, generating average data per minute of the flow and the connection number, and putting the average data into a message queue; data in the message queue is received by the monitoring service and the number of containers is adjusted according to the size of the received data. The invention can realize automatic deployment of services and automatic expansion and contraction of containers based on monitoring data, thereby realizing automatic management of data receiving and analysis.
Description
Technical Field
The invention relates to the technical field of data receiving and analyzing, in particular to a data receiving and analyzing method based on a container.
Background
At present, the receiving and analysis of data of various networking devices are not based on containerization, and operations such as service deployment, capacity expansion, capacity reduction and the like are generally performed in a manual mode. The existing manual operation mode easily causes deployment errors, capacity cannot be expanded in time after data traffic is increased, and capacity cannot be contracted in time after the data traffic is reduced, so that data receiving and analyzing processing are not facilitated.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a data receiving and analyzing method based on a container, which realizes automatic deployment of services, automatic expansion and contraction of the container based on monitoring data and further realizes automatic management of data receiving and analysis.
The data receiving and analyzing method based on the container comprises the following steps:
providing a standard container mirror image, wherein basic components required by the operation of data receiving and analyzing services are packaged in the standard container mirror image;
the automatic deployment platform automatically makes user codes into container mirror images based on Webhook through the standardized container mirror images, and pushes the container mirror images to a central library;
automatically deploying the container image through the automated deployment platform;
monitoring the flow and the connection number of each data receiving and analyzing service based on the container mirror image, generating average data per minute of the flow and the connection number, and putting the average data into a message queue;
receiving data in the message queue through a monitoring service and adjusting the number of containers according to the size of the data received from the message queue.
Further, the receiving, by the monitoring service, data in the message queue and adjusting the number of containers according to the size of the data received from the message queue includes:
receiving data in the message queue through a monitoring service;
when the received data is smaller than a preset maximum threshold and larger than a preset minimum threshold, updating the cached latest data in the database;
and when the received data is larger than the maximum threshold or smaller than the minimum threshold, inquiring the latest data in the database, and adjusting the number of containers according to the size of the latest data in the database.
Further, the adjusting the number of containers according to the size of the latest data in the database includes:
updating the most recent data in the database when the most recent data in the database is less than the maximum threshold and greater than the minimum threshold;
when the latest data in the database is larger than the maximum threshold or smaller than the minimum threshold, checking the difference value of the data time stamps in the database;
when the difference value is within two minutes, directly discarding the received data;
when the difference value exceeds two minutes and the latest data in the database is greater than the maximum threshold value, updating the latest data in the database and triggering capacity expansion service to increase the number of containers;
and when the difference value exceeds two minutes and the latest data in the database is less than the minimum threshold value, updating the latest data in the database and triggering the capacity reduction service to reduce the number of containers.
Preferably, the number of containers increased by the capacity expansion service is equal to a ratio of the current latest data in the database to the maximum threshold.
Preferably, the triggered condensed content service reduces the number of containers by only one container at a time.
The method comprises the steps that a user code is automatically made into a container mirror image through an automatic deployment platform based on Webhook and pushed to a central library; automatically deploying container mirror images through an automatic deployment platform; monitoring the flow and the connection number of each data receiving and analyzing service, generating average data per minute and putting the average data into a message queue; data in the message queue is received by the monitoring service and the number of containers is adjusted according to the size of the received data. Therefore, automatic deployment of services can be realized, automatic expansion and contraction of containers can be realized based on the monitoring data, and further, automatic management of data receiving and analysis can be realized.
Drawings
FIG. 1 is a flow chart of a container-based data receiving and parsing method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating automated manufacturing and deployment of a container image by an automated deployment platform.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating a container-based data receiving and parsing method according to the embodiment, which specifically includes the following steps:
s101, providing a standard container mirror image.
S102, the automatic deployment platform automatically makes the user code into a container mirror image based on Webhook through standardized container mirror images, and pushes the container mirror image into a central library.
And S103, automatically deploying the container mirror image through the automatic deployment platform.
It should be noted that the standard container image packages basic components required for the operation of the data receiving and parsing service. The automatic deployment platform is a gitlab platform. The steps are the process of automatically making and deploying the container mirror image based on Webhook according to the code submitted by the user based on the gitlab platform.
The process of automatically making and deploying the container mirror image is shown in fig. 2, firstly codes submitted to other branches by a user are merged to a master branch through a gitlab platform, then a tag corresponding to the container mirror image is generated based on the master branch, then the container mirror image is automatically made based on Webhook, and finally the container mirror image is automatically deployed through an automatic deployment platform and subjected to automatic testing.
S104, monitoring the flow and the connection number of each data receiving and analyzing service based on the container mirror image, generating the average data per minute of the flow and the connection number, and putting the average data into a message queue.
And S105, receiving the data in the message queue through the monitoring service and adjusting the number of the containers according to the size of the data received from the message queue.
It should be noted that the receiving data in the message queue by the monitoring service and adjusting the number of containers according to the size of the data received from the message queue includes:
receiving data in a message queue through a monitoring service; when the received data is smaller than a preset maximum threshold and larger than a preset minimum threshold, updating the cached latest data in the Redis database;
when the received data is larger than a maximum threshold or smaller than a minimum threshold, querying the latest data in a Redis database; when the latest data in the Redis database is smaller than the maximum threshold and larger than the minimum threshold, updating the latest data in the Redis database; when the latest data in the Redis database is larger than a maximum threshold or smaller than a minimum threshold, checking the difference value of the data time stamps in the Redis database;
when the difference value of the data timestamps in the Redis database is within two minutes, directly discarding the received data; when the difference value exceeds two minutes and the latest data in the Redis database is greater than the maximum threshold value, updating the latest data in the Redis database and triggering capacity expansion service to increase the number of containers;
and when the difference value exceeds two minutes and the latest data in the database is less than the minimum threshold value, updating the latest data in the database and triggering the capacity reduction service to reduce the number of containers.
And when the capacity expansion service is triggered to increase the number of the containers, the increased number of the containers is equal to the ratio of the current latest data in the Redis database to the maximum threshold. When the reduction service is triggered to reduce the number of containers, only one container instance is reduced at a time to ensure smooth reduction of the service.
The method in the embodiment automatically makes the user code into a container mirror image through an automatic deployment platform based on Webhook and pushes the container mirror image to a central library; automatically deploying container mirror images through an automatic deployment platform; monitoring the flow and the connection number of each data receiving and analyzing service, generating average data per minute and putting the average data into a message queue; data in the message queue is received by the monitoring service and the number of containers is adjusted according to the size of the received data. Therefore, automatic deployment of services can be realized, automatic expansion and contraction of containers can be realized based on the monitoring data, and further, automatic management of data receiving and analysis can be realized.
Furthermore, it will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method or computer program product. Accordingly, embodiments of the present invention may take the form of an embodiment incorporating software aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should also be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (4)
1. A container-based data receiving and parsing method, the method comprising:
providing a standard container mirror image, wherein basic components required by the operation of data receiving and analyzing services are packaged in the standard container mirror image;
the automatic deployment platform automatically makes user codes into container mirror images based on Webhook through the standardized container mirror images, and pushes the container mirror images to a central library;
automatically deploying the container image through the automated deployment platform;
monitoring the flow and the connection number of each data receiving and analyzing service based on the container mirror image, generating average data per minute of the flow and the connection number, and putting the average data into a message queue;
receiving data in the message queue through a monitoring service and adjusting the number of containers according to the size of the data received from the message queue;
the receiving, by the monitoring service, data in the message queue and adjusting the number of containers according to the size of the data received from the message queue includes:
receiving data in the message queue through a monitoring service;
when the received data is smaller than a preset maximum threshold and larger than a preset minimum threshold, updating the cached latest data in the database;
and when the received data is larger than the maximum threshold or smaller than the minimum threshold, inquiring the latest data in the database, and adjusting the number of containers according to the size of the latest data in the database.
2. The method for container-based data reception and parsing of claim 1, wherein the adjusting the number of containers according to the size of the latest data in the database comprises:
updating the most recent data in the database when the most recent data in the database is less than the maximum threshold and greater than the minimum threshold;
when the latest data in the database is larger than the maximum threshold or smaller than the minimum threshold, checking the difference value of the data time stamps in the database;
when the difference value is within two minutes, directly discarding the received data;
when the difference value exceeds two minutes and the latest data in the database is greater than the maximum threshold value, updating the latest data in the database and triggering capacity expansion service to increase the number of containers;
and when the difference value exceeds two minutes and the latest data in the database is less than the minimum threshold value, updating the latest data in the database and triggering the capacity reduction service to reduce the number of containers.
3. The container-based data receiving and parsing method of claim 2 wherein the number of containers added when the trigger volume expansion service increases the number of containers is equal to a ratio of current latest data in the database to the maximum threshold.
4. The container-based data reception and parsing method of claim 2 wherein the trigger reduction service reduces the number of containers by only one container at a time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810207667.7A CN110278218B (en) | 2018-03-14 | 2018-03-14 | Container-based data receiving and analyzing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810207667.7A CN110278218B (en) | 2018-03-14 | 2018-03-14 | Container-based data receiving and analyzing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110278218A CN110278218A (en) | 2019-09-24 |
| CN110278218B true CN110278218B (en) | 2022-02-15 |
Family
ID=67958330
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810207667.7A Active CN110278218B (en) | 2018-03-14 | 2018-03-14 | Container-based data receiving and analyzing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110278218B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103023969A (en) * | 2012-11-15 | 2013-04-03 | 北京搜狐新媒体信息技术有限公司 | Cloud platform scheduling method and system |
| CN105119952A (en) * | 2015-07-07 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Method and system for automatically and flexibly assigning resource under cloud platform |
| CN107220100A (en) * | 2016-03-22 | 2017-09-29 | 中国移动（深圳）有限公司 | One kind exploitation O＆M method, device and cloud computing PaaS platform |
| CN107688322A (en) * | 2017-08-31 | 2018-02-13 | 天津中新智冠信息技术有限公司 | A kind of containerization management system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9684571B2 (en) * | 2013-05-01 | 2017-06-20 | Netapp, Inc. | Namespace mirroring in an expandable storage volume |
| US10079719B2 (en) * | 2016-06-29 | 2018-09-18 | International Business Machines Corporation | Automatically tuning middleware in a mobilefirst platform running in a docker container infrastructure |
| CN106330576B (en) * | 2016-11-18 | 2019-10-25 | 北京红马传媒文化发展有限公司 | The method of containerization micro services automatic telescopic and migration scheduling, system and equipment |
| CN107329800B (en) * | 2017-06-29 | 2019-11-05 | 北京北信源软件股份有限公司 | A kind of middleware application management-control method and device based on Docker platform |
| CN107391279B (en) * | 2017-07-31 | 2020-08-25 | 浪潮云信息技术股份公司 | Message queue container creation method and device and message queue container |
-
2018
- 2018-03-14 CN CN201810207667.7A patent/CN110278218B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103023969A (en) * | 2012-11-15 | 2013-04-03 | 北京搜狐新媒体信息技术有限公司 | Cloud platform scheduling method and system |
| CN105119952A (en) * | 2015-07-07 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Method and system for automatically and flexibly assigning resource under cloud platform |
| CN107220100A (en) * | 2016-03-22 | 2017-09-29 | 中国移动（深圳）有限公司 | One kind exploitation O＆M method, device and cloud computing PaaS platform |
| CN107688322A (en) * | 2017-08-31 | 2018-02-13 | 天津中新智冠信息技术有限公司 | A kind of containerization management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110278218A (en) | 2019-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7401065B2 (en) | System for downloading contents, and client terminal for downloading contents from contents server | |
| CN110865842B (en) | OTA upgrading method and equipment | |
| US20070192158A1 (en) | Performing scheduled device management | |
| CN110995836A (en) | Log management method, device, storage medium and device based on Internet of vehicles platform | |
| CN101909282B (en) | Triggering method, device and system of terminal operation | |
| CN108415708B (en) | Application program installation package size processing method, device, equipment and storage medium | |
| US7908524B2 (en) | Storage medium readable by a machine tangible embodying event notification management program and event notification management apparatus | |
| US20210216301A1 (en) | Algorithm update method and apparatus | |
| CN110278218B (en) | Container-based data receiving and analyzing method | |
| CN111309696A (en) | Log processing method and device, electronic equipment and readable medium | |
| CN110769057A (en) | Automatic upgrading method, device and system based on cascade system | |
| CN107153508B (en) | Data storage method and device | |
| CN113205130B (en) | Data auditing method and device, electronic equipment and storage medium | |
| US20100319048A1 (en) | Publication of television content to television distribution sites | |
| CN107948937B (en) | Passenger flow volume statistical method and device | |
| CN113900723B (en) | Common platform updating method, network device and computer readable storage medium | |
| US8374593B2 (en) | Method of providing mobile application | |
| CN112042136B (en) | Broadcast receiver for infotainment system and method of operating the same | |
| CN107357864B (en) | File processing method and terminal | |
| CN112148479B (en) | Data processing method, device and computer equipment based on Siddi log audit | |
| CN111061888B (en) | Image acquisition method and system | |
| CN112532660A (en) | Data synchronization method, device and network management system | |
| CN116226138B (en) | Block chain-based information vulnerability processing method and device | |
| CN114090112B (en) | Method and device for loading configuration file, storage medium and electronic device | |
| CN113891262A (en) | Information sending method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |