CN110278218A - A kind of data receiver and analytic method based on container - Google Patents
A kind of data receiver and analytic method based on container Download PDFInfo
- Publication number
- CN110278218A CN110278218A CN201810207667.7A CN201810207667A CN110278218A CN 110278218 A CN110278218 A CN 110278218A CN 201810207667 A CN201810207667 A CN 201810207667A CN 110278218 A CN110278218 A CN 110278218A
- Authority
- CN
- China
- Prior art keywords
- data
- container
- mirror image
- database
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0273—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/29—Flow control; Congestion control using a combination of thresholds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a kind of data receiver and analytic method based on container comprising: volumetric standard mirror image is provided, basic module needed for data receiver and analysis service operation have been packed in the volumetric standard mirror image；Automatically dispose platform is based on Webhook and personal code work is made as to container mirror image automatically, and is pushed in central library；Pass through automatically dispose platform automatic deployment container mirror image；The flow and connection number of each data receiver and analysis service are monitored, the average data per minute of flow and connection number is generated and the average data is put into message queue；The data in message queue are received by monitoring service and according to the size adjustment container quantity of the data received.The present invention may be implemented the automatic deployment of business and be realized based on monitoring data to the automatic dilatation of container and capacity reducing, and then realize the automatic management to data receiver and parsing.
Description
Technical field
The present invention relates to data receiver and analytic technique field more particularly to a kind of data receivers and parsing based on container
Method.
Background technique
Reception for the data of various networked devices at present and parsing are based on containerization not yet, typically use hand
Dynamic mode carries out the operation such as service deployment, dilatation and capacity reducing.And existing this manual manipulation mode easily causes deployment
Mistake, and after data traffic increase cannot timely dilatation, equally data traffic reduction after can not timely capacity reducing, thus
It is unfavorable for the reception and dissection process of data.
Summary of the invention
Place, the present invention provide a kind of data receiver and solution based on container in view of the above-mentioned deficiencies in the prior art
Analysis method is realized the automatic deployment of business and is realized based on monitoring data to the automatic dilatation of container and capacity reducing, and then realization pair
The automatic management of data receiver and parsing.
The data receiver and analytic method based on container, comprising:
Volumetric standard mirror image is provided, has been packed in the volumetric standard mirror image needed for data receiver and analysis service operation
Basic module；
Personal code work is made as by the standardised container mirror image based on Webhook by automatically dispose platform automatically
Container mirror image, and the container mirror image is pushed in central library；
Pass through container mirror image described in the automatically dispose platform automatic deployment；
Flow and connection number based on each data receiver of the container mirror-image monitoring and analysis service, generate the flow
And connection number average data per minute and the average data is put into message queue；
The data in the message queue are received by monitoring service and according to the number received from the message queue
According to size adjustment container quantity.
Further, described data in the message queue to be received by monitoring service and according to from the message queue
In the size adjustment container quantity of data that receives, comprising:
The data in the message queue are received by monitoring service；
When the data received are less than preset max-thresholds and are greater than preset minimum threshold, institute in more new database
The latest data of caching；
When the data received are greater than the max-thresholds or are less than the minimum threshold, inquire in the database
Latest data, and according to the size adjustment container quantity of latest data in the database.
Further, the size adjustment container quantity according to latest data in the database, comprising:
When latest data is less than the max-thresholds and is greater than the minimum threshold in the database, the number is updated
According to the latest data in library；
When latest data is greater than the max-thresholds or is less than the minimum threshold in the database, then described in inspection
The difference of data time stamp in database；
When the difference is in two minutes, then the data received are directly abandoned；
When the difference is more than that latest data is greater than the max-thresholds in two minutes and the database, then institute is updated
It states the latest data in database and triggers dilatation service and increase number of containers；
When the difference is more than that latest data is less than the minimum threshold in two minutes and the database, then institute is updated
It states the latest data in database and triggers capacity reducing service and reduce number of containers.
Preferably, the triggering dilatation service increases the increased number of containers of number of containers when institute and is equal in the database
The ratio of current latest data and the max-thresholds.
Preferably, once only reduce by a container when triggering capacity reducing service reduces number of containers.
The present invention is based on Webhook by automatically dispose platform and personal code work is made as to container mirror image automatically, and pushes away
It is sent in central library；And pass through automatically dispose platform automatic deployment container mirror image；Monitor each data receiver and analysis service
Flow and connection number, generate average data per minute and average data be put into message queue；It is connect by monitoring service
Receive the data in message queue and according to the size adjustment container quantity of the data received.So as to realize the automatic of business
It disposes and is realized based on monitoring data to the automatic dilatation of container and capacity reducing, and then realize the automation to data receiver and parsing
Management.
Detailed description of the invention
Fig. 1 is the flow chart of the data receiver and analytic method provided in an embodiment of the present invention based on container；
Fig. 2 makes automatically for automatically dispose platform and the flow diagram of deployment container mirror image.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiments of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people
Member's every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
Referring to Fig. 1, Fig. 1 shows the flow chart of the data receiver and analytic method based on container in the present embodiment,
This method specifically includes the following steps:
S101 provides volumetric standard mirror image.
Personal code work is made as by standardised container mirror image based on Webhook by S102, automatically dispose platform automatically
Container mirror image, and container mirror image is pushed in central library.
S103 passes through automatically dispose platform automatic deployment container mirror image.
It should be noted that having been packed in above-mentioned standard container mirror image basic needed for data receiver and analysis service operation
Component.Above-mentioned automatically dispose platform is gitlab platform.Above-mentioned steps are to be submitted based on gitlab platform according to user
Code is made automatically based on Webhook and the process of deployment container mirror image.
Wherein, the process of automatic production and deployment container mirror image is as shown in Fig. 2, first mentioned user by gitlab platform
The code for being sent to other branches is merged into master branch, is then based on master branch and generates tag corresponding to container mirror image,
It is then based on Webhook and makes container mirror image automatically, finally by automatically dispose platform automatic deployment container mirror image and carry out
Automatic test.
S104, flow and connection number based on each data receiver of container mirror-image monitoring and analysis service, generate flow and
Average data is simultaneously put into message queue by the average data per minute of connection number.
S105 receives the data in message queue by monitoring service and according to the data received from message queue
Size adjustment container quantity.
It should be noted that receiving the data in message queue above by monitoring service and being connect according to from message queue
The size adjustment container quantity of the data received, comprising:
The data in message queue are received by monitoring service；When the data received are less than preset max-thresholds and big
When preset minimum threshold, the latest data cached in Redis database is updated；
When the data received are greater than max-thresholds or are less than minimum threshold, the newest number in Redis database is inquired
According to；When latest data is less than max-thresholds and is greater than minimum threshold in Redis database, update in Redis database most
New data；When latest data is greater than max-thresholds or is less than minimum threshold in Redis database, then Redis database is checked
The difference of middle data time stamp；
When the difference of data time stamp in Redis database is in two minutes, then the data received are directly abandoned；When
When above-mentioned difference is more than that latest data is greater than max-thresholds in two minutes and Redis database, then update in Redis database
Latest data simultaneously triggers dilatation service increase number of containers；
When above-mentioned difference is more than that latest data is less than minimum threshold in two minutes and database, then more in new database
Latest data simultaneously triggers capacity reducing service reduction number of containers.
Wherein, when triggering dilatation service and increasing number of containers the increased number of containers of institute be equal in Redis database when
The ratio of preceding latest data and above-mentioned max-thresholds.Once only reduce by a container when triggering capacity reducing service and reducing number of containers
Example is to guarantee the steady capacity reducing serviced.
Method in the present embodiment is based on Webhook by automatically dispose platform and personal code work is made as container automatically
Mirror image, and be pushed in central library；And pass through automatically dispose platform automatic deployment container mirror image；Monitor each data receiver and
The flow and connection number of analysis service generate average data per minute and average data are put into message queue；Pass through prison
Control service receives the data in message queue and according to the size adjustment container quantity of the data received.So as to realize industry
The automatic deployment of business is simultaneously realized based on monitoring data to the automatic dilatation of container and capacity reducing, and then is realized to data receiver and parsing
Automatic management.
In addition, it should be understood by those skilled in the art that, the embodiment of the present invention can provide as method or computer program
Product.Therefore, the form of the embodiment in conjunction with software aspects can be used in the embodiment of the present invention.Moreover, the embodiment of the present invention can adopt
It wherein include computer-usable storage medium (the including but not limited to magnetic of computer usable program code used in one or more
Disk storage, CD-ROM, optical memory etc.) on the form of computer program product implemented.
It should also be noted that, herein, the terms "include", "comprise" or its any other variant are intended to non-
It is exclusive to include, so that process, method, article or terminal device including a series of elements are not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or terminal
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in process, method, article or the terminal device for including the element.
These are only the preferred embodiment of the present invention, is not intended to restrict the invention, for those skilled in the art
For member, the invention may be variously modified and varied.All within the spirits and principles of the present invention, it is made it is any modification,
Equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (5)
1. a kind of data receiver and analytic method based on container, which is characterized in that the described method includes:
Volumetric standard mirror image is provided, has been packed in the volumetric standard mirror image basic needed for data receiver and analysis service operation
Component；
Personal code work is made as container based on Webhook by the standardised container mirror image by automatically dispose platform automatically
Mirror image, and the container mirror image is pushed in central library；
Pass through container mirror image described in the automatically dispose platform automatic deployment；
Flow and connection number based on each data receiver of the container mirror-image monitoring and analysis service, generate the flow and company
It connects several average datas per minute and the average data is put into message queue；
The data in the message queue are received by monitoring service and according to the data received from the message queue
Size adjustment container quantity.
2. data receiver and analytic method based on container as described in claim 1, which is characterized in that described to be taken by monitoring
Business receives the data in the message queue and according to the size adjustment container number of the data received from the message queue
Amount, comprising:
The data in the message queue are received by monitoring service；
When the data received are less than preset max-thresholds and are greater than preset minimum threshold, cached in more new database
Latest data；
When the data received are greater than the max-thresholds or are less than the minimum threshold, inquire newest in the database
Data, and according to the size adjustment container quantity of latest data in the database.
3. data receiver and analytic method based on container as claimed in claim 2, which is characterized in that described according to the number
According to the size adjustment container quantity of latest data in library, comprising:
When latest data is less than the max-thresholds and is greater than the minimum threshold in the database, the database is updated
In latest data；
When latest data is greater than the max-thresholds or is less than the minimum threshold in the database, then the data are checked
The difference of data time stamp in library；
When the difference is in two minutes, then the data received are directly abandoned；
When the difference is more than that latest data is greater than the max-thresholds in two minutes and the database, then the number is updated
According to the latest data in library and trigger dilatation service increase number of containers；
When the difference is more than that latest data is less than the minimum threshold in two minutes and the database, then the number is updated
According to the latest data in library and trigger capacity reducing service reduction number of containers.
4. data receiver and analytic method based on container as claimed in claim 3, which is characterized in that the triggering dilatation clothes
The increased number of containers of institute is equal in the database currently latest data and the max-thresholds when business increases number of containers
Ratio.
5. data receiver and analytic method based on container as claimed in claim 3, which is characterized in that the triggering capacity reducing clothes
Business once only reduces by a container when reducing number of containers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810207667.7A CN110278218B (en) | 2018-03-14 | 2018-03-14 | Container-based data receiving and analyzing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810207667.7A CN110278218B (en) | 2018-03-14 | 2018-03-14 | Container-based data receiving and analyzing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110278218A true CN110278218A (en) | 2019-09-24 |
| CN110278218B CN110278218B (en) | 2022-02-15 |
Family
ID=67958330
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810207667.7A Active CN110278218B (en) | 2018-03-14 | 2018-03-14 | Container-based data receiving and analyzing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110278218B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103023969A (en) * | 2012-11-15 | 2013-04-03 | 北京搜狐新媒体信息技术有限公司 | Cloud platform scheduling method and system |
| US20140330787A1 (en) * | 2013-05-01 | 2014-11-06 | Netapp, Inc. | Namespace mirroring in an expandable storage volume |
| CN105119952A (en) * | 2015-07-07 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Method and system for automatically and flexibly assigning resource under cloud platform |
| CN106330576A (en) * | 2016-11-18 | 2017-01-11 | 北京红马传媒文化发展有限公司 | Automatic scaling and migration scheduling method, system and device for containerization micro-service |
| CN107220100A (en) * | 2016-03-22 | 2017-09-29 | 中国移动（深圳）有限公司 | One kind exploitation O＆M method, device and cloud computing PaaS platform |
| CN107329800A (en) * | 2017-06-29 | 2017-11-07 | 北京北信源软件股份有限公司 | A kind of middleware application management-control method and device based on Docker platforms |
| CN107391279A (en) * | 2017-07-31 | 2017-11-24 | 山东浪潮云服务信息科技有限公司 | A kind of message queue container creation method, device and message queue container |
| US20180006886A1 (en) * | 2016-06-29 | 2018-01-04 | International Business Machines Corporation | Automatically tuning middleware in a mobilefirst platform running in a docker container infrastructure |
| CN107688322A (en) * | 2017-08-31 | 2018-02-13 | 天津中新智冠信息技术有限公司 | A kind of containerization management system |
-
2018
- 2018-03-14 CN CN201810207667.7A patent/CN110278218B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103023969A (en) * | 2012-11-15 | 2013-04-03 | 北京搜狐新媒体信息技术有限公司 | Cloud platform scheduling method and system |
| US20140330787A1 (en) * | 2013-05-01 | 2014-11-06 | Netapp, Inc. | Namespace mirroring in an expandable storage volume |
| CN105119952A (en) * | 2015-07-07 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Method and system for automatically and flexibly assigning resource under cloud platform |
| CN107220100A (en) * | 2016-03-22 | 2017-09-29 | 中国移动（深圳）有限公司 | One kind exploitation O＆M method, device and cloud computing PaaS platform |
| US20180006886A1 (en) * | 2016-06-29 | 2018-01-04 | International Business Machines Corporation | Automatically tuning middleware in a mobilefirst platform running in a docker container infrastructure |
| CN106330576A (en) * | 2016-11-18 | 2017-01-11 | 北京红马传媒文化发展有限公司 | Automatic scaling and migration scheduling method, system and device for containerization micro-service |
| CN107329800A (en) * | 2017-06-29 | 2017-11-07 | 北京北信源软件股份有限公司 | A kind of middleware application management-control method and device based on Docker platforms |
| CN107391279A (en) * | 2017-07-31 | 2017-11-24 | 山东浪潮云服务信息科技有限公司 | A kind of message queue container creation method, device and message queue container |
| CN107688322A (en) * | 2017-08-31 | 2018-02-13 | 天津中新智冠信息技术有限公司 | A kind of containerization management system |
Non-Patent Citations (2)
| Title |
|---|
| ISO/IEC/IEEE 24765:2017(E): "《ISO/IEC/IEEE International Standard - Systems and software engineering -- Vocabulary》", 《IEEE》 * |
| 何松林等: "基于Docker的跨主机容器集群宿主机调度策略研究", 《工业控制计算机》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110278218B (en) | 2022-02-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP2008527566A (en) | Ground-based software tool to control redundant management exchange operations | |
| CN105338078A (en) | Data storage method and device used for storing system | |
| CN108337127B (en) | Application performance monitoring method, system, terminal and computer readable storage medium | |
| KR102220712B1 (en) | Dynamic condition management apparatus and method to respond to real time in a factory | |
| CN104320325A (en) | Information push method and device | |
| CN106790454A (en) | Deferred information method for pushing and device | |
| CN112306369A (en) | Data processing method, device, server and storage medium | |
| CN104468222A (en) | Method, device and system for reporting log information | |
| CN110222073A (en) | A kind of method and relevant apparatus of data query | |
| CN113806651A (en) | Data caching method, device, server and storage medium | |
| CN110278218A (en) | A kind of data receiver and analytic method based on container | |
| US11252121B2 (en) | Message sending method and terminal device | |
| CN101783797A (en) | Alarm pushing method and server thereof | |
| CN114301760A (en) | Test method, test device, electronic equipment and storage medium | |
| CN104239538B (en) | A kind of snapshot-type log compression method, system and device | |
| CN113312321A (en) | Abnormal monitoring method for traffic and related equipment | |
| GB2397404A (en) | Automatic selection of encoding parameters for the transmission of media objects | |
| CN105187490B (en) | A kind of transfer processing method of internet of things data | |
| CN115460124B (en) | Method, device, equipment and storage medium for optimizing transmission link across machine room | |
| US20110276839A1 (en) | Remote communication system and method | |
| CN103078905A (en) | Communication management method of GPS (Global Position System) terminal | |
| CN103778218A (en) | Cloud computation-based standard information consistency early warning system and method | |
| CN115061947A (en) | Resource management method, device, equipment and storage medium | |
| CN111683261B (en) | Live video stream scheduling method and system based on big data | |
| CN112181671B (en) | Method and device for processing delay message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |