CN111435566B - Traffic data acquisition method, device and system - Google Patents
Traffic data acquisition method, device and system Download PDFInfo
- Publication number
- CN111435566B CN111435566B CN201910027606.7A CN201910027606A CN111435566B CN 111435566 B CN111435566 B CN 111435566B CN 201910027606 A CN201910027606 A CN 201910027606A CN 111435566 B CN111435566 B CN 111435566B
- Authority
- CN
- China
- Prior art keywords
- traffic data
- data
- receiving end
- traffic
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
- G08G1/0125—Traffic data processing
Landscapes
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Telephonic Communication Services (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a traffic data acquisition method, a device and a system, relates to the technical field of computers, and mainly aims to realize that one server is in butt joint with a plurality of data manufacturers, a plurality of servers are mutually backed up, and the acquisition of GPS data is dynamically adjusted. The main technical scheme of the invention is as follows: receiving connection information of at least one traffic data source distributed by a server; collecting traffic data according to the connection information; detecting the acquisition amount of the traffic data according to a preset frequency, and judging whether the acquisition amount exceeds a maximum threshold value; if the maximum collection quantity is exceeded, disconnecting at least one traffic data source according to a preset rule so as to enable the collection quantity to be lower than the maximum threshold value. The invention is used for collecting traffic data.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a traffic data acquisition method, a device and a system.
Background
With the development of economy and the advancement of technology, motor vehicles become one of the most important vehicles for public trip. However, the road facilities in the city cannot meet the high-speed growth of the motor vehicles due to limited space resources, so that the traffic jam phenomenon becomes more and more serious. Therefore, the demand for real-time traffic information is more and more urgent. According to the real-time road conditions published by some map manufacturers, people can sense the congestion conditions of road sections in real time, and adopt a congestion avoidance strategy, so that unnecessary road loss is reduced, and great convenience is brought to the daily life of people. The release of real-time traffic road conditions mainly includes performing road matching and road condition calculation according to real-time satellite Positioning (GPS) points acquired by vehicles, and finally converting the real-time traffic road conditions into corresponding real-time road conditions.
For the acquisition of real-time GPS points, the method mainly depends on mobile phone users of taxies, private cars and map manufacturers at present. Different map manufacturers acquire and transmit GPS points, so that GPS data access parties need to store the GPS data by adopting independent servers according to different communication standards, and in order to ensure the draft availability of the servers, the common method is to serve cold backup, namely for one set of protocol standards, a master backup is realized through a ZooKeeper, the backup monitors the running state of a host, and when the host is abnormal, the host is taken over for working. However, with the increase of GPS data manufacturers, respective protocol standards are also different, and when a service cold standby mode is used for each manufacturer, the number of server requirements is multiplied, and for some GPS data manufacturers with smaller returned data volume, a set of data access equipment is separately deployed for the GPS data manufacturers, which also causes waste of service resources.
Disclosure of Invention
In view of the above problems, the present invention provides a method, an apparatus and a system for acquiring traffic data, and mainly aims to realize that one server is connected to a plurality of data manufacturers, and a plurality of servers are backed up with each other, so as to dynamically adjust the acquisition of GPS data.
In order to achieve the purpose, the invention mainly provides the following technical scheme:
in a first aspect, the present invention provides a traffic data collecting method, where the method is used at a data receiving end, and specifically includes:
receiving connection information of at least one traffic data source distributed by a server;
collecting traffic data according to the connection information;
detecting the acquisition amount of the traffic data according to a preset frequency, and judging whether the acquisition amount exceeds a maximum threshold value;
if the acquisition quantity exceeds the maximum threshold value, disconnecting at least one traffic data source according to a preset rule so as to enable the acquisition quantity to be lower than the maximum threshold value.
In a second aspect, the present invention provides a traffic data collecting method, where the method is used at a server, and specifically includes:
distributing connection information of at least one traffic data source for each data receiving terminal according to the registered data receiving terminal information;
monitoring whether a traffic data source connected with each data receiving end is disconnected;
if the traffic data source is disconnected, determining a new data receiving end for the disconnected traffic data source according to a preset strategy;
and sending the connection information to the new data receiving end.
In a third aspect, the present invention provides a traffic data collecting device, where the device is arranged in a data receiving end, and specifically includes:
the connection information receiving unit is used for receiving the connection information of at least one traffic data source distributed by the server;
the data acquisition unit is used for acquiring traffic data according to the connection information received by the connection information receiving unit;
the acquisition quantity judging unit is used for detecting the acquisition quantity of the traffic data according to a preset frequency and judging whether the acquisition quantity exceeds a maximum threshold value;
and the connection disconnection unit is used for disconnecting at least one traffic data source according to a preset rule when the acquisition amount judgment unit determines that the acquisition amount exceeds the maximum threshold value, so that the acquisition amount is lower than the maximum threshold value.
In a fourth aspect, the present invention provides a traffic data collecting device, where the traffic data collecting device is disposed in a server, and specifically includes:
the data source distribution unit is used for distributing the connection information of at least one traffic data source for each data receiving terminal according to the registered data receiving terminal information;
the connection monitoring unit is used for monitoring whether the traffic data source connected with each data receiving end is disconnected;
the determining unit is used for determining a new data receiving end for the disconnected traffic data source according to a preset strategy when the connection monitoring unit monitors that the disconnected traffic data source exists;
and the sending unit is used for sending the connection information to the new data receiving end determined by the determining unit.
In a fifth aspect, the present invention provides a traffic data collection system, comprising: the system comprises a server and at least two data receiving ends, wherein the data receiving ends are used for executing the traffic data acquisition method in the first aspect, and the server is used for executing the traffic data acquisition method in the second aspect.
In another aspect, the present invention provides a storage medium, where the storage medium is used to store a computer program, where the computer program controls a data receiving end where the storage medium is located to execute the traffic data acquisition method according to the first aspect, or controls a service end where the storage medium is located to execute the traffic data acquisition method according to the second aspect when running.
In another aspect, the present invention provides a processor for executing a computer program, wherein the computer program executes the method for acquiring traffic data according to the first aspect or the method for acquiring traffic data according to the second aspect.
By means of the technical scheme, the traffic data acquisition method, the traffic data acquisition device and the traffic data acquisition system provided by the invention have the advantages that when the data receiving end normally receives data of one traffic data source, the data receiving end is simultaneously used as a standby machine of other data receiving ends, and the data of a plurality of traffic data sources are received according to the indication of the server end. And the local machine can disconnect part of the connection in real time through judging the load so as to ensure the stable work of the local machine, and the disconnected traffic data source can be sensed by the server end and distributed to other data receiving ends to continue receiving the traffic data, thereby realizing the dynamic adjustment of the traffic data acquisition. In the process of collecting traffic data, the server only senses the disconnected traffic data sources of each data receiving end and informs the other data receiving end of collecting the data of the traffic data sources according to a preset strategy, and the data receiving ends need to determine the quantity of the collected traffic data sources according to the load conditions of the data receiving ends and are connected with the corresponding traffic data sources based on the indication of the server.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 shows a flowchart of a traffic data collection method for a data receiving end according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating another traffic data collection method for a data receiving end according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method for implementing traffic data collection by using a scheduling server according to an embodiment of the present invention;
fig. 4 is a flowchart illustrating a traffic data collection method for a server according to an embodiment of the present invention;
fig. 5 is a block diagram illustrating a traffic data collecting apparatus for a data receiving end according to an embodiment of the present invention;
fig. 6 is a block diagram illustrating another traffic data collecting apparatus for a data receiving end according to an embodiment of the present invention;
fig. 7 is a block diagram illustrating a traffic data collecting apparatus for a server according to an embodiment of the present invention;
fig. 8 is a block diagram illustrating another traffic data collection apparatus for a server according to an embodiment of the present invention;
fig. 9 is a block diagram showing a traffic data acquisition system according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
The embodiment of the invention provides a traffic data acquisition method which is applied to a data receiving end and used for receiving traffic data provided by a GPS data manufacturer. The specific steps are shown in fig. 1 and comprise:
In the invention, the data receiving end can simultaneously receive the traffic data provided by a plurality of traffic data sources (GPS data manufacturers), and different communication protocols can be selected through different data ports to download and analyze the traffic data in order to distinguish different traffic data sources.
In addition, when the server side in this step allocates the traffic data sources, the traffic data sources can be divided into two cases, one is that when the data receiving side starts the service to start downloading the traffic data, i.e. when the service starts the initial downloading, the traffic data sources allocated by the server are allocated according to all the traffic data sources in the system, that is, the data receiving side at this time is the receiving host of the traffic data sources; and the other is the traffic data source distributed by the server in the receiving process of the data receiving end, and at the moment, the data receiving end is a receiving standby machine of the traffic data sources.
And 102, collecting traffic data according to the connection information.
And the data receiving end establishes long connection with the traffic data source according to the connection information and acquires traffic data based on the long connection. It should be noted that, in the present invention, the establishment of the long connection does not limit the initiator of the connection, that is, after determining the corresponding relationship between the data receiving end and the traffic data source, which party initiates the connection can be set according to the requirements of the actual situation.
The preset frequency can be set by self-definition according to actual requirements, and the setting of the maximum threshold is determined by the configuration of the data receiving end.
The data receiving end periodically detects the load of the self-downloaded traffic data to ensure the stability of the work of the local machine, once the maximum threshold value is exceeded, the self load needs to be reduced, the step 104 is specifically executed, and if the maximum threshold value is not exceeded, no processing is carried out, and the next detection is waited.
And 104, disconnecting at least one traffic data source according to a preset rule so that the acquisition amount is lower than a maximum threshold value.
When the load of the data receiving end exceeds the maximum threshold value, the data receiving end can achieve the purpose of reducing the load by reducing the number of the connected traffic data sources.
The preset rule is used for determining the mode of disconnecting the currently connected traffic data source, and the specific content of the preset rule comprises the determination of whether the traffic data source needs to be disconnected and which traffic data source needs to be disconnected.
When the connected traffic data source is unique, the preset rule only needs to judge whether the traffic data source needs to be disconnected. For example, the traffic data source may be disconnected to reconnect one or more traffic data sources with smaller acquisition amount, or may be temporarily kept connected to determine whether the acquisition amount of the traffic data source continuously exceeds the maximum threshold, and then determine whether to disconnect the traffic data source. In addition, for the condition that only one traffic data source is connected currently, the management personnel can be prompted to carry out manual processing.
When the connected traffic data sources are not unique, the preset rule needs to select the traffic data source to be disconnected from the plurality of connected traffic data sources, at this time, the preset rule can be set according to actual requirements, for example, the connection of the traffic data sources can be disconnected one by one, whether the residual acquisition amount exceeds the maximum threshold value is calculated every time the connection is disconnected, and if the residual acquisition amount exceeds the maximum threshold value, the connection is disconnected again until the residual acquisition amount is lower than the maximum threshold value; or calculating the data volume exceeding the maximum threshold, selecting one or more traffic data sources with the total acquisition volume higher than the data volume, disconnecting the traffic data sources, and enabling the load of the data receiving end to meet the requirement through one-time operation.
According to the specific implementation manner in the embodiment, when the traffic data collection method applied to the data receiving end is used as a host to collect traffic data provided by a traffic data source, the collected traffic data source can be increased according to the distribution of the service end, the load of the host can be detected periodically, and when the load is too high, the connection of part of the traffic data source is actively disconnected so that the load meets the threshold requirement, so that the dynamic adjustment of the load is realized, and the stable operation of the service is ensured. And for the disconnected traffic data source, the server side reconnects the appointed standby data receiving end and collects traffic data, so that the multiple data receiving ends are standby mutually, and the utilization efficiency of processing resources is improved.
To further explain the traffic data collection method shown in fig. 1 in detail, in particular, the specific steps of the data receiving end in the process of performing traffic data collection in combination with the service end in the traffic data collection system are as shown in fig. 2, and the method includes:
The step is executed when the system is initialized, and the data receiving end uses the identification information to register in the service end, so that the service end counts the number of all the data receiving ends in the system. The specific content of the identification information is not limited, and may be a network address of the device or a device identification number of the device.
Furthermore, the server side can also send the configuration information of the equipment to the server side along with the identification information, so that the server side can allocate a traffic data source to the equipment according to the configuration information.
After the server is successfully registered, the data receiving end receives the connection information of at least one traffic data source distributed by the server.
And step 203, establishing long connection with the corresponding traffic data source according to the connection information.
And step 204, collecting traffic data through the long connection and the preset data receiving port.
The data receiving end distinguishes a plurality of traffic data sources through a plurality of preset data receiving ports, and therefore the traffic data collected from different traffic data sources can be ensured to determine collected communication protocols and data analysis modes according to downloaded ports.
The above steps are initialization operations executed when the system is started, and in short, the server determines the collected traffic data source of each data receiving end, and the data receiving ends start collecting traffic data. The subsequent step is a concrete implementation mode that the data receiving ends are mutually standby in the process of collecting the traffic data.
And step 205, in the process of collecting the traffic data, receiving a data port monitoring instruction distributed by the server according to dynamic perception.
And the data port monitoring instruction is used for monitoring a data port of the scheduling server so as to be connected with a traffic data source. In the embodiment of the invention, the scheduling server is connected between the traffic data source and the data receiving end, the scheduling server is connected with the traffic data source through the appointed data port, the data receiving end realizes the connection with the traffic data source by monitoring the data port, and the data port is intelligently monitored by one data receiving end at the same time, so that for one data port, when the originally monitored data receiving end stops monitoring (the data receiving end is disconnected from the traffic data source), the server end sends the monitoring instruction of the data port to a new data receiving end through dynamic sensing, so that the new data receiving end continues to monitor the data port, establishes the connection with the traffic data source and collects traffic data.
And step 206, monitoring the corresponding data port according to the data port monitoring instruction, and collecting traffic data provided by a traffic data source corresponding to the data port.
The implementation of the above steps 205 and 206 is shown in fig. 3, where the LVS in the figure is a Linux virtual server, which is an implementation manner of a scheduling server, and it is assumed that the LVS determines a connection relationship with a GPS data manufacturer 2 (a traffic data source) by monitoring an 8080 port, when the receiving server a monitors the 8080 port, the receiving server a acquires traffic data of the GPS data manufacturer 2, and when the receiving server a breaks the connection due to a fault or a load problem, the monitoring on the 8080 port is stopped at the same time, and at this time, the server obtains the disconnection between the receiving server a and the GPS data manufacturer 2 by dynamic sensing, so as to send an instruction for monitoring the 8080 port to the receiving server B, and monitor the port by the receiving server B, thereby implementing the connection with the GPS data manufacturer 2 and acquiring the traffic data.
It is emphasized that steps 205, 206 are performed at different stages of the system execution, the former during the acquisition of traffic data and the latter during system initialization, as compared to steps 202 to 204. Alternatively, in steps 205 and 206, the data sink is the backup of the traffic data source, and in steps 202 to 204, the data sink is the host of the traffic data source. Through the steps, the data receiving end can be used as a standby machine of other data receiving ends in the process of collecting the traffic data, and the server end is used for sensing and triggering the data receiving end to be used as the standby machine to collect the traffic data.
And step 207, detecting the acquisition amount of the traffic data according to the preset frequency, and judging whether the acquisition amount exceeds a maximum threshold value.
And step 208, judging whether the currently connected traffic data source is unique.
And 209, disconnecting at least one traffic data source according to a preset rule so that the acquisition amount is lower than a maximum threshold value.
In this embodiment, one implementation of the preset rule is to select a disconnected connection according to a weight of each traffic data source, where the weight may be a manually preset value or may be calculated by a preset parameter, and for the preset parameter, the preset parameter at least includes an acquisition amount and/or a data quality of the traffic data source, the acquisition amount can measure a total amount of data provided by the traffic data source, and the data quality can measure an effectiveness degree of the data provided by the traffic data source. In specific implementation, the traffic data sources can be sorted from low to high according to the weights of all the traffic data sources, and then the connection of the traffic data source with the smallest weight is disconnected one by one according to the sorting until the sum of the acquisition amounts corresponding to all the traffic data sources of the remaining connections is lower than the maximum threshold value, so that the adjustment of the connection is completed.
Corresponding to the traffic data acquisition method applied to the data receiving end described in fig. 1 and fig. 2, an embodiment of the present invention further provides a traffic data acquisition method for a service end, where the method is used to sense a disconnected connection in the data receiving end and allocate a new data receiving end, and the specific steps are as shown in fig. 4, and include:
The registration is a mode that the server side is used for counting the number of the data receiving ends, and through registration operation, the server side can determine the number of the data receiving ends for executing traffic data collection, so that at least one traffic data source is distributed to each data receiving end according to respective configuration and processing capacity, and the data receiving end is determined to be a host of the traffic data sources. Meanwhile, the connection information of the traffic data sources is sent to the data receiving end, so that the data receiving end is connected with the traffic data sources and collects traffic data.
It should be noted that this step is performed at system initialization, that is, during the system starts to perform data acquisition, the server does not accept the newly registered data receiver. And if a new data receiving end needs to be added, the acquisition operation of the system needs to be stopped, the initialization operation needs to be executed again, and a traffic data source needs to be distributed.
In the process of collecting traffic data, the server side monitors each data receiving end in the system, judges whether a disconnected traffic data source exists in the data receiving ends, and if yes, executes step 303; if not, no processing is performed. The invention does not limit the period of the monitoring operation, and the specific period interval can be adjusted according to the requirements of practical application.
When the data receiving end is disconnected from the traffic data source, the data receiving end is actively operated according to the load condition of the data receiving end, and the data receiving end is not actively informed to the server end, so that the server end needs to actively monitor the data receiving end, and in practical application, the server end can dynamically sense the connection state in the data receiving end through the Zookeeper service.
In addition, when there is disconnection at the data receiving end, there are two main reasons for this, one is partial disconnection due to excessive load at the data receiving end, and the other is all disconnection due to failure at the data receiving end. Therefore, the server side can judge the working state of the data receiving end according to the number of the disconnected connections. When the data receiving end fails, all the traffic data source connections need to be transferred to the standby data receiving end.
The preset strategy is mainly used for judging the matching degree of the current data processing capacity of the data receiving end and the traffic data acquisition amount provided by the unconnected traffic data source. One possible embodiment of the preset policy is to determine a new data receiving end according to the acquisition amount of the disconnected traffic data source and the load information of other data receiving ends, for example, in fig. 3, the acquisition amount of the disconnected GPS data is 10M, and accordingly, the service end needs to determine the load condition of the receiving server B to determine whether it has the processing capability of receiving 10M traffic data, if so, determine that the receiving server B is the new data receiving end, and if not, determine other data receiving ends until the new data receiving end is determined, and if no new data receiving end is matched in the system, generate a prompt message to manually process the disconnected connection.
In addition, the preset policy may also be set as a pre-designated data receiving end, that is, a spare machine of the receiving server a connected to the GPS data manufacturer 2 is designated as the receiving server B, and once the disconnection is sensed by the server, the receiving server B is notified to establish the connection with the GPS data manufacturer 2.
Further, the server identifies the device type of the new data receiving end when determining the new data receiving end, and in practical application, the device type is mainly divided into a user end and a server end.
When the device type is the user side, it is the implementation shown in fig. 4, that is, the connection information is directly sent to the user side.
When the device type is a server side, in practice, a scheduling server is generally arranged between a traffic data source and a data receiving end, which is realized by a plurality of LVSs, and the LVS is used to receive traffic data of the traffic data source and forward the traffic data to the data receiving end which monitors the designated data port. Therefore, when the new data receiving end is a server, the server end acquires the scheduling server corresponding to the new data receiving end and sends a data port monitoring instruction for monitoring the data port specified by the scheduling server to the new data receiving end.
The implementation shown in fig. 4 above describes in detail how the server side perceives the disconnection of the connection and determines that a new data receiving end performs collection when the data receiving end collects traffic data.
Further, as an implementation of the methods shown in fig. 1 and fig. 2, an embodiment of the present invention provides a traffic data collecting device, where the traffic data collecting device is disposed in a data receiving end, and is used to collect traffic data and store the traffic data locally. For convenience of reading, details in the foregoing method embodiments are not described in detail again in this apparatus embodiment, but it should be clear that the apparatus in this embodiment can correspondingly implement all the contents in the foregoing method embodiments. As shown in fig. 5, the apparatus specifically includes:
a connection information receiving unit 41, configured to receive connection information of at least one traffic data source allocated by the server;
a data acquisition unit 42 for acquiring traffic data according to the connection information received by the connection information receiving unit 41;
the acquisition amount judging unit 43 is configured to detect the acquisition amount of the traffic data acquired by the data acquisition unit 42 according to a preset frequency, and judge whether the acquisition amount exceeds a maximum threshold;
a connection disconnection unit 44, configured to, when the acquired amount judgment unit 43 determines that the acquired amount exceeds the maximum threshold, disconnect at least one traffic data source according to a preset rule so that the acquired amount is lower than the maximum threshold.
Further, as shown in fig. 6, the apparatus further includes:
a monitoring instruction receiving unit 45, configured to receive a data port monitoring instruction distributed by the server according to dynamic sensing, where the data port monitoring instruction is used to monitor a data port of the scheduling server to connect to a traffic data source;
a data port monitoring unit 46, configured to monitor a corresponding data port according to the data port monitoring instruction received by the monitoring instruction receiving unit 45, so that the data acquisition unit 42 acquires traffic data provided by a traffic data source corresponding to the data port.
Further, as shown in fig. 6, the connection disconnection unit 44 includes:
the connection number judging module 441 is used for judging whether the currently connected traffic data source is unique;
a first disconnection module 442, configured to disconnect at least one traffic data source when the connection quantity determination module determines that the connected traffic data sources are not unique.
Further, as shown in fig. 6, the connection disconnection unit 44 includes:
the data source sorting module 443 is used for sorting from low to high according to the weights of all traffic data sources, wherein the weights are at least determined by the acquisition amount of the traffic data;
and the second disconnection module 444 is configured to disconnect the traffic data sources with the smallest weights one by one according to the sorting of the data source sorting module 443 until the sum of the acquisition amounts corresponding to all connected traffic data sources is lower than the maximum threshold.
It should be noted that, in practical applications, the first disconnection module 442 and the second disconnection module 444 may be disposed in the same module.
Further, as shown in fig. 6, the data acquisition unit 42 includes:
a connection creating module 421, configured to create a long connection with a corresponding traffic data source according to the connection information;
and a data collection module 422, configured to collect traffic data through the long connection established by the connection creation module 421 and a preset data receiving port.
Further, as shown in fig. 6, the apparatus further includes:
an identification information sending unit 47, configured to send the identification information of the data receiving end to the server, so that the server registers the data receiving end as a node.
Further, as an implementation of the method shown in fig. 4, an embodiment of the present invention provides a traffic data collecting device, which is disposed in a service, and is configured to sense a disconnected connection in a data receiving end and allocate a new data receiving end. For convenience of reading, details in the foregoing method embodiments are not described in detail again in this apparatus embodiment, but it should be clear that the apparatus in this embodiment can correspondingly implement all the contents in the foregoing method embodiments. As shown in fig. 7, the apparatus specifically includes:
a data source allocating unit 51, configured to allocate connection information of at least one traffic data source to each data receiving end according to the registered data receiving end information;
a connection monitoring unit 52, configured to monitor whether the traffic data source connected to each data receiving end is disconnected;
a determining unit 53, configured to determine a new data receiving end for a disconnected traffic data source according to a preset policy when the connection monitoring unit 52 monitors that the disconnected traffic data source exists;
a sending unit 54, configured to send connection information to the new data receiving end determined by the determining unit 53.
Further, as shown in fig. 8, the apparatus further includes:
a determining unit 55, configured to determine a device type of the new data receiving end, where the device type includes a user side and a server side;
the sending unit 54 is further configured to, when the determining unit 55 determines that the device type is the user side, execute an operation of sending connection information to the new data receiving end;
a monitoring instruction sending unit 56, configured to, when the determining unit 55 determines that the device type is a server side, obtain a scheduling server corresponding to the new data receiving end, and send a data port monitoring instruction to the new data receiving end, where the scheduling server is configured to receive traffic data of a traffic data source and forward the traffic data through the data port.
Further, the determining unit 53 is further configured to determine a new data receiving end according to the acquisition amount of the disconnected traffic data source and the load information of other data receiving ends when it is monitored that all the registered data receiving ends are disconnected.
Further, by combining the above traffic data collection method and apparatus, an embodiment of the present invention further provides a traffic data collection system, as shown in fig. 9, which is composed of a plurality of traffic data sources 61, at least two data receiving terminals 62, and a service terminal 63. The data receiving end executes the traffic data collection method in the embodiment shown in fig. 1 or fig. 2, and the service end executes the traffic data collection method in the embodiment shown in fig. 4.
In the system, the server 63 monitors the connection state of each data receiving end 62 and the traffic data source 63, and once the disconnection condition is sensed, the standby data receiving end 62 is started to reestablish the connection with the disconnected traffic data source. Moreover, each data receiving end can simultaneously receive traffic data of a plurality of traffic data sources through different data ports, so that the multiplication of the number of the data receiving ends caused by different communication protocols of data manufacturers is avoided.
Further, in the system, a scheduling server 64 is further included, and the scheduling server is configured to receive traffic data of a traffic data source and forward the traffic data through a designated data port, so as to monitor a data receiving end of the data port to receive the traffic data. The specific description of the scheduling server is already explained in the embodiment shown in fig. 2, and is not repeated here.
In summary, the traffic data acquisition method, the traffic data acquisition device and the traffic data acquisition system according to the embodiments of the present invention achieve that when a data receiving end normally receives data of one traffic data source, the data receiving end simultaneously serves as a backup for other data receiving ends to receive data of multiple traffic data sources according to an instruction from a server. And the data receiving end can disconnect part of the connection in real time through judging the load so as to ensure the stable work of the local machine, and the disconnected traffic data source can be sensed by the service end and distributed to other data receiving ends to continue receiving the traffic data, so that the dynamic adjustment of traffic data acquisition is realized, and the data receiving ends are mutually standby. Compared with the existing cold standby mode, in the process of collecting traffic data, the service end only senses the disconnected traffic data sources of each data receiving end and informs the other data receiving end of collecting the data of the traffic data sources according to the preset strategy, the data receiving ends need to determine the quantity of the collected traffic data sources according to the load conditions of the data receiving ends and connect the corresponding traffic data sources based on the indication of the service end, and therefore the framework of the traffic data collecting system formed by the service end and the data receiving ends can greatly reduce the required equipment quantity, the functions of the server end are simplified, the data quantity of interaction among systems is reduced, the data receiving ends can concentrate more processing resources on collecting the traffic data, and the collecting efficiency of the system is improved.
Further, an embodiment of the present invention further provides a storage medium, where the storage medium is used to store a computer program, where when the computer program runs, the data receiving end where the storage medium is located is controlled to execute the traffic data acquisition method shown in fig. 1 or fig. 2, or the service end where the storage medium is located is controlled to execute the traffic data acquisition method shown in fig. 4.
In addition, an embodiment of the present invention further provides a processor, where the processor is configured to execute a computer program, where the computer program executes the traffic data acquisition method described in fig. 1 or fig. 2, or executes the traffic data acquisition method described in fig. 4.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It will be appreciated that the relevant features of the method, apparatus and system described above are referred to one another. In addition, "first", "second", and the like in the above embodiments are for distinguishing the embodiments, and do not represent merits of the embodiments.
It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In addition, the memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both permanent and non-permanent, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (22)
1. A traffic data acquisition method, which is used for a data receiving end, comprises the following steps:
receiving connection information of at least one traffic data source distributed by a server;
collecting traffic data according to the connection information;
detecting the acquisition amount of the traffic data according to a preset frequency, and judging whether the acquisition amount exceeds a maximum threshold value;
if the traffic data source is not connected with the new data receiving end, the server side monitors whether the traffic data source connected with each data receiving end is disconnected, and if the traffic data source is disconnected, the server side determines the new data receiving end for the disconnected traffic data source according to a preset strategy and sends connection information to the new data receiving end.
2. The method of claim 1, further comprising:
receiving a data port monitoring instruction distributed by a server according to dynamic sensing, wherein the data port monitoring instruction is used for monitoring a data port of a scheduling server so as to connect a traffic data source;
and monitoring the corresponding data port according to the data port monitoring instruction, and acquiring traffic data provided by a traffic data source corresponding to the data port.
3. The method according to claim 1, wherein disconnecting at least one traffic data source according to a preset rule specifically comprises:
judging whether the currently connected traffic data source is unique;
if the traffic data source connected is not unique, the connection of at least one traffic data source is disconnected.
4. The method of claim 1, wherein disconnecting at least one traffic data source according to a preset rule further comprises:
sorting from low to high according to the weights of all traffic data sources, wherein the weights are at least determined by the acquisition amount of the traffic data;
and disconnecting the traffic data sources with the minimum weight one by one according to the sequence until the sum of the acquisition quantities corresponding to all the connected traffic data sources is lower than the maximum threshold value.
5. The method of claim 1, wherein collecting traffic data based on the connection information comprises:
establishing long connection with a corresponding traffic data source according to the connection information;
and collecting traffic data through the long connection and a preset data receiving port.
6. The method according to any one of claims 1-4, further comprising:
and sending the identification information of the data receiving end to the server so that the server registers the data receiving end as a node.
7. A traffic data acquisition method is used for a server side and comprises the following steps:
distributing connection information of at least one traffic data source to each data receiving end according to the registered data receiving end information so that the data receiving ends acquire traffic data according to the connection information, detecting the acquisition amount of the traffic data according to preset frequency, judging whether the acquisition amount exceeds a maximum threshold value or not, and if the acquisition amount exceeds the maximum threshold value, disconnecting the connection of at least one traffic data source according to a preset rule so that the acquisition amount is lower than the maximum threshold value;
monitoring whether a traffic data source connected with each data receiving end is disconnected;
if the traffic data source is disconnected, determining a new data receiving end for the disconnected traffic data source according to a preset strategy;
and sending the connection information to the new data receiving end.
8. The method of claim 7, further comprising:
judging the equipment type of the new data receiving end, wherein the equipment type comprises a user end and a server end;
when the equipment type is the user side, executing the operation of sending connection information to the new data receiving end;
and when the equipment type is a server side, acquiring a scheduling server corresponding to the new data receiving end, and sending a data port monitoring instruction to the new data receiving end, wherein the scheduling server is used for receiving traffic data of a traffic data source and forwarding the traffic data through the data port.
9. The method according to claim 7 or 8, characterized in that the method further comprises:
and when all the registered data receiving ends are monitored to be disconnected, determining a new data receiving end according to the acquisition amount of the disconnected traffic data source and the load information of other data receiving ends.
10. A traffic data collection device, said device being disposed in a data receiving end, said device comprising:
the connection information receiving unit is used for receiving the connection information of at least one traffic data source distributed by the server;
the data acquisition unit is used for acquiring traffic data according to the connection information received by the connection information receiving unit;
the acquisition quantity judging unit is used for detecting the acquisition quantity of the traffic data according to a preset frequency and judging whether the acquisition quantity exceeds a maximum threshold value;
and the connection disconnection unit is used for disconnecting at least one traffic data source according to a preset rule when the acquisition amount judgment unit determines that the acquisition amount exceeds the maximum threshold value, so that the acquisition amount is lower than the maximum threshold value, the server monitors whether the traffic data source connected with each data receiving end is disconnected, and if the connection is disconnected, a new data receiving end is determined for the disconnected traffic data source according to a preset strategy, and connection information is sent to the new data receiving end.
11. The apparatus of claim 10, further comprising:
the monitoring instruction receiving unit is used for receiving a data port monitoring instruction distributed by the server according to dynamic sensing, and the data port monitoring instruction is used for monitoring a data port of the dispatching server so as to be connected with a traffic data source;
and the data port monitoring unit is used for monitoring the corresponding data port according to the data port monitoring instruction received by the monitoring instruction receiving unit so as to facilitate the data acquisition unit to acquire the traffic data provided by the traffic data source corresponding to the data port.
12. The apparatus of claim 10, wherein the connection disconnection unit comprises:
the connection quantity judging module is used for judging whether the currently connected traffic data source is unique;
and the first connection disconnection module is used for disconnecting at least one traffic data source when the connection quantity judgment module determines that the connected traffic data sources are not unique.
13. The apparatus of claim 10, wherein the connection disconnection unit comprises:
the data source sequencing module is used for sequencing from low to high according to the weight of all traffic data sources, and the weight is at least determined by the acquisition amount of the traffic data;
and the second disconnection module is used for disconnecting the traffic data sources with the minimum weight one by one according to the sequence of the data source sequencing module until the sum of the acquisition amount corresponding to all the connected traffic data sources is lower than the maximum threshold value.
14. The apparatus of claim 10, wherein the data acquisition unit comprises:
the connection establishing module is used for establishing long connection with a corresponding traffic data source according to the connection information;
and the data acquisition module is used for acquiring traffic data through the long connection established by the connection establishing module and a preset data receiving port.
15. The apparatus according to any one of claims 10-13, further comprising:
and the identification information sending unit is used for sending the identification information of the data receiving end to the server so that the server can register the data receiving end as a node.
16. A traffic data collection device, the device disposed in a server, the device comprising:
the data source distribution unit is used for distributing connection information of at least one traffic data source to each data receiving end according to the registered data receiving end information so that the data receiving ends collect traffic data according to the connection information, detecting the collection quantity of the traffic data according to preset frequency, judging whether the collection quantity exceeds a maximum threshold value or not, and if the collection quantity exceeds the maximum threshold value, disconnecting the connection of at least one traffic data source according to preset rules so that the collection quantity is lower than the maximum threshold value;
the connection monitoring unit is used for monitoring whether the traffic data source connected with each data receiving end is disconnected;
the determining unit is used for determining a new data receiving end for the disconnected traffic data source according to a preset strategy when the connection monitoring unit monitors that the disconnected traffic data source exists;
and the sending unit is used for sending the connection information to the new data receiving end determined by the determining unit.
17. The apparatus of claim 16, further comprising:
the judging unit is used for judging the equipment type of the new data receiving end, wherein the equipment type comprises a user end and a server end;
the sending unit is further configured to, when the determining unit determines that the device type is the user side, execute an operation of sending connection information to the new data receiving end;
and the monitoring instruction sending unit is used for acquiring a scheduling server corresponding to the new data receiving end and sending a data port monitoring instruction to the new data receiving end when the judging unit determines that the equipment type is the server end, and the scheduling server is used for receiving traffic data of a traffic data source and forwarding the traffic data through the data port.
18. The apparatus according to claim 16 or 17, wherein the determining unit is further configured to determine a new data sink according to the acquisition amount of the disconnected traffic data source and the load information of other data sinks when monitoring all the disconnections from the registered data sinks.
19. A traffic data collection system, the system comprising: the system comprises a server and at least two data receiving ends, wherein the data receiving ends are used for executing the traffic data acquisition method in any one of claims 1 to 6, and the server is used for executing the traffic data acquisition method in any one of claims 7 to 9.
20. The traffic data collection system of claim 19, further comprising:
the scheduling server is used for receiving the traffic data of a traffic data source and forwarding the traffic data through a specified data port so as to monitor that a data receiving end of the data port receives the traffic data.
21. A storage medium, characterized in that the storage medium is used for storing a computer program, wherein the computer program controls a data receiving end of the storage medium to execute the traffic data collecting method according to any one of claims 1 to 6 when running, or controls a service end of the storage medium to execute the traffic data collecting method according to any one of claims 7 to 9.
22. A processor for running a computer program, wherein the computer program is running for performing the traffic data collecting method according to any one of claims 1-6 or for performing the traffic data collecting method according to any one of claims 7-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910027606.7A CN111435566B (en) | 2019-01-11 | 2019-01-11 | Traffic data acquisition method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910027606.7A CN111435566B (en) | 2019-01-11 | 2019-01-11 | Traffic data acquisition method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111435566A CN111435566A (en) | 2020-07-21 |
| CN111435566B true CN111435566B (en) | 2022-08-16 |
Family
ID=71580331
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910027606.7A Active CN111435566B (en) | 2019-01-11 | 2019-01-11 | Traffic data acquisition method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111435566B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101621373A (en) * | 2009-06-22 | 2010-01-06 | 许继集团有限公司 | Network communication method of centralized control station system, preferable shift method and dynamic routing method |
| CN102081845A (en) * | 2011-01-25 | 2011-06-01 | 北京世纪高通科技有限公司 | Preprocessing method and device of traffic information |
| CN107729205A (en) * | 2017-08-22 | 2018-02-23 | 国家电网公司 | Fault handling method and device for operation system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5942374B2 (en) * | 2011-09-30 | 2016-06-29 | ソニー株式会社 | Information processing apparatus, communication system, and information processing apparatus control method |
-
2019
- 2019-01-11 CN CN201910027606.7A patent/CN111435566B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101621373A (en) * | 2009-06-22 | 2010-01-06 | 许继集团有限公司 | Network communication method of centralized control station system, preferable shift method and dynamic routing method |
| CN102081845A (en) * | 2011-01-25 | 2011-06-01 | 北京世纪高通科技有限公司 | Preprocessing method and device of traffic information |
| CN107729205A (en) * | 2017-08-22 | 2018-02-23 | 国家电网公司 | Fault handling method and device for operation system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111435566A (en) | 2020-07-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106375420B (en) | Server cluster intelligent monitoring system and method based on load balancing | |
| US9532359B2 (en) | Resource allocation method and device | |
| CN111475276A (en) | Task management method and device based on edge calculation | |
| CN109802986B (en) | Equipment management method, system, device and server | |
| CN101895472A (en) | Traffic flow load sharing method and processing method and corresponding device and system | |
| CN109769029B (en) | Communication connection method based on electricity consumption information acquisition system and terminal equipment | |
| CN113364682B (en) | Data transmission method and device, storage medium and electronic device | |
| CN112671928A (en) | Equipment centralized management architecture, load balancing method, electronic equipment and storage medium | |
| CN113746821B (en) | Intelligent device control method and device, storage medium and electronic device | |
| CN115277598B (en) | Method and device for scheduling computing power resources and computer readable storage medium | |
| EP3813335B1 (en) | Service processing methods and systems based on a consortium blockchain network | |
| CN102316483A (en) | Method and device for ensuring quality of service (QoS) of application service in evolution-data optimized (EVDO) system | |
| CN105933271B (en) | Data processing method and device based on encryption machine | |
| CN101262479A (en) | A network file share method, server and network file share system | |
| CN111435566B (en) | Traffic data acquisition method, device and system | |
| CN113407340A (en) | Service control system, gateway service method, service request forwarding method and device | |
| JP6091333B2 (en) | Network management apparatus and network management method | |
| CN112118275A (en) | Overload processing method, Internet of things platform and computer readable storage medium | |
| CN110635927B (en) | Node switching method, network node and network system | |
| CN113535402A (en) | Load balancing processing method and device based on 5G MEC and electronic equipment | |
| CN113190347A (en) | Edge cloud system and task management method | |
| CN116055496B (en) | Monitoring data acquisition method and device, electronic equipment and storage medium | |
| CN111158899A (en) | Data acquisition method, data acquisition device, task management center and task management system | |
| CN114710494B (en) | Application management method, device and storage medium | |
| CN111669280B (en) | Message transmission method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |