CN114722006A - Data transmission control method, device, equipment and storage medium - Google Patents
Data transmission control method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114722006A CN114722006A CN202210272035.5A CN202210272035A CN114722006A CN 114722006 A CN114722006 A CN 114722006A CN 202210272035 A CN202210272035 A CN 202210272035A CN 114722006 A CN114722006 A CN 114722006A
- Authority
- CN
- China
- Prior art keywords
- call ticket
- line
- call
- data transmission
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 75
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000004590 computer program Methods 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 238000012544 monitoring process Methods 0.000 claims description 8
- 238000012546 transfer Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 description 11
- 230000003139 buffering effect Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- CYTYCFOTNPOANT-UHFFFAOYSA-N Perchloroethylene Chemical compound ClC(Cl)=C(Cl)Cl CYTYCFOTNPOANT-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Telephonic Communication Services (AREA)
Abstract
The embodiment of the invention discloses a method, a device, equipment and a storage medium for controlling data transmission. Wherein, the method comprises the following steps: acquiring a call ticket file sent by a data acquisition system in real time, and determining the number of call single lines in the call ticket file; determining the average input rate of the call ticket rows in a preset data transmission period according to the number of the call ticket rows; outputting the single line of words to a KAFKA cluster according to the average input rate. The output of the voice single line is controlled according to the input rate of the voice single line, the problem of large data quantity jitter in a single time period is solved, data loss is avoided, smooth fluctuation of the data quantity is realized, and the efficiency and the precision of data transmission are improved.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method, an apparatus, a device, and a storage medium for controlling data transmission.
Background
In a telecommunication collection system, an xDR (x Detail Record) ticket is key data for data transmission between systems of each layer. The underlying data collection layer typically uses files as a bearer for xDR tickets, while the upper application layer typically uses KAFKA (kaffka) as a high-performance message queue. There is therefore a need for a KAFKA producer to provide a forwarding service that forwards xDR file content piece by piece into a KAFKA cluster.
In the prior art, when a ticket file is acquired, the ticket file can be processed by centralized calculation, and a KAFKA producer can output all files as soon as possible. But this approach would push data pressure directly to the upper layer applications, and this peak of bursts is a challenge for KAFKA clusters or KAFKA consumers.
The data output speed of the KAFKA producer can also be set to a preset value, but under the condition that the collection layer ticket file is increased, data backlog exists, service data is delayed and lost, and the efficiency and the precision of data transmission are influenced.
Disclosure of Invention
The embodiment of the invention provides a method, a device, equipment and a storage medium for controlling data transmission, which are used for improving the efficiency and the precision of data transmission.
According to an aspect of the present invention, there is provided a method of controlling data transmission, the method being performed by a kaff card KAFKA producer, comprising:
acquiring a call ticket file sent by a data acquisition system in real time, and determining the number of call single lines in the call ticket file;
determining the average input rate of the call ticket rows in a preset data transmission period according to the number of the call ticket rows;
outputting the single line of words to a KAFKA cluster according to the average input rate.
According to another aspect of the present invention, there is provided a control apparatus for data transmission, the apparatus being provided on a KAFKA producer, comprising:
the phone single line quantity determining module is used for acquiring a phone bill file sent by a data acquisition system in real time and determining the number of phone single lines in the phone bill file;
the average input rate determining module is used for determining the average input rate of the call ticket rows in a preset data transmission period according to the number of the call ticket rows;
a single line output module to output the single line to a KAFKA cluster according to the average input rate.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of controlling data transmission according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a control method for data transmission according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, the number of the single lines input in the data transmission period is determined by acquiring the call ticket file in the preset data transmission period, so that the average input rate of the single lines is obtained. The output rate of the call leg is controlled according to the average input rate so that the call leg can be pushed to the KAFKA cluster by the KAFKA producer at a smooth output speed. By determining the average input rate to adjust the output rate of the single line, it is ensured that the output rate is maintained in a relatively stable state. Even under the condition that the call ticket file is suddenly increased or suddenly reduced, the call ticket file can be output at an average input rate, and the pressure of an upper application is reduced. If the average input rate changes, the output rate can be adaptively changed, the problem of data backlog caused by the increase of the ticket file is solved, the delay and the loss of service data are avoided, and the efficiency and the precision of data transmission are improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for controlling data transmission according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a control method for data transmission according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a control apparatus for data transmission according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing the control method of data transmission according to the embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "current," "target," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a schematic flow chart of a data transmission control method according to an embodiment of the present invention, which is applicable to a case where call ticket data collected in a bottom layer is transmitted to an upper layer application, and the method can be executed by a data transmission control device, which can be implemented in a hardware and/or software form, and can be configured in a KAFKA producer. As shown in fig. 1, the method includes:
s110, a call ticket file sent by the data acquisition system is obtained in real time, and the number of call single lines in the call ticket file is determined.
The data acquisition system is an acquisition layer positioned at the bottom layer in the telecommunication platform and can acquire communication data such as call ticket files of users. The call ticket file may be an xDR call ticket file, and the xDR call ticket file may include information such as an identity, a location, and a network condition of a communication party. One call ticket file can comprise one or more call ticket lines, the KAFKA producer can send each call ticket line in the call ticket file to the KAFKA consumer, and the KAFKA consumer can also be a KAFKA cluster. The data acquisition system can acquire the ticket files in real time, for example, a plurality of ticket files can be acquired within one second. The KAFKA producer can acquire the ticket file sent by the data acquisition system in real time. After the call ticket file is obtained, the call ticket file can be analyzed, and the number of call single lines in the call ticket file is determined.
In this embodiment, optionally, the obtaining of the ticket file sent by the data acquisition system in real time includes: reading a call bill file on a hard disk of a data acquisition system in real time; and transmitting the ticket file to a preset monitoring disk directory, and carrying out memory loading operation on the ticket file in the monitoring disk directory.
Specifically, the KAFKA producer can obtain the ticket file in real time. The data acquisition system stores the acquired call ticket file on the hard disk, and the KAFKA producer reads the xDR call ticket file on the hard disk in the data acquisition system in real time. And transmitting the read call ticket file to a preset monitoring disk directory, namely storing the read call ticket file in a disk. And then, the ticket file is subjected to memory loading operation, namely, the ticket file is stored into the memory from the disk, so that the ticket file is effectively stored, the transmission interruption of a single line of the call is avoided, and the data transmission efficiency and precision are improved. The process of storing the ticket file from the data acquisition system to the memory is the input process of the ticket file, and after the memory is successfully loaded, the ticket line data in the ticket file can be output.
In this embodiment, optionally, determining the number of phone bill lines in the phone bill file includes: and decoding the call ticket file in the memory, and decoding each line of call tickets in the call ticket file according to a preset call ticket format to obtain the number of call single lines.
Specifically, a call ticket format of a call ticket file is preset, after the call ticket file is stored in the memory, the call ticket file in the memory is decoded according to the call ticket format, each call ticket in the call ticket file is obtained, and the number of the call tickets is determined. The KAFKA producer obtains new call ticket files in real time, and the number of call ticket lines in each call ticket file can be different. The KAFKA producer can update all the telephone single-line quantities acquired in the current data transmission period in real time every time a new telephone bill file is acquired. The method has the advantages that the xDR call ticket file is a converged xDR call ticket data set, and the set comprises a plurality of xDR call single lines which can reach millions. Whereas the KAFKA producer is a real-time data interface, the smallest unit of data that can be transferred is a single piece of data, i.e. a single row of data. Therefore, the call ticket file is decoded into a call ticket line, and the data transmission of the KAFKA producer is realized.
And S120, determining the average input rate of the single voice line in the preset data transmission period according to the number of the single voice line.
The data transmission period is preset, and is a preset time period. The number of lines that have been acquired within a data transmission cycle, i.e. within a predetermined time period, is determined in real time. Based on the number of call lines and the predetermined time period, the average rate at which the KAFKA producer takes the call lines, i.e. the average rate at which the call lines are input to the KAFKA producer, is determined as the average input rate during the data transmission period. For example, the data transmission period has a time period length of N, which may be 60 seconds. The number of call ticket lines transmitted in one minute is M, and the rate of acquiring the call ticket file per second is obtained by dividing the number of call ticket lines by the length of the time period and is used as the average input rate. The average input rate may be S, M/N, and the unit of S may be lps (line persec).
And S130, outputting the call ticket row to the KAFKA cluster according to the average input rate.
Wherein, after obtaining the average input rate, the average input rate is used as the output rate for the single line of the KAFKA producer output words. That is, the output rate of the KAFKA producer is kept consistent with the average input rate. The KAFKA producer may determine the current output rate by monitoring the number of talk spurts that it outputs during a data transmission period, for example, if the average input rate is 100 talk spurts per second and the data transmission period is one minute, the KAFKA producer may send 6000 talk spurts during the data transmission period, making the output rate the average input rate. The KAFKA producer may communicate with the KAFKA cluster and send the ticket line to the KAFKA cluster.
The number of the input lines of the call ticket line of the call ticket file in a period of time is captured in real time, and the average input rate is calculated, so that the call ticket line can be automatically and synchronously sent to a KAFKA producer, and the number of the output call ticket lines is equal to the number of the input call ticket lines in a period of time. Therefore, the problem of large data size jitter in a single time period is solved, and the KAFKA producer realizes a self-adaptive speed limit adjusting process to smooth the peak flow of the KAFKA cluster.
The embodiment of the invention determines the number of the single lines of speech input in the data transmission period by acquiring the call ticket file in the preset data transmission period, and obtains the average input rate of the single lines of speech. The output rate of the call leg is controlled according to the average input rate so that the call leg can be pushed to the KAFKA cluster by the KAFKA producer at a smooth speed. By determining the average input rate to adjust the output rate of the single line, it is ensured that the output rate is maintained in a relatively stable state. Even under the condition that the call ticket file is suddenly increased or suddenly reduced, the call ticket file can be output at an average input rate, and the pressure applied by an upper layer is reduced. If the average input rate changes, the output rate can be adaptively changed, the problem of data backlog caused by the increase of the ticket file is solved, the delay and the loss of service data are avoided, and the efficiency and the precision of data transmission are improved.
Example two
Fig. 2 is a flowchart illustrating a data transmission control method according to a second embodiment of the present invention, which is an optional embodiment based on the above embodiments, and the method can be executed by a data transmission control device.
In this embodiment, the call ticket row is output to the KAFKA cluster according to the average input rate, which can be detailed as: determining the average input rate as an output rate threshold for the single line; determining the current single-line output rate in the current data transmission period according to a preset speed limit algorithm, and judging whether the current single-line output rate is smaller than an output rate threshold value or not; and if so, outputting the call ticket line to the KAFKA cluster.
As shown in fig. 2, the method includes:
s210, a call ticket file sent by the data acquisition system is obtained in real time, and the number of call single lines in the call ticket file is determined.
And S220, determining the average input rate of the single voice line in the preset data transmission period according to the number of the single voice lines.
And S230, determining the average input speed as the output speed threshold of the dialogue line.
Wherein, after obtaining the average input rate, the average input rate is used as an output rate threshold for a single line of KAFKA producer output words. That is, the maximum output rate of the line of tickets of the KAFKA producer is the average input rate. For example, an average input rate of 1000lps, the KAFKA producer outputs a maximum of 1000 lines of tickets in 1 second.
S240, according to a preset speed limit algorithm, determining the current call list output rate in the current data transmission period, and judging whether the current call list output rate is smaller than an output rate threshold value.
The method comprises the steps of presetting a speed limit algorithm, wherein the speed limit algorithm is used for determining the current single-line output rate of the call and enabling the current single-line output rate not to exceed an output rate threshold. For example, the rate limiting algorithm may be a token bucket algorithm. The length of the data transmission period is set in advance, and for example, the data transmission period may be one minute. An output rate threshold may be determined for each data transfer cycle, and the number of tokens in the bucket is fixed during the current data transfer cycle, the fixed number of tokens being the determined output rate threshold. Inquiring the number of the vacant tokens in the token bucket in real time, if the vacant tokens exist, indicating that the single-row output rate of the current speech is less than the output rate threshold value; if no empty token exists, the current single-line output speed is equal to or larger than the output speed threshold value.
And S250, if yes, outputting the call ticket row to the KAFKA cluster.
If the current call ticket row output rate is determined to be smaller than the output rate threshold value, the algorithm can return a call ticket row releasing instruction and output the current call ticket row to the KAFKA cluster or the KAFKA consumer.
In this embodiment, optionally, after determining whether the current call ticket line output rate is smaller than the output rate threshold, the method further includes: and if the current call ticket row output rate is equal to or greater than the output rate threshold value, storing the call ticket row in a preset call ticket row buffer queue.
Specifically, if it is determined that the current call ticket line output rate is equal to or greater than the output rate threshold, the algorithm may return a call ticket line forbidding instruction. A call list buffer queue is preset, forbidden call list lines can be temporarily stored in the call list buffer queue to wait for subsequent release, and the control of the current call list output rate is realized. The method has the advantages that when the output rate of the current call ticket row is equal to or greater than the output rate threshold value, the output of the call ticket row is stopped in time, and the output rate of the current call ticket row is consistent with the average input rate. The output xDR call ticket line number is equal to the average input xDR call ticket line number in a period of time through automatic sensing and automatic synchronization, so that the problem of large data quantity jitter in a single period of time caused by a file xDR data set input mode is solved, and the peak flow of the KAFKA cluster is smoothed. Through the single-line buffer queue, the backlog or loss of the xDR call ticket data can be avoided, and the efficiency and the precision of data transmission are improved.
In this embodiment, optionally, before storing the call ticket row in the preset call ticket row buffer queue, the method further includes: acquiring the current length of a call ticket line buffering queue, and judging whether the current length of the call ticket line buffering queue is equal to or greater than a preset length threshold value; if not, storing the call ticket row in a preset call ticket row buffer queue.
Specifically, a length threshold of the call single line buffer queue is preset, for example, the length threshold of the call single line buffer queue is 10000 call single lines. Before storing the call list in the call list buffer queue, determining the current length of the call list buffer queue. And judging whether the current length is equal to or greater than a length threshold value, namely judging whether a new call ticket line can be stored in the call ticket line buffer queue. And if the current length is equal to or greater than the length threshold value, determining that the call ticket row buffer queue is full and storing a new call ticket row, and directly sending the new call ticket row to the KAFKA cluster or KAFKA consumer by the KAFKA producer. If the current length is smaller than the preset length threshold value, determining that the residual space exists in the call ticket row buffer queue, and storing the call ticket row in the call ticket row buffer queue to wait for subsequent release. The method has the advantages that the length of the call list line buffer queue is prevented from being too long by setting the length threshold of the call list line buffer queue, the use of the memory is guaranteed not to exceed the expected limit, data loss is avoided, and the efficiency and the precision of data transmission are improved.
In this embodiment, optionally, the method further includes: determining whether a call ticket row which is not output exists in a call ticket row buffer queue according to a preset buffer queue check period; if yes, executing a preset speed limit algorithm, determining the current call list output rate in the current data transmission period, and judging whether the current call list output rate is smaller than an output rate threshold value; and if so, sending the call ticket row in the call ticket row buffer queue to the KAFKA cluster.
Specifically, a buffer queue check period may be preset, and for example, a single line of buffer queues may be checked every 2 seconds. Whether the call ticket row which is not output exists in the call ticket row buffering queue can be checked, namely whether the call ticket row exists in the call ticket row buffering queue is determined. If not, the single-line buffer queue of the conversation is not required to be released in a single line; if yes, whether the call ticket row in the single-row buffer queue can be released needs to be determined. S240 can be executed, an output rate threshold value under the current data transmission period is determined, the current single-line output rate under the current data transmission period is determined according to a preset speed limit algorithm, whether the current call ticket line output rate is smaller than the output rate threshold value or not is judged, if yes, the single-line call ticket can be released, and the call ticket lines in the call ticket line buffer queue are sent to the KAFKA cluster; if not, the call list line is continuously kept in the call list line buffer queue. The method has the advantages that the method can timely clear the call list in the call list buffer queue, so that a space can be reserved in the call list buffer queue for storing a new subsequent forbidden call list, the rate control of data transmission is realized, the situation that the call list buffer queue is always occupied to cause data loss is avoided, and the precision and the efficiency of data transmission are improved.
The embodiment of the invention determines the number of the single lines of speech input in the data transmission period by acquiring the call ticket file in the preset data transmission period, and obtains the average input rate of the single lines of speech. The average input rate is used as an output rate threshold value, so that the output rate is consistent with the average input rate, and the call ticket row can be pushed to the KAFKA cluster by the KAFKA producer at a smooth speed. By determining the average input rate to adjust the output rate of the single line, it is ensured that the output rate is maintained in a relatively stable state. Even under the condition that the call ticket file is suddenly increased or suddenly reduced, the call ticket file can be output at an average input rate, and the pressure of an upper application is reduced. If the average input rate changes, the output rate can be adaptively changed, the problem of data backlog caused by the increase of the ticket file is solved, the delay and the loss of service data are avoided, and the efficiency and the precision of data transmission are improved.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a control apparatus for data transmission according to a third embodiment of the present invention, the apparatus being configured on a KAFKA producer. As shown in fig. 3, the apparatus includes:
the phone single line quantity determining module 301 is configured to obtain a phone bill file sent by a data acquisition system in real time, and determine the number of phone single lines in the phone bill file;
an average input rate determining module 302, configured to determine an average input rate of a call ticket row in a preset data transmission period according to the number of the call ticket rows;
and a call ticket row output module 303, configured to output the call ticket row to the KAFKA cluster according to the average input rate.
Optionally, the module 301 for determining the number of call ticket lines includes:
the call ticket file reading unit is used for reading the call ticket file on the hard disk of the data acquisition system in real time;
and the ticket file storage unit is used for transmitting the ticket file to a preset monitoring disk directory and carrying out memory loading operation on the ticket file in the monitoring disk directory.
Optionally, the module 301 for determining the number of call ticket lines further includes:
and the ticket file decoding unit is used for decoding the ticket files in the memory and decoding each line of tickets in the ticket files according to a preset ticket format to obtain the number of single lines of tickets.
Optionally, the call ticket line output module 303 includes:
an output rate threshold determination unit for determining the average input rate as an output rate threshold for a speech line;
the current single-line speech output rate judging unit is used for determining the current single-line speech output rate in the current data transmission period according to a preset speed limit algorithm and judging whether the current single-line speech output rate is smaller than the output rate threshold value or not;
an output unit, configured to output the single file to the KAFKA cluster if yes.
Optionally, the call ticket line output module 303 further includes:
and the single-line call buffering unit is used for storing the call ticket line in a preset call ticket line buffering queue if the current call ticket line output rate is equal to or greater than the output rate threshold value after judging whether the current call ticket line output rate is less than the output rate threshold value.
Optionally, the call ticket line output module 303 further includes:
a buffer queue length judging unit, configured to obtain a current length of the call ticket line buffer queue before storing the call ticket line in a preset call ticket line buffer queue, and judge whether the current length of the call ticket line buffer queue is equal to or greater than a preset length threshold;
if not, storing the call ticket row in a preset call ticket row buffer queue.
Optionally, the apparatus further comprises:
the buffer queue checking module is used for determining whether the call ticket line buffer queue has a call ticket line which is not output according to a preset buffer queue checking period;
the speed judging module is used for executing the current single-line speech output speed under the current data transmission period according to a preset speed limiting algorithm if the current single-line speech output speed is smaller than the output speed threshold value;
and if yes, the call ticket row sending module is used for sending the call ticket rows in the call ticket row buffer queue to the KAFKA cluster.
The embodiment of the invention determines the number of the single lines of speech input in the data transmission period by acquiring the call ticket file in the preset data transmission period, and obtains the average input rate of the single lines of speech. The output rate of the call leg is controlled according to the average input rate so that the call leg can be pushed to the KAFKA cluster by the KAFKA producer at a smooth speed. By determining the average input rate to adjust the output rate of the single line, it is ensured that the output rate is maintained in a relatively stable state. Even under the condition that the call ticket file is suddenly increased or suddenly reduced, the call ticket file can be output at an average input rate, and the pressure of an upper application is reduced. If the average input rate changes, the output rate can be adaptively changed, the problem of data backlog caused by the increase of the ticket file is solved, the delay and the loss of service data are avoided, and the efficiency and the precision of data transmission are improved.
The control device for data transmission provided by the embodiment of the invention can execute the control method for data transmission provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
FIG. 4 shows a schematic block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
In some embodiments, the control method of data transfer may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the control method of data transmission described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured by any other suitable means (e.g., by means of firmware) to perform the control method of the data transfer.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of controlling data transfer, the method being performed by a kaffk KAFKA producer, the method comprising:
acquiring a call ticket file sent by a data acquisition system in real time, and determining the number of call single lines in the call ticket file;
determining the average input rate of the call ticket rows in a preset data transmission period according to the number of the call ticket rows;
outputting the single line of words to a KAFKA cluster according to the average input rate.
2. The method of claim 1, wherein obtaining the ticket file sent by the data acquisition system in real time comprises:
reading a call ticket file on a hard disk of a data acquisition system in real time;
and transmitting the call ticket file to a preset monitoring disk directory, and carrying out memory loading operation on the call ticket file in the monitoring disk directory.
3. The method of claim 2, wherein determining the number of phone bill lines in the phone bill file comprises:
and decoding the call ticket file in the memory, and decoding each line of call tickets in the call ticket file according to a preset call ticket format to obtain the number of call single lines.
4. The method of claim 1, wherein outputting the single line of words to a KAFKA cluster according to the average input rate comprises:
determining the average input rate as an output rate threshold for a single line;
determining the current single-line speech output rate in the current data transmission period according to a preset speed limit algorithm, and judging whether the current single-line speech output rate is smaller than the output rate threshold value or not;
if so, output the single line of words to the KAFKA cluster.
5. The method of claim 4, after determining whether the current single-line output rate is less than the output rate threshold, further comprising:
and if the current single-line call output rate is equal to or greater than the output rate threshold, storing the call ticket line in a preset call ticket line buffer queue.
6. The method of claim 5, wherein before storing the line of tickets in a preset line of tickets buffer queue, further comprising:
acquiring the current length of the call ticket line buffer queue, and judging whether the current length of the call ticket line buffer queue is equal to or greater than a preset length threshold value;
if not, storing the call ticket row in a preset call ticket row buffer queue.
7. The method of claim 5, further comprising:
determining whether a phone bill line which is not output exists in the phone bill line buffer queue according to a preset buffer queue check period;
if yes, executing the preset speed limit algorithm, determining the current single-line output rate of the current speech in the current data transmission period, and judging whether the current single-line output rate is smaller than the output rate threshold value;
and if so, sending the call ticket row in the call ticket row buffer queue to the KAFKA cluster.
8. A control device for data transmission, said device being configured on a KAFKA producer, said device comprising:
the phone single line quantity determining module is used for acquiring a phone bill file sent by a data acquisition system in real time and determining the number of phone single lines in the phone bill file;
the average input rate determining module is used for determining the average input rate of the call ticket rows in a preset data transmission period according to the number of the call ticket rows;
a single line output module to output the single line to a KAFKA cluster according to the average input rate.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of controlling data transmission according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions for causing a processor to implement the control method of data transmission of any one of claims 1-7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210272035.5A CN114722006A (en) | 2022-03-18 | 2022-03-18 | Data transmission control method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210272035.5A CN114722006A (en) | 2022-03-18 | 2022-03-18 | Data transmission control method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114722006A true CN114722006A (en) | 2022-07-08 |
Family
ID=82237396
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210272035.5A Pending CN114722006A (en) | 2022-03-18 | 2022-03-18 | Data transmission control method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114722006A (en) |
-
2022
- 2022-03-18 CN CN202210272035.5A patent/CN114722006A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110808922B (en) | Message processing method and device, storage medium and electronic equipment | |
| CN114095438B (en) | Data transmission method, apparatus, device, storage medium and computer program product | |
| CN116661960A (en) | Batch task processing method, device, equipment and storage medium | |
| CN115237635A (en) | Message management method, device, platform and storage medium | |
| CN114153614A (en) | Memory management method and device, electronic equipment and automatic driving vehicle | |
| CN114722006A (en) | Data transmission control method, device, equipment and storage medium | |
| CN114564149B (en) | Data storage method, device, equipment and storage medium | |
| CN115629903A (en) | Task delay monitoring method, device, equipment and storage medium | |
| CN115967677A (en) | Flow control method, device and equipment for service call and storage medium | |
| CN113360736A (en) | Method and device for capturing internet data | |
| CN115442432B (en) | Control method, device, equipment and storage medium | |
| CN114785740B (en) | Token management and information forwarding method, device, equipment and storage medium | |
| CN114553894B (en) | Data synchronization method, device, system and storage medium | |
| CN117082083B (en) | Data storage method, device and medium based on distributed Internet of things architecture | |
| CN118626233B (en) | Multi-acceleration card multi-task scheduling method and medium based on distributed parallel big model | |
| CN114793216B (en) | Token management and information sending method and device, electronic equipment and storage medium | |
| CN113691823B (en) | Method, device, equipment, storage medium and program product for processing resource file | |
| CN115801763A (en) | File transmission method and device, electronic equipment and storage medium | |
| CN117851045A (en) | Resource scheduling method, device and medium for message data processing | |
| CN117119064A (en) | Request data processing method, device, equipment and storage medium | |
| CN116633881A (en) | Bandwidth control method, device, equipment and medium | |
| CN114661496A (en) | Message processing method, device, electronic equipment, storage medium and product | |
| CN114637584A (en) | Method, device, equipment and medium for adjusting parallelism of data processing task | |
| CN115801718A (en) | Message processing method and device, electronic equipment and message processing system | |
| CN115904550A (en) | Service request processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |