WO2019027476A1

WO2019027476A1 - Augmented diameter protocol

Info

Publication number: WO2019027476A1
Application number: PCT/US2017/045599
Authority: WO
Inventors: Ranjan Sharma; Maryse Gardella
Original assignee: Nokia Solutions And Networks Oy
Priority date: 2017-08-04
Filing date: 2017-08-04
Publication date: 2019-02-07

Abstract

Various communication systems may benefit from an improved Diameter protocol. For example, certain communication systems may benefit from improved transmission of accounting records in a virtualized machine environment using an augmented Diameter protocol. A method, in certain embodiments, may include transmitting at least one accounting record from a charging triggering function to a charging data function. The method may also include receiving an indication at the charging triggering function from the charging data function. In addition, the method may include pausing any action on the at least one accounting record at the charging triggering function until a duration time.

Description

TITLE:

AUGMENTED DIAMETER PROTOCOL

BACKGROUND:

Field:

[0001] Various communication systems may benefit from an improved Diameter protocol. For example, certain communication systems may benefit from improved transmission of accounting records in a virtualized machine environment using an augmented Diameter protocol.

Description of the Related Art:

[0002] Third generation partnership project (3 GPP) Long Term Evolution (LTE) technology, such as IP Multimedia Core Network Subsystem (IMS), may utilize a Diameter protocol to facilitate charging users for consumed network resources. The Diameter protocol may be used during offline charging, in which charging information for network resource usage is collected concurrently with that resource usage. In other words, as a user equipment uses the network, the Diameter protocol may be used by a network element to create an accounting record including a list of network resources consumed by the user equipment. This accounting record is then used by internet service providers (ISPs) or a network operator to bill or charge customers for using the network.

[0003] To help create the accounting record, a network entity may use an integrated charging trigger function (CTF) to send the accounting record while the user equipment consumes network resources. The accounting record can be sent from a CTF to a charging data function (CDF), which is located in the network entity or in a different network entity. The CDF is used to transfer the accounting record to the billing domain of the network operator or the ISP, which may use the accounting record to charge subscribers and/or other operators for the consumed network resources.

[0004] In physical network deployment, solution providers aim to deploy robust hardware with redundancy and resiliency so that end-to-end message delivery related to the accounting records maintains a high level of reliability. The Diameter protocol runs on Transmission Control Protocol (TCP), which provides inherit reliability over connection-less protocols like the User Datagram Protocol (UDP). While the Diameter protocol is still used when network functions are virtualized, the associated reliability of the commodity hardware used for the virtualized network functions does not compare well with the physical deployments. To counter the lower reliability of this commodity hardware, when an error occurs or when functionality of the virtualized network is lost, one or more virtual machines within the virtualized network function can be rebooted, rebuild, and migrated or evacuated to other host(s) to revive them. The above reboot, rebuild, and migrate recovery methods for the virtual machines within the virtualized network function are performed quicker than a comparable recovery method for the physical deployments.

[0005] As discussed above, the CTF in the network entity will send the accounting record to the CDF. When the CTF does not receive an acknowledgement from the CDF, the CTF retransmits the accounting record to the CDF. If an acknowledgment is not received after the retransmissions, the CTF can failover and to an alternative CDF peer. This failover of session accounting records results in partial session reporting to more than one CDF. Each CDF will then generate an incomplete charging data record, which may contain duplicate or partial information that can potentially conflict with the charging data records from another CDF. The billing systems of the ISPs and network operators are unable to reconcile the conflicts created by the incomplete records, and often discard these incomplete charging data records, instead of properly billing users for the network resources they had consumed.

SUMMARY

[0006] According to certain embodiments, an apparatus may include at least one memory including computer program code, and at least one processor. The at least one memory and the computer program code may be configured, with the at least one processor, to cause the apparatus at least to transmit at least one accounting record from a charging triggering function to a charging data function. The at least one memory and the computer program code may also be configured, with the at least one processor, to cause the apparatus at least to receive an indication at the charging triggering function from the charging data function. In addition, the at least one memory and the computer program code may be configured, with the at least one processor, to cause the apparatus at least to pause any action on the at least one accounting record at the charging triggering function for a duration of time.

[0007] A method, in certain embodiments, may include transmitting at least one accounting record from a charging triggering function to a charging data function. The method may also include receiving an indication at the charging triggering function from the charging data function. In addition, the method may include pausing any action on the at least one accounting record at the charging triggering function for a duration time.

[0008] An apparatus, in certain embodiments, may include means for transmitting at least one accounting record from a charging triggering function to a charging data function. The apparatus may also include receiving an indication at the charging triggering function from the charging data function. In addition, the apparatus may include pausing any action on the at least one accounting record at the charging triggering function for a duration of time.

[0009] According to certain embodiments, a non-transitory computer-readable medium encoding instructions that, when executed in hardware, perform a process. The process may include transmitting at least one accounting record from a charging triggering function to a charging data function. The process may also include receiving an indication at the charging triggering function from the charging data function. In addition, the process may include pausing any action on the at least one accounting record at the charging triggering function for a duration of time.

[0010] According to certain other embodiments, a computer program product may encode instructions for performing a process. The process may include transmitting at least one accounting record from a charging triggering function to a charging data function. The process may also include receiving an indication at the charging triggering function from the charging data function. In addition, the process may include pausing any action on the at least one accounting record at the charging triggering function for a duration of time.

[0011] According to certain embodiments, an apparatus may include at least one memory including computer program code, and at least one processor. The at least one memory and the computer program code may be configured, with the at least one processor, to cause the apparatus at least to receive at least one accounting record at a charging data function from a charging triggering function. The at least one memory and the computer program code may also be configured, with the at least one processor, to cause the apparatus at least to send an indication to the charging triggering function from the charging data function. In addition, the at least one memory and the computer program code may also be configured, with the at least one processor, to cause the apparatus at least to process the at least one accounting record for a duration of time.

[0012] A method, in certain embodiments, may include receiving at least one accounting record at a charging data function from a charging triggering function. The method may also include sending an indication to the charging triggering function from the charging data function. In addition, the method may include processing the at least one accounting record for a duration of time.

[0013] An apparatus, in certain embodiments, may include means for receiving at least one accounting record at a charging data function from a charging triggering function. The apparatus may also include means for sending an indication to the charging triggering function from the charging data function. In addition, the apparatus may include means for processing the at least one accounting record for a duration of time.

[0014] According to certain embodiments, a non-transitory computer-readable medium encoding instructions that, when executed in hardware, perform a process. The process may include receiving at least one accounting record at a charging data function from a charging triggering function. The process may also include sending an indication to the charging triggering function from the charging data function. In addition, the process may include processing the at least one accounting record for a duration of time.

[0015] According to certain other embodiments, a computer program product may encode instructions for performing a process. The process may include receiving at least one accounting record at a charging data function from a charging triggering function. The process may also include sending an indication to the charging triggering function from the charging data function. In addition, the process may include processing the at least one accounting record for a duration of time.

BRIEF DESCRIPTION OF THE DRAWINGS:

[0016] For proper understanding of the invention, reference should be made to the accompanying drawings, wherein:

[0017] Figure 1 illustrates a signal flow diagram according to certain embodiments.

[0018] Figure 2 illustrates a signal flow diagram according to certain embodiments.

[0019] Figure 3 illustrates a signal flow diagram according to certain embodiments.

[0020] Figure 4 illustrates a flow diagram according to certain embodiments.

[0021] Figure 5 illustrates a flow diagram according to certain embodiments.

[0022] Figure 6 illustrates a system according to certain embodiments.

DETAILED DESCRIPTION:

[0023] Certain embodiments may provide for an augmented Diameter protocol that may help to prevent the generation of conflicting and/or incomplete charging data records. Some embodiments, for example, may allow the CDF to send an indication, such as a semi-acknowledgment or a half acknowledgment, informing the CTF that it is attempting to process the accounting records. The receiving/sending the indication may initiate a timer at the CTF and/or the CDF. During which the CTF may pause or may be prevented from taking any actions related to the specific accounting records during the duration of the timer. This pausing of future actions by the CTF may allow for the virtualized machine, which may be an integral part of the CDF, enough time to reboot and/or recover to properly process the received accounting record. In some embodiments, pausing the CTF from taking additional actions relating to the accounting records may prevent the creation of incomplete, conflicting, or duplicating accounting records.

[0024] Interactions between the CTF and the CDF may be based on end-to-end reliable delivery of at least one accounting record. The CTF originates the at least one accounting record in order to create an accounting charging record of the user equipment consumption of network resources. The CTF may gather, collect, or accumulate data regarding the consumption of network resources by the user equipment. In certain embodiments the CTF, or the network entity in which the CTF is located, may receive measurements relating to network resource usage directly from the user equipment. Once the at least one accounting record is created, the CTF may transmit the at least one accounting record to the CDF. The CDF may be located either in the same network entity as the CTF or in a different network entity.

[0025] Upon receiving the at least one accounting record, the CDF may attempt to process the at least one accounting record. Processing the at least one accounting record may mean that the CDF attempts to commit or store the received accounting record (ACR) to a memory of the network entity in which the CDF is located. The memory, for example may be a non- volatile memory. In other embodiments, processing the at least one ACR may mean that the CDF is attempting to commit or store the at least one ACR in a database of the network entity in which it resides, or any in the database of a different network entity. Once the at least one accounting record is properly committed or stored, the CDF may send an acknowledgment to the CTF. The acknowledgment may inform the CTF that the at least one accounting record was reliably committed or stored. Based on the received acknowledgment, the CTF may remove/delete/release the at least one accounting record from the outgoing buffer of the CTF. [0026] At any given point in time, the CTF may be in communication with several CDFs, and may wait to receive an indication or an acknowledgment from one or more CDFs. In other words, a single CTF may send the at least one accounting record to one or more CDFs. The CTF may receive one or more acknowledgments for each accounting record sent to the one or more CDFs. In some embodiments, the acknowledgement received by the CTF from the one or more CDFs may be in the form of an accounting answer (ACA), which may include a return code signifying the successful delivery, parsing, and storage of the at least one accounting record properly or correctly at the CDF. In certain embodiments, the ACA may include either an acknowledgment or a negative acknowledgment.

[0027] In other embodiments, the CTF may receive a negative acknowledgment, in the form of an ACA, which includes return codes that signify transient and/or permanent errors that result in an inability to process the at least one accounting record at the one or more CDFs. In response to a ACA including return codes indicating the successful delivery and parsing of the at least one accounting record, the CTF may remove the corresponding at least one accounting record from its outgoing buffer. The CTF may then proceed to send at least one other accounting record waiting to be transmitted to the one or more CDFs.

[0028] The Internet Engineering Task Force (IETF) Request for Comments 6733, titled Diameter Base Protocol, provides that a CTF may wait for an ACA from one or more CDFs. IETF RFC 6733 is hereby incorporated by reference. After the wait period, the CTF may retransmit the same at least one accounting record to the same CDF. The number to retransmissions by the CTF may depend on a specific implementation of the network entity. These retransmissions of the at least one accounting record may be sent along with an indication that the at least one accounting record has previously been transmitted. For example, the indication may be included in a retransmission bit (T-bit) in a Diameter header of the retransmission message including the retransmitted at least one accounting record.

[0029] In certain embodiments, the CTF may retransmit the at least one accounting record for a maximum number of times, and still not receive a response. The maximum number of times the CTF may retransmit the accounting record may depend on the implementation. The CTF may then failover to an alternative peer, which may be another CDF that is known to the CTF. In some other embodiments, the CTF may discover another CDF via a fully qualified domain network (FQDN) resolution from a domain name system (DNS). The failover to another CDF, as well as the retransmission of the at least one accounting record to the initial CDF, may result in incomplete or conflicting accounting recordings that are split amongst different CDFs. In other words, a single CDF may not possess a comprehensive accounting record that includes all of the network resources used by a given user. Consequently, each CDF generates incomplete accounting records.

[0030] Due to the splitting of the accounting records amongst two or more different CDFs, the downstream billing system, to which the CDF forwards the accounting records, is expected to reconcile the different accounting records and generate a single complete accounting record. However, there is no way to practically extricate all of the different accounting records from the two or more CDFs into a single comprehensive accounting record. Because the split accounting records may include duplicative information in some containers, it cannot be determined by the downstream billing system which containers include redundant information.

[0031] In other words, some of the entries within the accounting records may be erroneously duplicated. The CDFs have no mechanism to make the billing system aware of the erroneous duplication included in the forwarded accounting records. In order to prevent charging customers for the duplicative accounting records, ISPs and network operators simply discard these incomplete accounting records, leading to a loss of revenue.

[0032] In physical network deployment, which does not utilize virtual machines to perform various functions, the various network entities used were compliant with industry standards to provide a high level of availability and resilience. When virtualizing their networks, however, network operators utilize commodity hardware to operate the virtualized machines. In come embodiments, the CDF and/or the CTF may be virtualized network functions operating on virtual machines. The commodity hardware, on which the virtualized machines operate, does not have to comply with industry standards, and may have a lower resiliency.

[0033] Some embodiments may depend on a consensus protocol, or another similar protocol or variation thereof, such as Paxos or Raft. An operating condition of the consensus protocol may be that greater than 50% of the nodes are available. The consensus protocol may also work better with an odd number of starting nodes, for example 2N+1. In certain embodiments, the component that is dependent on the consensus protocol, for example a database used by the CDF, may be the same component that imparts reliability to the storage of accounting records. A further discussion regarding the use of the database by the CDF is presented below.

[0034] In other protocols, which are applied in virtual machines, for example an OpenStack protocol, the nodes may reside in different availability zones (AZs). For example, a network may have two AZs, a first AZ may include N+l nodes, while a second AZ may include N nodes. When the first AZ is lost, the cluster of nodes stops selecting a master node, because it loses more than 50% of its nodes. The virtualization of the network, allows for the creation of a new virtual machine or the rebooting of a failed virtual machine within a short amount of time. In certain embodiments, the reboot or recovery of the virtual machine may be completed within 10 seconds. This quick recovery or reboot may allow for a quick recovery of lost nodes in the first AZ, and a restoration of the needed 50% nodes threshold.

[0035] It may therefore be helpful for the Diameter protocol to be augmented to have a timed wait state, in which the CTF pauses future actions related to the accounting record, to allow for the recovery of failed virtualized machines or to allow new virtualized machines to be created. During the time the CTF pauses, the CDF may recover and properly process the received at least one accounting record, thereby alleviating the need for the retransmission of the at least one accounting record. In certain embodiments, the CDFs may use a database to store the received at least one accounting record. To provide for reliable data persistence, the CDF may have the ability to check-point data records in nonvolatile disks, for example. The ability to check-point data records maybe similar to Cinder Volume in OpenStack. In some other embodiments, the CDF may be able to create a replica of the received data, and follow a master-slave replication strategy for storing the records.

[0036] In some embodiments, the database may be distributed across multiple virtual machines, which are placed in more than one availability zone so that failure of a virtual machine or a complete availability zone does not affect the data availability. The database may, in certain embodiments, meet at least 2 of the following 3 criteria: consistency, availability, and partition tolerance. Consistency may mean that each read gets the most recent value from the latest write, and availability may mean that each read may be answered with a non-error response without the guarantee that the most recent value from the latest write is retired in the response. Partition tolerance may mean that the system operates despite losses and inter-node communication delays.

[0037] In certain embodiments, the database may meet the consistency and availability requirements, via data sharing and master-slave replication across OpenStack AZs. However, the database may not be able to meet the partition tolerance criteria, and the database cannot continue after a sufficient number of nodes are lost. Distributed databases, on the other hand, use a consensus protocol that elects a master node or a cluster manager that keeps information about sharing, availability of nodes, identification of master and slave shards of the database.

[0038] The consensus protocol, for example, may be a Paxos, Raft, or any other variant. The protocol may ensure that a master node can be selected if at least N+ 1 nodes are available in a cluster of 2N+1 nodes. The master node, which may only be chosen upon meeting of a quorum, may be responsible for promoting a slave shard to a master shard, in case the master shard for the data set is lost due to a failure of a host or a virtual machine on the host. The creating, reading, updating, and deleting (CRUD) operations of the virtual machines may be carried out on the master shards, and the master shards may be responsible for replication at the salve shards.

[0039] The following is an example in which a CDF cannot properly process received accounting records due to a database quorum loss resulting from an availability zone loss. The database may be referred to as memory below. In one embodiment, five nodes may be selected to represent the distributed database (2n+l = 5, or n=2). This node count may be fault tolerant up to an N number of nodes. A database may therefore sustain its functionality up to the point where 3 functional nodes remain. Because the database may be distributed, the nodes are deployed in two availability zones. In order for the database to remain operative, the quorum of 3 nodes, which is greater than 50% of the total nodes, has to be maintained.

[0040] From the operator's perspective, the nodes are distributed across at least two availability zones such that one AZ includes N nodes and the other AZ includes N+l nodes. In the operational node, the loss of the AZ including N+l nodes leaves only the other AZ including N operational nodes, which means that the quorum is lost. In other words, if there are any slave shards in the remaining N nodes, they cannot be promoted to master status, and the CRUD operations on the data set may no longer be possible.

[0041] With the database no longer operational, the CDF cannot reliably commit or store the incoming accounting records into persistent memory provided by the database, and must refuse the incoming at least one accounting record. For ongoing charging sessions, for which at least one accounting record start and one or more accounting record interim messages have already been received at the CDF, and responded to via the ACA, the database quorum loss would lead to generation of incomplete accounting records, as the CDF would refuse subsequent messages for the accounting session. The sending CTF may then failover to another CDF, different thank the original CDF, and transmit the accounting records to the another CDF.

[0042] In addition to the above embodiments involving loss of quorum, other possible reasons for the inability of the CDF to properly process the at least one accounting record may be, for example, a temporary unavailability of the database writer or congestion at the CDF. The CTF may then failover to another or a different CDF. As discussed above, this may result in the generation of incomplete accounting records, and cause revenue loss for the ISP or network operator. The downstream billing system is not able to mediate or reconcile the incomplete accounting records in a manner that is non-trivial and not error-prone.

[0043] The Diameter protocol was developed with the expected resilience of the transport protocol in mind, with TCP providing the requisite failsafe mechanism for the transport. The protocol also demands a per message acknowledgment, address retransmission, and may allow for failover when necessary. The implementation of the network using virtualized machines, however, erodes the expected resiliency of the Diameter protocol. When a CDF sends an acknowledgement, via an ACA, the CTF may discard the accounting record in the outgoing buffer and move on to process the next messages in the outgoing buffer.

[0044] When the CTF does not receive the ACA from the CDF, the CTF retransmits the at least one accounting record to the CDF for a configurable number of maximum times. After the configurable number of times, the CTF may execute a failover to another CDF. This indicates a binary nature of the response mechanism, in which the ACA is neither received nor not received by the CTF. In certain embodiments, therefore, the CDF may send or transmit a third state acknowledgments, referred to as an indication, which indicates to the CTF whether the message including the at least one accounting record has been reliability received or not.

[0045] Some embodiments allow for an augmented Diameter protocol in which a CDF, or a network entity in which the CDF resides, transmits to the CTF an indication in the form of a semi-acknowledgment or a half acknowledgment. In other words, the indication transmitted to the CTF from the CDF may indicate that while the at least one accounting record has been received at the CDF, the at least one accounting record is currently being processed by the CDF. As such, the indication informs that the CTF that the CDF is attempting to properly process the received at least one accounting record. Being properly processed means that the CDF has been reliably committed or stored the at least one account record to the database or the memory, for example a non- volatile memory.

[0046] Certain embodiments may augment the Diameter protocol to include an indication, such as a semi-acknowledgment or a half acknowledgment. When the CTF receives the indication from the CDF the CTF may initiate or trigger a timer. A similar or identical timer may be kept at the CDF. The CTF, or the network entity in which the CTF is located, may utilize a differential buffer, in some embodiments. The buffer may differentiate between those at least one accounting records for which an indication has been received from the CDF, and those at least one accounting records for which no indication has been received.

[0047] Once the indication is received by the CTF from the CDF, the CTF may become aware that the CDF cannot currently commit or store the at least one accounting record to the database or the non-volatile memory, but that the CDF, or the network entity in which the CDF resides, is "trying" to process the at least one accounting record. The purpose of the indication is to indicate to the CTF not to retransmit the at least one accounting record either to the original CDF, from which the indication was received, or to another/alternative CDF. In addition, since the indication is a non-final response, it also instructs the CTF not to discard the message from the outgoing buffer. The indication, as such, initiates or triggers the timer, and the CTF knows to retain the message in an outgoing buffer for a time period controlled by a per-message timer. The timer, in other words, may be specific to the accounting record sent from the CTF to the CDF.

[0048] The length of the timer, in certain embodiments, may be dependent on the ongoing traffic and/or a count of accounting records per average session reported. For example, if the CDF is expected to handle 100 thousand accounting records per second, the normal buffer size may be derived based on the average session holding time. The average session holding time may then be increased to account for the percentage of accounting records that are held in memory for a longer time. Certain embodiments may balance the amount of memory versus how long can we hold the accounting records in memory. In some other embodiments, the length of the timer may be based on the average reboot time of a virtual machine. With a 30 second average reboot time, the buffer may be set to at least two or three times that interval, with the virtual machine memory size being accounted for as well.

[0049] When the network functions are performed or conducted in a virtualized environment, the recovery mechanism for a virtualized node may be quicker than the recovery mechanism for a physical node. During the duration of the timer, the database or the non-volatile memory may recover, and the CDF can properly commit and/or store the at least one accounting record in the database and/or memory. The CDF may then signal to the CTF to release the at least one accounting record from the outgoing buffer of the CTF. Once the CDF has acknowledged that it properly processed the at least one accounting record, the CTF can remove the message from the outgoing buffer.

[0050] In certain embodiments, the recovery of the database or memory, which may mean the recovery or reboot of the virtualized machine, may not occur during the duration of the timer at the CTF and/or the CDF. When the timer expires, a mutual agreement may occur between the CTF and the CDF, and the CTF may proceed to a failover procedure. Instead of retransmitting the at least one accounting record to the same CDF, however, the CTF may transmit the at least one accounting record to another or an alternative CDF, which is different from the original CDF. Because of the augmented Diameter protocol, the CTF may avoid retransmitting the at least one accounting record to the CDF, thereby minimizing the number of incomplete or partial accounting records received at the billing system.

[0051] Figure 1 illustrates a flow diagram according to certain embodiments. As shown in the specific embodiment of Figure 1, the CDF may properly process the at least one accounting record by storing the received at least one accounting record in the database or the memory. The proper processing may occur during the given duration of the timer. In step 110, a network entity (NE) in which the CTF resides 101 may sent a message to CDF 102. The CTF may be remote, in some embodiments, meaning that the CTF may reside in a different network entity than the CDF. The message may include at least one accounting record. The at least one accounting record may include information about the network resource utilization of a user on a per session basis.

[0052] In step 110, CDF 102 receives the message including the at least one accounting record on the radio frequency reference point supported by a Diameter front-end. The CDF 102 may then attempt to process the received message to a back-end database or memory 103, in step 111. In other words, CDF 102 may attempt to store and/or commit the at least one accounting record to database or memory 103. Database (DB) 103 may be located in the same network entity in which CDF 102 is location or a different network entity. When the processing is successful, DB 103 may send an acknowledgement to CDF 102, in step 112. In step 113, CDF 102 may respond with an acknowledgment in the form of an ACA to CTF 101. Upon receiving the ACA, CTF 101 may remove the at least one accounting record from its outgoing buffer or queue of messages. CTF 101 may then transmit subsequent messages and/or accounting records.

[0053] In some embodiments, CTF 101 may send multiple messages from its outgoing buffer one after another, within a short period of time. For example, if the CTF has a buffer size of ten thousand messages, then all ten thousand messages may be transmitted. CTF 101 may then clear the transmitted messages from the outgoing buffer upon receiving a positive acknowledgement from CDF 102. The Diameter protocol may provide for a per-message acknowledgment. When the ACA is sent back to CTF 101, it is accompanied by a Diameter code that specifies one of the following results: lxxx - Informational, 2xxx - Success, 3xxx - Protocol Errors, 4xxx - Transient Errors, 5xxx - Permanent Errors. In step 114, CTF 101 may send a message including the at least one accounting record to CDF 102. CDF 102 may then attempt to commit the at least one accounting record to non- volatile memory, meaning that CDF 102 attempts to store the at least one accounting record in DB 103 or non- volatile memory in step 115.

[0054] However, as seen in Figure 1, the DB quorum may be lost. For example, if DB 103 is spread between two AZs, and three or more nodes are lost in a first AZ, CDF 102 may be unable to write, store, and/or commit the at least one accounting record in DB or non-volatile memory 103. In step 116, a non-response for the DB write from step 115 may be indicated. . In step 117, the Diameter front- end, CDF 102, may provide a transient error indication to CTF 101, since the at least one accounting record cannot be committed to stable storage due to the loss of quorum.

[0055] In certain embodiments, in order to overcome the deficiencies of message 4002, in step 117 message 4004 Diameter Queued for Write may be transmitted from CDF 102 to CTF 101. Message 4004 may be related to CDF 102 receiving the at least one accounting record, and may indicate that CDF 102 is unable to write, store, and/or commit the at least one accounting record to stable storage. CDF 102 may continue to process the at least accounting record, meaning that CDF 102 may continue to retry or reattempt to store the at least one accounting record in the non-volatile memory or the database. Message 4004 may include an indication to CTF 101 to pause any action associated with the at least one corresponding accounting record. For example, the indication may indicate to CTF 101 that the at least one accounting message sent to CDF 102 should not be dequeued from the CTF' s 101 outgoing buffer. The indication may also command CTF 101 not to retransmit the at least one accounting record to the same CDF or a different CTF as a retransmission. In other words, the CTF retains the at least one accounting records in its outgoing queue or buffer. Two timers may then be initiated at CTF 101 and CDF 102.

[0056] In some embodiments, a first timer (Tl) may be started at CDF 102, when ACA including message 4004, is sent to CTF 101. A second timer (T2) timer may be started at CTF 101 upon receiving message 4004 from CDF 102. Both timers Tl and T2 may be started on the CDF and the CTF on a per-message or a per accounting record basis. The duration of the timers may be configurable or determined by the ISPs or the network operators. First timer Tl may define the duration for which the CDF should attempt to process the at least one accounting record, while second timer T2 may define the duration in which the CTF maintains the at least one accounting record in its outgoing queue/buffer.

[0057] The configured duration the timers, in certain embodiments, may be commensurate with the network operator's ability to restore the database or memory quorum. In some embodiments there may be an N: 1 cardinality rule between CTFs and CDFs, meaning that for every one CTF there may be more than one CDF, Tl may be set to a smaller duration than T2. The following equation may, in some embodiments, therefore apply: Tl < T2. For example, Tl may be 15 minutes while T2 may be 20 minutes. The timer may be predetermined by the ISP or the network operator, and the duration of the timer may be pre- stored in the CTF and/or CDF.

[0058] As shown in Figure 1, after receiving AC A in step 117, CTF 101 is aware that the at least one accounting record is being processed by CDF 102, and that CTF 101 should not retransmit the message to the same or another Diameter peer or the same or another CDF. Given that thousands of sessions may be in progress at any time, certain embodiments may refine the Diameter protocol so that only those messages which may result in incomplete at least one accounting record are treated according to the above described embodiments, such as the embodiment shown in Figure 1. In some embodiments at least one interim accounting record and at least one accounting record stops may be subject to the above embodiments. In certain embodiments, an attribute value pair (A VP), referred to as an accounting record type, may be included in the accounting record. The accounting record type may define the following type of records: 0 - Event, 1 - Start, 2 - Interim, 3 - Stop. Type 2 and type 3 records may utilize a type 4004 message.

[0059] In some other embodiments, the 4004 Diameter return code may be included in an ACA-interim or ACA-stop message. In this embodiments, the CTF may hold the at least one accounting record in its outgoing buffer or queue. In addition, the 4004 Diameter return code may not retransmit the at least one accounting records from CTF 101. CTF 101 also starts a count-down timer, for example second timer T2, with a duration configured by the operator for the message or the at least one accounting record. The at least one accounting records may be kept in the outgoing buffer of the CTF while the CDF is processing the at least one accounting record. [0060] A shown in Figure 1, the quorum of the virtual nodes in the first availability zone may be restored or recovered before first timer Tl expires. In step 118, CDF 102 may properly store the at least one accounting record in DB or non- volatile memory 103. An acknowledgment may then be sent from DB 103 to CDF 102. CDF 102 may then send a positive acknowledgement, for example ACA, via 2xxx, in step 120, and CTF 101 can dequeue the corresponding accounting message from its buffer. The acknowledgment may also be received by CTF 101 within the duration of timer T2.

[0061] When too many messages accumulate at CTF 101 and/or at CDF 102, which may occur when the database quorum is not restored for a prolonged period, then another defensive measure may be taken. In certain embodiments, when the timer expires the CDF may discard the accumulated messages waiting to be committed to memory. The CDF may then indicate to the CTF to send the messages in the buffer to the alternative peers or CDFs, with the retransmission bit being unset in the Diameter header. In such an embodiments, a failure message may then be sent to the CTFs.

[0062] Figure 2 illustrates a flow diagram according to certain embodiments. Steps 210-217 in Figure 2 may be similar to steps 110-117 in Figure 1, including CTF 201, CDF 202, and DB or non- volatile memory 203. Unlike Figure 1, however, message 4005 Diameter Queue Write Timeout in Figure 2 may be sent from CDF 202 to CTF 201 in step 218. Message 4005 may indicate to a CTF 201, or the network entity in which CTF 201 resides, that the CDF 202 has not been able to store/commit the at least one accounting record to DB 203, but that the duration of first timer Tl has expired. CTF 201 may then be permitted to retransmit the at least one accounting record to CDF 202 or to an alternative or another CDF, as shown in Figure 3. Since duration of first timer Tl is set to a shorter duration than second timer T2, the ACA sent in step 218, which may include message 4005, may trigger the early expiration of second timer T2. Upon sending the type 4005 message in step 218, CDF 202 may also release the message from its own memory.

[0063] Figure 3 illustrates a flow diagram according to certain embodiments. Steps 310-317 in Figure 3 may be similar to steps 110-117 in Figure 1, including CTF 301, CDF1 302, and database 303. As shown in Figure 3, the virtual machine running CDF 1 302 may fail momentarily near the expiration of one or more Tl timers, meaning that it may be unable to indicate a disposition to CTF 301 for the queued records. After both the durations of first timer Tl and second timer T2 expire, CTF 301 may then assume an implicit release, and retransmit the at least one accounting record to an alternative or another CDF 304, also referred to as CDF2, in step 318. The at least one accounting record is not retransmitted to CDF 302, also referred to original CDF1, instead being retransmitted to CDF 304 as part of a failover. In step 319, an acknowledgement, in the form of an ACA message, may be sent from CDF 304 to CTF 301. The embodiment shown in Figure 3 prevents CTF 301 from retransmitting the at least one accounting record to a CDF that has previously been unable to properly process the at least one accounting record.

[0064] Figure 4 illustrates a flow diagram according to certain embodiments. Specifically, Figure 4 illustrates an embodiment of a CTF or a network entity including the CTF. As discussed above, the CTF may be a virtualized network function operated in one or more virtual machines hosted on one or more compute resources. In step 410, the CTF may transmit at least one accounting record to a CDF. The CDF may then attempt to process the at least one accounting record. In certain embodiments, the CTF may receive an indication from the CDF, as shown in step 420. The indication may be a semi-acknowledgement or a half acknowledgment that indicates to the CTF that the at least one accounting record has been received at the charging data function, but that the at least one accounting record is being processed at the CDF. Being processed or processing the at least one accounting record may mean that the CDF is attempting to store, commit, or write the at least one accounting record to a database or a memory, for example, a non- volatile memory.

[0065] Upon receiving the indication, the CTF in certain embodiments may initiate a timer. The CTF may pause any action relating to the at least one accounting record for a duration of time, as shown in step 430. The duration of time may be determined by the timer. In other words, the CTF may retain the at least one accounting record in an outgoing buffer of the CTF during the duration of the timer. In certain embodiments, the CTF may receive an acknowledgement from the CDF that the at least one accounting record has been properly stored or committed to memory (as shown in Figure 1). The acknowledgement may end the duration of the timer. As shown in step 440, the CTF may then release the at least one accounting record from the outgoing buffer/queue, since the at least one accounting record was properly stored or committed to the memory or the database by the CDF. In such embodiments, the timer at the CTF and/or CDF may have a duration that is greater than a recovery time of the virtualized node in which the CDF resides.

[0066] In certain other embodiments, the CTF may receive a negative acknowledgement from the CDF that the at least one accounting record has not been properly committed to memory (as shown in Figure 2). The negative acknowledgement may end the duration of the timer. As shown in step 450, the CTF may retransmit the at least one accounting record to the CDF. In other embodiments, the CTF may transmit the at least one accounting record to another CDF as part of a failover, as shown in step 460. In some other embodiments, upon the expiration of the timer the CTF may either retransmit the at least one accounting record to the CDF, or another CDF as part of a failover (as shown in Figure 3), or release the at least one accounting record from the outgoing buffer without retransmitting the at least one accounting record.

[0067] Figure 5 illustrates a flow diagram according to certain embodiments. Specifically, Figure 5 illustrates an embodiment of a CDF or a network entity including the CDF. As discussed above, the CDF may be a virtualize network function operated in a virtual machine. In step 510, the CDF may receive at least one accounting record from the CTF. The CDF may then send an indication to the CTF, as shown in step 520. In certain embodiments, the indication may initiate a timer. The timer may be initiated at the CTF and/or the CDF. In step 530, the CDF processes the at least one accounting report for a duration of time. The duration of time may be determined by the timer. Processing may mean attempting to store or commit the at least one accounting report to a memory or a database.

[0068] In certain embodiments, the CDF may send an acknowledgement to the CTF that the at least one accounting report has been properly processed, as shown in step 540 and in Figure 1. The duration of time may have a greater duration than a recovery time of a virtualized node in which the charging data function resides. In step 550, the CDF may send a negative acknowledgement from the CTF that the at least one accounting report has not been properly processed (as shown in Figure 2). The negative acknowledgement ends the duration of the timer. In certain embodiments, the CDF may then receive a retransmission of the at least one accounting record from the CTF. In other embodiments, the CTF upon receiving the negative acknowledgement may transmit the at least one accounting record to another CDF, as shown in step 460 of Figure 4.

[0069] Figure 6 illustrates a system according to certain embodiments. It should be understood that each signal or block in Figures 1, 2, 3, 4, and 5, may be implemented by various means or their combinations, such as hardware, software, firmware, one or more processors and/or circuitry. In one embodiment, a system may include several devices, such as, for example, a network entity 620 or a user equipment 610. The system may include more than one UE 610 and more one network node 620, although only one access node shown for the purposes of illustration. The network entity may be a network node, an access node, a base station, a 5G NodeB (5G-NB), server, host, or any of the other access or network node discussed herein. The network entity 620 may include at least one CTF and/or at least one CDF. The CTF and/or the CDF included in the network entity 620 may be virtualized functions performed by a virtual machine operated in network entity 620.

[0070] Each of these devices may include at least one processor or control unit or module, respectively indicated as 611 and 621. At least one memory may be provided in each device, and indicated as 612 and 622, respectively. The memory may include computer program instructions or computer code contained therein. One or more transceiver 613 and 623 may be provided, and each device may also include an antenna, respectively illustrated as 614 and 624. Although only one antenna each is shown, many antennas and multiple antenna elements may be provided to each of the devices. Higher category UEs generally include multiple antenna panels. Other configurations of these devices, for example, may be provided. For example, network node 620 and UE 610 may be additionally configured for wired communication, in addition to wireless communication, and in such a case antennas 614 and 624 may illustrate any form of communication hardware, without being limited to merely an antenna.

[0071] Transceivers 613 and 623 may each, independently, be a transmitter, a receiver, or both a transmitter and a receiver, or a unit or device that may be configured both for transmission and reception. In other embodiments, the UEs or the network node may have at least one separate receiver or transmitter. The transmitter and/or receiver (as far as radio parts are concerned) may also be implemented as a remote radio head which is not located in the device itself, but in a mast, for example. The operations and functionalities may be performed in different entities, such as nodes, hosts or servers, in a flexible manner. In other words, division of labor may vary case by case. One possible use is to make a network node deliver local content. One or more functionalities may also be implemented as virtual application(s) in software that can run on a server. A beamformer may be a type of transceiver.

[0072] A user device or user equipment 610 may be a mobile station (MS) such as a mobile phone or smart phone or multimedia device, a computer, such as a tablet, provided with wireless communication capabilities, personal data or digital assistant (PDA) provided with wireless communication capabilities, portable media player, digital camera, pocket video camera, navigation unit provided with wireless communication capabilities or any combinations thereof. The CTF in network entity 620 may collect and/or receive network resource usage information from user equipment 610, which the CTF may then use to create the at least one accounting report.

[0073] In some embodiments, an apparatus, such as a network entity, may include means for carrying out embodiments described above in relation to Figures 1, 2, 3, 4, and 5. In certain embodiments, at least one memory including computer program code can be configured to, with the at least one processor, cause the apparatus at least to perform any of the processes described herein.

[0074] Processors 611 and 621 may be embodied by any computational or data processing device, such as a central processing unit (CPU), digital signal processor (DSP), application specific integrated circuit (ASIC), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), digitally enhanced circuits, or comparable device or a combination thereof. The processors may be implemented as a single controller, or a plurality of controllers or processors.

[0075] For firmware or software, the implementation may include modules or unit of at least one chip set (for example, procedures, functions, and so on). Memories 612 and 622 may independently be any suitable storage device, such as a non-transitory computer-readable medium. A hard disk drive (HDD), random access memory (RAM), flash memory, or other suitable memory may be used. The memories may be combined on a single integrated circuit as the processor, or may be separate therefrom. Furthermore, the computer program instructions may be stored in the memory and which may be processed by the processors can be any suitable form of computer program code, for example, a compiled or interpreted computer program written in any suitable programming language. The memory or data storage entity is typically internal but may also be external or a combination thereof, such as in the case when additional memory capacity is obtained from a service provider. The memory may be fixed or removable.

[0076] The memory and the computer program instructions may be configured, with the processor for the particular device, to cause a hardware apparatus such as network entity 620 or UE 610, to perform any of the processes described above (see, for example, Figures 1, 2, 3, 4, and 5). Therefore, in certain embodiments, a non-transitory computer-readable medium may be encoded with computer instructions or one or more computer program (such as added or updated software routine, applet or macro) that, when executed in hardware, may perform a process such as one of the processes described herein. Computer programs may be coded by a programming language, which may be a high- level programming language, such as objective-C, C, C++, C#, Java, etc., or a low-level programming language, such as a machine language, or assembler. Alternatively, certain embodiments may be performed entirely in hardware.

[0077] Furthermore, although Figure 6 illustrates a system including a network entity 620 and UE 610, certain embodiments may be applicable to other configurations, and configurations involving additional elements, as illustrated and discussed herein. For example, multiple user equipment devices and multiple network entities may be present, or other nodes providing similar functionality, such as nodes that combine the functionality of a user equipment and an network entity, such as a relay node. The UE 610 may likewise be provided with a variety of configurations for communication other than communication network node 720. For example, the UE 610 maybe configured for device-to-device, machine to machine, or vehicle-to-vehicle communication.

[0078] The above embodiments may provide for significant improvements to the functioning of a network and/or to the functioning of the network entities within the network. Specifically, certain embodiments can augment the Diameter protocol in order to reduce the number of incomplete accounting reports in the network. The network may include virtualized machines that perform the functions of the CTF and/or the CDF. In some of the above embodiments, the CTF may receive an indication that may prevent the CTF from repeatedly retransmitting to the CDF accounting reports. Reducing the number of incomplete accounting reports will allow ISPs and network operators to more accurately account and charge for network resource usage, while minimizing lost profits stemming from incomplete accounting reports.

[0079] The features, structures, or characteristics of certain embodiments described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, the usage of the phrases "certain embodiments," "some embodiments," "other embodiments," or other similar language, throughout this specification refers to the fact that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. Thus, appearance of the phrases "in certain embodiments," "in some embodiments," "in other embodiments," or other similar language, throughout this specification does not necessarily refer to the same group of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0080] One having ordinary skill in the art will readily understand that the invention as discussed above may be practiced with steps in a different order, and/or with hardware elements in configurations which are different than those which are disclosed. Therefore, although the invention has been described based upon these preferred embodiments, it would be apparent to those of skill in the art that certain modifications, variations, and alternative constructions would be apparent, while remaining within the spirit and scope of the invention.

[0081] Partial Glossary

[0082] 3 GPP Third generation partnership project

[0083] LTE Long Term Evolution

[0084] IMS IP Multimedia Core Network Subsystem

[0085] CTF Charging Trigger Function

[0086] CDF Charging Data Function

[0087] TCP Transmission Control Protocol

[0088] UDP User Datagram Protocol

[0089] ACR Accounting Record

[0090] ACA Accounting Answer

[0091] IETF Internet Engineering Task Force

[0092] AZs Availability Zones

[0093] CRUD Creating, Reading, Updating, and Deleting Operations

[0094] NE Network Entity

Claims

WE CLAIM:

1. A method comprising:

transmitting at least one accounting record from a charging triggering function to a charging data function;

receiving an indication at the charging triggering function from the charging data function; and

pausing any action relating to the at least one accounting record at the charging triggering function for a duration of time.

2. The method according to claim 1, wherein the receiving of the indication initiates a timer, and wherein the duration of time is determined by the timer.

3. The method according to claim 1 or 2, wherein the indication is placed in a Diameter header of the at least one accounting record.

4. The method according to any of claims 1-3, wherein the indication is a semi-acknowledgement or a half acknowledgment that indicates that the at least one accounting record has been received at the charging data function, but that the at least one accounting record is still being processed by the charging data function.

5. The method according to any of claims 1-4, further comprising:

retaining the at least one accounting record in an outgoing buffer of the charging triggering function during the duration of time.

6. The method according to any of claims 1-5, at least one of the charging triggering function or the charging data function is located in a virtualized node.

7. The method according to any of claims 1-6, further comprising:

receiving an acknowledgement at the charging triggering function from the charging data function that the at least one accounting record has been properly committed to reliable memory; and

releasing the at least one accounting record from the outgoing buffer of the charging triggering function.

8. The method according to claim 7, wherein the duration time is greater than a recovery time of a virtualized node in which the charging data function resides.

9. The method according to any of claims 1-8, further comprising:

receiving a negative acknowledgement at the charging triggering function from the charging data function that the at least one accounting record has not been properly stored or committed to a database or a memory, wherein the negative acknowledgement ends the duration of time.

10. The method according to claim 9, further comprising at least one: retransmitting the at least one accounting record from the charging triggering function to the charging data function; or

transmitting the at least one accounting record from the charging triggering function to another charging data function as part of a failover.

11. The method according to any of claims 1-10, further comprising: releasing the at least one accounting record from the outgoing buffer of the charging triggering function upon expiration of the duration of time.

12. The method according to any of claims 1-11, further comprising at least one:

retransmitting the at least one accounting record from the charging triggering function to the charging data function upon expiration of the duration of time; or

transmitting the at least one accounting record from the charging triggering function to another charging data function as part of a failover upon expiration of the duration of time.

13. A method comprising:

receiving at least one accounting record at a charging data function from a charging triggering function;

sending an indication to the charging triggering function from the charging data function; and

processing the at least one accounting record for a duration time.

14. The method according to claim 13, wherein the sending of the indication initiates a timer, and wherein the duration of time is determined by the timer.

15. The method according to claim 13 or 14, wherein the indication is a semi-acknowledgement or a half acknowledgment that indicates that the at least one accounting record has been received at the charging data function, but that the at least one accounting record is still being processed.

16. The method according to any of claims 13-15, wherein the processing comprises attempting to commit or store the at least one accounting report to a memory or a database.

17. The method according to any of claims 13-16, wherein the charging data function does not receive an additional accounting record during the duration of the timer.

18. The method according to any of claims 13-17, at least one of the charging triggering function or the charging data function is located in a virtualized node.

19. The method according to any of claims 13-18, further comprising: sending an acknowledgement from the charging data function to the charging triggering function that the at least one accounting record has been properly processed.

20. The method according to claim 19, wherein the duration of time is greater than a recovery time of a virtualized node in which the charging data function resides.

21. The method according to any of claims 13-20, further comprising: sending a negative acknowledgement from the charging data function to the charging triggering function that the at least one accounting record has not been properly processed, wherein the negative acknowledgement ends the duration of time.

22. The method according to claim 21, further comprising:

receiving a retransmission of the at least one accounting record from the charging triggering function at the charging data function.

23. An apparatus comprising:

at least one processor; and

at least one memory including computer program code,

wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform a process, the process including the method according to any of claims 1-22.

24. A non-transitory computer-readable medium encoding instructions that, when executed in hardware, perform a process, the process including the method according to any of claims 1-22.

. An apparatus comprising means for performing a process, the process including the method according to any of claims 1-22.

26. A computer program product encoding instructions for performing a process, the process including the method according to any of claims 1-22.

27. A computer program product embodied in a non-transitory computer- readable medium and encoding instructions that, when executed in hardware, perform a process, the process including the method according to any of claims 1-22.