EP4690899A1

EP4690899A1 - Methods, apparatus and computer programs

Info

Publication number: EP4690899A1
Application number: EP23716495.9A
Authority: EP
Inventors: Alvaro VALCARCE RIAL
Original assignee: Nokia Solutions and Networks Oy
Current assignee: Nokia Solutions and Networks Oy
Priority date: 2023-03-31
Filing date: 2023-03-31
Publication date: 2026-02-11
Also published as: WO2024199670A1

Abstract

A network node apparatus for a cellular network, the network node apparatus comprising: means for receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate across user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; the apparatus comprising: means for training a neural network based on the information.

Description

METHOD, APPARATUS AND COMPUTER PROGRAM FIELD The present application relates to a method, apparatus, system and computer program. BACKGROUND A communication system can be seen as a facility that enables communication sessions between two or more entities such as user terminals, base stations and/or other nodes by providing carriers between the various entities involved in the communications path. A communication system can be provided for example by means of a communication network and one or more compatible communication devices. The communication sessions may comprise, for example, communication of data for carrying communications such as voice, video, electronic mail (email), text message, multimedia and/or content data and so on. Non- limiting examples of services provided comprise two-way or multi-way calls, data communication or multimedia services and access to a data network system, such as the Internet. In a wireless communication system at least a part of a communication session between at least two stations occurs over a wireless link. Examples of wireless systems comprise public land mobile networks (PLMN), satellite-based communication systems and different wireless local networks, for example wireless local area networks (WLAN). Some wireless systems can be divided into cells, and are therefore often referred to as cellular systems. A user can access the communication system by means of an appropriate communication device or terminal. A communication device of a user may be referred to as user equipment (UE) or user device. A communication device is provided with an appropriate signal receiving and transmitting apparatus for enabling communications, for example enabling access to a communication network or communications directly with other users. The communication device may access a carrier provided by a station, for example a base station of a cell, and transmit and/or receive communications on the carrier. The communication system and associated devices typically operate in accordance with a given standard or specification which sets out what the various entities associated with the system are permitted to do and how that should be achieved. Communication protocols and/or parameters which shall be used for the connection are also typically defined. One example of a communications system is UTRAN (3G radio). Other examples of communication systems are the long-term evolution (LTE) of the Universal Mobile Telecommunications System (UMTS) radio-access technology and so-called 5G or New Radio (NR) networks. NR is being standardized by the 3rd Generation Partnership Project (3GPP). SUMMARY According to a first aspect, there is provided an apparatus comprising network node apparatus for a cellular network, the network node apparatus comprising: means for receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate across user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; the apparatus comprising: means for training a neural network based on the information. According to some examples, each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and the apparatus comprises: means for sending, to the base station node and before receiving the information, current Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and wherein the apparatus comprises: means for sending, to a validation base station node, a first set of Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the first set of Q network parameters to determine a first average reward value between an initial time step and a final time step; means for sending, to the validation base station node, a second set of Q network parameters, the second set of Q network parameters determined based on training the first set of Q network parameters, wherein the base station node uses the second set of Q network parameters to determine a second average reward value between the initial time step and the final time step; means for receiving the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step; means for validating the training based on a comparison of the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step. According to a second aspect, there is provided a method comprising: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; the method comprising: training a neural network based on the information. According to some examples, each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending, to the base station node and before receiving the information, current Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending, to the base station node and before receiving the information, current Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending, to a validation base station node, a first set of Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the first set of Q network parameters to determine a first average reward value between an initial time step and a final time step; sending, to the validation base station node, a second set of Q network parameters, the second set of Q network parameters determined based on training the first set of Q network parameters, wherein the base station node uses the second set of Q network parameters to determine a second average reward value between the initial time step and the final time step; receiving the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step; validating the training based on a comparison of the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step. According to a third aspect, there is provided an apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and cause the apparatus at least to: train a neural network based on the information. According to some examples, each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending, to the base station node and before receiving the information, current Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to some examples, training the neural network based on the information comprises training using Q network reinforcement learning, and the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: sending each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending, to the base station node and before receiving the information, current Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to some examples, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: training the neural network based on the information comprises training using Q network reinforcement learning, and the method comprises: sending, to a validation base station node, a first set of Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the first set of Q network parameters to determine a first average reward value between an initial time step and a final time step; sending, to the validation base station node, a second set of Q network parameters, the second set of Q network parameters determined based on training the first set of Q network parameters, wherein the base station node uses the second set of Q network parameters to determine a second average reward value between the initial time step and the final time step; receiving the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step; validating the training based on a comparison of the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step. According to a fourth aspect there is provided an apparatus comprising: circuitry for: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and: training a neural network based on the information. According to a fifth aspect there is provided a computer program comprising instructions for causing an apparatus to perform at least the following: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and: training a neural network based on the information. According to a sixth aspect there is provided a computer program comprising instructions stored thereon for performing at least the following: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and: training a neural network based on the information. According to a seventh aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the following: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and: training a neural network based on the information. According to an eighth aspect there is provided a non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and: training a neural network based on the information. According to a ninth aspect there is provided a base station node apparatus comprising: means for sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node apparatus at a respective origin time step; an action performed by the base station node apparatus at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node apparatus based on a change of a function of a data rate across user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to some examples, each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, a neural network is trained by the network node apparatus based on the information by using Q network reinforcement learning, and wherein the apparatus comprises: means for receiving, from the network node apparatus and before sending the information, current Q network parameters being used in the Q network reinforcement learning; means for using the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to a tenth aspect there is provided an apparatus comprising: at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node apparatus at a respective origin time step; an action performed by the base station node apparatus at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node apparatus based on a change of a function of a data rate across user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to some examples, each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, a neural network is trained by the network node apparatus based on the information by using Q network reinforcement learning, and the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: receiving, from the network node apparatus and before sending the information, current Q network parameters being used in the Q network reinforcement learning; using the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to an eleventh aspect there is provided a method performed by a base station node comprising: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to some examples, each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. According to some examples, a neural network is trained by the network node apparatus based on the information by using Q network reinforcement learning, and the method comprises: receiving, from the network node apparatus and before sending the information, current Q network parameters being used in the Q network reinforcement learning; using the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. According to some examples, the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. According to some examples, the value of the reward function is zero when a threshold number of total time steps have been taken. According to some examples, the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. According to some examples, each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. According to a twelfth aspect there is provided an apparatus comprising circuitry for: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to a thirteenth aspect there is provided a computer program comprising instructions for causing an apparatus to perform at least the following: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to a fourteenth aspect there is provided a computer program comprising instructions stored thereon for performing at least the following: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to a fifteenth aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the following: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to a sixteenth aspect there is provided a non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. According to an aspect, there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the method according to any of the preceding aspects. In the above, many different embodiments have been described. It should be appreciated that further embodiments may be provided by the combination of any two or more of the embodiments described above. DESCRIPTION OF FIGURES Embodiments will now be described, by way of example only, with reference to the accompanying Figures in which: Figure 1 shows an example representation of a time-frequency radio resource grid; Figure 2 shows an example state/action transition diagram; Figure 3 shows an example network; Figure 4 shows an example method flow diagram; Figure 5 shows an example apparatus; Figure 6 shows an example apparatus; Figure 7 shows a method flow diagram; Figure 8 shows a schematic representation of a non-volatile memory medium storing instructions which when executed by a processor allow a processor to perform one or more of the steps of the method of Figures 4 and 7. DETAILED DESCRIPTION In the following certain embodiments are explained with reference to mobile communication devices capable of communication via a wireless cellular system and mobile communication systems serving such mobile communication devices. According to some examples, a method and apparatus is provided for allocating time- frequency radio resources to User Equipment’s (UEs) in wireless systems. This task may be carried out by a Medium Access Control (MAC) scheduler, where typical scheduling algorithms include variations of Proportional Fair (PF), Exponential Rule (EXP), Modified Largest Weighted Delay First (MLWDF), etc. According to some examples, deep reinforcement learning (DRL) is used as an algorithm for some sequential decision-making tasks such as videogames and robotics. DRL algorithms are usually data hungry and take long to train. Examples use a DRL-based scheduler that leverages data collection from numerous eNBs/gNBs to reduce training time and avoid the tuning requirements of other schedulers. Since the number of radio resources is limited, scheduling algorithms strike tradeoffs between the bitrates expected by the UEs, UE instantaneous radio conditions, historic QoS of UEs, etc. Achieving the right balance between all these requirements is challenging and many schedulers must be tuned to adapt to changes in other parts of the RAN. Tuning such a fundamental part of an eNB/gNB is not easy especially if not done by the engineers who originally built the scheduler. This gives rise to problems of code inheritance. Furthermore, many of the utility functions used in resource allocation lead to non-deterministic polynomial- time (NP)-hard optimization problems. As a result, non-optimal heuristics are often the solution to which MAC engineers turn to. According to some examples, radio resource allocation is formulated as a Markov Decision Process (MDP). The size of the action space is reduced by time-stepping through resource blocks (in uplink) or through resource block groups (in downlink). For example, time-stepping can be performed through Resource Block Groups each Transmission Time Interval (TTI). This ensures that that the DRL action space is as low as the maximum number of UEs in RRC- CONNECTED mode supported by the eNB/gNB. The number of UEs to schedule on each TTI is not imposed, thus supporting long time-horizon strategies that respect QoS latency budgets. According to some examples, an MDP is solved using a distributed DRL technique that does not require stack assumptions or constraints. Using the distributed approach, MAC schedulers can be generalize to more radio environments than when based on single eNB/gNB approaches. Therefore the number of scenarios where the solution can provide a benefit is increased. Scheduling data from multiple eNBs/gNBs (actor eNBs/gNBs) can be collected and stored in a so-called Replay memory, from which a GPU-powered learner software feeds to train its own Artificial Neural Network (ANN). The data collection requires no modification to the scheduling policies used by the actor eNBs/gNBs. The method described herein can be used for downlink or uplink scheduling. Consider an example downlink time-frequency resource grid with N Physical Resource Blocks (PRBs) of Figure 1, where some PRBs carry data bits (shown with small circles in a PRB), others control bits (shown with small crosses in a PRB), and others control & data bits (shown as blank PRBs). On each subframe, the MAC scheduler allocates each and all these PRBs to active UEs (i.e. UEs that are in RRC CONNECTED mode). The MAC scheduler takes these decisions considering various metrics about the current conditions of the system and the UE requirements. At a first time step t=0 the MAC scheduler performs an action, which may comprise assigning a PRB 0 to an active UE or deciding not to allocate a PRB to a UE. The time step then moves to the next step, where an action is performed for the next PRB. When the time step has been advanced to t=N-1, an action is performed for PRB N-1, where N is the number of PRBs. The time step is advanced to t=N and an action is performed for PRB 0. This is shown at 100 of Figure 1. A MDP is used where the agent (e.g. the MAC scheduler) begins in an initial state ^^₀201a and then takes a scheduling decision 203a that lands it on a new state ^^₁201b (see Figure 2). This procedure is repeated until all resources have been allocated or a maximum number of time steps have been reached. Time-stepping dynamics can be performed as two nested sweeps, first over the subframes, and then in each subframe, over the PRBs. Let us define ^^ _^^ as the index of the subframe where PRBs are being allocated in time step t, and ^^ _^^ as the index of the PRB to be allocated. Their values are defined as follows: where ^^₀ ≜ 0 and ^^₀ ≜ 1. The concept of legal/illegal actions is described below. ^^ is defined as the number of subframes during PRBs must be allocated. The final state is defined as the state in which ^^ _^^ = 0 and ^^ _^^ = ^^ + 1, or ^^ = ^^ ∙ ^^ ∙ ^^, whichever happens first. ^^ is a positive integer. In some examples, ^^=2. The time step leading up to the final state is defined as the final step. The sequence of time steps between the initial and the final states is called an episode. For the sake of time averaging, ^^ should not be too small (e.g.1), or else the performance evaluation would be too noisy and unreliable. On the other hand, it should not be too large either, or the learning algorithm may be too slow. A scheduling decision 203b may be taken at ^^₁201b that moves the agent to a new state ^^₂ 201c. According to some examples, the current conditions of the system are aggregated into a state ^{vector. The state vector at time step ^^ may be defined as ^^ ^^ =} _{[ ^^1 … ^^ ^^, ^^1 … ^^ ^^ , ^^, ^^, ^^, ^^, ^^0,0 … ^^ ^^−1,0, ^^0,1 … ^^ ^^−1,0 … ^^ ^^−1, ^^−1], where the following} definitions apply: ^ ^^ is the maximum number of UEs that can be in RRC CONNECTED mode in the cell at any given time. ^ ^^ _^^ ∈ [0…15] is the most recently available wideband Channel Quality Indicator (CQI) received from the u^th connected UE, where a higher value indicates better channel conditions for that UE. In addition, this invention defines a value of 0 to indicate that the UE is not connected. ^ ^^ _^^ ∈ ^[0,1^] is the buffer occupancy of the u^th connected UE and it takes value 0 if no downlink data is available for the UE or 1 if it is. ^ ^^ ∈ [1… ^^ + 1] is the index of the subframe where PRBs are being allocated in the current time step. ^ ^^ ∈ [0… ^^ − 1] is the index of the PRB being allocated in the current time step. ^ ^^ ∈ [0…9] is the current subframe number as per the LTE or 5GNR standard within a 10 ms radio frame. ^ ^^ _{^^, ^^} ∈ [0… indicates the index of the UE to which the ^^ ^{^^ℎ} resource has been allocated during the ^^ ^{^^ℎ} subframe. A value of 0 indicates that the resource has not yet been allocated to any UE. It should be noted that in some examples, ^^ _^^ may have more or fewer terms than those listed above. At each time step, the MAC scheduler decides to which UE the current PRB is to be allocated to. These decisions are the so-called actions that the scheduler can take. The action space is thus defined as ^^ = [0, 1 … ^^ − 1]. With this definition, selecting action ^^ is an attempt to allocate the current PRB to the u^th UE. It may not be possible, on each time step, to allocate the current PRB to any UE. Such an attempt may be defined as an illegal action. This is because, depending on the current aggregation level chosen by the Link Adaptation algorithm for the current and the previous UEs, there may not be Control Channel Elements (CCEs) left to assign a new Physical Downlink Control Channel (PDCCH) for a new UE on the current TTI. The learning algorithm used by this invention trains the scheduler to avoid this problem by providing zero reward if the scheduler cannot allocate all PRBs after a maximum of ^^ ∙ ^^ ∙ ^^ time steps, where ^^ is a positive integer. When ^^=2, a by-product of this approach is that the learned data allocation policies are compatible with the PDCCH allocation constraints where one subframe has two scheduling slots. A behavioural policy comprise a mapping from states to actions, and can be used to indicate which action to take on a given state. An action-value method can be used to evaluate different actions and select which action to take. On each time step the scheduler uses the current state to calculate the so-called Q value of each and all actions. According to some examples, a trained scheduler (e.g., one deployed in a commercial product) shall choose the action whose estimated Q value is highest. In some examples, a training scheduler may sometimes choose a non-optimal action for the purpose of exploring the state space. According to some examples, the Q function is approximated using a Dueling neural network architecture as described in “Dueling Network Architectures for Deep Reinforcement Learning”, Z. Wang et. Al., Apr 2016” with one hidden layer. The hidden layer may have a Dueling network has two streams: one advantage stream, and one value stream. Considering this, an example network architecture is: ^ 1 fully-connected input layer of size equal to the number of features in the state vector. ^ 1 fully-connected hidden layer with the following streams: o 1 Advantage stream of size 256 neurons. o 1 Value stream of size 256 neurons. ^ 1 fully-connected output layer of size ^^ neurons. After taking an action, a scheduler receives a reward, which is a scalar number that evaluates how good the state transition from ^^ _^^ to ^^ _^^+1 was. The sum of all rewards received during an episode is known as the return. According to some examples, the return can be discounted or not by means of a discount factor. Due to the intrinsic randomness of the wireless channel and of the UE traffic requirements, the sequence of rewards received during two episodes may vary even if the initial states were the same. As such, expected return can be more interesting than the instantaneous reward from a policy evaluation viewpoint. During training, a scheduler’s policy is modified to optimize the expected return. According to some examples, the reward function received by the scheduler on time step ^^ as: where ^^ _^^ is the normalized downlink data rate experienced by the k^th UE averaged over the past episode (i.e. during the past ^^ subframes). This reward function has the objective of maximizing the minimum data rate across all UEs. This is a utility function for the allocation of radio resources to Non-GBR (Guaranteed Bit Rate) UEs. It should be noted that in other examples, other reward functions may be used. In some examples, these reward functions may be based on a data rate across UEs in the network. Training a neural network can be used as a process to improve an agent’s performance over time. According to some examples, instead of executing the resource intensive training algorithms on an eNB/gNB such as base station nodes 305a and 305b of Figure 3, all training is centralized on a GPU-capable machine 309 which is not collocated with the base station node 305a, 305b. Furthermore, instead of using MDP transitions from only one eNB/gNB, data collected from multiple eNBs/gNBs via network 307 is used for training (see Figure 3). In general, training data can then be collected from large portions of live networks without changes to the current commercial MAC scheduling policies due to the off-policy learning properties of Q-learning. According to some examples, Prioritized Experience Replay as described in “Reinforcement Learning Scheduler for Vehicle-to-Vehicle Communications Outside Coverage”, T. Sahin et al., Dec 2018, IEEE Vehicular Networking Conference (VNC) is used, which accelerates learning by retraining more often on the most interesting MDP transitions (i.e. those with a higher prediction error). At each time step, the MAC scheduler in an eNB/gNB (e.g., base station 305a, 305b) executes an action (i.e. it allocates a PRB to a UE or determines not to assign a PRB to a UE). This moves the scheduler from an initial state to a new state. This transition can be constructed as a data tuple ( ^^ _^^ , ^^ _^^ , ^^ _{^^+ ^^}, ^^ _{^^+ ^^}, ^^( ^^ _^^), ^^( ^^ _{^^+ ^^})), where ^^ is the size of the eligibility trace for Temporal Difference (TD) bootstraping and ^^( ^^) is the Q value estimate done by the eNB/gNB on state ^^. These transitions can be sent from the base stations to device 309. According to some examples, these transitions are buffered locally at base station nodes 305a and 305b and periodically sent in lots to the central Experience Replay Memory 313 of device 309 for training using learner (GPU) 311. Each gNB/eNB 305a, 305b can periodically request the latest Q network parameters from the learner in order to calculate the Q values. Note that this is an inference computation, and that no training takes place at the eNBs/gNBs 305a and 305b. Therefore, no hardware accelerator for training Artificial Neural Networks is required on the Radio Access Network (RAN), as it is provided at device 309. Base station nodes may be provided with backhaul links from the eNBs/gNBs 305a and 305b to device 309. In some examples, the experience transitions to be sent to the Replay Memory 313 have backhaul capacity requirements on the same order of magnitude as network logs. This can be taken into account when configuring the network. On the centralized learning facilities at device 309, the learner software can collect transition batches from the Experience Replay Memory via prioritized sampling (see “Prioritized Experience Replay”, T. Schaul et. al., Feb 2016) and applies root mean squared propagation (RMSProp) on them to change the current Q network parameters. These updates are compatible with most deep reinforcement learning algorithms, for example Double Deep Q Network (DQN) architecture. According to some examples, for evaluating learning progress, a so-called validation base station node can be configured. This is a base station (e.g. eNB/gNB) that also collects the latest Q network parameters from learner 311. However, the transitions generated by the validation base station shall not be sent to the Experience Replay Memory 313. Learning progress can be assessed through a gradual increase in the average reward per episode for this validation agent. Figure 4 shows an example method flow that can take place at a scheduler (e.g. MAC scheduler) of a base station (e.g. eNB/gNB 305a, 305b). Figure 4 uses a method of Q learning, but in other examples other learning methods may be used. At 420, t=0 (we are at an initial time step) and p=0 (assignment of PRB 0 is being considered). At 422, a state vector ^^ _^^ is built for time t. At 424, an ANN is used to approximate a value of an expected cumulative reward (Q-value) for the state vector ^^ _^^ for all actions belonging to an action space A. This may be performed using ANN parameters (Q network parameters) received from a central DRL device (such as device 309). Based on the approximation at 424, at 426 the scheduler selects an action ^^ _^^ that maximises the expected cumulative reward (Q-value) for the state vector ^^ _^^. At 428, based on the determination at 426, the scheduler allocates the pth PRB to the UE indicated by ^^ _^^. This may be a UE controlled by the base station node comprising the scheduler. In some examples, if the action is illegal, the PRB is allocated to no UE. The method then moves on to a subsequent time step t+1 and the scheduler builds new state vector ^^ _^^+1. At 430, a reward based on the data-rate of UEs controlled by the base station node is determined. This is a scalar number that evaluates how good the transition from t to t+1 was. This may be, for example, a measure of the minimum date rate across all UEs controlled by the base station node. At 432, the scheduler sends a state transition data tuple ( ^^ _^^ , ^^ _^^ , ^^ _^^+1, ^^ _^^+1, ^^⁽ ^^ _^^ ⁾, ^^( ^^ _^^+1)) to replay memory. According to some examples, this may happen after multiple time steps such that the data tuple covers a state transition over more than one time step. A learning device can use this state transition data tuple to train a neural network to maximise Q (or in other examples that e.g., don’t use Q learning, to maximise a reward function). According to some examples, multiple state transition data tuples are sent to a DRL scheduler from a plurality of base stations (e.g., from the schedulers of each base station). The DRL scheduler may use state transition data tuples to train an ANN. In some examples, this is performed using Q network reinforcement learning. At 434, t is increased by 1 (i.e., the scheduler moves from an origin time step to a subsequent time step) and p is increased to ( ^^ + 1) ^^ ^^ ^^ ^^. The method returns to 422 and repeats. An origin time step may comprise any time step that is before a subsequent time step. An origin time step may comprise a time step that is earlier relative to a subsequent time step. Using hyperparameters as shown in Table 1, various architectures of the Q neural network in a simulated LTE cell were tested. The simulation used random seeds that are disjoint from those used during training. Small ANNs (i.e.32 neurons on each stream of the hidden layer) perform only slightly better than a random agent. Optimal performance was obtained with a network of size 256, with larger networks suffering from overfitting. The optimal configuration reached 93% of the performance of a Proportional Fair scheduler at a fraction of the complexity and maintenance effort. These results were obtained for the max-min reward function described above, and the concept described by this invention may be easily extended to other utility functions (e.g. max mean throughput, etc). The hyperparameters used for these experiments are summarized in Table 1. Actor (MAC scheduler) Discount factor 0.99 n 3 Parameters collection period 400 steps Learner Batch size 512 Learning rate 1E-5 Number of training epochs 1E6 DQN target network update period 2500 epoch Experience Replay Memory Priority exponent ^^ 0.6 Importance sampling exponent ^^ 0.4 Size 4E6 transitions Prioritization type Proportional Neural network Architecture Dueling Hidden layers size Advantage: 256x1 Value: 256x1 Table 1: Main hyperparameters By using experience data from numerous base stations (eNBs/gNBs), training is accelerated compared to previous approaches to MDP and DRL based MAC scheduling. Further, adding support for new deployment scenarios does not require writing the scheduler, only retraining the scheduler. This facilitates code development by new developers. According to some examples, the learning device can be trained with data from non-ML commercial schedulers without changes to their policies, Current commercial schedulers can continue to run as usual, with the new DRL scheduler learns in real-time form their experience. Using ANN code accounts for non-intuitive and non-linear relationships between scheduler inputs and target utility function to replace code repositories of MAC schedulers, which have traditionally been built as expert systems due to their high complexity. Traditional approaches to time-frequency resource allocation encounter challenges when applied to the massive bandwidths of mmWave bands. On the other hand, using the DRL scheduler as described herein scales naturally to these bands by means of larger ANN sizes. Figure 5 illustrates an example of a control apparatus 500 for controlling a function of the network as illustrated in Figure 3. The control apparatus may comprise at least one random access memory (RAM) 511a, at least on read only memory (ROM) 511b, at least one processor 512, 513 and an input/output interface 514. The at least one processor 512, 513 may be coupled to the RAM 511a and the ROM 511b. The at least one processor 512, 513 may be configured to execute an appropriate software code 515. The software code 515 may for example allow to perform one or more steps to perform one or more of the present aspects. The software code 515 may be stored in the ROM 511b. The control apparatus 500 may be interconnected with another control apparatus 500 controlling another function of the RAN or the core network. Figure 6 illustrates an example of a terminal 600, such as a UE. The terminal 600 may be provided by any device capable of sending and receiving radio signals. Non-limiting examples comprise a user equipment, a mobile station (MS) or mobile device such as a mobile phone or what is known as a ’smart phone’, a computer provided with a wireless interface card or other wireless interface facility (e.g., USB dongle), a personal data assistant (PDA) or a tablet provided with wireless communication capabilities, a machine-type communications (MTC) device, an Internet of things (IoT) type communication device or any combinations of these or the like. The terminal 600 may provide, for example, communication of data for carrying communications. The communications may be one or more of voice, electronic mail (email), text message, multimedia, data, machine data and so on. The terminal 600 may receive signals over an air or radio interface 607 via appropriate apparatus for receiving and may transmit signals via appropriate apparatus for transmitting radio signals. In Figure 6 transceiver apparatus is designated schematically by block 606. The transceiver apparatus 606 may be provided for example by means of a radio part and associated antenna arrangement. The antenna arrangement may be arranged internally or externally to the mobile device. The terminal 600 may be provided with at least one processor 601, at least one memory ROM 602a, at least one RAM 602b and other possible components 603 for use in software and hardware aided execution of tasks it is designed to perform, including control of access to and communications with access systems and other communication devices. The at least one processor 601 is coupled to the RAM 602b and the ROM 602a. The at least one processor 601 may be configured to execute an appropriate software code 608. The software code 608 may for example allow to perform one or more of the present aspects. The software code 608 may be stored in the ROM 602a. The processor, storage and other relevant control apparatus can be provided on an appropriate circuit board and/or in chipsets. This feature is denoted by reference 604. The device may optionally have a user interface such as key pad 605, touch sensitive screen or pad, combinations thereof or the like. Optionally one or more of a display, a speaker and a microphone may be provided depending on the type of the device. Figure 7 shows an example method flow. In some examples, the method of Figure 7 may be performed by a central learning device such as device 309. The device may comprise a DRL device. At 750, the method comprises receiving, from at least one base station node, information comprising a plurality of data tuples. Each respective data tuple may comprise a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate across user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step. According to some examples, the information may also comprise Q values determined by each base station at the origin time step and the subsequent time step. At 752, the method comprises training a neural network based on the information received at 750. After the neural network is trained, the DRL device performing the method of Figure 7 may share the parameters to the trained neural network with one or more base stations. These base stations may then use these parameters to assign PRBs to UEs. Figure 8 shows a schematic representation of non-volatile memory media 800a (e.g. computer disc (CD) or digital versatile disc (DVD)) and 800b (e.g. universal serial bus (USB) memory stick) storing instructions and/or parameters 802 which when executed by a processor allow the processor to perform one or more of the steps of the above-described methods. It should be understood that the apparatuses may comprise or be coupled to other units or modules etc., such as radio parts or radio heads, used in or for transmission and/or reception. Although the apparatuses have been described as one entity, different modules and memory may be implemented in one or more physical or logical entities. It is noted that whilst some embodiments have been described in relation to 5G networks, similar principles can be applied in relation to other networks and communication systems. Therefore, although certain embodiments were described above by way of example with reference to certain example architectures for wireless networks, technologies and standards, embodiments may be applied to any other suitable forms of communication systems than those illustrated and described herein. It is also noted herein that while the above describes example embodiments, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention. As used herein, “at least one of the following: <a list of two or more elements>” and “at least one of <a list of two or more elements>” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements. In general, the various embodiments may be implemented in hardware or special purpose circuitry, software, logic or any combination thereof. Some aspects of the disclosure may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto. While various aspects of the disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and/or digital hardware circuit(s) with software/firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device. The embodiments of this disclosure may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and/or macros, may be stored in any apparatus-readable data storage medium and they comprise program instructions to perform particular tasks. A computer program product may comprise one or more computer- executable components which, when the program is run, are configured to carry out embodiments. The one or more computer-executable components may be at least one software code or portions of it. Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD. The physical media is a non-transitory media. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM). The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may comprise one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), FPGA, gate level circuits and processors based on multi core processor architecture, as non-limiting examples. Embodiments of the disclosure may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate. The scope of protection sought for various embodiments of the disclosure is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the disclosure. The foregoing description has provided by way of non-limiting examples a full and informative description of the exemplary embodiment of this disclosure. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this disclosure will still fall within the scope of this invention as defined in the appended claims. Indeed, there is a further embodiment comprising a combination of one or more embodiments with any of the other embodiments previously discussed.

Claims

CLAIMS 1. A network node apparatus for a cellular network, the network node apparatus comprising: means for receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate across user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; the apparatus comprising: means for training a neural network based on the information. 2. An apparatus as claimed in claim 1, wherein each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. 3. An apparatus according to claim 2, wherein training the neural network based on the information comprises training using Q network reinforcement learning, and wherein the apparatus comprises: means for sending, to the base station node and before receiving the information, current Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. 4. An apparatus according to any preceding claim, wherein the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. 5. An apparatus according to any preceding claim, wherein the value of the reward function is zero when a threshold number of total time steps have been taken. 6. An apparatus according to claim 5, wherein the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. 7. An apparatus according to any preceding claim, wherein each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. 8. An apparatus according to any preceding claim, wherein training the neural network based on the information comprises training using Q network reinforcement learning, and wherein the apparatus comprises: means for sending, to a validation base station node, a first set of Q network parameters being used in the Q network reinforcement learning, wherein the base station node uses the first set of Q network parameters to determine a first average reward value between an initial time step and a final time step; means for sending, to the validation base station node, a second set of Q network parameters, the second set of Q network parameters determined based on training the first set of Q network parameters, wherein the base station node uses the second set of Q network parameters to determine a second average reward value between the initial time step and the final time step; means for receiving the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step; means for validating the training based on a comparison of the first average reward value between the initial time step and the final time step and the second average reward value between the initial time step and the final time step. 9. A method comprising: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change in minimum data rate of user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; the method comprising: training a neural network based on the information. 10. A computer program comprising instructions stored thereon for performing at least the following: receiving, from one or more base station nodes, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node of the one or more base station nodes at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate across user equipment devices controlled by the base station node apparatus between the respective origin time step and the respective subsequent time step; and the computer program comprising instructions stored thereon for performing: training a neural network based on the information. 11. A base station node apparatus comprising: means for sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node apparatus at a respective origin time step; an action performed by the base station node apparatus at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node apparatus based on a change of a function of a data rate across user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. 12. An apparatus as claimed in claim 11, wherein each data tuple further comprises: a value of an expected cumulative reward determined by the base station node for the first state vector; a value of the expected cumulative reward determined by the base station node for the second state vector. 13. An apparatus according to claim 12, wherein a neural network is trained by the network node apparatus based on the information by using Q network reinforcement learning, and wherein the apparatus comprises: means for receiving, from the network node apparatus and before sending the information, current Q network parameters being used in the Q network reinforcement learning; means for using the current Q network parameters to determine the value of the expected cumulative reward for the first state vector and the value of the expected cumulative for the second state vector. 14. An apparatus according to any of claims 11 to 13, wherein the action performed by the base station node at the respective origin time step comprises allocating a physical resource block to a user equipment of the user equipment devices controlled by the base station node. 15. An apparatus according to any of claims 11 to 14, wherein the value of the reward function is zero when a threshold number of total time steps have been taken. 16. An apparatus according to claim 15, wherein the threshold number of total time steps is an integer multiple of a product of a number of available physical resource blocks and a number of subframes in which the available physical resource blocks must be allocated. 17. An apparatus according to any of claims 11 to 16, wherein each state vector comprises at least one of: a maximum number of user equipment devices that can be in a radio resource configuration connected mode in the system controlled by the base station node; one or more channel quality indicators for one or more respective user equipment devices; one or more buffer occupancy values for the one or more respective user equipment devices; an index of a subframe where physical resource blocks are being allocated in a current time step; an index of the physical resource block being allocated in the current time step; a current subframe number; an index of a user equipment to which a physical resource block has been allocated to. 18. A method performed by a base station node comprising: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by the base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate of user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step. 19. A computer program comprising instructions stored thereon for performing at least the following: sending, to a network node apparatus, information comprising a plurality of data tuples, each respective data tuple comprising: a first state vector of a system controlled by a base station node at a respective origin time step; an action performed by the base station node at the respective origin time step; a second state vector of the system at a respective subsequent time, the respective subsequent time being later than the respective origin time step, the second state vector being a consequence of the action performed by the base station node at the respective origin time step; a value of a reward function determined by the base station node based on a change of a function of a data rate across user equipment devices controlled by the base station node between the respective origin time step and the respective subsequent time step.