KR20230146416A - Method of scheduling resource block, and electronic device performing the method - Google Patents

Abstract

A method for scheduling resource blocks and an electronic device that performs the method are disclosed. An electronic device according to various embodiments includes a processor and a memory electrically connected to the processor and storing instructions that can be executed by the processor, and the processor, when the instruction is executed, sends a message to a plurality of terminals. Based on the CQI index (channel quality information index), an expected transmission rate (achievable rate) when a resource block is allocated to the plurality of terminals is obtained, and the expected transmission rate is input into a learned neural network model, and the plurality of terminals are assigned an achievable rate. A schedule for allocating the resource block to a terminal is output, and the neural network model collects learning CQI indices for the plurality of terminals, and based on the learning CQI index, the resource block is provided to the plurality of terminals. Obtain the learning expected data rate when allocated, and use the input expected learning data rate to maximize the total data rate for the plurality of terminals, and establish a fairness condition in which the fairness index for the plurality of terminals is set. It can be learned to output the schedule that satisfies it.

Description

Method for scheduling resource blocks, and electronic device for performing the method {METHOD OF SCHEDULING RESOURCE BLOCK, AND ELECTRONIC DEVICE PERFORMING THE METHOD}

The present disclosure relates to a method for scheduling resource blocks according to various embodiments, and an electronic device that performs the method.

In order to distribute limited time-frequency resources to user equipment (UE), resource blocks (RB) are scheduled for the UE in various ways. Scheduling is performed to allocate limited resource blocks to users based on data about the channel environment and requirements between the base station and users.

If the network fails to schedule resource blocks efficiently, large delays may occur for UEs requiring low-latency services, and the quality of service (QoS) of users requiring high-capacity communication may not be met, leading to problems in the entire network. Communication quality may deteriorate.

Methods for allocating resource blocks include Round Robin (RR), Best-CQI, and PF. The RR scheduling method allocates the same communication resources by periodically allocating RBs to all users, ensuring fair resource allocation for all users.

The Best-CQI scheduling method allocates RBs to users with the best channels based on CQI (channel quality information) received from users for each RB. When scheduling in this way, users with the best channel environment are always scheduled with priority, so high performance can be achieved for the entire network.

PF scheduling is a technique to allocate RBs so that no user has poor performance by considering the trade-off between the performance of the entire network and the performance of individual users.

If resource blocks are scheduled for user terminals using a fixed algorithm, resources may not be properly allocated and resources may not be properly responded to in rapidly changing channel and network situations.

When allocating resources according to the RR, Best CQI, and PF methods, fairness is satisfied for users, but the performance of each user and the entire network is low, or the performance of each user and the entire network is high. , it may not satisfy fairness, or it may be difficult to immediately respond to rapidly changing channel and network environments due to fixed parameter values.

According to various embodiments, a scheduling method for allocating resource blocks to each user terminal so that the overall system transmission rate is high and fairness for each user is satisfied, and an electronic device that performs the method, can be provided.

According to various embodiments, a scheduling method that can allocate resource blocks to user terminals in consideration of various environmental changes such as network environment or channel environment and an electronic device that performs the method can be provided.

According to various embodiments, a scheduling method that allocates resource blocks based on an artificial neural network (ANN) in consideration of environmental changes and can increase system transmission rate while satisfying fairness, and an electronic device that performs the method, Devices and neural network model learning methods can be provided.

An electronic device according to various embodiments includes a processor and a memory electrically connected to the processor and storing an instruction that can be executed by the processor, and the processor, when the instruction is executed, relates to a plurality of terminals. Based on the CQI index (channel quality information index), the expected transmission rate (achievable rate) when a resource block is allocated to the plurality of terminals is obtained, and the expected transmission rate is input into the learned neural network model, and the plurality of terminals Outputs a schedule for allocating the resource block, and the neural network model collects learning CQI indices for the plurality of terminals, and allocates the resource block to the plurality of terminals based on the learning CQI index. Obtain the learning expected data rate when the expected learning data rate is input, and use the input expected learning data rate to maximize the total data rate for the plurality of terminals, and satisfy the fairness condition set by the fairness index for the plurality of terminals. It can be learned to output the schedule.

An electronic device according to various embodiments includes a processor and a memory electrically connected to the processor and storing an instruction that can be executed by the processor, and the processor, when the instruction is executed, relates to a plurality of terminals. Based on the CQI index (channel quality information index), obtain the current state including the average transmission rate of each of the plurality of terminals, the transmission rate, and the average fairness index for the plurality of terminals, and DQN (deep Q- Using a neural network model learned according to a learning method of (network), an action for allocating the resource block to the plurality of terminals according to the current state is determined, and the neural network model maximizes the reward in the current state. It is learned to output the action, and the reward can be determined based on a reward function according to constraints set for allocating the resource block to the plurality of terminals.

The scheduling method according to various embodiments is based on a CQI index (channel quality information index) for a plurality of terminals, an average transmission rate for each of the plurality of terminals, a transmission rate, and an average fairness index for the plurality of terminals. ) and an action for allocating the resource blocks to the plurality of terminals according to the current state using a neural network model learned according to the learning method of DQN (deep Q-network). Including an operation of determining, wherein the neural network model is learned to output the action that maximizes the reward in the current state, and the reward is determined according to a constraint condition set for allocating the resource block to the plurality of terminals. It can be determined based on the reward function according to the reward function.

A neural network model learning method according to various embodiments includes the operation of allocating resource blocks for a set time to a plurality of terminals and collecting learning data including the current state, action, reward, and next state, and learning input to the neural network model. An operation of determining a Q value according to data and an operation of training the neural network model to output the action that maximizes the reward in the current state using a loss calculated based on the Q value, The reward may be determined based on a reward function according to constraints set for allocating the resource block to the plurality of terminals.

According to various embodiments, by providing a scheduling method and an electronic device for performing the method, the optimal solution of a pre-obtained high-complexity algorithm is learned through a supervised learning-based scheduling technique, thereby significantly lowering execution complexity and enabling efficient scheduling. can do. In addition, through an unsupervised learning-based scheduling technique, RBs can be effectively scheduled to the UE by learning given constraints without the process of collecting learning data.

According to various embodiments, a scheduling method and an electronic device for performing the method are provided, utilizing a CQI index that changes over time through a reinforcement learning-based algorithm (e.g., DQN, DDQN) while satisfying several constraints. Scheduling that maximizes the average transmission rate is possible. Scheduling is possible by satisfying constraints that cannot be reflected in existing Best-CQI and RR algorithms, and in particular, scheduling metrics can be improved by satisfying given fairness.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
FIG. 2 is a diagram illustrating an operation of an electronic device outputting a schedule according to various embodiments.
FIG. 3 is a diagram illustrating the operation of a scheduling method performed by an electronic device according to various embodiments.
Figure 4 is a diagram showing an operation of a learning device according to various embodiments to learn a neural network model based on a supervised learning method.
Figure 5 shows a method in which a learning device according to various embodiments trains a neural network model based on an unsupervised learning method.
FIG. 6 is a diagram illustrating a supervised learning-based neural network model learning method performed by a learning device according to various embodiments.
FIG. 7 is a diagram illustrating a method of learning a neural network model based on unsupervised learning performed by a learning device according to various embodiments.
Figure 8 is a diagram showing the network structure of a neural network model learned based on supervised learning according to various embodiments.
Figure 9 is a diagram showing the network structure of a neural network model learned based on unsupervised learning according to various embodiments.
Figure 10 is a diagram showing an activation function of a neural network model learned based on unsupervised learning according to various embodiments.
Figure 11 is a diagram showing the performance of an electronic device using a neural network model learned according to various embodiments.
FIG. 12 is a diagram illustrating an operation of an electronic device outputting an action according to various embodiments.
FIG. 13 is a diagram illustrating the operation of a scheduling method performed by an electronic device according to various embodiments.
Figure 14 is a diagram showing an operation of a learning device training a neural network model according to various embodiments.
FIG. 15 is a diagram illustrating the operation of a neural network model learning method performed by a learning device according to various embodiments.
FIG. 16 is a diagram illustrating an operation in which an electronic device divides a plurality of terminals or resource blocks into subgroups according to various embodiments.
Figure 17 is a diagram showing the performance of a neural network model learned according to various embodiments.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

FIG. 2 is a diagram illustrating an operation of the electronic device 200 to output a schedule according to various embodiments.

Referring to FIG. 2, the processor 220 of an electronic device (e.g., the processor 120 of FIG. 1) according to various embodiments may receive a CQI index (channel quality information index) for a plurality of terminals. For example, the electronic device 200 may identify the distance of each of a plurality of terminals.

For example, the processor 220 of the electronic device 200 may obtain an achievable rate when a resource block (RB) is allocated to a plurality of terminals. For example, the electronic device 200 may calculate the expected transmission rate for each user and resource block based on the CQI index and spectral efficiency according to the modulation coding scheme (MCS). For example, spectral efficiency (or bandwidth efficiency) can refer to the rate at which a communication system can transmit over a given bandwidth.

For example, the processor 220 of the electronic device 200 may calculate spectral efficiency according to the CQI index and MCS. The electronic device may calculate the expected transmission rate when a resource block is allocated to each of a plurality of terminals based on spectral efficiency.

For example, the processor 220 of the electronic device 200 may input the expected data rate into the learned neural network model 210 and output a schedule for allocating resource blocks to a plurality of terminals. For example, the electronic device 200 may allocate resource blocks to a plurality of terminals according to the output schedule, and the plurality of terminals may perform communication according to the allocated resource blocks.

For example, the neural network model 210 may be trained to output a schedule using the input expected learning transmission rate. For example, the neural network model 210 may be trained to output a schedule that satisfies fairness conditions in which the sum data rate for a plurality of terminals is maximized and a fairness index for a plurality of terminals is set.

For example, the processor 220 of the electronic device 200 outputs a schedule using the learned neural network model 210, so the output schedule has the maximum sum data rate for a plurality of terminals and meets the set fairness conditions. You can be satisfied. The electronic device can allocate resource blocks to a plurality of terminals according to the output schedule, thereby increasing the system transmission rate and simultaneously allocating resource blocks to satisfy fairness conditions.

FIG. 3 is a diagram illustrating the operation of a scheduling method performed by an electronic device (e.g., the electronic device 200 of FIG. 2) according to various embodiments.

Referring to FIG. 3, the electronic device 200 according to various embodiments may obtain an expected data rate based on CQI indices for a plurality of terminals in operation 305. For example, the processor of the electronic device 200 (e.g., the processor 220 in FIG. 2) may calculate the expected transmission rate for each terminal and resource block based on the CQI index and spectrum efficiency according to MCS.

For example, in operation 310, the electronic device 200 inputs the expected transmission rate into a learned neural network model (e.g., the neural network model 210 of FIG. 2) and outputs a schedule for allocating resource blocks to a plurality of terminals. You can.

For example, the neural network model 210 may be trained to maximize the sum data rate for a plurality of terminals and output a schedule that satisfies set fairness conditions. For example, allocating resource blocks to a plurality of terminals to maximize the sum data rate and satisfy set fairness conditions may mean a set constraint condition.

For example, the electronic device 200 may output a schedule that satisfies the set constraint conditions using the neural network model 210 learned to output a schedule that satisfies the set constraint conditions. For example, the electronic device 200 can allocate resource blocks to a plurality of terminals according to the output schedule, thereby increasing the total transmission rate of the entire network and improving fairness for each terminal.

FIG. 4 is a diagram illustrating an operation in which a learning device 400-1 trains a neural network model 410-1 based on a supervised learning method according to various embodiments.

For example, the learning device 400-1 shown in FIG. 4 can learn the neural network model 410-1 based on a supervised learning method. For example, the neural network model 210 used by the electronic device 200 of FIG. 2 may refer to the neural network model 410-1 trained by the learning device 400-1 shown in FIG. 4.

As an example, the learning device 400-1 may identify CQI indices for a plurality of terminals using the processor 420 (eg, processor 120 of FIG. 1). The processor 420 of the learning device 400-1 may use the CQI index to calculate the expected transmission rate when a resource block is allocated to a plurality of terminals. For example, the learning device 400-1 may calculate the expected transmission rate substantially the same as the electronic device 200 of FIG. 2.

For example, the CQI index collected by the learning device 400-1 is collected for training of the neural network model 410-1, and may be referred to as a learning CQI index. For example, the expected data rate calculated by the learning device 400-1 is for training the neural network model 410-1 and may be referred to as the expected learning data rate.

For example, the learning device 400-1 uses the learned expected transmission rate to output a schedule that satisfies the fairness condition in which the sum of the transmission rates for a plurality of terminals is maximum and the fairness index for the plurality of terminals is set, using a neural network. The model 410-1 can be trained.

For example, scheduling that ensures fairness of transmission rates for multiple terminals while maximizing the total transmission rate of the entire network for multiple terminals involves allocating resource blocks to multiple terminals to satisfy Equation 1 below: It can mean.

[Equation 1]

In Equation 1 above, is the transmission rate of the kth terminal, is a variable indicating the status in which the kth terminal is allocated to the mth resource block (e.g., 0 is the unscheduled state, 1 is the scheduled state), may mean a set fairness index (e.g., jain fairness index).

In Equation 1 above, the fairness index for multiple terminals is It can mean, and the fairness condition for multiple terminals is the fairness index The fairness index set by It can mean something more.

For example, in Equation 1 above, According to the conditions, there can be only one terminal allocated to each resource block. In Equation 1 above, According to the conditions, the transmission rate for each terminal to satisfy the fairness conditions The calculated value must be greater than or equal to the set fairness index.

For example, the learning device 400-1 may provide a variable indicating the allocation status of a resource block for each terminal that satisfies the conditions of Equation 1 above. The neural network model 410-1 can be trained to output . For example, the schedule in which the neural network model 410-1 satisfies Equation 1 above Learning to output may mean learning to output a schedule that maximizes the sum data rate for a plurality of terminals and satisfies the fairness conditions for which the fairness index for a plurality of terminals is set. For example, the schedule states that resource blocks are allocated to all multiple terminals. It may mean (k= 1, 2, ..., K, n= 1, 2, ..., N).

For example, based on the expected data rate, the learning device 400-1 may calculate a correct answer schedule that maximizes the total data rate for a plurality of terminals and satisfies the fairness index. For example, the learning device 400-1 can calculate a correct answer schedule that satisfies Equation 1 above.

As another example, the learning device 400-2 may calculate the correct answer schedule according to Equation 2 below. Equation 2 is the binary variable of Equation 1 above It may mean a condition that relaxes the conditions of . For example, the learning device 400-2 may calculate the correct answer schedule using a method such as CVX (convex programming) according to the relaxed conditions of Equation 2 below.

[Equation 2]

For example, the neural network model 410-1 may include a deep neural network (DNN) structure and an activation function. For example, the learning device 410-1 may train the neural network model 410-1 using the loss 440 calculated using a loss function.

For example, the learning device 400-1 may input the expected transmission rate into the neural network model 410-1 and output the schedule 430. For example, the learning device 400-1 may calculate the loss 440 using the correct answer schedule and schedule according to the loss function. For example, the learning device 400-1 may train the neural network model 410-1 to minimize the loss 440. For example, the loss calculated according to the loss function may mean the schedule output from the neural network model 410-1 430 and the mean squared error (mse) of the calculated correct answer schedule.

The learning device 400-1 of FIG. 4 provides resource blocks to a plurality of terminals so that, for example, the total transmission rate of the entire network is maximized and the fairness index for the plurality of terminals satisfies the set fairness conditions under set limiting conditions. The neural network model 410-1 can be trained to output the allocated schedule 430.

Figure 5 shows a method by which the learning device 400-2 according to various embodiments trains the neural network model 410-2 based on an unsupervised learning method.

Referring to FIG. 5, a learning device 400-2 according to various embodiments may learn a neural network model 410-2 based on an unsupervised learning method. For example, the neural network model 210 used by the electronic device 200 of FIG. 2 may refer to the neural network model 410-2 trained by the learning device 400-2 shown in FIG. 5.

For example, the processor 420 of the learning device 400-2 (e.g., the processor 120 of FIG. 1) may identify CQI indices for a plurality of terminals. For example, the processor 420 of the learning device 400-2 may calculate the expected transmission rate based on the CQI index. For example, with respect to the operation of the learning device 400-2 in FIG. 5 to identify the CQI index and calculate the expected data rate, the electronic device 200 in FIG. 2 or the learning device 400-1 in FIG. 4 The description of the operation of identifying the CQI index and calculating the expected data rate can be applied substantially the same.

For example, the CQI index collected by the learning device 400-2 of FIG. 5 is for training the neural network model 410-2, and may be referred to as a learning CQI index. For example, the expected transmission rate calculated by the learning device 400-2 is for training the neural network model 410-2, and may be referred to as the expected learning transmission rate.

For example, the learning device 400-2 of FIG. 5 uses a neural network model to output a schedule 430 that satisfies the fairness condition in which the sum data rate for a plurality of terminals is maximum and the fairness index for a plurality of terminals is set. (410-2) can be learned. For example, the learning device 400-2 may train the neural network model 410-2 to output a schedule 430 that satisfies the constraints of Equation 1 above.

For example, the learning device 400-2 may input the expected transmission rate into the neural network model 410-2 and output the schedule 430. For example, the learning device 400-2 may calculate the loss 440 based on the output schedule 430.

For example, the neural network model 410-2 may include an activation function for outputting a schedule such that a resource block is allocated to only one of a plurality of terminals. For example, the schedule output through the activation function of the neural network model 410-2 meets the conditions of Equation 1 above. As such, the status in which a resource block is allocated to each terminal can be set to have one of two values, such as 0 or 1.

For example, the learning device 400-2 may calculate the loss 440 based on the loss function as shown in Equation 3 below. For example, according to Equation 3 below, if the size of the sum transmission rate according to the schedule 430 and the loss have a negative correlation and the fairness condition is not satisfied, the learning device 400-2 uses the schedule 430 ), the loss can be calculated based on a loss function set so that the fairness index and the loss have a negative correlation.

[Equation 3]

In Equation 3 above, silver and fairness (e.g., in Equation 3 ) Because the range of values is different, hyper parameters for normalization, In Equation 1 above, It may mean a hyperparameter corresponding to .

In Equation 3 above, The higher the sum, the smaller the loss, so the sum transmission rate and loss can have a negative correlation.

In Equation 3 above, if the fairness condition is not satisfied, for example, the fairness index The fairness index is set to the size of If smaller than, Is It can be. Fairness Index As is larger, the size of the loss can be smaller, and the fairness index and loss can have a negative correlation.

In Equation 3 above, if the fairness condition is satisfied, for example, the fairness index The fairness index is set to the size of If smaller than, can be 0.

For example, the learning device 400-2 can train the neural network model 410-2 using the loss 440 calculated according to Equation 3 above. For example, the learning device 400-2 may train the neural network model 410-2 to minimize the loss 440. For example, training the neural network model 410-2 to minimize the loss 440 calculated according to Equation 3 maximizes the sum transmission rate of the entire network and satisfies the set fairness conditions. This may mean learning -2).

FIG. 6 illustrates a method of learning a supervised learning-based neural network model (e.g., the neural network model 410-1 of FIG. 4) performed by a learning device (e.g., the learning device 400-1 of FIG. 4) according to various embodiments. This is the drawing shown.

Referring to FIG. 6, the learning device 400-1 according to various embodiments may obtain an expected data rate based on CQI indices for a plurality of terminals in operation 605. For example, the learning device 400-1 may calculate the expected transmission rate for each terminal and resource block based on the CQI index and spectrum efficiency according to MCS.

For example, the CQI index and expected transmission rate are training data for training the neural network model 410-1, and may be called learning CQI index and learning expected transmission rate, respectively.

For example, in operation 610, the learning device 400-1 may calculate the correct answer schedule based on the expected transmission rate. For example, the learning device 400-1 may calculate a correct answer schedule that satisfies set constraints.

For example, the set constraint condition is as shown in Equation 1 above, and the correct answer schedule can be calculated so that the sum data rate for a plurality of terminals is maximized, and the fairness index for the plurality of terminals is greater than or equal to the set fairness index. In Equation 1 above, the variable representing the scheduling status for the resource block of each terminal may be either 0 or 1, and each resource block may be allocated to only one terminal.

For example, the learning device 400-1 may calculate the correct answer schedule based on a constraint condition in which the condition of the variable representing the scheduling state for the resource block of each terminal is relaxed, as shown in Equation 2 above.

For example, the learning device 400-1 may input the expected transmission rate into a neural network model in operation 615 and output a schedule (e.g., schedule 430 in FIG. 4). For example, the learning device 400-1 may train the neural network model 410-1 based on the schedule and the correct answer schedule in operation 620.

For example, the learning device 400-1 may calculate the loss based on the schedule and the correct answer schedule and train the neural network model 410-1 to minimize the loss. For example, loss can refer to the correct answer schedule and the mse (mean squared error) of the schedule.

FIG. 7 shows learning of an unsupervised learning-based neural network model (e.g., the learning device 410-2 of FIG. 5) performed by a learning device (e.g., the learning device 400-2 of FIG. 5) according to various embodiments. This is a drawing showing the method.

Referring to FIG. 7, a learning device 400-2 according to various embodiments may obtain an expected data rate based on CQI indices for a plurality of terminals. For example, the description of operation 605 of FIG. 6 may be applied substantially identically to operation 705.

For example, in operation 710, the learning device inputs the expected transmission rate into the neural network model 410-2, and in operation 715, processes the output of the last layer of the neural network model 410-2 with an activation function to create a schedule (e.g. The schedule 430 of FIG. 5 can be output.

For example, the activation function processes the output of the continuous range of the last layer of the neural network model 410-2, and sets a condition regarding a variable indicating the state in which a resource block is allocated to each terminal in Equation 1 ( ), the schedule can have one of two values.

For example, the learning device 400-2 may train the neural network model 410-2 in operation 720 based on the loss calculated according to the schedule 430 and the constraint conditions. For example, if the constraint condition is to maximize the sum data rate of the entire network and satisfy the fairness conditions set by the fairness index for a plurality of terminals, the loss function related to the sum data rate and fairness condition as shown in Equation 3 above The loss can be calculated accordingly.

For example, the learning device 400-2 trains the neural network model 410-2 based on a loss function set according to a constraint condition, so that the schedule 430 output by the neural network model 410-2 is divided into a plurality of schedules. It is possible to maximize the sum data rate for the terminal and satisfy fairness conditions.

FIG. 8 is a diagram showing the network structure of a neural network model (e.g., the neural network model 410-1 in FIG. 4) learned based on supervised learning according to various embodiments.

Referring to FIG. 8, the neural network model 410-1 may include a DNN structure including a plurality of layers and an activation function (eg, Softmax in FIG. 8). For the neural network model 410-1 having the network structure shown in FIG. 8, a learning device (e.g., the learning device 400-1 in FIG. 4) creates the neural network model 410-1 based on a supervised learning method. It can be learned.

For example, the learning device 400-1 may calculate the correct answer schedule based on the expected transmission rate according to Equation 1 or Equation 2 above. The learning device 400-1 inputs the expected transmission rate into the neural network model 410-1 and uses the output schedule (e.g., schedule 430 in FIG. 4) and the loss calculated based on the correct answer schedule to create the neural network model. (410-1) can be learned.

FIG. 9 is a diagram illustrating the network structure of a neural network model (e.g., the neural network model 410-2 of FIG. 5) learned based on unsupervised learning according to various embodiments.

Referring to FIG. 9, the neural network model 410-2 may include a DNN structure including a plurality of layers and an activation function 450 (eg, Activation function in FIG. 9). For the neural network model 410-2 having the network structure shown in FIG. 9, a learning device (e.g., the learning device 400-2 in FIG. 5) creates the neural network model 410-2 based on an unsupervised learning method. can be learned.

For example, the learning device 400-2 uses the activation function 450, and in the neural network model 410-2, a variable indicating the state in which a resource block is allocated to each terminal is set to one of two values: For example, a schedule with a value such as 0 or 1 (e.g., schedule 430 in FIG. 5) can be output.

The learning device 400-2 may calculate a loss (eg, loss 440 in FIG. 5) based on a loss function according to set constraints. For example, as shown in Equation 3 above, the loss function may be set so that the total transmission rate of the entire network according to the output schedule 430 and the loss 440 have a negative correlation.

For example, as shown in Equation 3 above, the loss function may be set so that when the fairness index according to the schedule 430 is less than or equal to the set fairness index, the fairness index and the loss 440 have a negative correlation. For example, if the fairness index according to the schedule 430 is less than or equal to the set fairness index, it may be understood that the set fairness condition is not satisfied.

For example, the loss 440 is calculated according to the loss function according to the set constraints, the neural network model 410-2 is trained to minimize the loss 440, and a schedule 430 that satisfies the set constraints is created. The neural network model 410-2 can be trained to output.

FIG. 10 is a diagram illustrating an activation function 450 of a neural network model (e.g., the neural network model 410-2 of FIGS. 5 and 9) learned based on unsupervised learning according to various embodiments.

In Figure 10, It can mean representing all information of . For example, X may mean the output of the last layer of the neural network model 410-2 in FIG. 9.

Referring to FIG. 10, the activation function 450 according to various embodiments may refer to a function that processes the output X of the last layer of the neural network model 410-2 by repeating Equation 4 below and normalization n times. there is.

[Equation 4]

In Equation 4 above, may refer to the temperature parameter that sets the slope according to the input of the exponential function.

For example, the output X of the last layer of the neural network model 410-1 is processed according to the activation function 450, and the output schedule from each , can have one of two values, for example 0 or 1.

FIG. 11 shows an electronic device (e.g., the electronic device of FIG. 2) using a neural network model (e.g., the neural network model 410-1 of FIG. 4 and the neural network model 410-2 of FIG. 5) learned according to various embodiments. This is a diagram showing the performance of (200)).

Figure 11 is a diagram showing the performance of a neural network model 410-1 learned based on a supervised learning method and the performance of a neural network model 410-2 learned based on an unsupervised learning method.

For example, the CDF (cumulative distribution function) in Figure 11 is based on 25 resource blocks, 4 number of terminals, and 100,000 samples. This is an example showing the results.

For example, Table 1 below shows the parameters of the neural network model 410-1 learned based on supervised learning, and Table 2 shows the parameters of the neural network model 410-2 learned based on unsupervised learning.

[Table 1]

[Table 2]

Tables 3 and 4 below show the performance of the neural network model 410-1 learned based on the supervised learning method of FIG. 11 and the performance of the neural network model 410-2 learned based on the unsupervised learning method.

[Table 3]

[Table 4]

Referring to Figure 11, As the probability of satisfying increases (in Tables 3 and 4) ), for example, when resource blocks are allocated to a plurality of terminals according to each method, the probability that the fairness index for a plurality of terminals satisfies the fairness condition increases, and the existing method (e.g., Figure 11, In the case of Round Robin, Best CQI, and Convex Solver in Tables 3 and 4, it can be seen that the sum throughput (Sum throughput in Tables 3 and 4) decreases. Referring to Table 3 and Table 4, for existing methods (RR, Best-CQI), are 0.4949 and 0.0001, respectively, which does not guarantee fairness.

Referring to FIG. 11 and Tables 3 and 4, neural network models 410-1 and 420-2 learned according to various embodiments (e.g., Supervised, Unsupervised in FIG. 11, supervised learning in Tables 3 and 4, In the case of unsupervised learning, the sum transfer rate can be increased while guaranteeing given fairness conditions.

FIG. 12 is a diagram illustrating an operation of the electronic device 1200 to output an action according to various embodiments.

Referring to FIG. 12, the processor 1220 (e.g., the processor 120 of FIG. 1) of the electronic device 1200 (e.g., the electronic device 101 of FIG. 1) according to various embodiments is connected to a plurality of terminals. You can identify the CQI index for

For example, the electronic device 1200 may calculate the current state based on the CQI index. For example, the current state may include the average transmission rate of each of the plurality of terminals, the transmission rate, and the average fairness index for the plurality of terminals.

For example, the electronic device 1200 may calculate the transmission rate in substantially the same manner as the operation of the electronic device 200 in FIG. 2 to calculate the expected transmission rate.

For example, the processor 1220 of the electronic device 1200 sets the average transmission rate as shown in Equation 5 below: can be calculated. For example, the average transfer rate may mean the average transmission rate of the kth user at the nth time index.

[Equation 5]

In Equation 5 above, may mean the transmission rate of the kth user at the nth time index. is the average transmission rate It may mean the window time (window time size) of .

For example, the electronic device 1200 has an average transmission rate for each terminal. You can calculate the average fairness index using . For example, the average fairness index for multiple terminals is It can be calculated as

For example, the electronic device 1200 may input the current state into the neural network model 1210 and output an action 1240. For example, action 1240 may refer to a schedule for allocating resource blocks to a plurality of terminals during a set time.

For example, the neural network model 1210 may be learned according to a deep Q-network (DQN) learning method. For example, the neural network model 1210 may be learned using training data including the current state, action, reward, and next state. For example, the neural network model 1210 may be trained to output actions that maximize rewards by inputting training data. For example, the neural network model 1210 may calculate the Q-value according to the input current state and output an action 1240 regarding the maximum Q-value among the Q-values.

For example, a reward may be determined based on a reward function according to constraints set for allocating resource blocks to a plurality of terminals.

For example, the limiting conditions are i) the sum of the average data rates for multiple terminals is maximum, ii) the data rates for multiple terminals are greater than or equal to the set quality of service (QoS), and iii) each of the multiple terminals has This may be to ensure that the number of resource blocks that can be allocated is less than or equal to the set number of resource blocks, and iv) that the average fairness index for a plurality of terminals is greater than or equal to the set fairness index.

For example, the reward function may be determined based on i) the sum of average transmission rates for a plurality of terminals, ii) a fairness index for a plurality of terminals, iii) a set QoS and a set maximum number of resource blocks.

FIG. 13 is a diagram illustrating the operation of a scheduling method performed by an electronic device (e.g., the electronic device 1200 of FIG. 12) according to various embodiments.

Referring to FIG. 13, the electronic device 1200 according to various embodiments may obtain the current state based on the CQI index in operation 1305. For example, the current state may include the average transmission rate, transmission rate, and average fairness index for a plurality of terminals.

For example, the electronic device 1200 may use the neural network model learned in operation 1310 (e.g., the neural network model 1210 of FIG. 12) to determine an action for allocating resource blocks to a plurality of terminals according to the current state. there is.

For example, the neural network model 1210 may be trained to output an action that satisfies set constraint conditions. For example, the neural network model 1210 may be learned using learning data including the current state, action, reward, and next state (s, a, r, s') according to the DQN learning technique.

For example, neural network model 1210 can be trained to maximize rewards. For example, the reward may be determined by a reward function set to have a high value when a constraint condition is satisfied.

For example, the limiting condition may be to allocate resource blocks to a plurality of terminals so that the sum of the average transmission rates for the plurality of terminals is maximized and the average fairness index for the plurality of terminals is greater than or equal to a set fairness index. For example, the limiting condition may be to allocate resource blocks such that resource blocks are allocated below the maximum number of resource blocks set for each terminal and the transmission rate of each terminal satisfies the set minimum QoS condition.

FIG. 14 is a diagram illustrating an operation of a learning device 1400 (e.g., the electronic device 101 of FIG. 1 ) to learn a neural network model 1410 according to various embodiments.

Referring to FIG. 14, a learning device 1400 according to various embodiments may train a neural network model 1410 using training data. For example, the learning device 1400 may train the neural network model 1410 to output an action 1440 that satisfies Equation 6 below.

[Equation 6]

In Equation 6 above, is the average transmission rate of the kth terminal at the nth time index, is a variable indicating the state in which the mth resource block is allocated to the kth terminal at the nth time index (e.g., if 0, the resource block is not allocated, if 1, the resource block is allocated), is the set QoS condition, is a set fairness index (e.g. jain fairness index), is the transmission rate of the kth terminal at the nth time index, may mean the maximum number of resource blocks that can be allocated to one terminal.

For example, the learning device 1400 may train the neural network model 1410 to output an action 1440 that satisfies the constraints of Equation 6 above. The limiting conditions of Equation 6 are i) the sum of the average transmission rates for multiple terminals is maximum, ii) each resource block is assigned to only one terminal, and iii) the number of resource blocks that can be assigned to one terminal The maximum number is Below, iv) Transmission rate of each terminal this satisfies the QoS conditions set above, and v) the fairness index with the average fairness index set It can mean something more.

For example, the transmission rate of the kth terminal at the nth time index in Equation 6 above Can be calculated as in Equation 7 below.

[Equation 7]

In Equation 7 above, may mean the expected transmission rate when the mth resource block is allocated to the kth terminal at the nth time index. For example, the learning device 1400 performs an operation in which the electronic device 200 of FIG. 2 and the learning devices 400-1 and 400-2 of FIGS. 4 and 5 calculate an expected transmission rate using the CQI index. Expected transfer rates to be substantially the same can be calculated.

For example, the processor 1420 of the learning device 1400 (e.g., the processor 120 of FIG. 1) may train the neural network model 1410 using training data stored in the memory 1430. For example, learning data (s, a, r, s') may mean the current state, action, reward, and next state, respectively.

As an example, the neural network model 1410 may be learned according to the DQN learning method. For example, in the learning data (s, a, r, s'), the state of the environment changes from the current state s to the next state s' according to the action of the agent in the given environment, Rewards r according to actions can be collected using r.

For example, the agent's actions on the environment can be performed for a plurality of episodes and the learning data can be stored in the memory 1430. For example, memory 1430 may be referred to as replay memory. For example, at each time step the agent -Actions can be performed using a greedy method.

For example, learning data can be collected by performing an agent's action on the environment and storing the reward and next state according to the action in the memory 1430. Because the agent's action according to action a in the current state of the environment s affects the entire system, it may correspond to a finite Markov decision process (MDP).

For example, the learning device 1400 may perform a mini batch update by sampling the learning data stored in the memory 1430 using various methods, such as random sampling or sampling using prioritized experience replay (PER).

For example, the learning device 1400 may train the neural network model 1410 to estimate a function that determines the Q-value according to the DQN learning method. For example, the learning device 1400 can extract the Q-value as a result by using the current state as an input. For example, the neural network model 1410 may include a plurality of neural networks (NNs) and determine the Q-value.

For example, for action a performed on an agent, immediate and future rewards can be received from the environment. For example, immediate reward may refer to immediate reward that occurs for action a performed by the agent, and future reward may refer to reward to the future environment that appears due to action a.

For example, the agent's goal may mean maximizing rewards, e.g., updating the Q-value to maximize immediate and future rewards.

[Equation 8]

Equation 8 above is the current state action in Q-value for By updating, is the learning rate of Q-value between 0 and 1, is the discount factor for future compensation, is the maximum future reward, can mean immediate compensation.

For example, the neural network model 1410 can perform the process of selecting and evaluating actions using each network. For example, the neural network model 1410 may include a Q network 1410-1 and a target Q network 1410-2. In Equation 8 above, is a parameter of the Q network (1410-1), may mean the parameters of the target Q network 1410-2. For example, the target Q network 1410-2 may refer to a network that copies the parameters of the Q network at certain steps.

For example, the Q network 1410-1 may refer to a network for outputting Q-value, and the target Q network 1410-2 may refer to a network for evaluating actions. For example, the learning device 1400 inputs the current state and action (s, a) of the learning data to the Q network 1410-1 to obtain a Q-value can be output.

For example, the learning device 1400 inputs the next state (s') of the learning data into the target Q network 1410-2, can be output. for example, may mean the maximum value of future compensation for updating the Q-value in Equation 8 above. for example, may mean a value for evaluating an action.

For example, the Q-value that can be calculated using the Q network 1410-1 and the target Q network 1410-2 can be calculated as shown in Equation 9 below.

[Equation 9]

In Equation 9 above, is the Q-value (e.g. in Equation 8) ), can mean immediate compensation.

As an example, the reward function can be set as shown in Equation 10 below.

[Equation 10]

In Equation 10, F1, F2, and F2 may mean hyper parameters. For example, F1 is the sum of average transfer rates and set average fairness index It may refer to a variable for scaling the size of .

For example, F2 is the average fairness index with the average fairness index set If it is larger, it may mean a variable to respond to cases where fairness conditions are satisfied.

For example, F3 may mean a parameter corresponding to QoS conditions and a limit on the maximum number of resource blocks.

Table 5 below is a table showing the ranges of hyperparameters F1, F2, and F3 of Equation 10 according to an embodiment.

[Table 5]

In Table 5 above, the requirements for determining the range of F2 are as shown in Equation 6 above. It can mean a condition.

In Table 5 above, the requirements for determining the range of F3 are as shown in Equation 6 above. conditions and It can mean a condition.

According to Equation 10 above, the reward function is the sum of the average transmission rates , average fairness index , may be determined according to the set maximum number of resource blocks and the set QoS (e.g., hyperparameter F3).

The reward determined according to the reward function of Equation 10 may be determined as a high value if it satisfies the constraints of Equation 10, and may be determined as a low value if it does not satisfy the constraints of Equation 10. . For example, in the learning data (s, a, r, s'), if the result of action a performed in the current state s satisfies the constraint condition according to Equation 6 above, the reward satisfies the constraint condition. A larger value may be given than the reward value in case of not doing so.

The reward function of Equation 10 is set according to the constraints of Equation 6, and the reward function may be set differently from Equation 10 according to the set constraints.

For example, when allocating resource blocks to multiple terminals, various variables such as CQI, type of service, QoS, buffer status, priority, retransmission, and scheduling period can be considered. For example, when considering priority, unlike the constraints in Equation 6 above, the reward function can be set so that a high reward is given when allocating resource blocks according to priority.

For example, the learning device 1400 can calculate the loss as shown in Equation 11 below.

[Equation 11]

For example, the learning device 1400 may train the neural network model 1410 to minimize the loss according to Equation 11 above. Training the neural network model 1410 to minimize the loss according to Equation 11 may mean training the neural network model 1410 to maximize the reward according to Equation 10.

For example, in Equation 11 above, is the reward r of the training data, Mokpo Q Network (1410-2), may refer to data output from the Q network 1410-1.

For example, the reward may be determined according to a reward function (e.g., Equation 8) set according to the constraints of Equation 6. Accordingly, in the above example, the action 1440 output by the neural network model 1410 learned to maximize the reward may satisfy the constraint condition according to Equation 6.

As an example, the learning device 1400 may be trained to output an action 1440 that maximizes the reward. For example, the learning device 1400 learns the neural network model 1410 to output an action 1440 that maximizes the sum data rate for a plurality of terminals and ensures that the average fairness index of the plurality of terminals is greater than or equal to the set fairness index. You can do it. For example, in the action 1440 output by the learned neural network model 1410, the number of resource blocks that can be allocated to each of a plurality of terminals is less than or equal to the set number of resource blocks, and the average fairness index for the plurality of terminals This may be to ensure that is greater than or equal to the set fairness index.

For example, the learning device 1400 may sample learning data using prioritized experience replay (PER). PER refers to a technique of performing a mini batch update by randomly sampling learning data stored in the memory 1430, and sampling the stored data according to priority. For example, if you continue to learn using only the training data that minimizes the loss in Equation 11, updates are continuously made only when the loss is replayed, so there is no opportunity to repeat it again in case the initial loss is large. This may result in over-fitting problems for specific transitions due to insufficient learning from diverse experiences.

As an example, the learning device may sample learning data according to Equation 12 below. For example, Equation 12 may mean the probability of applying uniform random sampling and sampling techniques that consider priority.

[Equation 12]

In Equation 12 above, is the sampling probability of the ith transition among k transitions, may refer to a parameter that determines how much priority sampling is performed. For example, equation 12 is If is 0, uniform sampling, If is 1, it may mean greedy-prioritization sampling.

In Equation 12 above, is a parameter that is set to be proportional according to the loss in Equation 11 using a proportional prioritization method, and can be set so that the visit probability of all transitions is not 0.

For example, prioritized replay according to Equation 12 above tends to result in biased information. For example, the learning device 1400 may correct the bias value using a weight according to Equation 13 below.

[Equation 13]

In Equation 13 above, is the IS weight (importance-sampling weight), may refer to a parameter that adjusts the bias value. for example, increases linearly at each learning stage and can be set to 1 at the last stage of learning.

For example, the learning device uses the parameters of Equation 12 and Equation 13 , By adjusting the size of the gradient of the loss, the stability and performance of the learned neural network model 1410 can be improved.

FIG. 15 is a diagram illustrating the operation of a method for learning a neural network model (e.g., the neural network model 1410 of FIG. 14 ) performed by a learning device (e.g., the learning device 1400 of FIG. 14 ) according to various embodiments.

Referring to FIG. 15, the learning device 1400 according to various embodiments may allocate resource blocks to a plurality of terminals for a set time in operation 1505 and collect learning data. For example, learning data can be collected by performing the agent's actions on the environment and including current state, action, reward, and next state.

For example, the learning device 1400 may determine the Q value according to learning data input to the neural network model 1410 in operation 1510. For example, the neural network model may include a Q network 1410-1 and a target Q network 1410-2.

For example, the learning device 1400 may use the loss calculated based on the Q value in operation 1515 to train the neural network model 1410 to output an action that maximizes the reward in the current state for learning.

For example, the learning device 1400 may calculate the loss according to Equation 11 using the output of the Q network 1410, the output of the target Q network, and the reward of the learning data. The learning device 1400 may train the neural network model 1410 to minimize loss.

FIG. 16 illustrates an operation of an electronic device (e.g., the electronic device 1200 of FIG. 12) dividing a plurality of terminals or resource blocks into subgroups (e.g., sub CQI tables 1620 and 1630) according to various embodiments. This is the drawing shown.

For example, the space of states and actions may increase depending on the number of terminals to which resource blocks can be allocated or the number of resource blocks. As the number of terminals or the number of resource blocks increases, the cost of time and calculation amount required for learning a neural network model (e.g., neural network model 1410 in FIG. 14) increases.

Since the number of terminals and/or the number of resource blocks may vary depending on the communication situation, it may be necessary to reduce the cost required for learning the neural network model 1410 and to utilize the neural network model 1410 that can be applied to each case. there is.

For example, a learning device (e.g., learning device 1400 in FIG. 14) may divide the large-dimensional CQI table 1610 into a plurality of sub CQI tables 1620 and 1630. The sub CQI tables 1620 and 1630 shown in FIG. 16 illustrate the case of dividing a resource block. However, unlike the embodiment shown in FIG. 16, a sub CQI table is generated by dividing a plurality of terminals or a plurality of terminals are divided. A sub CQI table can be created by dividing the terminal and resource blocks.

For example, the learning device 1400 can learn a plurality of neural network models using each sub CQI table (1620, 1630). For example, the learning device 1400 may learn neural network model 1 using the sub CQI table 1620 and neural network model 2 using the sub CQI table 1630.

For example, the electronic device 1200 may output an action to allocate a plurality of resource blocks to a terminal using a plurality of learned neural network models. For example, the electronic device 1200 may identify CQI indices for a plurality of terminals and divide the CQI table 1610 into a plurality of sub CQI tables 1620 and 1630, as shown in FIG. 16. The divided sub CQI tables 1620 and 1630 may each be called subgroups.

For example, the electronic device 1200 may output an action by inputting the sub CQI table 1620 into learned neural network model 1, and output an action by inputting the sub CQI table 1630 into learned neural network model 2. there is. The electronic device 1200 may combine the actions output from learned neural network models 1 and 2 and output an action for allocating the entire resource block to a plurality of terminals.

In an embodiment different from the embodiment shown in FIG. 16, the electronic device 1200 or the learning device 1400 may divide the CQI table 1610 into a plurality of terminals and determine a sub CQI table. For example, the CQI table for a plurality of UEs 1 to 6 can be divided to determine a sub CQI table for UEs 1 to 3 and a sub CQI table for UEs 4 to 6.

For example, when a sub CQI table is generated by dividing a plurality of terminals, the processors 1220 and 1420 of the electronic device 1200 or the learning device 1400 may prevent resource blocks from being allocated to the same terminal.

For example, the processor 1220 of the electronic device 1200 may output an action to prevent resource blocks from being allocated to the same terminal. For example, the processor 1420 of the learning device 1400 may train the neural network model 1410 to output an action to prevent resource blocks from being allocated to the same terminal.

In FIG. 16, the electronic device 1200 of FIG. 12 or the learning device 1400 of FIG. 14 divides the CQI table 1610 into sub CQI tables 1620 and 1630, and creates the sub CQI tables 1620 and 1630. An operation of learning a plurality of neural network models using a plurality of neural network models or an operation of outputting an action using a plurality of neural network models is described, but is not limited thereto.

For example, for the electronic device 200 of FIG. 2 or the learning devices 400-1 and 400-2 of FIGS. 4 and 5, the operation of the electronic device 1200 or the learning device 1400 of FIG. 16 The description can be applied practically in the same way.

FIG. 17 is a diagram showing the performance of a neural network model (e.g., the neural network model 1410 of FIG. 14) learned according to various embodiments.

The graph shown in FIG. 17 represents the sum throughput, fairness index, and UE throughput for a plurality of terminals, respectively.

For example, the neural network model 1410 can be learned according to the parameters in Table 6 below.

[Table 6]

Referring to FIG. 17, a neural network model 1410 learned according to various embodiments shows improved performance compared to an existing algorithm or method for allocating resource blocks. Although the sum data rate according to the existing Best-CQI method is higher than the sum data rate of the learned neural network model (1410), it does not satisfy the fairness conditions and minimum QoS conditions.

Referring to FIG. 17, it can be seen that the learned neural network model 1410 is similar to the PF or Max-min method and has a fairness index higher than target fairness.

Table 7 below is a table showing the probability of not satisfying the minimum QoS (outage probability). In Table 7, it can be seen that about 2.1% (e.g., DDQN+PER) of the actions output by the electronic device 1200 using the neural network model 1410 do not satisfy the minimum QoS condition, and higher performance compared to other methods. It can be confirmed that it has .

[Table 7]

An electronic device 200 (e.g., the electronic device 101 of FIG. 1) according to various embodiments is electrically connected to a processor 220 (e.g., the processor 120 of FIG. 1) and the processor 220, It includes a memory (e.g., memory 130 in FIG. 1) that stores instructions that can be executed by the processor, and the processor 220, when the instruction is executed, determines the CQI index (channel quality) for a plurality of terminals. Based on the information index, the expected transmission rate (achievable rate) when resource blocks are allocated to the plurality of terminals is obtained, and the expected transmission rate is calculated using the learned neural network model 210 (e.g., neural network model 410 in FIG. 4) -1), input to the neural network model 410-2) of FIG. 5 to output a schedule for allocating the resource block to the plurality of terminals, and the neural network model 210 to the plurality of terminals. Collect a learning CQI index regarding the learning CQI index, obtain a learning expected data rate when the resource block is allocated to the plurality of terminals based on the learning CQI index, and use the input learning expected data rate to obtain a learning expected data rate for the plurality of terminals. It can be learned to output the schedule that maximizes the total data rate and satisfies the fairness conditions set by the fairness index for the plurality of terminals.

The neural network model 210 maximizes the sum data rate for the plurality of terminals based on the expected data rate, calculates a correct answer schedule that satisfies the fairness condition, and uses the expected data rate and the correct answer schedule. Therefore, it can be learned based on a supervised learning method.

The neural network model 210 may be learned based on an unsupervised learning method.

The neural network model 210 may include an activation function for outputting the schedule such that the resource block is allocated to only one of the plurality of terminals.

The neural network model 210 has a negative correlation between the size of the sum data rate according to the schedule and the loss, and when the fairness condition is not satisfied, the fairness index according to the schedule and the loss have a negative correlation. It can be learned based on a loss function set to have.

The electronic device 1200 (e.g., the electronic device 101 of FIG. 1) according to various embodiments is electrically connected to a processor 1220 (e.g., the processor 120 of FIG. 1) and the processor, and the processor It includes a memory (e.g., memory 130 in FIG. 1) that stores instructions that can be executed by, and the processor 1220, when the instruction is executed, CQI index (channel quality information index) for a plurality of terminals. ) Based on the average transmission rate of each of the plurality of terminals, the current state including the transmission rate and the average fairness index for the plurality of terminals is obtained, and according to the learning method of DQN (deep Q-network) Using the learned neural network model 1210 (e.g., the neural network model 1410 in FIG. 14), an action for allocating the resource block to the plurality of terminals is determined according to the current state, and the neural network model ( 1210) is learned to output the action that maximizes the reward in the current state, and the reward may be determined based on a reward function according to a constraint condition set to allocate the resource block to the plurality of terminals. .

The limiting condition is that the sum of the average data rates for the plurality of terminals is maximum, the data rate for the plurality of terminals is greater than or equal to a set quality of service (QoS), and the data rate allocated to each of the plurality of terminals is The number of available resource blocks can be set to be less than or equal to the set maximum number of resource blocks, and the average fairness index for the plurality of terminals can be set to be greater than or equal to the set fairness index.

The reward may be determined according to a reward function determined based on the sum of the average data rates for the plurality of terminals, the average fairness index for the plurality of terminals, the set QoS, and the set maximum number of resource blocks.

The processor 1220 determines a plurality of subgroups 1620 and 1630 by dividing at least one of the plurality of terminals or the resource block according to the number of the plurality of terminals or the number of the resource blocks, The current state for each of the plurality of subgroups 1620 and 1630 may be input to the neural network model 1210, and the action for each of the plurality of subgroups may be output.

The neural network model 1210 can be learned using a Q network 1410-1 for outputting the Q value according to the action and a target Q network 1410-2 for evaluating the action.

The scheduling method according to various embodiments is based on a CQI index (channel quality information index) for a plurality of terminals, an average transmission rate for each of the plurality of terminals, a transmission rate, and an average fairness index for the plurality of terminals. ) and allocate the resource blocks to the plurality of terminals according to the current state using a neural network model 1210 learned according to the learning method of DQN (deep Q-network) and the operation of acquiring the current state including and determining an action to take, wherein the neural network model 1210 is learned to output the action that maximizes the reward in the current state, and the reward allocates the resource block to the plurality of terminals. It may be determined based on the reward function according to the constraints set for this purpose.

The limiting condition is that the sum of the average data rates for the plurality of terminals is maximum, the data rate for the plurality of terminals is greater than or equal to a set quality of service (QoS), and the data rate allocated to each of the plurality of terminals is The number of available resource blocks can be set to be less than or equal to the set maximum number of resource blocks, and the average fairness index for a plurality of terminals can be set to be greater than or equal to the set fairness index.

The reward may be determined according to a reward function determined based on the sum of the average data rates for the plurality of terminals, the average fairness index for the plurality of terminals, the set QoS, and the set maximum number of resource blocks.

The scheduling method further includes determining a plurality of subgroups 1620 and 1630 by dividing at least one of the plurality of terminals or the resource block according to the number of the plurality of terminals or the number of the resource block. The operation of determining the action includes inputting the current state for each of the plurality of subgroups 1620 and 1630 into the neural network model 1210 and outputting the action for each of the plurality of subgroups. can do.

The neural network model can be learned using a DDQN model, a Q network for outputting the action, and a target network for evaluating the action.

A neural network model learning method according to various embodiments includes the operation of allocating resource blocks for a set time to a plurality of terminals and collecting learning data including the current state, action, reward, and next state, and the neural network model 1410. The neural network model outputs the action 1440 of determining the Q value according to the input learning data and the action 1440 of maximizing the reward in the current state using the loss 1450 calculated based on the Q value. It includes an operation of learning (1410), and the reward may be determined based on a reward function according to a constraint condition set for allocating the resource block to the plurality of terminals.

The limiting condition is that the sum of the average data rates for the plurality of terminals is maximum, the data rate for the plurality of terminals is greater than the set quality of service (QoS), and the data rate that can be assigned to each of the plurality of terminals is the maximum. It is possible to ensure that the number of resource blocks is less than or equal to the set maximum number of resource blocks, and that the average fairness index for the plurality of terminals is greater than or equal to the set fairness index.

The neural network model learning method includes determining a sampling probability based on a first parameter regarding whether to sample the learning data according to priority, and using a weight according to a second parameter that increases linearly for each learning step. , an operation of correcting a bias value of the sampling probability, and an operation of sampling the learning data using the sampling probability corrected according to the bias value.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Claims (18)

In electronic devices,
processor; and
Memory electrically connected to the processor and storing instructions that can be executed by the processor
Including,
The processor,
When the command is executed, an achievable rate is obtained when a resource block is allocated to the plurality of terminals based on a CQI index (channel quality information index) for the plurality of terminals,
Input the expected data rate into a learned neural network model to output a schedule for allocating the resource blocks to the plurality of terminals,
The neural network model is,
Collect learning CQI indices for the plurality of terminals,
Based on the learning CQI index, obtain a learning expected data rate when the resource block is allocated to the plurality of terminals,
Using the input expected learning data rate, the schedule is learned to output the schedule that maximizes the total data rate for the plurality of terminals and satisfies the fairness condition in which the fairness index for the plurality of terminals is set. Device. According to paragraph 1,
The neural network model is,
Based on the expected data rate, calculate a correct schedule that maximizes the sum data rate for the plurality of terminals and satisfies the fairness condition,
An electronic device that is trained based on a supervised learning method using the expected transmission rate and the correct answer schedule. According to paragraph 1,
The neural network model is,
An electronic device that learns based on unsupervised learning methods. According to paragraph 3,
The neural network model is,
An electronic device comprising an activation function for outputting the schedule so that the resource block is allocated to only one of the plurality of terminals. According to paragraph 1,
The neural network model is,
The size and loss of the sum data rate according to the schedule have a negative correlation,
When the fairness condition is not satisfied, the electronic device learns based on a loss function set so that the fairness index according to the schedule and the loss have a negative correlation. In electronic devices,
processor; and
Memory electrically connected to the processor and storing instructions that can be executed by the processor
Including,
The processor,
When the command is executed, based on a CQI index (channel quality information index) for a plurality of terminals, including an average transmission rate of each of the plurality of terminals, a transmission rate, and an average fairness index for the plurality of terminals. obtain the current state,
Using a neural network model learned according to a deep Q-network (DQN) learning method, determine an action for allocating resource blocks to the plurality of terminals according to the current state,
The neural network model is,
Learned to output the action that maximizes the reward in the current state,
The above rewards are,
An electronic device determined based on a reward function according to constraints set to allocate the resource block to the plurality of terminals. According to clause 6,
The above limiting conditions are:
The sum of the average data rates for the plurality of terminals becomes the maximum,
The transmission rate for the plurality of terminals is greater than or equal to a set quality of service (QoS),
Ensure that the number of resource blocks that can be allocated to each of the plurality of terminals is less than or equal to the set maximum number of resource blocks,
An electronic device that ensures that the average fairness index for the plurality of terminals is greater than or equal to a set fairness index. According to clause 6,
The above rewards are,
An electronic device determined according to a reward function determined based on the sum of the average data rates for the plurality of terminals, the average fairness index for the plurality of terminals, a set QoS, and a set maximum number of resource blocks. According to clause 6,
The processor,
Depending on the number of the plurality of terminals or the number of the resource blocks, dividing at least one of the plurality of terminals or the resource block to determine a plurality of subgroups,
An electronic device that inputs the current state for each of the plurality of subgroups into the neural network model and outputs the action for each of the plurality of subgroups. According to clause 6,
The neural network model is,
An electronic device that is learned using a Q network for outputting the Q value according to the action and a target Q network for evaluating the action. Based on the CQI index (channel quality information index) for a plurality of terminals, an operation of obtaining a current state including an average transmission rate of each of the plurality of terminals, a transmission rate, and an average fairness index for the plurality of terminals. ; and
An operation of determining an action for allocating resource blocks to the plurality of terminals according to the current state using a neural network model learned according to the learning method of DQN (deep Q-network)
Including,
The neural network model is,
Learned to output the action that maximizes the reward in the current state,
The above rewards are,
A scheduling method determined based on a reward function according to constraints set to allocate the resource blocks to the plurality of terminals. According to clause 11,
The above limiting conditions are:
The sum of the average data rates for the plurality of terminals becomes the maximum,
The transmission rate for the plurality of terminals is greater than or equal to a set quality of service (QoS),
Ensure that the number of resource blocks that can be allocated to each of the plurality of terminals is less than or equal to the set maximum number of resource blocks,
A scheduling method that ensures that the average fairness index for a plurality of terminals is greater than or equal to a set fairness index. According to clause 11,
The above rewards are,
A scheduling method determined according to a reward function determined based on the sum of the average data rates for the plurality of terminals, the average fairness index for the plurality of terminals, the set QoS, and the set maximum number of resource blocks. According to clause 11,
An operation to determine a plurality of subgroups by dividing at least one of the plurality of terminals or the resource block according to the number of the plurality of terminals or the number of the resource blocks.
It further includes,
The operation that determines the action is,
A scheduling method for inputting the current state for each of the plurality of subgroups into the neural network model and outputting the action for each of the plurality of subgroups. According to clause 11,
The neural network model is,
A scheduling method that is learned using a DDQN model, a Q network for outputting the action, and a target network for evaluating the action. An operation of allocating resource blocks for a set time to a plurality of terminals and collecting learning data including current state, action, reward, and next state;
An operation of determining a Q value according to learning data input to a neural network model; and
An operation of training the neural network model to output the action that maximizes the reward in the current state, using the loss calculated based on the Q value.
Including,
The above rewards are,
A method of learning a neural network model, which is determined based on a reward function according to constraints set to allocate the resource blocks to the plurality of terminals. According to clause 16,
The above limiting conditions are:
The sum of the average data rates for the plurality of terminals becomes the maximum,
The transmission rate for the plurality of terminals is higher than the set quality of service (QoS),
Ensure that the number of resource blocks that can be allocated to each of the plurality of terminals is less than or equal to the set maximum number of resource blocks,
A neural network model learning method that ensures that the average fairness index for the plurality of terminals is greater than or equal to a set fairness index. According to clause 16,
determining a sampling probability based on a first parameter regarding whether to sample the learning data according to priority;
Correcting a bias value of the sampling probability using a weight according to a second parameter that increases linearly in each learning step; and
An operation of sampling the learning data using the sampling probability corrected according to the bias value.
A neural network model learning method further comprising:

Images