KR102234049B1 - Receiver, system and method for adaptive modulation based on reinforcement learning - Google Patents

Abstract

According to the present invention, a receiver, to which a reinforcement learning-based adaptive modulation/demodulation way is applied, comprises: a receiving unit for estimating a state of a channel by receiving a pilot signal from a transmitter; a modulation/demodulation selecting unit for selecting a modulation/demodulation way according to the state of the channel based on reinforcement learning; and a feedback unit for transmitting information of the selected modulation/demodulation way to the transmitter, wherein the modulation/demodulation selecting unit adjusts a boundary value of a signal-to-noise ratio of the channel and gives a reward in a direction of maximizing a spectral efficiency value so as to perform reinforcement learning.

Description

Receiver, system and method for adaptive modulation and demodulation based on reinforcement learning {RECEIVER, SYSTEM AND METHOD FOR ADAPTIVE MODULATION BASED ON REINFORCEMENT LEARNING}

The present invention relates to a receiver, a system, and a method for adaptive modulation and demodulation based on reinforcement learning, and the receiver, system, and method for determining a modulation/demodulation method based on reinforcement learning according to a state of a communication channel. It is about.

The conventional adaptive modem system has an inefficient level of allocation criteria for modems. In addition, there are studies that have been approached through various optimization techniques previously, but it is difficult to solve the optimal solution in a clear form, and numerical analysis techniques and methods based on experience are the main ones.

In the field of wireless communication, artificial intelligence techniques are widely applied to automated selection tasks. However, in the case of supervised learning among artificial intelligence techniques, a vast amount of data is required when training an artificial intelligence model, and insufficient data degrades the learning rate of the artificial intelligence model.

On the other hand, the reinforcement learning technique among artificial intelligence techniques is a method in which the reinforcement learning model is executed and rewards are given without additional external data, and learning in the direction of maximizing the reward.

On the other hand, the conventional reinforcement learning-based adaptive modulation and demodulation technique has a problem of complexity of an algorithm and difficulty in deriving an optimal solution accordingly (see Non-Patent Document 1).

In addition, the conventional deep Q network learning technique has problems of algorithm instability and divergence (see Non-Patent Document 2).

J. P. Leite, P. H. P. de Carvalho and R. D. Vieira, "A flexible framework based on reinforcement learning for adaptive modulation and coding in OFDM systems," Proc. 2012 IEEE Wireless Communications and Networking Conference (WCNC), pp. 809-814, Shanghai, China, Apr. 2012. V. Mnih, et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, Feb. 2015. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA, USA: MIT Press, 2018. D. P. Kingma and J. L. Ba, "ADAM: A method for stochastic optimization," Proc. 3rd International Conference for Learning Representations, pp. 1-41, San Diego, USA, 2015. S. T. Chung and A. J. Goldsmith, "Degrees of freedom in adaptive modulation: A unified view," IEEE Trans. on Communications, vol. 49, no. 9, pp. 1561-1571, Sept. 2001. X. Song and J. Cheng, "Quantifying the accuracy of high SNR BER approximation of MPSK in fading channels," Proc. 2014 27th Biennial Symposium on Communications (QBSC), pp. 105-108, Kingston, ON, Canada, June 2014. M. K. Simon and M.-S. Alouini, Digital Communication over Fading Channels: A Unified Approach to Performance Analysis, Wiley-Interscience, 2000.

In order to solve the above-described problem, the present invention provides a receiver, a system, and a method for maximizing transmission efficiency by selecting a criterion for selecting a modulation/demodulation method suitable for a channel state through reinforcement learning and allocating an optimal modulation and demodulation method. There is a purpose to do it.

Other objects not specified of the present invention may be additionally considered within a range that can be easily deduced from the following detailed description and effects thereof.

According to an aspect of the present embodiment for achieving the above object, a receiver to which an adaptive modulation and demodulation method based on reinforcement learning is applied, comprising: a receiver configured to receive a pilot signal from a transmitter to estimate a channel state; A modulation/demodulation selection unit for selecting a modulation/demodulation method according to the state of the channel based on reinforcement learning; And a feedback unit that transmits information of the selected modulation and demodulation method to the transmitter, and the modulation and demodulation selection unit adjusts a boundary value of a signal-to-noise ratio of a channel and performs reinforcement learning by giving a reward in a direction maximizing a spectral efficiency value.

The reinforcement learning is learning by a deep Q network algorithm, and a state of the deep Q network algorithm is set as a threshold value for assigning the modulation scheme, and an action is adjusted to raise or lower the threshold value. Set.

The modulation/demodulation selection unit is characterized in that at the start of every trial, when the bit error probability exceeds a preset bit error probability reference value, the threshold value corresponding to the optimal spectral efficiency value is retrieved from the state-value buffer and the trial is started. do.

In addition, according to another aspect of the present embodiment, there is provided a method for modulation and demodulation based on reinforcement learning by a receiver, the method comprising: receiving a pilot signal from a transmitter; Estimating the state of the channel; Selecting a modulation/demodulation method according to the state of the channel based on reinforcement learning; And transmitting information of a selected modulation/demodulation method to the transmitter, wherein the reinforcement learning adjusts a boundary value of a signal-to-noise ratio of a channel to give a reward in a direction of maximizing a spectral efficiency value.

In addition, according to another aspect of the present embodiment, the reinforcement learning-based adaptive modulation and demodulation system includes: a transmitter for transmitting a pilot signal; And a receiver configured to receive the pilot signal to estimate a channel state, select a modem based on the channel state based on reinforcement learning, and transmit information of the selected modem to the transmitter, wherein the receiver comprises a channel Reinforcement learning is performed by adjusting the boundary value of the signal-to-noise ratio of and giving a reward in the direction of maximizing the spectral efficiency value.

As described above, according to the embodiments of the present invention, transmission efficiency can be maximized by selecting a criterion for selecting a modulation/demodulation method suitable for a channel state through reinforcement learning and allocating an optimal modulation/demodulation method. .

In addition, according to embodiments of the present invention, there is an effect of deriving a boundary value having an optimal spectral efficiency value within a probability density distribution range of a signal-to-noise ratio of a channel and a predetermined bit error probability limitation condition.

In addition, according to the embodiments of the present invention, there is an effect of rapidly converging to an optimal modulation/demodulation method boundary value.

Even if it is an effect not explicitly mentioned herein, the effect described in the following specification expected by the technical features of the present invention and the provisional effect thereof are treated as described in the specification of the present invention.

1 is a block diagram schematically showing the configuration of an adaptive modem system based on reinforcement learning according to embodiments of the present invention.
2 is a flowchart illustrating a modulation and demodulation method of an adaptive modulation and demodulation system based on reinforcement learning according to embodiments of the present invention.
3 is a block diagram schematically showing the configuration of a modulation/demodulation selection unit based on reinforcement learning of a receiver according to embodiments of the present invention.
4 is a flowchart illustrating a modulation and demodulation method of a modulation/demodulation selection unit based on reinforcement learning of a receiver according to embodiments of the present invention.
5 is a graph showing a simulation result of a modulation system based on reinforcement learning according to embodiments of the present invention.
6 is a graph showing a simulation result in which an optimal boundary value of a modulation system based on reinforcement learning according to embodiments of the present invention is not called.
7 is a graph showing a simulation result in which an optimal boundary value of a modulation system based on reinforcement learning according to embodiments of the present invention is called.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

Unless otherwise defined, all terms used herein including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present application. Does not.

Hereinafter, an adaptive modulation and demodulation system based on reinforcement learning will be described with reference to the accompanying drawings.

1 is a block diagram schematically showing the configuration of an adaptive modem system based on reinforcement learning according to embodiments of the present invention.

An adaptive modulation and demodulation system based on reinforcement learning includes a transmitter 100 and a receiver 200.

The transmitter 100 includes a transmission unit 110 and a modulator 120, and the receiver 200 includes a reception unit 210, a channel detection unit 220, a modem selection unit 230, a feedback unit 240, and a demodulation unit. It may include (250).

The communication system may further include other components required for wireless communication, may include an antenna that emits electromagnetic waves, and may include a filter or amplifier required for signal processing.

The transmitter 110 transmits a pilot signal.

The modulator 120 modulates a signal based on the modulation/demodulation method information received from the feedback unit 230 to be described later.

The receiver 210 receives a pilot signal from the transmitter 100.

The channel detector 220 estimates a channel state based on the pilot signal. The estimated channel is derived in the form of a probability density distribution function.

Various wireless communication channel models can be applied to the channel according to the embodiment of the present invention, and for example, the Rayleigh fading channel model can be applied to the channel according to the embodiment of the present invention.

The modulation/demodulation selection unit 230 selects a modulation/demodulation method according to the state of the channel based on reinforcement learning. The modulation/demodulation selection unit 230 performs reinforcement learning by adjusting the boundary value of the signal-to-noise ratio of the channel and giving a reward in the direction of maximizing the spectral efficiency value. Reinforcement learning is learning by a deep Q network algorithm, and the state of the deep Q network algorithm is set as a threshold value to which the modulation method is assigned, and an action is set as an adjustment to raise or lower the threshold value. do.

Preferably, the threshold value is a reference value for dividing the signal-to-noise ratio probability density distribution function of the channel into a plurality of modulation/demodulation scheme allocation regions.

At the start of each trial, the modem selection unit 230 fetches a threshold value corresponding to the pre-stored optimal spectral efficiency value and starts the trial.

The modem and demodulation selection unit 230 calculates a bit error probability for a signal-to-noise ratio for a channel, and when the bit error probability exceeds a preset bit error probability reference value, corresponds to the optimal spectral efficiency value from the state-value buffer. It loads the threshold value and starts the trial.

When the modulation/demodulation selection unit 230 starts every trial and when the bit error probability exceeds a preset reference value, by calling the threshold value having the highest spectral efficiency value among the result values that have been executed before and starting the trial, further The reinforcement learning model converges quickly.

The feedback unit 240 transmits the selected modulation/demodulation method information to the transmitter 100. When the transmitter 100 fails to receive a signal certifying that the modem information has been smoothly received, the feedback unit 240 may continuously repeatedly transmit the selected modem information.

The demodulation unit 250 demodulates a signal received from the transmitter 100 and extracts data according to the selected modulation/demodulation method.

2 is a flowchart illustrating a modulation and demodulation method of an adaptive modulation and demodulation system based on reinforcement learning according to embodiments of the present invention.

The transmitter transmits a pilot signal for estimating the channel condition.

The receiver receives a pilot signal from the transmitter (S210).

The receiver estimates the state of the channel (S220). Since channel estimation using a pilot signal is a conventional technique, a detailed description will be omitted.

The receiver selects a modulation and demodulation method according to the state of the channel based on reinforcement learning (S230). A method of selecting a modulation/demodulation method according to the state of a channel based on reinforcement learning will be described later with reference to FIGS. 3 to 7.

The receiver transmits the selected modulation/demodulation method information to the transmitter (S240).

The transmitter receives the modem information, modulates and transmits the data according to the modem (S250).

The receiver receives the modulated data and demodulates it using a demodulation method corresponding to the modulation method (S260).

3 is a block diagram schematically showing the configuration of a modulation/demodulation selection unit based on reinforcement learning of a receiver according to embodiments of the present invention.

The modulation/demodulation selection unit may include a state-value buffer 211, an environment model 212, an iterative memory 213, a main network 214, a target network 215, and an optimization model 216.

The state-value buffer 211 stores an optimal spectral efficiency value for each trial and a modulation/demodulation allocation boundary value in the signal-to-noise ratio domain.

The environmental model 320 adjusts state, action, and reward.

In the environmental model 320, a state is a threshold value to which the modulation scheme is assigned, an action is an adjustment of raising or lowering the threshold value, and a compensation is given in the direction of maximizing the spectral efficiency value. In addition, the environmental model 320 sets a penalty if the boundary is in an invalid state.

S={s1, s2, s3} is a state value, and is a threshold value assigned to a modem in a wireless communication environment. The unit of the threshold value is dB. Modulation and demodulation methods are allocated for each segmented area based on the boundary value in the area of the signal-to-noise ratio of the channel.

In the reinforcement learning model according to an embodiment of the present invention, the signal-to-noise ratio area is set from 0 dB to 25 dB, and the boundary values are set to three, thereby adopting four modulation and demodulation methods. Four modulation and demodulation methods can be set to QPSK, 8PSK, 16PSK, and 32PSK. For example, a QPSK scheme may be allocated between 0 dB and s1 dB, and an 8PSK scheme may be allocated between s2 dB and s3 dB.

A={a1, a2, ..., a6} is an action value, which is an action that the reinforcement learning model can take. Each of the two actions sequentially corresponds to three threshold values. For example, if the reinforcement learning model takes the action of a1, the value of s1 is adjusted downward. Conversely, if the reinforcement learning model takes the action of a2, the value of s1 is adjusted upward. In a similar way, other actions act in response to the threshold.

R={r1, r2} is a reward value, which is a reward that the reinforcement learning model can receive.

Compensation is given through r1 when performance improvement is made, and penalty is given through r2 when the threshold value is not valid. In this case, the penalty is a negative compensation value. In order to effectively prevent going into an invalid state, the size of the penalty can be set larger than the size of the compensation. For example, the size may be set to r2=1.25*r1.

The state in which the boundary is not valid may be set to satisfy three conditions. The first condition corresponds to a case in which the next state in the deep Q network model is less than or equal to the previous state, so that the high boundary invades the lower boundary. That is, the first condition is a case in which a higher modem threshold value violates a lower threshold value.

The second condition is a case where the boundary exceeds the preset signal-to-noise ratio range. For example, the range of the signal-to-noise ratio may be set to 0 dB to 25 dB. In this case, the second condition may be out of 0 dB to 25 dB.

The third condition is a case where the neighboring boundary value is set to be narrower than a preset interval. For example, the third condition may be a case where an interval between neighboring boundary values is narrower than 3 dB.

The repetition memory 213 stores s _t , a _t , r _t , and s _t+1 information. The subscript t represents a discrete unit of time, and each element is given a value that fits the unit of time described in the environmental model.

The main network 214 and the target network 215 are composed of an artificial neural network structure. The artificial neural network structure is a network with connected layers and is a model that learns weights and biases.

The main network 214 interacts with the environment to store the result of the action and the next state in the iterative memory 213, and randomly sample it at each trial period designated by the system.

The optimization model 216 defines a loss function based on the output values of the main network 214 and the target network 215. Before defining the loss function, first define the Q function. The Q function is expressed as in Equation 1.

(Refer to Non-Patent Document 3)

[Equation 1]

Q ^* is the optimal Q function value, d is the reduction variable, and π is the behavior judgment policy of the reinforcement learning model. That is, it represents the optimal value of the cumulative reward that the reinforcement learning model can receive. If the loss function is defined according to the Q function, it is expressed as in Equation 2.

[Equation 2]

이다.

는 메인 네트워크 내의 파라미터를 의미한다. 대상 네트워크의 Q 함수값은

으로 나타낸다.

는 대상 네트워크 내의 파라미터를 의미한다. 최적화 알고리즘으로 비특허문헌 4의 Adam optimizer를 적용할 수 있다.The value of the Q function of the main network is

to be.

Denotes a parameter in the main network. The value of the Q function of the target network is

Represented by

Denotes a parameter in the target network. As an optimization algorithm, the Adam optimizer of Non-Patent Document 4 can be applied.

The repetition memory 213 performs random mini-batch sampling to the main network 214 and copying parameters from the main network 214 to the target network 215 at every trial period designated by the system.

4 is a flowchart illustrating a modulation and demodulation method of a modulation/demodulation selection unit based on reinforcement learning of a receiver according to embodiments of the present invention.

In step S410, the communication system performs parameter initialization.

In step S420, it is checked whether the reinforcement learning model is the first implementation. If the reinforcement learning model is the first trial in step S420, the process proceeds to step S430, the optimal boundary value among the existing trials is fetched from the state-value buffer, and then the process proceeds to step S440.

In step S420, if the reinforcement learning model is not the first implementation, the process proceeds to step S440 and a series of boundary value adjustment processes are performed a specified number of times (M).

In the next step S450, spectral efficiency and bit error probability are calculated.

The spectral efficiency is expressed as Equation 3. (Refer to Non-Patent Document 5)

[Equation 3]

D/B means the average data rate per reference bandwidth, m represents the number of bits per symbol, γ is the instantaneous signal-to-noise ratio, and pc(γ) is the probability density distribution of the signal-to-noise ratio of the satellite communication channel. Represents a function. The bit error probability can be calculated through Equation 4 and Equation 5. (Refer to Non-Patent Document 6 and Non-Patent Document 7) [Equation 4]

Pe represents the bit error probability, and Pe(γ) represents the bit error probability in the AWGN (Additive White GAUSSIAN Noise) channel.

[Equation 5]

Equation 5 is the bit error probability in the AWGN channel of the MPSK modulation and demodulation method. By substituting Equation 5 into Equation 1, the bit error probability of each modulation/demodulation section can be calculated.

In step S460, the reinforcement learning model adjusts the boundary value. The boundary value is adjusted based on the environmental model. After adjusting the threshold, in step S470, a bit error probability is calculated for the signal-to-noise ratio for the channel. It is checked whether the bit error probability calculated in step S480 exceeds a preset bit error probability reference value. In an embodiment of the present invention, the bit error probability criterion is set ^{to 10 -3.} However, this is only an example and is not limited thereto.

As a result of checking in step S480, if the bit error probability for the channel exceeds the preset bit error probability reference value, the process proceeds to step S490, the optimal boundary value is loaded from the previously implemented results, and a penalty is given to the boundary value in step S500. .

As a result of checking in step S480, if the bit error probability for the channel does not exceed a preset bit error probability reference value, the process proceeds to step S510 to check whether the spectral efficiency is improved. If it is determined that the spectral efficiency is improved as a result of the check in step S510, the process proceeds to step S530 and compensation is given to the threshold value. do. In step S540, information of the boundary value, action value, cumulative compensation, and changed boundary value is stored in the iterative memory, and the process proceeds to step S550 to store the optimal boundary value during execution.

In the next step S560, it is checked whether the trial has been implemented every C times, and if it has been implemented C times, the process proceeds to step S570 to perform optimization and network parameter adjustment. In the reinforcement learning model, C can be set to 10, and optimization and network parameter adjustment can be performed every 10 trials. Optimization and network parameter adjustment (S570) is performed through an optimization model, and the adjusted parameters of the main network are copied to the target network.

If it has not been executed C times in step S560, the process proceeds to step S580.

In step S580, it is checked whether the trial has been completed, and if the number of adjustments remains, since the trial has not been completed, the process proceeds to step S440 to check the number of adjustments and then proceeds again (S590). If the enforcement is ended in step S580, the process proceeds to step S420.

5 is a graph showing a simulation result in which an optimal boundary value of a modulation system based on reinforcement learning of a receiver according to embodiments of the present invention is not called.

Referring to Fig. 5, the maximum spectral efficiency value in each run is shown. It can be seen that the maximum spectral efficiency value per episode is improved by 0.8649 bps/Hz for the initial state randomly selected in the following areas. The initial state is randomly selected between: 0~8dB,: 8~17dB, and: 17~25dB.

6 is a graph showing a simulation result of not loading an optimal boundary value of a modulation system based on reinforcement learning according to embodiments of the present invention, and FIG. 7 is a diagram of a modulation system based on reinforcement learning according to embodiments of the present invention. This is a graph showing the simulation result with the optimal boundary value loaded.

Referring to FIG. 6, if the optimal boundary value is not called during the existing trial when the trial starts and the bit error probability exceeds the reference, it can be seen that stable performance improvement is not performed and the performance improvement value is also low.

Referring to FIG. 7, it can be seen that the bit error probability criterion set in advance according to the reinforcement learning model is observed. In addition, it can be seen that stable performance improvement is achieved by comparing FIG. 7 with FIG. 6.

Although the components included in the transmitter and the receiver are shown separately in FIG. 1, a plurality of components may be combined with each other to be implemented as at least one module. Components are connected to a communication path connecting a software module or a hardware module inside the device and operate organically with each other. These components communicate using one or more communication buses or signal lines.

The transmitter and receiver may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general purpose or specific purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The transmitter and receiver may be mounted in software, hardware, or a combination thereof on a computing device provided with hardware elements. A computing device is a variety of devices including all or part of a communication device such as a communication modem for performing communication with various devices or communication networks, a memory storing data for executing a program, and a microprocessor for calculating and commanding by executing a program. Can mean

In FIGS. 2 and 4, each process is described as sequentially executing, but this is only illustrative, and those skilled in the art are shown in FIGS. 2 and 4 without departing from the essential characteristics of the embodiment of the present invention. By changing the described order, executing one or more processes in parallel, or adding other processes, various modifications and variations may be applied.

The operation according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. Computer-readable medium refers to any medium that has participated in providing instructions to a processor for execution. The computer-readable medium may include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. Computer programs may be distributed over networked computer systems to store and execute computer-readable codes in a distributed manner. Functional programs, codes, and code segments for implementing this embodiment may be easily inferred by programmers in the art to which this embodiment belongs.

The above description is merely illustrative of the technical idea of the present invention, and a person of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Accordingly, the embodiments implemented in the present invention are not intended to limit the technical idea of the present invention, but to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: transmitter
200: receiver
210: receiver
220: channel detection unit
230: Modem and demodulation selection unit
240: feedback unit
250: demodulation unit

Claims (15)

A receiver for estimating a channel state by receiving a pilot signal from a transmitter;
A modulation/demodulation selection unit for selecting a modulation/demodulation method according to the state of the channel based on reinforcement learning; And
A feedback unit for transmitting information of the selected modulation and demodulation method to the transmitter;
Including,
When the modulation and demodulation selector divides according to the number of modulation and demodulation methods based on the signal-to-noise ratio of the channel, the signal-to-noise ratio, which is the boundary between each channel, is used as a boundary value, and the boundary value is adjusted to compensate in a direction of maximizing the spectral efficiency value. Reinforcement learning by giving (reward),
The reinforcement learning is learning by a deep Q network algorithm, characterized in that a state of the deep Q network algorithm is set as the boundary value, and an action is set as an adjustment of raising or lowering the boundary value. Reinforcement learning-based adaptive modulation and demodulation method is applied. delete The receiver of claim 1, wherein the boundary value is a reference value for dividing the signal-to-noise ratio probability density distribution function of the channel into a plurality of modulation/demodulation scheme allocation regions. The method of claim 1,
The modulation/demodulation selection unit includes a state-value buffer for storing an optimal spectral efficiency value for a boundary value,
The modulation and demodulation selection unit, at the start of each trial, calls a boundary value corresponding to an optimal spectral efficiency value from the state-value buffer and starts the trial. The method of claim 1,
And the modem selector sets a penalty when the threshold value is not valid. The receiver to which the reinforcement learning-based adaptive modulation and demodulation method is applied. The method of claim 5,
The state in which the threshold value is not valid is a state in which the next state is less than or equal to the previous state, so that a high boundary violates a lower boundary, a state in which the threshold value exceeds a preset signal-to-noise ratio range, and a state between neighboring boundary values. The receiver to which the reinforcement learning-based adaptive modulation and demodulation method is applied, characterized in that at least one of the states in which the range of is set to be narrower than the preset interval. The method of claim 1,
The modulation/demodulation selection unit includes a state-value buffer for storing an optimal spectral efficiency value for a boundary value,
The modem selector calculates a bit error probability for a signal-to-noise ratio for a channel, and when the bit error probability exceeds a preset bit error probability reference value, a threshold value corresponding to an optimal spectral efficiency value from the state-value buffer Reinforcement learning-based adaptive modulation and demodulation method is applied, characterized in that it calls and starts execution. In the modulation and demodulation method based on reinforcement learning by a receiver,
Receiving a pilot signal from a transmitter;
Estimating the state of the channel from the received pilot signal;
Selecting a modulation/demodulation method according to the state of the channel based on reinforcement learning; And
Transmitting information of the selected modulation and demodulation method to the transmitter;
Including,
The reinforcement learning is learning by a deep Q network algorithm, and a state of the deep Q network algorithm is set as a threshold value, an action is set as an adjustment to raise or lower the threshold value, and the threshold value It gives a reward in the direction of maximizing the spectral efficiency value by adjusting
The boundary value is a signal-to-noise ratio that becomes a boundary between each channel when partitioning according to the number of modulation and demodulation methods based on a signal-to-noise ratio of a channel. delete The method of claim 8,
The step of selecting the modulation and demodulation method
An adaptive modulation and demodulation method based on reinforcement learning, characterized in that, at the start of each trial, the threshold value corresponding to the optimal spectral efficiency value stored in advance is called and the trial is started. The method of claim 8,
The step of selecting the modulation and demodulation method,
The bit error probability is calculated for the signal-to-noise ratio for the channel, and when the bit error probability exceeds the preset bit error probability reference value, the threshold value corresponding to the optimal spectral efficiency value stored in advance in the buffer is called and execution is started. An adaptive modulation and demodulation method based on reinforcement learning, characterized in that. A transmitter for transmitting a pilot signal; And
Receiver for receiving the pilot signal, estimating the state of the channel, selecting a modem based on the state of the channel based on reinforcement learning, and transmitting information of the selected modem to the transmitter
Including,
When the receiver divides according to the number of modulation and demodulation methods based on the signal-to-noise ratio of the channel, the signal-to-noise ratio, which is the boundary between the channels, is used as a boundary value, and the boundary value is adjusted to compensate in the direction of maximizing the spectral efficiency value. Reinforcement learning by giving (reward),
The reinforcement learning is learning by a deep Q network algorithm, characterized in that a state of the deep Q network algorithm is set as the boundary value, and an action is set as an adjustment of raising or lowering the boundary value. A reinforcement learning-based adaptive modulation and demodulation system. The method of claim 12,
The transmitter receives the modulation and demodulation method information, modulates and transmits the data according to the modulation and demodulation method. The system of claim 13, wherein the receiver receives the data and demodulates the received data according to the modulation and demodulation method.
delete

Images