KR102420744B1 - IoT Packet Scheduling Control Method and Sink Node Device - Google Patents

Abstract

According to the present invention, an Internet of things packet scheduling control method comprises the following steps of: transmitting a state of a data packet currently waiting to be transmitted from a sensor node to a sink node; learning the state of the data packet of the sensor node by using reinforcement learning at the sink node; scheduling transmission of data packets from a plurality of sensor nodes according to a learned result; and transmitting, to each sensor node, a frame size to be implemented and the number of packets to be transmitted by each sensor node determined by scheduling.

Description

IoT Packet Scheduling Control Method and Sink Node Device

The present invention relates to a packet scheduling and frame control method using reinforcement learning in an IoT environment constituting a wireless sensor network (WSN), and a sink node apparatus for performing the same.

A wireless sensor network (WSN) is a network in which a plurality of sensor nodes are disposed in a sensing area, configure a network by themselves, and remotely transmit sensing information obtained by the sensor node.

Wireless sensor networks are being used in various fields such as environmental control in intelligent buildings or factories, automatic control of production processes, logistics management, product and information management in hospitals, remote sensing of patient conditions, and military control.

In the early days, sensor networks, which were primarily used for military operations, have been made smaller and higher in performance due to improvements in semiconductor technology, and larger and lower costs in memory capacity are realized. It is even before commercialization. The currently proposed sensor network technology application examples are really diverse, such as unmanned security, environmental monitoring that monitors the temperature or pollution level of a certain area or water body, remote meter reading, and facility monitoring, etc. Working applications are also proposed and some are being implemented.

The basic structure of a sensor network is a structure in which a plurality of network nodes with independent sensing and computing power are interconnected by a communication network, and the power of each node is supplied through a local battery located for each node. However, since the battery that supplies power to each node is a relatively small-capacity battery in consideration of the mobility of each node, there is an extremely limited disadvantage in energy use. has been The main trend of the study is to maximize the network survival time by reducing the number or amount of wireless communication between each node. Although research on this has been published, it has not been able to actually reduce the amount of data transmission.

In summary, since the wireless sensor network operates on a battery and cannot function when the battery power is exhausted, a transmission method to maximize energy efficiency is urgently required. However, the existing technology is limitedly applicable only to specific operating conditions, and the adaptability is insufficient for the case where the statistical characteristics change over time.

Republic of Korea Registration No. 10-0627328

An object of the present invention is to provide an IoT packet scheduling control method and sink node apparatus capable of maintaining high energy efficiency even in an environment in which statistical characteristics change over time.

Specifically, in the present invention, when a plurality of sensor nodes transmit data centering on a sink node, for the purpose of maximizing the lifetime of the sensor node, that is, the lifetime of the network, the length of the frame in which the sensor nodes are activated, and each sensor node during the frame. We propose an algorithm that determines the number of packets to be transmitted using reinforcement learning.

According to an aspect of the present invention, there is provided a method for controlling IoT packet scheduling, comprising: transmitting a state of a data packet currently waiting to be transmitted from a sensor node to a sink node; learning the state of the data packet of the sensor node using reinforcement learning at the sink node; scheduling data packet delivery from a plurality of sensor nodes according to the learned result; and transmitting a frame size to be fulfilled by each sensor node determined by scheduling and the number of packets to be transmitted to each sensor node.

Here, the method may further include the step of applying the transmission result by applying the frame size determined in transmission from each sensor node and the number of packets to be transmitted to the reinforcement learning.

Here, the method may further include designing a compensation function for data transmission of the sensor node.

Here, in the step of designing the reward function, a reward function for maximizing the network lifespan and transmission rate may be designed using a queue-learning technique.

Here, in the reinforcement learning, a delay time of a data packet transmitted from the sensor node to the sink node is applied as a state, and a data packet transmitted from the sensor node to the sink node is applied as an action. It may be a queue-learning technique that applies the frame length and the number of packets, and applies the quality of data packet delivery from the sensor node to the sink node as a reward.

Here, the state may be set according to the following equation.

state = (d_1, d_2, ... , d_k)

(d_k is the delay of the kth packet in the queue)

Alternatively, the state may be set according to the following equation.

state = (r_1, r_2, ... , r_k)

(r_k is the remaining delay time of the kth packet in the queue)

Here, the action may be set according to the following equation.

action = (L, n_1, n_2, ... n_N)

(L is the length of the frame, n_k is the number of packets that the kth sensor node will send in this frame)

Here, the reward may be set according to the following equation.

Reward = L x QoS

(L is the length of the frame, QoS is an index indicating how effective the number of packets to transmit for each sensor node indicated through actions)

Here, the scheduling may include: determining a frame size according to a learned result; And it is possible to determine the number of packets to be transmitted in each frame for each sensor node according to the learned result.

Here, in the scheduling step, reinforcement learning for a plurality of sensor nodes is performed individually for each sensor node, but when inconsistency in behavior occurs as a result of learning, priority is given to behavior derived from exploitation. can be given

Here, in the scheduling step, when a new sensor node is connected, a value table of a sensor node in which existing learning is most active may be copied.

A sink node apparatus according to another aspect of the present invention includes: a sensor node receiving unit for receiving a packet containing sensing data from each sensor node; a storage unit in which a compensation function designed for data transmission in each sensor node and state information of each sensor node are recorded; a sensor node transmitter for transmitting the determined frame size and number of packets to each sensor node; a reinforcement learning unit for learning the frame size and the number of packets for each sensor node state using reinforcement learning; and a scheduler that determines the frame size and the number of packets for each sensor node according to the learned result of the learning unit.

Here, in the reinforcement learning, a delay time of a data packet transmitted from the sensor node to the sink node is applied as a state, and a data packet transmitted from the sensor node to the sink node is applied as an action. It may be a queue-learning technique that applies the frame length and the number of packets, and applies the quality of data packet delivery from the sensor node to the sink node as a reward.

Here, the reinforcement learning unit continuously performs reinforcement learning for each of the sensor nodes in charge of the sink node device, and stores the reinforcement learning results completed up to the present time for each sensor node in the storage unit. It can be stored in the allocated area for

If the IoT packet scheduling control method and/or the sink node device according to the spirit of the present invention having the above configuration are implemented, there is an advantage in that high energy efficiency of the IoT device can be maintained even in an environment in which statistical characteristics change according to time.

The IoT packet scheduling control method and/or sink node device of the present invention has the advantage of maximizing the lifespan of a sensor by improving energy efficiency while satisfying QoS expressed by delay time.

1 is a conceptual diagram of a wireless sensor network communication environment according to an embodiment of the present invention;
2 is a conceptual diagram illustrating a communication method between a sink node and a plurality of sensor nodes according to an embodiment of the present invention.
3 is a conceptual diagram of a queue-learning-based frame length and packet scheduling according to an embodiment of the present invention;
4 is a flowchart illustrating an embodiment of a method for controlling IoT packet scheduling according to the spirit of the present invention.
5 is a block diagram illustrating an embodiment of a sink node apparatus capable of performing the IoT packet scheduling control method according to the spirit of the present invention.

In describing the present invention, terms such as first, second, etc. may be used to describe various components, but the components may not be limited by the terms. The terms are only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being connected or connected to another component, it may be directly connected or connected to the other component, but it can be understood that other components may exist in between. .

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression may include the plural expression unless the context clearly dictates otherwise.

In this specification, the terms include or include are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and includes one or more other features or numbers, It may be understood that the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded in advance.

In addition, shapes and sizes of elements in the drawings may be exaggerated for clearer description.

In the present invention, the length of a frame, which is a key parameter that determines energy efficiency, and the number of packets to be transmitted per frame are determined by using Q-learning, a learning algorithm of the reinforcement learning series among artificial intelligence techniques.

The Q-learning technique intelligently parallels learning and execution while enhancing adaptability to environmental changes and enabling more effective control.

First, a wireless sensor network to which the spirit of the present invention can be applied will be described.

1 is a conceptual diagram of a wireless sensor network communication environment according to an embodiment of the present invention.

2 is a conceptual diagram illustrating a communication method between a sync node and a plurality of sensor nodes according to an embodiment of the present invention.

In the figure, communication between the sink and the sensor node occurs in units of time called frames, and the sink sends a packet announcing the start of a frame, and in the packet, the length of the current frame and the number of packets to be transmitted for each sensor node are informed. After each node transmits a packet, it saves energy while maintaining a sleeping mode until the next frame starts.

3 is a conceptual diagram of a queue-learning-based frame length and packet scheduling according to an embodiment of the present invention.

The technology proposed in the present invention is to control the data transmission amount and frame length of sensor nodes by using Q-learning in an Internet of Things (IoT) environment in which a plurality of sensor nodes are operated. .

The sensor node transmits the generated sensing data to the sink node that collects it, and the sensing data has a delay time requirement to be transmitted within a certain time according to the characteristics of the data.

In consideration of this requirement for the sink node, the transmission amount of the sensor node should be appropriately scheduled. In addition, all sensor nodes maintain a sleep state after transmission (activation) within a frame time for the purpose of energy saving, and this type of frame is repeated while the sensor node operates.

The sink node performs energy saving by controlling the frame size for every frame. According to the idea of the present invention, the sink node performs learning for the purpose of maximizing the lifespan and transmission rate of the sensor node using the queue-learning technique, and based on this, the transmission amount (number of packets) of each sensor node and the frame Control the size.

4 is a flowchart illustrating an embodiment of a method for controlling IoT packet scheduling according to the spirit of the present invention.

The IoT packet scheduling control method according to the illustrated flowchart includes the steps of designing a compensation function for data transmission to a sensor node (S20); defining the state of the sensor node (S40); Transmitting the status of the packet currently waiting to be transmitted from the sensor node at the sink (S120); learning a node state and a packet state using a queue-learning technique in the sink (S140); determining the size of a frame (which is a kind of a collection of timeslots) according to cue-learning (S150); determining the number of packets to be transmitted in the current frame per each sensor node according to the queue-learning (S160); transmitting the frame size and the number of packets to be transmitted to the sensor nodes (S170); and performing queue-learning learning in consideration of the result of transmission (S180).

A compensation function for sensor node data transmission can be designed in the following way.

The reward function may be designed to be suitable for the reinforcement learning method to be applied, and in the description of the following examples, it will be concretely described as a case of applying a Q-learning technique known to have an excellent learning effect.

First, we will look at the process (S20) of designing a compensation function by applying a Q-learning technique to a wireless sensor network.

In the step of designing the compensation function (S20), the compensation function is designed by being specific to the type of the wireless sensor network system (which can be defined by a service company or equipment specification), or it is specific to the site where each wireless sensor network system is installed. A compensation function can be designed.

The process of designing a reward function generally includes a process of defining the state of the sensor node constituting the Q-learning algorithm. However, in the drawing, in order to emphasize the state of the sensor node as the main information exchanged between the sensor node and the sink node, the step of defining the state of the sensor node ( S40 ) is divided.

From the perspective of the operation of the wireless sensor network system, it is noted that the step of designing the compensation function ( S20 ) can be omitted as a process executed at the time of system construction. Similarly, the step ( S40 ) of defining the state of the sensor node from the viewpoint of operation of the wireless sensor network system may also be omitted.

A variety of reinforcement learning can be applied to the present invention, and it will be concretely described as a case where the Q-learning algorithm, which is easy to apply and has an excellent learning effect, is applied.

In order to configure the queue-learning algorithm, the state, action, and reward can be set as follows.

As a first state, each sensor node generates a packet for sensing data transmission, and the generated packet waits for transmission order in a queue of the sensor node. Each packet has a requirement to be delivered within a certain amount of time, which is called maximum latency. Each packet must be transmitted to the sink node before the maximum delay time has elapsed. If the packet is not transmitted even after the maximum delay time, it is removed from the queue of the sensor node and discarded. If k packets are queued in the sensor node, the delay time of each packet is transmitted to the sink node as a packet.

The state of the sensor node reflecting the delay time of each packet may be defined, for example, according to Equation 1 below.

[ Equation 1 ]

state = (d_1, d_2, ... , d_k)

In the above equation, d_k is the delay time of the k-th packet in the queue. As k increases, more information is provided, but the size of the state space S increases exponentially, so a method of reducing the state space, which will be described later, may be required.

When sending the status information composed of the delay time as packets, it is effective to divide the number by section and then send the section number in the form of an integer such as 0,1,2,..

As the second action, the sink node determines the best action by executing a queue-learning algorithm that considers the state and reward sent by each sensor node.

An action may be composed of, for example, two elements as shown in Equation 2 below.

[ Equation 2 ]

action = (L, n_1, n_2, ... n_N)

In the above equation, L is the (temporal) length of the frame, and n_k is the number of packets to be sent by the k-th sensor node in this frame. Actions determined by the sink node through queue-learning are loaded into packets and broadcast to all sensor nodes. Each sensor node that receives the behavior information checks the frame length and enters sleep until a new frame starts after its transmission is finished. In addition, packets are transmitted to the sink node as many as the number of packets presented in the action.

Third, the reward will be described.

The purpose of scheduling operational parameters is to extend network lifetime while meeting latency constraints. Therefore, the reward function should be designed to guide the learning process in line with the goal. For example, the compensation function in the framework may consist of two parts: network lifetime and QoS metrics expressed as delay limits.

In other words, the reward should contain the engineering purpose we want to achieve through Q-learning. An object of the method and apparatus proposed in the present invention is to maximize the lifetime of a sensor network and simultaneously transmit packets within the maximum delay time. In consideration of this, in the present invention, the compensation function may be set as in Equation 3 below.

[ Equation 3 ]

Reward = L x QoS

In the above equation, L is the length of the frame, and QoS is an index indicating how effective the number of packets to transmit for each sensor node indicated through an action is.

QoS can be applied in various ways depending on the application environment or purpose.

For example, as an example for the purpose of simplifying calculation, QoS according to Equation 4 below may be applied.

[ Equation 4 ]

QoS = 1, if there is no packet loss (due to delay timeout) in this frame,

= 0, if there is a packet loss (due to delay timeout) in this frame.

The type of applying QoS according to Equation 4 is designed to minimize packet loss by allocating a value of 0 when a packet is deleted due to an expired delay limit.

As another example, QoS according to Equation 5 below may be applied to QoS of a type that prefers to place an absolute value on reducing packet loss, unlike the QoS of Equation 4 above.

[ Equation 5 ]

QoS = n_t/ (n_t + n_t- + n_t^)

In the above equation, n_t is the number of packets to be transmitted by the sensor node in the current frame, n_t- is the number of packets discarded due to not receiving a schedule in the current frame, and n_t^ is the number of packets discarded because they arrive after the start of the frame and are not reported as a status to be.

In the designing of the reward function (S20), the above equations for the state, action, and reward constituting the above-described queue-learning algorithm may be determined/selected.

In this case, in the subsequent step of defining the state of the sensor node ( S40 ), parameters for Equation 1 may be set.

In addition, when a new sensor node is added to the already built sensor network, the step of defining the state of the sensor node with respect to the added sensor node ( S40 ) may be performed.

Next, in the step (S120) of transmitting the status of the packet currently waiting to be transmitted from the sensor node to the sink, each sensor node transmits delay time information of all or part of the packet waiting in the queue for transmission in Equation 6 below. It can be sent in a packet in the following format.

[ Equation 6 ]

state = (d_1, d_2, ... , d_k)

In the above equation, d_k is the delay time of the k-th packet in the queue. It is effective to assign a number by dividing the delay time by section, and then send the section number in the form of an integer such as 0,1,2,.. For example, in the case of dividing into 20 equal parts by maximizing 4 seconds, the value of the state may be assigned as shown in Table 1 below.

In addition, the state may be configured according to the following Equation 7 as the remaining time instead of the delay time.

[ Equation 7 ]

state = (r_1, r_2, ... , r_k)

In the above equation, r_k is the remaining delay time of the k-th packet in the queue. Residual time is a value obtained by subtracting the current delay time from the maximum delivery delay time given to a packet, and means that it must be transmitted within this time in order not to be discarded. For example, if the maximum delay time is 4 seconds, the states may be configured as shown in Table 2 below.

Next, reward information for performing reinforcement learning and transmission of reward information will be described.

The compensation value is essential information required to operate Q-learning, and the sensor node may calculate the compensation value based on Equation 8 below.

[ Equation 8 ]

Reward = L x QoS

The compensation value according to the above equation may be calculated by the sensor node, loaded in a packet, and transmitted to the sink node, or only the QoS value may be transmitted because the sink node already knows L (frame length).

Next, a step ( S140 ) of learning using the queue-learning technique will be described.

When the current state of Q-learning is s, learning proceeds according to the following procedure.

1. Initialization : current state s, action a

2. Parameters: epsilon, ALPHA, GAMMA

3. if unif(0,1) < epsilon: # Exploration when a random number between 0-1 is less than epsilon

4. a = random_action() # choose a random action

5. else : # exploitation (exploitation)

6. a = optimal_action(s) # Choose optimal action a. The action with the greatest value in the current state s. a = (frame length, number of packets to be transmitted by each sensor node)

7. new_s, r = step(a) # Execute action a and receive the changed state new_s and reward r from the sensor node as a result. Here, one step means one frame. New action instruction frame by frame

8. value[(new_s, a)] += (1-ALPHA) + (r + GAMMA *best_v(new_s)) * ALPHA # where best_v(new_s) is the maximum value of actions that can be taken in the new state new_s value of action

9. Goto 3: # Go to step 3 and repeat while the sensor node is alive

Next, one of the methods of saving state space will be described.

If the state is expressed as state = (d_1, d_2, ... , d_k) according to Equation 6, there is a relationship as in Equation 9 below.

[ Equation 9 ]

d_1 ≥ d_2 ≥ d_3 ... ≥ d_k

In addition, when the state is expressed in terms of the remaining delay time, it can be expressed as state = (r_1, r_2, ..., r_k) according to Equation 7, and also in this case, the relationship shown in Equation 10 is established.

[ Equation 10 ]

r_1 ≤ r_2 ≤ r_3 ... ≤ r_k

In order to operate a queue-learning algorithm, the entire state must be created. Considering the above properties, removing the impossible state can save memory and speed up execution. The impossible state is, for example, k=5 and expressed as a delay time, such as (3,4,1,2,5), which is an impossible state because a packet arriving later cannot have a long delay time.

Next, scheduling methods selectively overlapping and applicable will be described.

In the first scheduling method, queue-learning learning for multiple sensor nodes is performed individually for each sensor. That is, if there are N sensor nodes, there are N individual queue-learning modules in the sink node. This has an advantage over learning through integration considering the mobility of sensor nodes and the frequency of creation/death.

The second scheduling method prepares for the case of inconsistency in the behavior of individual queue-learning learning results for N sensor nodes. In this case, the priority lies in actions derived from exploitation, not exploration.

For example, if the behavior of each sensor node is derived as follows through queue-learning, the common frame length is determined by considering only sensor nodes 1, 2, and 3.

Sensor node 1: (L_1, n_1) <- exploitation

Sensor node 2: (L_2, n_2) <- exploitation

Sensor node 3: (L_3, n_3) <- exploitation

Sensor node 4: (L_4, n_4) <- exploration

Sensor node 5: (L_5, n_5) <- exploration

As for the method of determining the common frame length, in the above example, ① the minimum value of L_1, L_2, L_3 may be determined, or ② may be determined to use an L value that minimizes the mean-square error.

If the number of packets to be transmitted (n_1 + n_2 + ... + n_5) cannot be accommodated in L (frame length) determined by the above method, ① select a floor value by decreasing each n_i proportionally (for example, When a 10% reduction is required, it can be determined in the form of reducing the value of n from floor(0.9*n_i)), or ② when the remaining delay time of the node to be transmitted is relatively large.

The third scheduling method is to speed up the learning rate by copying the value table of the node where the existing learning is most active, rather than running queue-learning in a state where there is no learning when a new sensor node is connected.

In the step (S170) of transmitting the frame size and the number of packets to be transmitted to the sensor nodes, as the scheduling information generated by the scheduling method in the sink node, the frame size and the number of packets to be transmitted can be specified and transmitted to each sensor node. have.

For example, as a method of transmitting an action from a sink node, the sink node may start a frame with a packet that informs information about the action (frame length, number of packets to be transmitted). In this case, the number of packets to be scheduled may start with the above packet or may be scheduled by designating a sensor node in the form of Bluetooth polling.

Next, an operation in the sensor node that has received the scheduling information will be described.

Each sensor node transmits a packet loaded with sensing data by reflecting the frame size and the number of packets received from each sensor node, and after transmitting the packet, each sensor node maintains a sleeping mode until the next frame starts. will save energy.

Next, by reflecting the results of the sensing packet transmission reflecting the received frame size and the number of packets, the next state information of the sensor nodes for reinforcement learning is collected, and the sink node uses the collected next state information as reinforcement learning. It is reflected in the queue learning (S180).

Meanwhile, as a processing method for the zero packet in each sensor node, when there is no data to be sent to the sensor node, the state may be transmitted through a short control packet without data.

5 illustrates an embodiment of a sink node apparatus capable of performing the IoT packet scheduling control method according to the spirit of the present invention.

The illustrated sink node device includes: a sensor node receiving unit 120 for receiving a packet containing sensing data from each sensor node; a storage unit 180 in which a compensation function designed for data transmission in each sensor node and state information of each sensor node are recorded; a sensor node transmitter 160 that transmits a state of a packet currently waiting to be transmitted from each sensor node to the corresponding sensor node, and transmits the determined frame size and number of packets to each sensor node; a reinforcement learning unit 140 for learning the frame size and the number of packets for each sensor node state using a reinforcement learning (queue-learning) technique; and a scheduler 150 that determines the frame size and the number of packets for each sensor node according to the learned result of the learning unit.

The storage unit 180 may be implemented as a storage medium (eg, a flash memory) embedded in the sink node device.

A compensation function for data transmission in each sensor node required to perform the IoT packet scheduling control method according to the spirit of the present invention may be designed in an external management server or the like and recorded in the storage unit 180 . After the compensation function is designed, the state information of each sensor node applied by the corresponding compensation function may also be recorded in the storage unit 180 .

The sensor node transmitter 160 may perform a broadcast of notifying a frame.

Although the sensor node receiving unit 120 and the sensor node transmitting unit 160 may be implemented as a single hardware communication module, they are highly differentiated in the functions they perform, and are expressed as separate and independent components.

Depending on the implementation, the reinforcement learning unit 140 and the scheduler 150 may be composed of SW blocks executed by one single CPU or may be composed of different hardware.

In the latter case, the scheduler 150 may be a CPU that performs general control of the entire sink node device, and the reinforcement learning unit 140 may be dedicated hardware/software for reinforcement learning. For example, the reinforcement learning unit 140 may be a dedicated chip for performing reinforcement learning, or a communication module that requests reinforcement learning from an external online reinforcement learning server and receives the result.

The reinforcement learning unit 140 may perform reinforcement learning on each of the sensor nodes in charge of the sink node device. In this case, since reinforcement learning is continuously performed, the reinforcement learning results completed up to the current point in time for each sensor node may be stored in the storage unit 180 in an area allocated for each sensor node.

Those skilled in the art to which the present invention pertains should understand that the present invention can be embodied in other specific forms without changing the technical spirit or essential characteristics thereof, so the embodiments described above are illustrative in all respects and not restrictive. only do The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

120: sensor node receiver
140: reinforcement learning unit
150 : Scheduler
160: sensor node transmitter
180: storage

Claims (15)

A method for controlling IoT packet scheduling performed in a wireless sensor network system in which a plurality of sensor nodes transmit data centering on a sink node, the method comprising:
transmitting the state of the data packet currently waiting to be transmitted from the sensor node to the sink node;
learning the state of the data packet of the sensor node using reinforcement learning at the sink node;
scheduling data packet delivery from a plurality of sensor nodes according to the learned result; and
Transmitting the frame size to be implemented by each sensor node determined by scheduling and the number of packets to be transmitted to each sensor node
including,
In the scheduling step,
When a new sensor node is connected, an IoT packet scheduling control method that copies the value table of the sensor node that has the most active learning.
According to claim 1,
applying the transmission result by applying the frame size determined in transmission from each sensor node and the number of packets to be transmitted to the reinforcement learning;
Internet of Things packet scheduling control method further comprising a.
According to claim 1,
Designing a compensation function for data transmission of the sensor node
Internet of Things packet scheduling control method further comprising a.
4. The method of claim 3,
In the step of designing the compensation function,
An IoT packet scheduling control method that uses a queue-learning technique to design a reward function to maximize network lifespan and transmission rate.
According to claim 1,
The reinforcement learning is
Applying a delay time of a data packet transmitted from the sensor node to the sink node as a state,
Applying the frame length and the number of packets for data packets transmitted from the sensor node to the sink node as an action,
A method for controlling IoT packet scheduling, which is a queue-learning technique that applies the quality of data packet delivery from the sensor node to the sink node as a reward.
6. The method of claim 5,
The state (state) is an IoT packet scheduling control method set according to the following equation.
state = (d_1, d_2, ... , d_k)
(d_k is the delay of the kth packet in the queue)
6. The method of claim 5,
The state (state) is an IoT packet scheduling control method set according to the following equation.
state = (r_1, r_2, ... , r_k)
(r_k is the remaining delay time of the kth packet in the queue)
6. The method of claim 5,
The action (action) is an IoT packet scheduling control method set according to the following equation.
action = (L, n_1, n_2, ... n_N)
(L is the length of the frame, n_k is the number of packets that the kth sensor node will send in this frame)
6. The method of claim 5,
The reward is set according to the following equation.
Reward = L x QoS
(L is the length of the frame, QoS is an index indicating how effective the number of packets to transmit for each sensor node indicated through actions)
According to claim 1,
The scheduling step is
determining a frame size according to the learned result; and
Determining the number of packets to be transmitted in each frame for each sensor node according to the learned result
Internet of Things packet scheduling control method comprising a.
According to claim 1,
In the scheduling step,
Reinforcement learning for a plurality of sensor nodes is performed individually for each sensor node, but when there is a discrepancy in behavior as a result of learning, a method for controlling an IoT packet scheduling that gives priority to behavior derived from exploitation .
delete a sensor node receiving unit for receiving a packet containing sensing data from each sensor node;
a storage unit in which a compensation function designed for data transmission in each sensor node and state information of each sensor node are recorded;
a sensor node transmitter for transmitting the determined frame size and number of packets to each sensor node;
a reinforcement learning unit for learning the frame size and the number of packets for each sensor node state using reinforcement learning; and
A scheduler that determines the frame size and the number of packets for each sensor node according to the learned result of the learning unit
including,
The reinforcement learning unit,
A sink that continuously performs reinforcement learning for each of the sensor nodes in charge of the sink node device, and stores the reinforcement learning results completed up to the current time for each sensor node in an area allocated for each sensor node in the storage unit node device.
14. The method of claim 13,
The reinforcement learning is
Applying a delay time of a data packet transmitted from the sensor node to the sink node as a state,
Applying the frame length and the number of packets for data packets transmitted from the sensor node to the sink node as an action,
A sink node device, which is a queue-learning technique that applies the quality of data packet delivery from the sensor node to the sink node as a reward.
delete

Images