WO2023022251A1 - Method and apparatus for transmitting signal in wireless communication system - Google Patents

Abstract

As an example of the present disclosure, a method for operation of a terminal in a wireless communication system may comprise the steps of: performing image sensing via a first sensor; receiving neural network-related information from a base station; training a first neural network on the basis of the neural network-related information; and transmitting, to the base station, data regarding the result of training the first neural network. The first sensor does not perform image signal processing (ISP), and the neural network-related information may include information indicating whether or not the terminal has trained a neural network. The training of the first neural network may be training for processing, on the basis of artificial intelligence, data regarding the result of the image sensing.

Description

Signal transmission method and apparatus in wireless communication system

The following description relates to a wireless communication system, and relates to a signal transmission method related to an artificial intelligence-based special purpose virtual camera through federated learning in a wireless communication system.

A wireless access system is widely deployed to provide various types of communication services such as voice and data. In general, a wireless access system is a multiple access system capable of supporting communication with multiple users by sharing available system resources (bandwidth, transmission power, etc.). Examples of the multiple access system include a code division multiple access (CDMA) system, a frequency division multiple access (FDMA) system, a time division multiple access (TDMA) system, an orthogonal frequency division multiple access (OFDMA) system, and a single carrier frequency (SC-FDMA) system. division multiple access) system.

In particular, as many communication devices require large communication capacity, an enhanced mobile broadband (eMBB) communication technology compared to existing radio access technology (RAT) has been proposed. In addition, a communication system considering reliability and latency-sensitive services/UE (user equipment) as well as mMTC (massive machine type communications) providing various services anytime and anywhere by connecting multiple devices and objects has been proposed. . Various technical configurations for this have been proposed.

The present disclosure may provide a method and apparatus in which a terminal or a base station performs image signal processing (ISP) based on artificial intelligence in a wireless communication system.

The technical objects to be achieved in the present disclosure are not limited to the above-mentioned matters, and other technical problems not mentioned above are common knowledge in the art to which the technical configuration of the present disclosure is applied from the embodiments of the present disclosure to be described below. can be considered by those who have

As an example of the present disclosure, a method of operating a terminal in a wireless communication system includes performing image sensing through a first sensor, receiving neural network-related information from a base station, and generating a first neural network based on the neural network-related information. The method may include learning a network and transmitting the first neural network learning result data to a base station. The first sensor does not perform image signal processing (ISP), but the neural network-related information may include information indicating whether or not the terminal learns a neural network. Learning of the first neural network may be learning to process the image sensing result data based on artificial intelligence. The neural network related information may include weight information of the first neural network. The terminal may perform the first neural network learning only when the information indicating whether the neural network is learning or not indicates the first neural network learning. The terminal may receive second neural network learning result data from the base station. The terminal may perform image sensing through a second sensor, and the second sensor may perform ISP. The terminal may transmit ISP output data of the second sensor to the base station.

As an example of the present disclosure, a method of operating a base station in a wireless communication system includes transmitting neural network related information to a terminal, receiving first neural network learning result data from the terminal, and the first neural network learning result data. Learning a first reverse neural network based on , and learning a second neural network based on a result of learning the first reverse neural network. The reverse first neural network learning may perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process image sensing result data based on artificial intelligence. In the learning of the second neural network, a result of performing the ISP may be compared with data resulting from learning the first neural network. The neural network related information may include weight information of the first neural network. The base station may transmit second neural network learning result data to the terminal. The base station may include an image signal processor (ISP). The image signal processor (ISP) may perform image signal processing (ISP) based on the first reverse neural network learning result data. The base station may receive ISP output data from the terminal and learn the second neural network based on the ISP output data.

As an example of the present disclosure, in a wireless communication system, a terminal may include a transceiver and a processor connected to the transceiver. The processor may control the terminal to perform image sensing through the first sensor. The processor may control the transceiver to receive neural network related information from a base station. The processor may control the terminal to learn a first neural network based on the neural network related information. The processor may control the transceiver to transmit the first neural network learning result data to the base station. Here, the first sensor may not perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process the image sensing result data based on artificial intelligence.

As an example of the present disclosure, in a wireless communication system, a base station may include a transceiver and a processor connected to the transceiver. The processor may control the transceiver to transmit neural network related information to the terminal. The processor may control the base station to receive first neural network learning result data from the terminal. The processor may control the base station to learn a first reverse neural network based on the first neural network learning result data, and to learn a second neural network based on the first reverse neural network learning result. . The reverse first neural network learning may perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process image sensing result data based on artificial intelligence. In the learning of the second neural network, a result of performing the ISP may be compared with data resulting from learning the first neural network.

As an example of the present disclosure, an apparatus may include at least one memory and at least one processor functionally connected to the at least one memory. The at least one processor may control the device to perform image sensing through the first sensor. The at least one processor may control the device to receive neural network related information from a base station. The at least one processor may control the device to learn a first neural network based on the neural network related information. The at least one processor may control the device to transmit the first neural network learning result data to the base station. The first sensor may not perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process the image sensing result data based on artificial intelligence.

As an example of the present disclosure, a non-transitory computer-readable medium storing at least one instruction (instructions) includes the at least one instruction executable by a processor. can include The at least one command instructs the computer readable medium to perform image sensing through a first sensor, to receive neural network-related information from a base station, and to generate a first neural network based on the neural network-related information. Instructing to learn, and instructing the base station to transmit the first neural network learning result data. Here, the first sensor may not perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process the image sensing result data based on artificial intelligence.

The above-described aspects of the present disclosure are only some of the preferred embodiments of the present disclosure, and various embodiments in which the technical features of the present disclosure are reflected are detailed descriptions of the present disclosure to be detailed below by those of ordinary skill in the art. It can be derived and understood based on.

The following effects may be obtained by embodiments based on the present disclosure.

According to the present disclosure, a terminal or a base station may perform image signal processing (ISP) based on artificial intelligence.

According to the present disclosure, reliability of communication between sensors inside a device may be improved.

According to the present disclosure, a terminal can efficiently transmit and receive ISP-related data based on artificial intelligence.

Effects obtainable in the embodiments of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned are technical fields to which the technical configuration of the present disclosure is applied from the description of the following embodiments of the present disclosure. can be clearly derived and understood by those skilled in the art. That is, unintended effects according to implementing the configuration described in the present disclosure may also be derived by those skilled in the art from the embodiments of the present disclosure.

The accompanying drawings are provided to aid understanding of the present disclosure, and may provide embodiments of the present disclosure together with a detailed description. However, the technical features of the present disclosure are not limited to specific drawings, and features disclosed in each drawing may be combined with each other to form a new embodiment. Reference numerals in each drawing may mean structural elements.

1 is a diagram illustrating an example of a communication system applicable to the present disclosure.

2 is a diagram illustrating an example of a wireless device applicable to the present disclosure.

3 is a diagram illustrating another example of a wireless device applicable to the present disclosure.

4 is a diagram illustrating an example of a portable device applicable to the present disclosure.

5 is a diagram illustrating an example of a vehicle or autonomous vehicle applicable to the present disclosure.

6 is a diagram showing an example of AI (Artificial Intelligence) applicable to the present disclosure.

7 is a diagram illustrating a method of processing a transmission signal applicable to the present disclosure.

8 is a diagram showing an example of a communication structure that can be provided in a 6G system applicable to the present disclosure.

9 is a diagram showing an electromagnetic spectrum applicable to the present disclosure.

10 is a diagram illustrating a THz communication method applicable to the present disclosure.

11 is a diagram showing the structure of a perceptron included in an artificial neural network applicable to the present disclosure.

12 is a diagram illustrating an artificial neural network structure applicable to the present disclosure.

13 is a diagram illustrating a deep neural network applicable to the present disclosure.

14 is a diagram illustrating a convolutional neural network applicable to the present disclosure.

15 is a diagram illustrating a filter operation of a convolutional neural network applicable to the present disclosure.

16 is a diagram showing a neural network structure in which a circular loop applicable to the present disclosure exists.

17 is a diagram illustrating an operational structure of a recurrent neural network applicable to the present disclosure.

18a and 18b are diagrams illustrating an example of federated learning applicable to the present disclosure.

19 is a diagram illustrating an example of federated learning applicable to the present disclosure.

20 is a diagram illustrating an example of a federated learning procedure applicable to the present disclosure.

21 is a diagram illustrating an example of a terminal applicable to the present disclosure.

22 is a diagram illustrating an example of a terminal applicable to the present disclosure.

23 is a diagram illustrating an example of a terminal applicable to the present disclosure.

24 is a diagram illustrating an example of a terminal applicable to the present disclosure.

25 is a diagram illustrating an example of an operation procedure of a terminal applicable to the present disclosure.

26 is a diagram illustrating an example of a server, base station, or terminal procedure applicable to the present disclosure.

The following embodiments are those that combine elements and features of the present disclosure in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form not combined with other components or features. In addition, an embodiment of the present disclosure may be configured by combining some elements and/or features. The order of operations described in the embodiments of the present disclosure may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

In the description of the drawings, procedures or steps that may obscure the gist of the present disclosure have not been described, and procedures or steps that can be understood by those skilled in the art have not been described.

Throughout the specification, when a part is said to "comprising" or "including" a certain element, it means that it may further include other elements, not excluding other elements, unless otherwise stated. do. In addition, terms such as “… unit”, “… unit”, and “module” described in the specification mean a unit that processes at least one function or operation, which is hardware or software or a combination of hardware and software. can be implemented as Also, "a or an", "one", "the" and similar related words in the context of describing the present disclosure (particularly in the context of the claims below) Unless indicated or otherwise clearly contradicted by context, both the singular and the plural can be used.

In this specification, the embodiments of the present disclosure have been described with a focus on a data transmission/reception relationship between a base station and a mobile station. Here, a base station has meaning as a terminal node of a network that directly communicates with a mobile station. A specific operation described as being performed by a base station in this document may be performed by an upper node of the base station in some cases.

That is, in a network composed of a plurality of network nodes including a base station, various operations performed for communication with a mobile station may be performed by the base station or network nodes other than the base station. At this time, the 'base station' is a term such as a fixed station, Node B, eNode B, gNode B, ng-eNB, advanced base station (ABS), or access point. can be replaced by

In addition, in the embodiments of the present disclosure, a terminal includes a user equipment (UE), a mobile station (MS), a subscriber station (SS), a mobile subscriber station (MSS), It may be replaced with terms such as mobile terminal or advanced mobile station (AMS).

In addition, the transmitting end refers to a fixed and/or mobile node providing data service or voice service, and the receiving end refers to a fixed and/or mobile node receiving data service or voice service. Therefore, in the case of uplink, the mobile station can be a transmitter and the base station can be a receiver. Similarly, in the case of downlink, the mobile station may be a receiving end and the base station may be a transmitting end.

Embodiments of the present disclosure are wireless access systems, such as an IEEE 802.xx system, a 3rd Generation Partnership Project (3GPP) system, a 3GPP Long Term Evolution (LTE) system, a 3GPP 5G (5th generation) NR (New Radio) system, and a 3GPP2 system. It may be supported by at least one disclosed standard document, and in particular, the embodiments of the present disclosure are supported by 3GPP technical specification (TS) 38.211, 3GPP TS 38.212, 3GPP TS 38.213, 3GPP TS 38.321 and 3GPP TS 38.331 documents It can be.

In addition, embodiments of the present disclosure may be applied to other wireless access systems, and are not limited to the above-described systems. For example, it may also be applicable to a system applied after the 3GPP 5G NR system, and is not limited to a specific system.

That is, obvious steps or parts not described in the embodiments of the present disclosure may be described with reference to the above documents. In addition, all terms disclosed in this document can be explained by the standard document.

Hereinafter, preferred embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of the present disclosure, and is not intended to represent the only embodiments in which the technical configurations of the present disclosure may be practiced.

In addition, specific terms used in the embodiments of the present disclosure are provided to aid understanding of the present disclosure, and the use of these specific terms may be changed in other forms without departing from the technical spirit of the present disclosure.

The following technologies include code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), single carrier frequency division multiple access (SC-FDMA), and the like. It can be applied to various wireless access systems.

In the following, in order to clarify the following description, the description is based on the 3GPP communication system (e.g. (eg, LTE, NR, etc.), but the technical spirit of the present invention is not limited thereto. LTE is 3GPP TS 36.xxx Release 8 or later In detail, LTE technology after 3GPP TS 36.xxx Release 10 is referred to as LTE-A, and LTE technology after 3GPP TS 36.xxx Release 13 may be referred to as LTE-A pro. 3GPP NR may mean technology after TS 38.xxx Release 15. 3GPP 6G may mean technology after TS Release 17 and/or Release 18. "xxx" means a standard document detail number. LTE/NR/6G may be collectively referred to as a 3GPP system.

For background art, terms, abbreviations, etc. used in the present disclosure, reference may be made to matters described in standard documents published prior to the present invention. As an example, 36.xxx and 38.xxx standard documents may be referred to.

Communication systems applicable to the present disclosure

Although not limited thereto, various descriptions, functions, procedures, proposals, methods and / or operational flowcharts of the present disclosure disclosed in this document may be applied to various fields requiring wireless communication / connection (eg, 5G) between devices. there is.

Hereinafter, it will be exemplified in more detail with reference to the drawings. In the following drawings/description, the same reference numerals may represent the same or corresponding hardware blocks, software blocks or functional blocks unless otherwise specified.

1 is a diagram illustrating an example of a communication system applied to the present disclosure.

Referring to FIG. 1 , a communication system 100 applied to the present disclosure includes a wireless device, a base station, and a network. Here, the wireless device means a device that performs communication using a radio access technology (eg, 5G NR, LTE), and may be referred to as a communication/wireless/5G device. Although not limited thereto, the wireless device includes a robot 100a, a vehicle 100b-1 and 100b-2, an extended reality (XR) device 100c, a hand-held device 100d, and a home appliance. appliance) 100e, Internet of Thing (IoT) device 100f, and artificial intelligence (AI) device/server 100g. For example, the vehicle may include a vehicle equipped with a wireless communication function, an autonomous vehicle, a vehicle capable of performing inter-vehicle communication, and the like. Here, the vehicles 100b-1 and 100b-2 may include an unmanned aerial vehicle (UAV) (eg, a drone). The XR device 100c includes augmented reality (AR)/virtual reality (VR)/mixed reality (MR) devices, and includes a head-mounted device (HMD), a head-up display (HUD) installed in a vehicle, a television, It may be implemented in the form of smart phones, computers, wearable devices, home appliances, digital signage, vehicles, robots, and the like. The mobile device 100d may include a smart phone, a smart pad, a wearable device (eg, a smart watch, a smart glass), a computer (eg, a laptop computer), and the like. The home appliance 100e may include a TV, a refrigerator, a washing machine, and the like. The IoT device 100f may include a sensor, a smart meter, and the like. For example, the base station 120 and the network 130 may also be implemented as a wireless device, and a specific wireless device 120a may operate as a base station/network node to other wireless devices.

The wireless devices 100a to 100f may be connected to the network 130 through the base station 120 . AI technology may be applied to the wireless devices 100a to 100f, and the wireless devices 100a to 100f may be connected to the AI server 100g through the network 130. The network 130 may be configured using a 3G network, a 4G (eg LTE) network, or a 5G (eg NR) network. The wireless devices 100a to 100f may communicate with each other through the base station 120/network 130, but communicate directly without going through the base station 120/network 130 (e.g., sidelink communication). You may. For example, the vehicles 100b-1 and 100b-2 may perform direct communication (eg, vehicle to vehicle (V2V)/vehicle to everything (V2X) communication). In addition, the IoT device 100f (eg, sensor) may directly communicate with other IoT devices (eg, sensor) or other wireless devices 100a to 100f.

Wireless communication/ connection 150a, 150b, and 150c may be performed between the wireless devices 100a to 100f/base station 120 and the base station 120/base station 120. Here, wireless communication/connection includes various types of uplink/downlink communication 150a, sidelink communication 150b (or D2D communication), and inter-base station communication 150c (eg relay, integrated access backhaul (IAB)). This can be done through radio access technology (eg 5G NR). Through the wireless communication/ connection 150a, 150b, and 150c, a wireless device and a base station/wireless device, and a base station and a base station can transmit/receive radio signals to each other. For example, the wireless communication/ connections 150a, 150b, and 150c may transmit/receive signals through various physical channels. To this end, based on various proposals of the present disclosure, various configuration information setting processes for transmitting / receiving radio signals, various signal processing processes (eg, channel encoding / decoding, modulation / demodulation, resource mapping / demapping, etc.) At least a part of a resource allocation process may be performed.

Communication systems applicable to the present disclosure

2 is a diagram illustrating an example of a wireless device applicable to the present disclosure.

Referring to FIG. 2 , a first wireless device 200a and a second wireless device 200b may transmit and receive radio signals through various wireless access technologies (eg, LTE and NR). Here, {the first wireless device 200a, the second wireless device 200b} denotes the {wireless device 100x and the base station 120} of FIG. 1 and/or the {wireless device 100x and the wireless device 100x. } can correspond.

The first wireless device 200a includes one or more processors 202a and one or more memories 204a, and may further include one or more transceivers 206a and/or one or more antennas 208a. The processor 202a controls the memory 204a and/or the transceiver 206a and may be configured to implement the descriptions, functions, procedures, suggestions, methods and/or operational flow diagrams disclosed herein. For example, the processor 202a may process information in the memory 204a to generate first information/signal, and transmit a radio signal including the first information/signal through the transceiver 206a. In addition, the processor 202a may receive a radio signal including the second information/signal through the transceiver 206a and store information obtained from signal processing of the second information/signal in the memory 204a. The memory 204a may be connected to the processor 202a and may store various information related to the operation of the processor 202a. For example, memory 204a may perform some or all of the processes controlled by processor 202a, or instructions for performing the descriptions, functions, procedures, suggestions, methods, and/or flowcharts of operations disclosed herein. It may store software codes including them. Here, the processor 202a and the memory 204a may be part of a communication modem/circuit/chip designed to implement a wireless communication technology (eg, LTE, NR). The transceiver 206a may be coupled to the processor 202a and may transmit and/or receive wireless signals through one or more antennas 208a. The transceiver 206a may include a transmitter and/or a receiver. The transceiver 206a may be used interchangeably with a radio frequency (RF) unit. In the present disclosure, a wireless device may mean a communication modem/circuit/chip.

For example, the first wireless device may include a transceiver and a processor connected to the transceiver. The processor may control the first wireless device to perform image sensing through a first sensor. The processor may control the transceiver to receive neural network related information from a base station. The processor may control the first wireless device to learn a first neural network based on the neural network related information. The processor may control the transceiver to transmit the first neural network learning result data to the base station. Here, the first sensor may not perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process the image sensing result data based on artificial intelligence.

As another example, the first wireless device may include a transceiver and a processor connected to the transceiver. The processor may control the transceiver to transmit neural network related information to the terminal. The processor may control the first wireless device to receive first neural network learning result data from the terminal. The processor controls the first wireless device to learn a first reverse neural network based on the first neural network learning result data, and to learn a second neural network based on the first reverse neural network learning result. can do. The reverse first neural network learning may perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process image sensing result data based on artificial intelligence. In the learning of the second neural network, a result of performing the ISP may be compared with data resulting from learning the first neural network.

As another example, the first wireless device may include at least one memory and at least one processor functionally connected to the at least one memory. The at least one processor may control the first wireless device to perform image sensing through a first sensor. The at least one processor may control the first wireless device to receive neural network related information from a base station. The at least one processor may control the first wireless device to learn a first neural network based on the neural network related information. The at least one processor may control the first wireless device to transmit the first neural network learning result data to the base station. The first sensor may not perform image signal processing (ISP). The neural network related information may include information indicating whether or not the terminal learns the neural network. Learning of the first neural network may process the image sensing result data based on artificial intelligence.

The second wireless device 200b includes one or more processors 202b, one or more memories 204b, and may further include one or more transceivers 206b and/or one or more antennas 208b. The processor 202b controls the memory 204b and/or the transceiver 206b and may be configured to implement the descriptions, functions, procedures, suggestions, methods and/or operational flow diagrams disclosed herein. For example, the processor 202b may process information in the memory 204b to generate third information/signal, and transmit a radio signal including the third information/signal through the transceiver 206b. In addition, the processor 202b may receive a radio signal including the fourth information/signal through the transceiver 206b and store information obtained from signal processing of the fourth information/signal in the memory 204b. The memory 204b may be connected to the processor 202b and may store various information related to the operation of the processor 202b. For example, the memory 204b may perform some or all of the processes controlled by the processor 202b, or instructions for performing the descriptions, functions, procedures, suggestions, methods, and/or flowcharts of operations disclosed herein. It may store software codes including them. Here, the processor 202b and the memory 204b may be part of a communication modem/circuit/chip designed to implement a wireless communication technology (eg, LTE, NR). The transceiver 206b may be coupled to the processor 202b and may transmit and/or receive wireless signals through one or more antennas 208b. The transceiver 206b may include a transmitter and/or a receiver. The transceiver 206b may be used interchangeably with an RF unit. In the present disclosure, a wireless device may mean a communication modem/circuit/chip.

Hereinafter, hardware elements of the wireless devices 200a and 200b will be described in more detail. Although not limited to this, one or more protocol layers may be implemented by one or more processors 202a, 202b. For example, the one or more processors 202a and 202b may include one or more layers (eg, PHY (physical), MAC (media access control), RLC (radio link control), PDCP (packet data convergence protocol), RRC (radio resource) control) and functional layers such as service data adaptation protocol (SDAP). One or more processors 202a, 202b may generate one or more protocol data units (PDUs) and/or one or more service data units (SDUs) according to the descriptions, functions, procedures, proposals, methods, and/or operational flow charts disclosed herein. can create One or more processors 202a, 202b may generate messages, control information, data or information according to the descriptions, functions, procedures, proposals, methods and/or operational flow diagrams disclosed herein. One or more processors 202a, 202b generate PDUs, SDUs, messages, control information, data or signals (eg, baseband signals) containing information according to the functions, procedures, proposals and/or methods disclosed herein , may be provided to one or more transceivers 206a and 206b. One or more processors 202a, 202b may receive signals (eg, baseband signals) from one or more transceivers 206a, 206b, and descriptions, functions, procedures, suggestions, methods, and/or flowcharts of operations disclosed herein PDUs, SDUs, messages, control information, data or information can be obtained according to these.

One or more processors 202a, 202b may be referred to as a controller, microcontroller, microprocessor or microcomputer. One or more processors 202a, 202b may be implemented by hardware, firmware, software, or a combination thereof. For example, one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), one or more digital signal processing devices (DSPDs), one or more programmable logic devices (PLDs), or one or more field programmable gate arrays (FPGAs). may be included in one or more processors 202a and 202b. The descriptions, functions, procedures, proposals, methods and/or operational flowcharts disclosed in this document may be implemented using firmware or software, and the firmware or software may be implemented to include modules, procedures, functions, and the like. Firmware or software configured to perform the descriptions, functions, procedures, proposals, methods and/or operational flow charts disclosed in this document may be included in one or more processors 202a or 202b or stored in one or more memories 204a or 204b. It can be driven by the above processors 202a and 202b. The descriptions, functions, procedures, suggestions, methods and/or operational flow charts disclosed in this document may be implemented using firmware or software in the form of codes, instructions and/or sets of instructions.

For example, the second wireless device may be a non-transitory computer-readable medium storing at least one instruction. The second wireless device may include the at least one command executable by a processor. The at least one command instructs the second wireless device to perform image sensing through a first sensor, to receive neural network-related information from a base station, and to generate a first neural network based on the neural network-related information. Instructing to learn, and instructing the base station to transmit the first neural network learning result data. Here, the first sensor may not perform image signal processing (ISP). The neural network related information may include information indicating whether or not the second wireless device learns the neural network. Learning of the first neural network may process the image sensing result data based on artificial intelligence.

One or more memories 204a, 204b may be coupled to one or more processors 202a, 202b and may store various types of data, signals, messages, information, programs, codes, instructions and/or instructions. One or more memories 204a, 204b may include read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), flash memory, hard drive, registers, cache memory, computer readable storage media, and/or It may consist of a combination of these. One or more memories 204a, 204b may be located internally and/or externally to one or more processors 202a, 202b. In addition, one or more memories 204a, 204b may be connected to one or more processors 202a, 202b through various technologies such as wired or wireless connections.

One or more transceivers 206a, 206b may transmit user data, control information, radio signals/channels, etc. referred to in the methods and/or operational flow charts of this document to one or more other devices. One or more transceivers 206a, 206b may receive user data, control information, radio signals/channels, etc. referred to in descriptions, functions, procedures, proposals, methods and/or operational flow charts, etc. disclosed herein from one or more other devices. there is. For example, one or more transceivers 206a and 206b may be connected to one or more processors 202a and 202b and transmit and receive radio signals. For example, one or more processors 202a, 202b may control one or more transceivers 206a, 206b to transmit user data, control information, or radio signals to one or more other devices. In addition, one or more processors 202a, 202b may control one or more transceivers 206a, 206b to receive user data, control information, or radio signals from one or more other devices. In addition, one or more transceivers 206a, 206b may be coupled to one or more antennas 208a, 208b, and one or more transceivers 206a, 206b may be connected to one or more antennas 208a, 208b to achieve the descriptions, functions disclosed in this document. , procedures, proposals, methods and / or operation flowcharts, etc. can be set to transmit and receive user data, control information, radio signals / channels, etc. In this document, one or more antennas may be a plurality of physical antennas or a plurality of logical antennas (eg, antenna ports). One or more transceivers (206a, 206b) in order to process the received user data, control information, radio signal / channel, etc. using one or more processors (202a, 202b), the received radio signal / channel, etc. in the RF band signal It can be converted into a baseband signal. One or more transceivers 206a and 206b may convert user data, control information, and radio signals/channels processed by one or more processors 202a and 202b from baseband signals to RF band signals. To this end, one or more transceivers 206a, 206b may include (analog) oscillators and/or filters.

Wireless device structure applicable to the present disclosure

3 is a diagram illustrating another example of a wireless device applied to the present disclosure.

Referring to FIG. 3, a wireless device 300 corresponds to the wireless devices 200a and 200b of FIG. 2, and includes various elements, components, units/units, and/or modules. ) can be configured. For example, the wireless device 300 may include a communication unit 310, a control unit 320, a memory unit 330, and an additional element 340. The communication unit may include communication circuitry 312 and transceiver(s) 314 . For example, communication circuitry 312 may include one or more processors 202a, 202b of FIG. 2 and/or one or more memories 204a, 204b. For example, transceiver(s) 314 may include one or more transceivers 206a, 206b of FIG. 2 and/or one or more antennas 208a, 208b. The control unit 320 is electrically connected to the communication unit 310, the memory unit 330, and the additional element 340 and controls overall operations of the wireless device. For example, the control unit 320 may control electrical/mechanical operations of the wireless device based on programs/codes/commands/information stored in the memory unit 330. In addition, the control unit 320 transmits the information stored in the memory unit 330 to the outside (eg, another communication device) through the communication unit 310 through a wireless/wired interface, or transmits the information stored in the memory unit 330 to the outside (eg, another communication device) through the communication unit 310. Information received through a wireless/wired interface from other communication devices) may be stored in the memory unit 330 .

The additional element 340 may be configured in various ways according to the type of wireless device. For example, the additional element 340 may include at least one of a power unit/battery, an input/output unit, a driving unit, and a computing unit. Although not limited thereto, the wireless device 300 may be a robot (FIG. 1, 100a), a vehicle (FIG. 1, 100b-1, 100b-2), an XR device (FIG. 1, 100c), a mobile device (FIG. 1, 100d) ), home appliances (FIG. 1, 100e), IoT devices (FIG. 1, 100f), digital broadcasting terminals, hologram devices, public safety devices, MTC devices, medical devices, fintech devices (or financial devices), security devices, climate/ It may be implemented in the form of an environment device, an AI server/device (FIG. 1, 140), a base station (FIG. 1, 120), a network node, and the like. Wireless devices can be mobile or used in a fixed location depending on the use-case/service.

In FIG. 3 , various elements, components, units/units, and/or modules in the wireless device 300 may be entirely interconnected through a wired interface or at least partially connected wirelessly through the communication unit 310 . For example, in the wireless device 300, the control unit 320 and the communication unit 310 are connected by wire, and the control unit 320 and the first units (eg, 130 and 140) are connected wirelessly through the communication unit 310. can be connected Additionally, each element, component, unit/unit, and/or module within wireless device 300 may further include one or more elements. For example, the control unit 320 may be composed of one or more processor sets. For example, the control unit 320 may include a set of a communication control processor, an application processor, an electronic control unit (ECU), a graphic processing processor, a memory control processor, and the like. As another example, the memory unit 330 may include RAM, dynamic RAM (DRAM), ROM, flash memory, volatile memory, non-volatile memory, and/or combinations thereof. can be configured.

Mobile device to which the present disclosure is applicable

4 is a diagram illustrating an example of a portable device applied to the present disclosure.

4 illustrates a portable device applied to the present disclosure. A portable device may include a smart phone, a smart pad, a wearable device (eg, smart watch, smart glasses), and a portable computer (eg, a laptop computer). A mobile device may be referred to as a mobile station (MS), a user terminal (UT), a mobile subscriber station (MSS), a subscriber station (SS), an advanced mobile station (AMS), or a wireless terminal (WT).

Referring to FIG. 4 , a portable device 400 includes an antenna unit 408, a communication unit 410, a control unit 420, a memory unit 430, a power supply unit 440a, an interface unit 440b, and an input/output unit 440c. ) may be included. The antenna unit 408 may be configured as part of the communication unit 410 . Blocks 410 to 430/440a to 440c respectively correspond to blocks 310 to 330/340 of FIG. 3 .

The communication unit 410 may transmit/receive signals (eg, data, control signals, etc.) with other wireless devices and base stations. The controller 420 may perform various operations by controlling components of the portable device 400 . The controller 420 may include an application processor (AP). The memory unit 430 may store data/parameters/programs/codes/commands necessary for driving the portable device 400 . Also, the memory unit 430 may store input/output data/information. The power supply unit 440a supplies power to the portable device 400 and may include a wired/wireless charging circuit, a battery, and the like. The interface unit 440b may support connection between the mobile device 400 and other external devices. The interface unit 440b may include various ports (eg, audio input/output ports and video input/output ports) for connection with external devices. The input/output unit 440c may receive or output image information/signal, audio information/signal, data, and/or information input from a user. The input/output unit 440c may include a camera, a microphone, a user input unit, a display unit 440d, a speaker, and/or a haptic module.

For example, in the case of data communication, the input/output unit 440c acquires information/signals (eg, touch, text, voice, image, video) input from the user, and the acquired information/signals are stored in the memory unit 430. can be stored The communication unit 410 may convert the information/signal stored in the memory into a wireless signal, and directly transmit the converted wireless signal to another wireless device or to a base station. In addition, the communication unit 410 may receive a radio signal from another wireless device or base station and then restore the received radio signal to original information/signal. After the restored information/signal is stored in the memory unit 430, it may be output in various forms (eg, text, voice, image, video, or haptic) through the input/output unit 440c.

Types of wireless devices to which this disclosure is applicable

5 is a diagram illustrating an example of a vehicle or autonomous vehicle to which the present disclosure applies.

5 illustrates a vehicle or autonomous vehicle to which the present disclosure is applied. A vehicle or an autonomous vehicle may be implemented as a mobile robot, vehicle, train, manned/unmanned aerial vehicle (AV), ship, etc., and is not limited to a vehicle type.

Referring to FIG. 5 , a vehicle or autonomous vehicle 500 includes an antenna unit 508, a communication unit 510, a control unit 520, a driving unit 540a, a power supply unit 540b, a sensor unit 540c, and an autonomous driving unit. A portion 540d may be included. The antenna unit 550 may be configured as a part of the communication unit 510 . Blocks 510/530/540a to 540d respectively correspond to blocks 410/430/440 of FIG. 4 .

The communication unit 510 may transmit/receive signals (eg, data, control signals, etc.) with external devices such as other vehicles, base stations (eg, base stations, roadside base units, etc.), servers, and the like. The controller 520 may perform various operations by controlling elements of the vehicle or autonomous vehicle 500 . The controller 520 may include an electronic control unit (ECU).

6 is a diagram illustrating an example of an AI device applied to the present disclosure. As an example, AI devices include TVs, projectors, smartphones, PCs, laptops, digital broadcasting terminals, tablet PCs, wearable devices, set-top boxes (STBs), radios, washing machines, refrigerators, digital signage, robots, vehicles, etc. It may be implemented as a device or a movable device.

Referring to FIG. 6, the AI device 600 includes a communication unit 610, a control unit 620, a memory unit 630, an input/output unit 640a/640b, a running processor unit 640c, and a sensor unit 640d. can include Blocks 610 to 630/640a to 640d may respectively correspond to blocks 310 to 330/340 of FIG. 3 .

The communication unit 610 communicates wired and wireless signals (eg, sensor information, user data) with external devices such as other AI devices (eg, FIG. 1, 100x, 120, and 140) or AI servers (Fig. input, learning model, control signal, etc.) can be transmitted and received. To this end, the communication unit 610 may transmit information in the memory unit 630 to an external device or transmit a signal received from the external device to the memory unit 630 .

The controller 620 may determine at least one executable operation of the AI device 600 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. And, the controller 620 may perform the determined operation by controlling components of the AI device 600 . For example, the control unit 620 may request, retrieve, receive, or utilize data from the learning processor unit 640c or the memory unit 630, and may perform a predicted operation among at least one feasible operation or one determined to be desirable. Components of the AI device 600 may be controlled to execute an operation. In addition, the control unit 620 collects history information including user feedback on the operation contents or operation of the AI device 600 and stores it in the memory unit 630 or the running processor unit 640c, or the AI server ( 1, 140) can be transmitted to an external device. The collected history information can be used to update the learning model.

The memory unit 630 may store data supporting various functions of the AI device 600 . For example, the memory unit 630 may store data obtained from the input unit 640a, data obtained from the communication unit 610, output data of the learning processor unit 640c, and data obtained from the sensing unit 640. Also, the memory unit 630 may store control information and/or software codes required for operation/execution of the controller 620 .

The input unit 640a may obtain various types of data from the outside of the AI device 600. For example, the input unit 620 may obtain learning data for model learning and input data to which the learning model is to be applied. The input unit 640a may include a camera, a microphone, and/or a user input unit. The output unit 640b may generate an output related to sight, hearing, or touch. The output unit 640b may include a display unit, a speaker, and/or a haptic module. The sensing unit 640 may obtain at least one of internal information of the AI device 600, surrounding environment information of the AI device 600, and user information by using various sensors. The sensing unit 640 may include a proximity sensor, an illuminance sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, and/or a radar. there is.

The learning processor unit 640c may learn a model composed of an artificial neural network using learning data. The running processor unit 640c may perform AI processing together with the running processor unit of the AI server (FIG. 1, 140). The learning processor unit 640c may process information received from an external device through the communication unit 610 and/or information stored in the memory unit 630 . In addition, the output value of the learning processor unit 640c may be transmitted to an external device through the communication unit 610 and/or stored in the memory unit 630.

7 is a diagram illustrating a method of processing a transmission signal applied to the present disclosure. For example, the transmitted signal may be processed by a signal processing circuit. In this case, the signal processing circuit 700 may include a scrambler 710, a modulator 720, a layer mapper 730, a precoder 740, a resource mapper 750, and a signal generator 760. At this time, as an example, the operation/function of FIG. 7 may be performed by the processors 202a and 202b and/or the transceivers 206a and 206b of FIG. 2 . Also, as an example, the hardware elements of FIG. 7 may be implemented in the processors 202a and 202b and/or the transceivers 206a and 206b of FIG. 2 . As an example, blocks 710 to 760 may be implemented in the processors 202a and 202b of FIG. 2 . Also, blocks 710 to 750 may be implemented in the processors 202a and 202b of FIG. 2 and block 760 may be implemented in the transceivers 206a and 206b of FIG. 2 , and are not limited to the above-described embodiment.

The codeword may be converted into a radio signal through the signal processing circuit 700 of FIG. 7 . Here, a codeword is an encoded bit sequence of an information block. Information blocks may include transport blocks (eg, UL-SCH transport blocks, DL-SCH transport blocks). Radio signals may be transmitted through various physical channels (eg, PUSCH, PDSCH). Specifically, the codeword may be converted into a scrambled bit sequence by the scrambler 710. A scramble sequence used for scrambling is generated based on an initialization value, and the initialization value may include ID information of a wireless device. The scrambled bit sequence may be modulated into a modulation symbol sequence by modulator 720. The modulation method may include pi/2-binary phase shift keying (pi/2-BPSK), m-phase shift keying (m-PSK), m-quadrature amplitude modulation (m-QAM), and the like.

The complex modulation symbol sequence may be mapped to one or more transport layers by the layer mapper 730. Modulation symbols of each transport layer may be mapped to corresponding antenna port(s) by the precoder 740 (precoding). The output z of the precoder 740 can be obtained by multiplying the output y of the layer mapper 730 by the N*M precoding matrix W. Here, N is the number of antenna ports and M is the number of transport layers. Here, the precoder 740 may perform precoding after transform precoding (eg, discrete fourier transform (DFT)) on complex modulation symbols. Also, the precoder 740 may perform precoding without performing transform precoding.

The resource mapper 750 may map modulation symbols of each antenna port to time-frequency resources. The time-frequency resource may include a plurality of symbols (eg, CP-OFDMA symbols and DFT-s-OFDMA symbols) in the time domain and a plurality of subcarriers in the frequency domain. The signal generator 760 generates a radio signal from the mapped modulation symbols, and the generated radio signal can be transmitted to other devices through each antenna. To this end, the signal generator 760 may include an inverse fast fourier transform (IFFT) module, a cyclic prefix (CP) inserter, a digital-to-analog converter (DAC), a frequency uplink converter, and the like. .

The signal processing process for the received signal in the wireless device may be configured in reverse to the signal processing process 710 to 760 of FIG. 7 . For example, a wireless device (eg, 200a and 200b of FIG. 2 ) may receive a wireless signal from the outside through an antenna port/transceiver. The received radio signal may be converted into a baseband signal through a signal restorer. To this end, the signal restorer may include a frequency downlink converter, an analog-to-digital converter (ADC), a CP remover, and a fast fourier transform (FFT) module. Thereafter, the baseband signal may be restored to a codeword through a resource de-mapper process, a postcoding process, a demodulation process, and a de-scramble process. The codeword may be restored to an original information block through decoding. Accordingly, a signal processing circuit (not shown) for a received signal may include a signal restorer, a resource demapper, a postcoder, a demodulator, a descrambler, and a decoder.

6G communication system

6G (radio communications) systems are characterized by (i) very high data rates per device, (ii) very large number of connected devices, (iii) global connectivity, (iv) very low latency, (v) battery- It aims to lower energy consumption of battery-free IoT devices, (vi) ultra-reliable connectivity, and (vii) connected intelligence with machine learning capabilities. The vision of the 6G system can be four aspects such as "intelligent connectivity", "deep connectivity", "holographic connectivity", and "ubiquitous connectivity", and the 6G system can satisfy the requirements shown in Table 1 below. That is, Table 1 is a table showing the requirements of the 6G system.

Per device peak data rate	1 Tbps
E2E latency	1ms
Maximum spectral efficiency	100 bps/Hz
Mobility support	up to 1000 km/hr
Satellite integration	Fully
AI	Fully
Autonomous vehicles	Fully
XR	Fully
Haptic Communication	Fully

At this time, the 6G system is enhanced mobile broadband (eMBB), ultra-reliable low latency communications (URLLC), mMTC (massive machine type communications), AI integrated communication, tactile Internet (tactile internet), high throughput, high network capacity, high energy efficiency, low backhaul and access network congestion and improved data security ( can have key factors such as enhanced data security.

10 is a diagram illustrating an example of a communication structure that can be provided in a 6G system applicable to the present disclosure.

Referring to FIG. 10 , a 6G system is expected to have 50 times higher simultaneous wireless communication connectivity than a 5G wireless communication system. URLLC, a key feature of 5G, is expected to become a more mainstream technology by providing end-to-end latency of less than 1 ms in 6G communications. At this time, the 6G system will have much better volume spectral efficiency, unlike the frequently used area spectral efficiency. 6G systems can provide very long battery life and advanced battery technology for energy harvesting, so mobile devices in 6G systems may not need to be charged separately.

Core implementation technology of 6G system

- Artificial Intelligence (AI)

The most important and newly introduced technology for the 6G system is AI. AI was not involved in the 4G system. 5G systems will support partial or very limited AI. However, the 6G system will be AI-enabled for full automation. Advances in machine learning will create more intelligent networks for real-time communication in 6G. Introducing AI in communications can simplify and enhance real-time data transmission. AI can use a plethora of analytics to determine how complex target tasks are performed. In other words, AI can increase efficiency and reduce processing delays.

Time-consuming tasks such as handover, network selection, and resource scheduling can be performed instantly by using AI. AI can also play an important role in machine-to-machine, machine-to-human and human-to-machine communications. In addition, AI can be a rapid communication in the brain computer interface (BCI). AI-based communication systems can be supported by metamaterials, intelligent structures, intelligent networks, intelligent devices, intelligent cognitive radios, self-sustaining wireless networks, and machine learning.

Recently, there have been attempts to integrate AI with wireless communication systems, but these are focused on the application layer, network layer, and especially deep learning, wireless resource management and allocation. come. However, such research is gradually developing into the MAC layer and the physical layer, and in particular, attempts to combine deep learning with wireless transmission are appearing in the physical layer. AI-based physical layer transmission means applying a signal processing and communication mechanism based on an AI driver rather than a traditional communication framework in fundamental signal processing and communication mechanisms. For example, deep learning-based channel coding and decoding, deep learning-based signal estimation and detection, deep learning-based multiple input multiple output (MIMO) mechanism, It may include AI-based resource scheduling and allocation.

Machine learning may be used for channel estimation and channel tracking, and may be used for power allocation, interference cancellation, and the like in a downlink (DL) physical layer. Machine learning can also be used for antenna selection, power control, symbol detection, and the like in a MIMO system.

However, the application of DNN for transmission in the physical layer may have the following problems.

AI algorithms based on deep learning require a lot of training data to optimize training parameters. However, due to limitations in acquiring data in a specific channel environment as training data, a lot of training data is used offline. This is because static training on training data in a specific channel environment may cause a contradiction between dynamic characteristics and diversity of a radio channel.

In addition, current deep learning mainly targets real signals. However, the signals of the physical layer of wireless communication are complex signals. In order to match the characteristics of a wireless communication signal, further research is needed on a neural network that detects a complex domain signal.

Hereinafter, machine learning will be described in more detail.

Machine learning refers to a set of actions that train a machine to create a machine that can do tasks that humans can or cannot do. Machine learning requires data and a running model. In machine learning, data learning methods can be largely classified into three types: supervised learning, unsupervised learning, and reinforcement learning.

Neural network training is aimed at minimizing errors in the output. Neural network learning repeatedly inputs training data to the neural network, calculates the output of the neural network for the training data and the error of the target, and backpropagates the error of the neural network from the output layer of the neural network to the input layer in a direction to reduce the error. ) to update the weight of each node in the neural network.

Supervised learning uses training data in which correct answers are labeled in the learning data, and unsupervised learning may not have correct answers labeled in the learning data. That is, for example, learning data in the case of supervised learning related to data classification may be data in which each learning data is labeled with a category. Labeled training data is input to the neural network, and an error may be calculated by comparing the output (category) of the neural network and the label of the training data. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate is used in the early stages of neural network learning to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and a low learning rate can be used in the late stage to increase accuracy.

The learning method may vary depending on the characteristics of the data. For example, in a case where the purpose of the receiver is to accurately predict data transmitted by the transmitter in a communication system, it is preferable to perform learning using supervised learning rather than unsupervised learning or reinforcement learning.

The learning model corresponds to the human brain, and the most basic linear model can be considered. ) is called

The neural network cord used as a learning method is largely divided into deep neural networks (DNN), convolutional deep neural networks (CNN), and recurrent boltzmann machine (RNN). and this learning model can be applied.

Terahertz (THz) communication

THz communication can be applied in 6G systems. For example, the data transmission rate can be increased by increasing the bandwidth. This can be done using sub-THz communication with wide bandwidth and applying advanced massive MIMO technology.

9 is a diagram showing an electromagnetic spectrum applicable to the present disclosure. As an example, referring to FIG. 9 , THz waves, also known as sub-millimeter radiation, generally represent a frequency band between 0.1 THz and 10 THz with corresponding wavelengths in the range of 0.03 mm-3 mm. The 100 GHz-300 GHz band range (sub THz band) is considered a major part of the THz band for cellular communications. Adding to the sub-THz band mmWave band will increase 6G cellular communications capacity. Among the defined THz bands, 300 GHz-3 THz is in the far infrared (IR) frequency band. The 300 GHz-3 THz band is part of the broad band, but is at the border of the wide band, just behind the RF band. Thus, this 300 GHz-3 THz band exhibits similarities to RF.

The main characteristics of THz communications include (i) widely available bandwidth to support very high data rates, and (ii) high path loss at high frequencies (highly directional antennas are indispensable). The narrow beamwidth produced by the highly directional antenna reduces interference. The small wavelength of the THz signal allows a much larger number of antenna elements to be incorporated into devices and BSs operating in this band. This enables advanced adaptive array technology to overcome range limitations.

Terahertz (THz) wireless communication

10 is a diagram illustrating a THz communication method applicable to the present disclosure.

Referring to FIG. 10, THz wireless communication uses wireless communication using a THz wave having a frequency of approximately 0.1 to 10 THz (1 THz = 1012 Hz), and a terahertz (THz) band radio using a very high carrier frequency of 100 GHz or more. can mean communication. THz waves are located between RF (Radio Frequency)/millimeter (mm) and infrared bands, and (i) transmit non-metal/non-polarizable materials better than visible light/infrared rays, and have a shorter wavelength than RF/millimeter waves and have high straightness. Beam focusing may be possible.

Artificial Intelligence System

11 is a diagram showing the structure of a perceptron included in an artificial neural network applicable to the present disclosure. 12 is a diagram showing an artificial neural network structure applicable to the present disclosure.

As described above, an artificial intelligence system may be applied in a 6G system. At this time, as an example, the artificial intelligence system may operate based on a learning model corresponding to the human brain, as described above. In this case, a paradigm of machine learning using a neural network structure having a high complexity such as an artificial neural network as a learning model may be referred to as deep learning. In addition, the neural network cord used in the learning method is largely a deep neural network (DNN), a convolutional deep neural network (CNN), and a recurrent neural network (RNN). There is a way. At this time, as an example, referring to FIG. 11, the artificial neural network may be composed of several perceptrons. At this time, the input vector x={x ₁ , x ₂ , … , x _d } are input, the weight {W ₁ , W ₂ , . . . , W _d }, summing up all the results, and then applying the activation function σ(·) can be called a perceptron. The huge artificial neural network structure extends the simplified perceptron structure shown in FIG. 11, and the input vector can be applied to different multi-dimensional perceptrons. For convenience of description, an input value or an output value is referred to as a node.

Meanwhile, the perceptron structure shown in FIG. 11 can be described as being composed of a total of three layers based on input values and output values. An artificial neural network in which there are H number of (d + 1) dimensional perceptrons between ^{the 1st} layer and ^{the 2nd} layer and K number of (H + 1) dimensional perceptrons between the 2nd layer and the ^3rd layer is represented as shown in FIG. can

At this time, the layer where the input vector is located is called the input layer, the layer where the final output value is located is called the output layer, and all layers located between the input layer and the output layer are called hidden layers. For example, although three layers are disclosed in FIG. 12 , since the number of actual artificial neural network layers is counted excluding input layers, the artificial neural network illustrated in FIG. 12 can be understood as a total of two layers. The artificial neural network is composed of two-dimensionally connected perceptrons of basic blocks.

The above-described input layer, hidden layer, and output layer can be jointly applied to various artificial neural network structures such as CNN and RNN, which will be described later, as well as multi-layer perceptrons. As the number of hidden layers increases, the artificial neural network becomes deeper, and a machine learning paradigm that uses a sufficiently deep artificial neural network as a learning model may be referred to as deep learning. In addition, an artificial neural network used for deep learning may be referred to as a deep neural network (DNN).

13 is a diagram illustrating a deep neural network applicable to the present disclosure.

Referring to FIG. 13, the deep neural network may be a multi-layer perceptron composed of 8 hidden layers + 8 output layers. At this time, the multilayer perceptron structure can be expressed as a fully-connected neural network. In a fully-connected neural network, there is no connection relationship between nodes located on the same layer, and a connection relationship may exist only between nodes located on adjacent layers. DNN has a fully-connected neural network structure and is composed of a combination of multiple hidden layers and activation functions, so it can be usefully applied to identify the correlation characteristics between inputs and outputs. Here, the correlation characteristic may mean a joint probability of input and output.

14 is a diagram illustrating a convolutional neural network applicable to the present disclosure. 15 is a diagram illustrating a filter operation of a convolutional neural network applicable to the present disclosure.

For example, various artificial neural network structures different from the aforementioned DNN can be formed depending on how a plurality of perceptrons are connected to each other. At this time, in the DNN, nodes located inside one layer are arranged in a one-dimensional vertical direction. However, referring to FIG. 14, it can be assumed that the nodes are two-dimensionally arranged with w nodes horizontally and h nodes vertically. (convolutional neural network structure in FIG. 14). In this case, since a weight is added for each connection in the connection process from one input node to the hidden layer, a total of h×w weights should be considered. Since there are h×w nodes in the input layer, a total of h ² w ² weights may be required between two adjacent layers.

In addition, since the convolutional neural network of FIG. 14 has a problem in that the number of weights increases exponentially according to the number of connections, it can be assumed that there is a filter with a small size instead of considering all mode connections between adjacent layers. can For example, as shown in FIG. 15, a weighted sum and an activation function operation may be performed on a portion where filters overlap.

At this time, one filter has weights corresponding to the number of filters, and learning of weights can be performed so that a specific feature on an image can be extracted as a factor and output. In FIG. 15 , a 3×3 filter is applied to a 3×3 area at the top left of the input layer, and an output value obtained by performing a weighted sum and an activation function operation on a corresponding node may be stored in z ₂₂ .

At this time, the above-described filter is moved by a certain distance horizontally and vertically while scanning the input layer, and the weighted sum and activation function calculations are performed, and the output value can be placed at the position of the current filter. Since this operation method is similar to the convolution operation for images in the field of computer vision, the deep neural network of this structure is called a convolutional neural network (CNN), and the result of the convolution operation The hidden layer may be called a convolutional layer. In addition, a neural network including a plurality of convolutional layers may be referred to as a deep convolutional neural network (DCNN).

Also, in the convolution layer, the number of weights may be reduced by calculating a weighted sum including only nodes located in a region covered by the filter in the node where the current filter is located. This allows one filter to be used to focus on features for a local area. Accordingly, CNN can be effectively applied to image data processing in which a physical distance in a 2D area is an important criterion. Meanwhile, in the CNN, a plurality of filters may be applied immediately before the convolution layer, and a plurality of output results may be generated through a convolution operation of each filter.

Meanwhile, there may be data whose sequence characteristics are important according to data attributes. Considering the length variability and precedence relationship of these sequence data, input each element on the data sequence one by one at each time step, and input the output vector (hidden vector) of the hidden layer output at a specific time point together with the next element on the sequence A structure in which this method is applied to an artificial neural network can be referred to as a recurrent neural network structure.

16 is a diagram showing a neural network structure in which a circular loop applicable to the present disclosure exists. 17 is a diagram illustrating an operational structure of a recurrent neural network applicable to the present disclosure.

Referring to FIG. 16, a recurrent neural network (RNN) is an element {x ₁ ^(t) , x ₂ ^(t) , . , x _d ^(t) } into the fully connected neural network, the immediately preceding point in time t-1 is the hidden vector {z ₁ ^(t-1) , z ₂ ^(t-1) , . . . , z _H ^(t-1) } together to apply a weighted sum and an activation function. The reason why the hidden vector is transmitted to the next time point in this way is that information in the input vector at previous time points is regarded as being accumulated in the hidden vector of the current time point.

Also, referring to FIG. 17 , the recurrent neural network may operate in a predetermined sequence of views with respect to an input data sequence. At this time, the input vector at time point 1 {x ₁ ^(t) , x ₂ ^(t) , . . . , x _d ^(t) } is input to the recurrent neural network {z ₁ ⁽¹⁾ , z ₂ ⁽¹⁾ , . . . , z _H ⁽¹⁾ } is the input vector {x ₁ ⁽²⁾ , x ₂ ⁽²⁾ , . , x _d ⁽²⁾ }, the vector {z ₁ ⁽²⁾ , z ₂ ⁽²⁾ , . . . of the hidden layer through the weighted sum and activation function , z _H ⁽²⁾ } is determined. This process is at time point 2, time point 3, . . . , iteratively performed until time T.

Meanwhile, when a plurality of hidden layers are arranged in a recurrent neural network, it is referred to as a deep recurrent neural network (DRNN). Recurrent neural networks are designed to be usefully applied to sequence data (eg, natural language processing).

As a neural network core used as a learning method, in addition to DNN, CNN, and RNN, restricted Boltzmann machine (RBM), deep belief networks (DBN), and deep Q-Network It includes various deep learning techniques such as computer vision, voice recognition, natural language processing, and voice/signal processing.

Recently, attempts have been made to integrate AI with wireless communication systems, but these are application layer and network layer, especially in the case of deep learning, wireless resource management and allocation has been focused on the field. However, such research is gradually developing into a MAC layer and a physical layer, and in particular, attempts to combine deep learning with wireless transmission are appearing in the physical layer. AI-based physical layer transmission means applying a signal processing and communication mechanism based on an AI driver, not a traditional communication framework, in fundamental signal processing and communication mechanisms. For example, deep learning-based channel coding and decoding, deep learning-based signal estimation and detection, deep learning-based MIMO mechanism, AI-based resource scheduling ( scheduling) and allocation.

Specific embodiments of the present invention

A method for exchanging information between devices may include communication between a server and devices, and communication between sensors inside devices. Specifically, a method for exchanging information between autonomous driving devices, unmanned devices, or general devices may include communication between a server and devices and communication between sensors inside the devices.

Devices are increasingly equipped with sensors to operate like autonomous vehicles, unmanned drones, IoT or robots. For example, devices are increasingly equipped with sensors such as radar and cameras to perform autonomous driving operations. Accordingly, the amount of data processed by the device increases, job processing becomes complicated, and data loss due to image signal processing (ISP) may increase.

In addition, the autonomous driving device can identify surrounding objects. For example, an autonomous device may identify whether a nearby object is a person, mobile device, or object. In addition, the autonomous driving device may grasp the driving environment. For example, the self-driving device can grasp the surrounding environment and region by 3D. In addition, the autonomous driving device may detect surrounding objects or measure the speed of surrounding objects. Accordingly, the autonomous driving device may brake or prevent a collision with a surrounding object. An autonomous driving device needs to be equipped with the above functions. However, the functions described above require very complex calculations and accurate references to be derived. Sensors included in existing devices have performance optimization limitations. That is, the camera lens, image sensor, and ISP included in the existing device have performance optimization limitations. In addition, the RF frequency and processing speed of the radar included in the existing device have limitations in performance optimization. In addition, there is performance optimization for laser signal generation and detection of a radar included in an existing device.

In addition, changes in sensor specifications according to the product target of the device may cause an imbalance between the amount of information collected from the device and the work processing capability. Here, the product target may include concepts such as premium, high tier, and not tier. Sensors may include lenses, image sensors, ISP processing, radar/lidar, and the like. The above-described imbalance may cause degradation in the performance of the device to identify surrounding objects, grasp the driving environment, detect objects, or measure speed.

In addition, since existing autonomous driving devices or unmanned devices perform recognition and confirmation by transmitting ISP-processed data, data loss may occur due to ISP processing. Such data loss may result in performance deterioration of autonomous driving devices or unmanned devices.

Also, the information obtained by the devices from the camera may be general information-centric. Accordingly, devices having a specific purpose must reprocess information obtained from the camera. Thus, additional processing costs may be incurred for such processing.

This disclosure proposes an organic operating method between a device and a server. In addition, the present disclosure proposes a method of increasing reliability of communication between sensors inside a device. In addition, the present disclosure proposes a method of increasing resource efficiency by performing ISP processing by AI.

18a and 18b are diagrams illustrating an example of federated learning applicable to the present disclosure. Federated Learning (FL) is a machine learning technique in which multiple terminals or servers holding local data samples and information train an algorithm without terminals or servers exchanging data samples and information. This method allows terminals to build a common powerful machine learning model without sharing data and information. Thus, federated learning technology can solve important issues such as data privacy, data security, data access rights, and heterogeneous data access. Federated learning is performed in such a way that the terminal trains a local model for local data and information, the terminal passes parameters of the local model to the server so that the server can create a global model, and the terminal receives the global model from the server. It can be. Parameters of the local model may include weights and information of the deep neural network.

Servers can contain global models. Devices that communicate with the server may include a local model. The global model may include information generated by the server based on information received from a plurality of terminals. The local model may include information generated by the device based on information received from the server. Servers and devices may communicate through a variety of methods to transfer models to each other. For example, a server and a device may communicate by being wired or wirelessly connected to transmit models to each other.

Referring to FIGS. 18A and 18B , a diagram illustrating that a server receives parameter information from various terminals. A server may include a base station. The server 1808 may receive W ₁ from the terminal 1 1802 . In this specification, a terminal may include a wireless communication device such as a vehicle, an autonomous vehicle, or an unmanned device. Here, W ₁ may mean the weight of the neural network generated by terminal 1 through learning. Terminal 1 (1802) can receive the W ^BS from the server (1808). Here, W ^BS may mean the weight of the neural network generated by the server through learning.

Server 1808 may receive W ₂ from terminal 2 1804 . Here, W ₂ may mean the weight of the neural network generated by terminal 2 through learning. Terminal 2 (1804) can receive the W ^BS from the server (1808). Here, W ^BS may mean the weight of the neural network generated by the server through learning. Terminal 3 (1806) may also operate in the same manner as described above. In addition, the server 1808 may generate the W ^BS through learning based on weights received from a plurality of terminals as described above. W ^BS generated by the server can be expressed as in Equation 1 below.

[Equation 1]

Here, U may mean the number of terminals that provided weight information to the server.

19 is a diagram illustrating an example of federated learning applicable to the present disclosure. Referring to FIG. 19 , a vehicle may include a central server 1902 and a sensor camera 1904 . Also, the vehicle may include a plurality of sensor cameras. The central server 1902 can communicate with the sensor camera 1904. For example, the central server may communicate with the sensor camera by including an Integrated Access Backhaul-donor (IAB-donor) and an Integrated Access Backhaul-node (IAB-node). As a specific example, the central server may include a central unit (CU) of the IAB donor, and the sensor camera may include a distributed unit (DU) of the IAB donor. As described above with reference to FIG. 18 , the central server may generate weights W ^BS through learning using artificial intelligence based on information received from a plurality of sensor cameras. The sensor camera may generate weights through learning using artificial intelligence based on information received from the central server. The sensor cameras may transmit the weights W each generated to the central server.

20 is a diagram illustrating an example of a federated learning procedure applicable to the present disclosure. In step S2001, the device may update the local model using local data. For example, a device may receive a global model and perform training with stored local data. In step S2003, the device may transmit the updated local model to the server. For example, the device may transmit the calculated parameters to the server. The calculated parameters may include weights and information of the deep neural network. In step S2005, the server may update the global model through the received local model. The server may update parameters of the global model using parameters received from a plurality of devices. The server may complete the parameter update and evaluate the cost function. In step S2007, the server may transmit the updated global model to the device. For example, the server may transmit the global model to multiple devices to perform learning. The device and the server may repeat the above-described procedures until the values obtained from the above-described evaluation converge. Through this process, the device and the server can derive an optimal weight value. The cost function and the optimal weight value can be expressed as Equation 2 below.

[Equation 2]

d _k may mean a value (trained information) measured by the base station. x _k (w) may mean a value measured by terminal k. For example, x _k (w) can be expressed as f _NN (s _k ;). J(w) may mean a value obtained by adding all squares of differences between d _k and x _k (w). W ^* may mean an estimator that makes J(w) a minimum value.

21 is a diagram illustrating an example of a terminal applicable to the present disclosure. In this specification, a terminal may include a vehicle and a wireless communication device. The terminal may include a sensor 2102 and a central server 2104 . Also, the terminal may include a plurality of sensors. The central server 2104 may include an ISP 2108 and a neural network training unit (2110). Sensor 2102 and central server 2104 may communicate. As an example, the central server 2104 may communicate with a plurality of sensors. The sensor may include a camera sensor. A virtual neural network unit (2106) may be located inside or outside the sensor (2102).

As another example, the aforementioned central server 2104 may be replaced by a base station. That is, the ISP 2108 and the neural net learner 2110 may be located outside the terminal and may be included in the base station.

The sensor 2102 according to the present disclosure may not include an image signal processing (ISP) device. In addition, the sensor 2102 according to the present disclosure may process an input image according to a specific purpose through the virtual neural net unit 2106. For example, the sensor 2102 may extract only a person from an image excluding other objects through the virtual neural net unit 2106 . That is, the virtual neural net unit 2106 may process images according to a specific purpose. For example, the virtual neural net unit may process an image according to a specific purpose based on artificial intelligence. The sensor may process the image through the virtual neural net unit and transmit data to the central server 2104. The data transmitted by the sensor to the central server does not go through the ISP, so data loss can be reduced. In addition, a plurality of sensors included in the terminal may transmit data to the central server without processing images through the ISP. A plurality of sensors can minimize data loss by transmitting unprocessed data to the central server. Central server 2104 may include ISP 2108 . The central server can process the image through the ISP and compare it with the data output by the virtual neural net. The relevant procedures are described in detail below.

The sensor 2102 can send data output to the virtual neural net unit 2106. The virtual neural net unit 2106 may receive the data output of the sensor 2102 as an input. The virtual neural net unit 2106 may multiply data received as an input by a weight value and output the result. For example, the virtual neural net unit may output x _k by multiplying data received as an input by a weight w ^BS value, which is a neural net learning result. The central server 2104 may receive the result of the virtual neural net as an input. For example, the neural net learning unit 2110 may receive output data of the virtual neural net unit 2106 as an input. ISP 2108 may also receive data from sensor 2102 . The ISP may perform image processing on the input data. The neural net learner 2110 may compare the output data of the ISP 2108 with the learning result. For example, the neural net learning unit may obtain a difference between an output d _k of the ISP and a virtual neural net x _k value. The neural net learning unit may generate weight values based on these differences. The neural net learning unit may transmit the generated weight value to the virtual neural net unit. The neural net learning unit and the virtual neural net unit may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

22 is a diagram illustrating an example of a terminal applicable to the present disclosure. The terminal may include a sensor 2202 and a central server 2204 . Also, the terminal may include a plurality of sensors. The central server 2204 may include an ISP 2208, a neural network training unit (2210) and a reverse virtual neural network unit (2112). Sensor 2202 and central server 2204 can communicate with each other. As an example, the central server 2204 may communicate with a plurality of sensors. The sensor may include a camera sensor. The virtual neural network unit 2206 may be located inside or outside the sensor 2202 .

As another example, the aforementioned central server 2204 may be replaced by a base station. That is, the reverse virtual neural net unit 2212, the ISP 2208, and the neural net learning unit 2210 may be located outside the terminal and may be included in the base station.

Even if the central server 2204 according to the present disclosure does not directly receive output data of the sensor from the sensor 2202 , it may generate data having a specific purpose removed through the reverse virtual neural net unit 2212 . For example, the reverse virtual neural net unit may generate output data of a sensor. The relevant procedures are described in detail below.

The sensor 2202 can send data output to the virtual neural net unit 2206. The virtual neural net unit 2206 may receive the data output of the sensor 2202 as an input. The virtual neural net unit 2206 may multiply data received as an input by a weight value and output the result. For example, the virtual neural net unit may output x _k by multiplying data received as an input by a weight w ^BS value, which is a neural net learning result. The central server 2204 may receive the result of the virtual neural net as an input. For example, the neural net learning unit 2210 may receive output data of the virtual neural net unit 2206 as an input. The reverse virtual neural net unit 2212 may receive output data of the virtual neural net unit 2206 .

The reverse virtual neural net unit may remove the specific purpose of the image processed by the virtual neural net unit according to the specific purpose. For example, the reverse virtual neural net unit may generate data with a specific purpose removed based on artificial intelligence. Also, the reverse virtual neural net unit may process the received data to generate output data of the sensor. The reverse virtual neural net unit may transmit the generated data to the ISP, and the ISP 2208 may receive data from the reverse neural net unit 2212. The ISP may perform image processing on the input data.

The neural net learner 2210 may compare the output data of the ISP 2208 with the learning result. For example, the neural net learning unit may obtain a difference between an output d _k of the ISP and a virtual neural net x _k value. The neural net learning unit may generate weight values based on these differences. The neural net learning unit may transmit the generated weight value to the virtual neural net unit. The neural net learning unit and the virtual neural net unit may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

23 is a diagram illustrating an example of a terminal applicable to the present disclosure. The terminal may include a sensor 2302 and a central server 2304 . Also, the terminal may include a plurality of sensors. The central server 2304 may include a reverse virtual neural network unit 2308 and a neural network training unit 2310. Sensor 2302 and central server 2304 can communicate with each other. As an example, the central server 2304 may communicate with a plurality of sensors. The sensor may include a camera sensor. The virtual neural network unit 2306 may be located inside or outside the sensor 2302.

As another example, the aforementioned central server 2304 may be replaced by a base station. That is, the reverse virtual neural net unit 2308 and the neural net learning unit 2310 may be located outside the terminal and may be included in the base station.

The central server 2304 according to the present disclosure may perform ISP through the reverse virtual neural net unit 2308. Hereinafter, related procedures will be described in detail.

The sensor 2302 can send data output to the virtual neural net unit 2308. The virtual neural net unit 2306 may receive the data output of the sensor 2302 as an input. The virtual neural net unit 2306 may multiply data received as an input by a weight value and output the result. For example, the virtual neural net unit may output x _k by multiplying data received as an input by a weight w ^BS value, which is a neural net learning result. The central server 2304 may receive the result of the virtual neural net as an input. For example, the neural net learning unit 2310 may receive output data of the virtual neural net unit 2306 as an input. The reverse virtual neural net unit 2308 may receive output data of the virtual neural net unit 2306.

The reverse virtual neural net unit may remove the specific purpose of the image processed by the virtual neural net unit according to the specific purpose. For example, the reverse virtual neural net unit may generate data with a specific purpose removed based on artificial intelligence. Also, the reverse virtual neural net unit may process the received data to generate output data of the sensor.

Also, the reverse virtual neural network unit may perform image signal processing (ISP). Accordingly, the neural net learning unit 2310 may compare the output data of the reverse virtual neural net unit and the learning result. For example, the neural net learning unit may obtain a difference between an output d _k value of the reverse virtual neural net unit and a virtual neural net x _k value. The neural net learning unit may generate weight values based on these differences. The neural net learning unit may transmit the generated weight value to the virtual neural net unit. The neural net learning unit and the virtual neural net unit may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

24 is a diagram illustrating an example of a terminal applicable to the present disclosure. The terminal may include a first sensor 2402 and a second sensor 2404 . The first sensor may include a radar. The second sensor may include an ISP. The central server may include a neural network training unit (2408). As another example, another terminal may include the neural net learning unit 2408. As another example, the base station may include a neural net learning unit 2408. That is, the neural net learning unit 2408 may be located in another terminal, a central server, or a base station. The location of the neural net learning unit is not limited to the above-described embodiment. The first sensor 2402 and the central server may communicate with each other. The second sensor 2404 and the central server can communicate with each other. The virtual neural network unit 2406 may be located inside or outside the first sensor 2402 .

A terminal according to the present disclosure may include a first sensor 2402 that does not include an ISP and a second sensor 2404 that includes an ISP. The relevant procedures are described in detail below.

The first sensor 2402 may transmit data output to the virtual neural net unit 2406 . The virtual neural net unit 2406 may receive the data output of the first sensor 2402 as an input. The virtual neural net unit 2406 may multiply data received as an input by a weight value and output the result. For example, the virtual neural net unit may output x _k by multiplying data received as an input by a weight w ^BS value, which is a neural net learning result. The central server may receive the result of the virtual neural net as an input. For example, the neural net learning unit 2408 may receive output data of the virtual neural net unit 2406 as an input. Since the second sensor 2404 includes an ISP device, it can directly perform ISP. The neural net learner 2408 may receive data that has undergone ISP processing from the second sensor. For example, the neural net learner may receive the ISP-processed d _k value from the second sensor. Accordingly, the neural net learner may compare data received from the second sensor with a learning result. For example, the neural net learner may obtain a difference between an output d _k value of the second sensor and an output x _k value of the virtual neural net unit. The neural net learning unit may generate weight values based on these differences. The neural net learning unit may transmit the generated weight value to the virtual neural net unit. The neural net learning unit and the virtual neural net unit may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

25 is a diagram illustrating an example of an operation procedure of a terminal applicable to the present disclosure. In step S2501, the terminal may perform image sensing through a sensor. Also, the terminal may perform image sensing through a plurality of sensors. The sensor may include a camera sensor and may not include an image signal processor (ISP). That is, the terminal can perform image sensing without performing ISP.

In step S2503, the terminal may learn a first neural network based on image sensing. The terminal may process the sensed image according to a specific purpose by learning the first neural network. For example, the terminal may include a virtual neural network unit. In addition, the virtual neural net unit may learn the first neural network and process the sensed image according to a specific purpose.

For example, the sensor of the terminal may transmit output data to the virtual neural net unit of the terminal. The virtual neural network unit of the terminal may multiply input data by a weight value and output the result. That is, the terminal may output x _k by multiplying the neural net learning result by the weight w ^BS value.

In step S2505, the terminal may transmit the first neural network learning result. For example, the terminal may transmit the first neural network learning result to the central server base station inside the terminal. Here, the central server inside the terminal may include a device capable of performing ISP. As another example, the terminal may transmit the first neural network learning result to the base station. Here, the base station may include a device capable of performing ISP. As another example, the terminal may transmit the first neural network learning result to another terminal. Here, another terminal may include a device capable of performing ISP.

Image signal processing (ISP) may not be performed on data transmitted by a terminal to a central server, a base station, or another terminal. The terminal can reduce data loss by transmitting data not performed by the ISP. In addition, a plurality of sensors included in the terminal may transmit data for which ISP is not performed to the central server or the base station. Accordingly, data loss can be minimized.

Also, some of the plurality of sensors included in the terminal may perform ISP. The terminal may transmit ISP-performed data to a central server, a base station, or other terminals using sensors capable of performing the ISP.

The terminal may include a central server therein. A central server may perform an ISP. For example, the central server may perform ISP using an ISP device. As another example, the central server may perform ISP by learning a reverse virtual neural network.

The central server of the terminal may perform ISP through a reverse virtual neural network unit. Hereinafter, related procedures will be described in detail. The central server of the terminal may receive data transmitted by the virtual neural net unit of the terminal. The reverse virtual neural net unit of the central server may remove the specific purpose of the image processed by the virtual neural net unit according to the specific purpose. For example, the reverse virtual neural net unit may generate data with a specific purpose removed based on artificial intelligence. Also, the reverse virtual neural net unit may process the received data to generate output data of the sensor.

In addition, the reverse virtual neural network unit of the central server may perform image signal processing (ISP). If the reverse virtual neural network portion of the central server is capable of performing an ISP, the central server may not include an ISP device. In addition, the central server may additionally include an ISP device even when the reverse virtual neural network unit can perform ISP. If the central server additionally includes an ISP device, the ISP device of the central server may perform ISP on output data of the reverse virtual neural net unit. The ISP device may transmit output data of the reverse virtual neural net unit that performed the ISP to the neural net learning unit of the central server.

The neural net learning unit of the central server may learn the second neural network. For example, the neural net learning unit of the central server may compare the output data of the reverse virtual neural net unit and the second neural network learning result. Specifically, the neural net learning unit of the central server may obtain a difference between an output d _k value of the reverse virtual neural net unit and an output x _k value of the virtual neural net learning unit. The neural net learning unit may generate weight values based on these differences. The neural net learning unit may transmit the generated weight value to the virtual neural net unit. The neural net learning unit of the terminal and the virtual neural net unit of the terminal may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

As another example, the central server of the terminal may not include a reverse virtual neural net unit, but may include an ISP device and a neural net learning unit. Here, the central server may receive the result of the virtual neural net as an input. For example, the neural net learning unit may receive output data of the virtual neural net unit as an input. The ISP device can also receive data from the terminal's sensor. The ISP device may perform ISP on the input data. The central server may learn the second neural network. For example, the neural net learning unit of the central server may compare the output data of the ISP and the second neural network learning result. For example, the neural net learning unit may obtain a difference between an output dk value of the ISP and an output x _k value of the virtual neural net unit. The neural net learning unit may generate weight values based on these differences. The neural net learning unit may transmit the generated weight value to the virtual neural net unit. The neural net learning unit of the terminal and the virtual neural net unit of the terminal may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

26 is a diagram illustrating an example of a server, base station, or terminal procedure applicable to the present disclosure. Referring to FIG. 25 , a terminal may transmit a first neural network learning result to a central server, a base station, or other terminals. Procedures related to FIG. 26 will be described below.

In step S2601, the base station may receive a first neural network training result. For example, the base station may receive the result of the virtual neural net unit of the terminal as an input. That is, the neural net learning unit of the base station may receive output data of the virtual neural net unit of the terminal as an input. Also, the reverse virtual neural net unit of the base station may receive output data of the virtual neural net unit. The reverse virtual neural net unit may remove the specific purpose of the image processed by the virtual neural net unit according to the specific purpose. The reverse virtual neural net unit may generate data with a specific purpose removed based on artificial intelligence. The reverse virtual neural net unit may process the received data to generate output data of the sensor. Also, the reverse virtual neural network unit may perform image signal processing (ISP).

In step S2603, the reverse virtual neural network unit of the base station may learn a second neural network based on a learning result of the first neural network. For example, the neural net learning unit of the base station may compare the output data of the reverse virtual neural net unit and the first neural network learning result. Specifically, the neural net learning unit of the base station may obtain a difference between an output _dk value of the reverse virtual neural net unit and an output x _k value of the virtual neural net unit. The neural net learning unit of the base station may generate a weight value based on this difference.

In addition, the base station may additionally include an ISP device. In this case, ISP may be performed on the output data of the reverse virtual neural network unit by the ISP device.

In step S2605, the base station may transmit the second neural network learning result. For example, the neural net learning unit of the base station may transmit the generated weight value to the virtual neural net unit of the terminal. The neural net learning unit of the base station and the virtual neural net unit of the terminal may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

As another example, the base station may include an ISP device and a neural network learning unit. The base station may receive the result of the virtual neural net as an input. The neural net learning unit of the base station may receive output data of the virtual neural net unit of the terminal as an input. The ISP device of the base station can also receive data from the sensor of the terminal. The ISP device may perform ISP on the input data.

In step S2603, the base station may learn a second neural network based on a learning result of the first neural network. The neural net learning unit of the base station may compare the output data of the ISP and the first neural network learning result. For example, the neural net learning unit of the base station may obtain a difference between an output d _k value of the ISP and an output x _k value of the virtual neural net unit of the terminal. The neural net learning unit of the base station may generate a weight value based on this difference.

In step S2605, the base station may transmit the second neural network learning result. Specifically, the neural net learning unit of the base station may transmit the generated weight value to the virtual neural net unit of the terminal. The neural net learning unit of the base station and the virtual neural net unit of the terminal may repeat the above-described procedure until the weight values converge. As the procedure is repeated, the weight values can become more and more accurate.

It is obvious that examples of the proposed schemes described above may also be included as one of the implementation methods of the present disclosure, and thus may be regarded as a kind of proposed schemes. In addition, the above-described proposed schemes may be implemented independently, but may also be implemented in a combination (or merged) form of some proposed schemes. Information on whether the proposed methods are applied (or information on the rules of the proposed methods) may be defined so that the base station informs the terminal through a predefined signal (eg, a physical layer signal or a higher layer signal). .

The present disclosure may be embodied in other specific forms without departing from the technical ideas and essential characteristics described in the present disclosure. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present disclosure should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent range of the present disclosure are included in the scope of the present disclosure. In addition, claims that do not have an explicit citation relationship in the claims may be combined to form an embodiment or may be included as new claims by amendment after filing.

Embodiments of the present disclosure may be applied to various wireless access systems. As an example of various wireless access systems, there is a 3rd Generation Partnership Project (3GPP) or 3GPP2 system.

Embodiments of the present disclosure may be applied not only to the various wireless access systems, but also to all technical fields to which the various wireless access systems are applied. Furthermore, the proposed method can be applied to mmWave and THz communication systems using ultra-high frequency bands.

Additionally, embodiments of the present disclosure may be applied to various applications such as free-running vehicles and drones.

Claims (16)

1. In a method of operating a terminal in a wireless communication system,

   performing image sensing through the first sensor;

   Receiving neural network related information from a base station;

   learning a first neural network based on the neural network related information; and

   Transmitting the first neural network learning result data to a base station;

   The first sensor does not perform image signal processing (ISP),

   The neural network-related information includes information indicating whether or not the terminal learns the neural network;

   The learning of the first neural network processes the image sensing result data based on artificial intelligence.
2. According to claim 1,

   The method of claim 1 , wherein the neural network related information includes weight information of the first neural network.
3. According to claim 1,

   The method of performing the first neural network learning only when the information indicating whether the neural network has been trained indicates the first neural network learning.
4. According to claim 1,

   Receiving second neural network learning result data from the base station.
5. According to claim 1,

   A method of performing image sensing through a second sensor, wherein the second sensor performs ISP.
6. According to claim 5,

   Transmitting ISP output data of the second sensor to the base station.
7. In the method of operating a base station in a wireless communication system,

   Transmitting neural network related information to a terminal;

   receiving first neural network learning result data from the terminal;

   learning a first reverse neural network based on the first neural network learning result data; and

   Learning a second neural network based on a result of learning the first reverse neural network;

   The reverse first neural network learning performs image signal processing (ISP), and the neural network-related information includes information indicating whether or not the terminal learns the neural network;

   The learning of the first neural network processes image sensing result data based on artificial intelligence, and the learning of the second neural network compares an ISP performance result with the first neural network learning result data.
8. According to claim 7,

   The method of claim 1 , wherein the neural network related information includes weight information of the first neural network.
9. According to claim 7,

   The method of transmitting second neural network learning result data to the terminal.
10. According to claim 7,

    The method of claim 1, wherein the base station includes an image signal processor (ISP).
11. According to claim 10,

    The method of claim 1 , wherein the image signal processor (ISP) performs image signal processing (ISP) based on the first reverse neural network learning result data.
12. According to claim 7,

    The method of receiving ISP output data from the terminal, and learning the second neural network based on the ISP output data.
13. In a terminal in a wireless communication system,

    transceiver; and

    It includes a processor connected to the transceiver,

    the processor,

    Controlling the terminal to perform image sensing through a first sensor,

    Controls the transceiver to receive neural network related information from a base station;

    Control the terminal to learn a first neural network based on the neural network-related information;

    Controlling the transceiver to transmit the first neural network learning result data to the base station;

    The first sensor does not perform image signal processing (ISP),

    The neural network-related information includes information indicating whether or not the terminal learns the neural network;

    The learning of the first neural network processes the image sensing result data based on artificial intelligence.
14. In a base station in a wireless communication system,

    transceiver; and

    It includes a processor connected to the transceiver,

    the processor,

    Control the transceiver to transmit neural network related information to the terminal, and control to receive first neural network learning result data from the terminal;

    Control the base station to learn a first reverse neural network based on the first neural network learning result data, and to learn a second neural network based on the first reverse neural network learning result;

    The reverse first neural network learning performs image signal processing (ISP), and the neural network-related information includes information indicating whether or not the terminal learns the neural network;

    Wherein the learning of the first neural network processes image sensing result data based on artificial intelligence, and the learning of the second neural network compares an ISP performance result with the first neural network learning result data.
15. An apparatus comprising at least one memory and at least one processor functionally connected to the at least one memory, comprising:

    The at least one processor is the device,

    Control to perform image sensing through the first sensor,

    Control to receive neural network related information from a base station;

    Control to learn a first neural network based on the neural network related information;

    Control to transmit the first neural network learning result data to the base station;

    The first sensor does not perform image signal processing (ISP),

    The neural network-related information includes information indicating whether or not the terminal learns the neural network;

    The learning of the first neural network processes the image sensing result data based on artificial intelligence.
16. In a non-transitory computer-readable medium storing at least one instruction,

    comprising the at least one instruction executable by a processor;

    The at least one instruction, the device,

    Instruct to perform image sensing through the first sensor,

    Instructing to receive neural network related information from the base station;

    Instruct to learn a first neural network based on the neural network related information;

    instructing the base station to transmit the first neural network learning result data;

    The first sensor does not perform image signal processing (ISP),

    The neural network-related information includes information indicating whether or not the terminal learns the neural network;

    Wherein the learning of the first neural network processes the image sensing result data based on artificial intelligence.

Images