KR102382170B1 - Method and apparatus for data processing - Google Patents

Abstract

Embodiments provide a video streaming method and device. According to one embodiment, the image streaming method includes: an operation of setting, by a first terminal, a first terminal as a first host and a second terminal as a first guest in response to an image transmission request signal received from the second terminal; an operation of exchanging pairing information and terminal status information between the first terminal and the second terminal and performing pairing by the first terminal and the second terminal; an operation of setting, by the first terminal, information for transmitting the entire image in response to the image transmission request signal; an operation of terminating, by the first terminal, pairing between the first host and the first guest before transmitting the entire image; an operation of setting the first terminal as the second host in response to the image transmission request signal received from the third terminal, and the third terminal as a second guest, before terminating the transmission of the entire image by the first terminal; an operation of exchanging pairing information and terminal status information between the first terminal and the third terminal and performing pairing, by the first terminal and the third terminal; and an operation of receiving, by the third terminal, a part of the entire image from the second terminal, and receiving the remainder of the part of the entire image from the first terminal.

Description

Video streaming method and apparatus {METHOD AND APPARATUS FOR DATA PROCESSING}

Embodiments of the present invention relate to a technology for transmitting an image, and to a technology for transmitting an image using a terminal.

In order to implement a video conferencing service, an infrastructure capable of effectively transmitting and receiving video and audio data is required. It is possible to provide different types of video conferencing systems and services according to the configuration of the server and client systems according to the interworking structure of the transmitted and received images, the image processing method, and the type of the terminal. The MCU-based video conferencing service technology consists of a user terminal (end-point), a multipoint control unit (MCU), and a management system. A multi-point control device performs signal conditioning, video and audio mixing, and resource control to process video and audio received from different users. It is processed through a multi-point control unit. This video conferencing service requires remote communication technology and data processing through a server because it is premised on remote participants.

According to one embodiment, there is provided a video streaming system capable of streaming video faster with less resources by performing video streaming between a plurality of terminals using data communication without a server, for example, a communication method such as Bluetooth. do.

Technical problems to be achieved in the embodiments are not limited to those mentioned above, and other technical problems not mentioned may be considered by those of ordinary skill in the art from various embodiments to be described below. can

In a video streaming method according to an embodiment, in response to a video transmission request signal received from a second terminal by a first terminal, the first terminal is set as a first host, and the second terminal is set as a first guest setting action; exchanging pairing information and terminal status information between the first terminal and the second terminal and performing pairing by the first terminal and the second terminal; setting, by the first terminal, information for transmitting the entire video in response to the video transmission request signal; terminating, by the first terminal, pairing between the first host and the first guest before transmitting the entire video; Before terminating the transmission of the entire video by the first terminal, in response to a video transmission request signal received from the third terminal, the first terminal is set as a second host, and the third terminal is configured as a second host. setting as a guest; exchanging pairing information and terminal status information between the first terminal and the third terminal and performing pairing by the first terminal and the third terminal; and receiving, by the third terminal, a part of the entire image from the second terminal, and transmitting the remainder of the part of the entire image from the first terminal.

The operation of receiving a portion of the entire image from the second terminal and receiving the remainder of the portion of the entire image from the first terminal includes, by the first terminal, the second terminal using an image transmission prediction model. dividing a portion to be transmitted by a terminal and a remaining portion to be transmitted by the first terminal; and receiving, by the third terminal, the divided portion from the second terminal based on the division result, and receiving the remaining divided portion from the first terminal.

The image transmission prediction model includes an input layer, at least one hidden layer, and an output layer, and the training data consisting of a plurality of training images, terminal state information, network state information, and correct answer prediction information is the training data of the image transmission prediction model. It is input to an input layer and passes through the one or more hidden layers and output layers to output an output vector, the output vector is input to a loss function layer connected to the output layer, and the loss function layer is the output vector and each learning The loss value is output using a loss function that compares the correct vector to the data, and the parameters of the spatial information extraction model are learned in a direction in which the loss value becomes smaller,

[Equation]

The loss function follows the above equation, where N is the number of the plurality of training data, n is a natural number identifying the training data, k is a natural number identifying the value of the nth training data, nk is the nth It may mean the k-th value of the training data, t may mean correct answer data, y may mean the output vector, and E may mean a loss value.

The operation of dividing the part to be transmitted by the second terminal and the remaining part to be transmitted by the first terminal using the image transmission prediction model includes the first terminal, the second terminal, and the third terminal, respectively. acquiring terminal status information, network status information between the first terminal, the second terminal, and the third terminal; a first transmission rate of video transmission from the first terminal to the second terminal using the video transmission prediction model based on the quality information of the entire video, the status information of each terminal, and the network status information; estimating a second transmission rate of video transmission from the first terminal toward the third terminal and a third transmission rate of video transmission from the second terminal toward the third terminal; and a portion to be transmitted by the second terminal and the remaining portion to be transmitted by the first terminal based on the first transmission rate, the second transmission rate, the third transmission rate, and the playback rate of the third terminal. It may include an operation of dividing.

Divide the part to be transmitted by the second terminal and the remaining part to be transmitted by the first terminal based on the first transmission rate, the second transmission rate, the third transmission rate, and the reproduction rate of the third terminal The operation may include: determining a first division unit based on the first transmission rate, the second transmission rate, the third transmission rate, and the reproduction rate of the third terminal; dividing the entire image into a plurality of partial images including a first partial image, a second partial image, a third partial image, and a fourth partial image according to the first division unit; and allocating the first partial image and the third partial image to the first terminal, and allocating the second partial image and the fourth partial image to the second terminal.

The determining of the first division unit may include: comparing the first transmission rate and the third transmission rate; comparing the third transfer rate and the reproduction rate when the first transfer rate is greater than the third transfer rate; setting the first division unit as a first division length when a difference between the third transmission rate and the reproduction rate is equal to or greater than a first threshold value; and setting the first division unit as a second division length when the difference between the third transmission rate and the reproduction rate is less than a first threshold, wherein the first division length is greater than the second division length. can be small

The method includes: receiving, by the first terminal, an image quality change request signal from the third terminal; In response to the image quality change request signal, the second terminal from the first terminal using the image transmission prediction model based on the to-be-changed quality information of the entire image, the state information of each terminal, and the network state information Predicting a fourth transmission rate of video transmission from the terminal, a fifth transmission rate of video transmission from the first terminal to the third terminal, and a sixth transmission rate of video transmission from the second terminal toward the third terminal action to do; and a portion to be transmitted by the second terminal and the remaining portion to be transmitted by the first terminal based on the fourth transmission rate, the fifth transmission rate, the sixth transmission rate, and the reproduction rate of the third terminal. It may further include an operation of dividing.

A portion to be transmitted by the second terminal and the remaining portion to be transmitted by the first terminal are divided based on the fourth transmission rate, the fifth transmission rate, the sixth transmission rate and the reproduction rate of the third terminal. The operation may include: determining a second division unit based on the fourth transmission rate, the fifth transmission rate, the sixth transmission rate, and the reproduction rate of the third terminal; dividing the untransmitted image from among the entire image according to the second division unit into a plurality of partial images including a fifth partial image, a sixth partial image, a seventh partial image, and an eighth partial image; and allocating the fifth partial image and the seventh partial image to the first terminal, and allocating the sixth partial image and the eighth partial image to the second terminal.

The determining of the second division unit may include: comparing the first transmission rate and the third transmission rate; comparing the third transfer rate and the reproduction rate when the first transfer rate is greater than the third transfer rate; setting the second division unit as a third division length when a difference between the third transmission rate and the reproduction rate is equal to or greater than a second threshold; and setting the second division unit as a fourth division length when the difference between the third transmission rate and the reproduction rate is less than a second threshold, wherein the third division length is greater than the fourth division length. can be small

According to an embodiment, the video streaming system may stream video faster with fewer resources by performing video streaming between a plurality of terminals using data communication without a server, for example, a communication method such as Bluetooth.

Effects obtainable from the embodiments are not limited to the effects mentioned above, and other effects not mentioned are clearly derived and understood by those of ordinary skill in the art based on the detailed description below. can be

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to aid understanding of the embodiments, provide various embodiments and, together with the detailed description, explain technical features of the various embodiments.
1 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
2 is a diagram showing the configuration of a program according to an embodiment.
3 is a diagram for explaining an operation of a video streaming system according to an embodiment.
4 is a flowchart illustrating an operation of an image transmission method according to an embodiment.
5 is a diagram illustrating a result of virtualization by a video streaming system according to an embodiment.
6 is a flowchart illustrating an exemplary operation of an image transmission method according to an embodiment.
7 is a diagram illustrating a configuration of an image transmission apparatus according to an exemplary embodiment.

The following embodiments combine elements and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. In addition, various embodiments may be configured by combining some components and/or features. The order of operations described in various embodiments may be changed. Some features or features of one embodiment may be included in another embodiment, or may be replaced with corresponding features or features of another embodiment.

In the description of the drawings, procedures or steps that may obscure the gist of various embodiments are not described, and procedures or steps that can be understood at the level of those of ordinary skill in the art are also not described. did

Throughout the specification, when a part is said to "comprising or including" a certain component, it does not exclude other components unless otherwise stated, meaning that other components may be further included. do. In addition, terms such as "...unit", "...group", and "module" described in the specification mean a unit that processes at least one function or operation, which is hardware or software or a combination of hardware and software. can be implemented as Also, "a or an," "one," "the," and like related terms are used herein in the context of describing various embodiments (especially in the context of the claims that follow). Unless indicated otherwise or clearly contradicted by context, it may be used in a sense including both the singular and the plural.

Hereinafter, embodiments according to various embodiments will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of various embodiments, and is not intended to represent the only embodiments.

In addition, specific terms used in various embodiments are provided to help the understanding of various embodiments, and the use of these specific terms may be changed to other forms without departing from the technical spirit of various embodiments. .

1 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with at least one of the electronic device 104 and the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 may be included. In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be The electronic device 101 may also be referred to as a client, a terminal, or a peer.

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various image transmission or operation. According to an embodiment, as at least part of image transmission or operation, the processor 120 stores a command or data received from another component (eg, the sensor module 176 or the communication module 190 ) into the volatile memory 132 . may be stored in the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 may use less power than the main processor 121 or may be set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the co-processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used in a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . A sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a LAN (local area network) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 includes various technologies for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The server 108 is connected to the electronic device 101 and may provide a service to the connected electronic device 101 . In addition, the server 108 may store and manage various types of information of users who have joined as members by performing a membership registration procedure, and may provide various purchase and payment functions related to services. Also, the server 108 may share execution data of a service application executed in each of the plurality of electronic devices 101 in real time so that the service can be shared among users. The server 108 may have the same configuration as a typical web server (Web Server) or WAP server (WAP Server) in terms of hardware. However, in terms of software, it may include a program module that is implemented through any language, such as C, C++, Java, Visual Basic, or Visual C, and performs various functions. In addition, the server 108 is generally connected to an unspecified number of clients and/or other servers through an open computer network such as the Internet, receives a request for performing a task from a client or other server, and derives and provides the work result. It means a computer system and the computer software (server program) installed therefor. In addition, the server 108, in addition to the above-described server program, a series of application programs (Application Program) operating on the server 108 and in some cases, various databases (DB: Database, hereinafter " It should be understood as a broad concept including "DB". Accordingly, the server 108 categorizes member registration information, various information and data about the game, stores and manages it in a DB, and this DB may be implemented inside or outside the server 108 . In addition, the server 108 uses a server program that is provided in various ways according to operating systems such as DOS, Windows, Linux, UNIX, and Macintosh in general server hardware. As a representative example, a website used in a Windows environment, Internet Information Server (IIS), and CERN, NCSA, APPACH, etc. used in a Unix environment may be used. In addition, the server 108 may interwork with an authentication system and a payment system for user authentication of a service or a purchase payment related to a service.

The first network 198 and the second network 199 are a connection structure capable of exchanging information between respective nodes such as terminals and servers, or a network connecting the server 108 and the electronic devices 101 and 104 . (Network). The first network 198 and the second network 199 may include the Internet, a local area network (LAN), a wireless local area network (Wireless LAN), a wide area network (WAN), a personal area network (PAN), and 3G. , 4G, LTE, 5G, Wi-Fi, and the like. The first network 198 and the second network 199 may be the closed first network 198 and the second network 199 such as a LAN or WAN, but preferably an open type such as the Internet. The Internet is a TCP/IP protocol and several services existing in its upper layers, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP ( Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service) refer to the structure of the worldwide open computer first network 198 and second network 199 .

The database may have a general data structure implemented in a storage space (hard disk or memory) of a computer system using a database management program (DBMS). The database may have a data storage form in which data can be freely searched (extracted), deleted, edited, added, and the like. Database is a relational database management system (RDBMS) such as Oracle, Infomix, Sybase, DB2, or object-oriented database management such as Gemston, Orion, O2, etc. It can be implemented for the purpose of an embodiment of the present disclosure by using a system (OODBMS) and an XML native database such as Excelon, Tamino, and Sekaiju, and its function It may have an appropriate field or elements to achieve .

2 is a diagram showing the configuration of a program according to an embodiment.

2 is a block diagram 200 illustrating a program 140 in accordance with various embodiments. According to an embodiment, the program 140 executes an operating system 142 for controlling one or more resources of the electronic device 101 , middleware 144 , or an application 146 executable in the operating system 142 . may include Operating system 142 may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least some of the programs 140 are, for example, preloaded into the electronic device 101 at the time of manufacture, or an external electronic device (eg, the electronic device 102 or 104 ), or a server (eg, the electronic device 102 or 104 ) when used by a user ( 108)) or may be updated. All or part of the program 140 may include a neural network.

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, a process, memory, or power) of the electronic device 101 . The operating system 142 may additionally or alternatively include other hardware devices of the electronic device 101 , for example, the input module 150 , the sound output module 155 , the display module 160 , and the audio module 170 . , sensor module 176 , interface 177 , haptic module 179 , camera module 180 , power management module 188 , battery 189 , communication module 190 , subscriber identification module 196 , or It may include one or more driver programs for driving the antenna module 197 .

The middleware 144 may provide various functions to the application 146 so that functions or information provided from one or more resources of the electronic device 101 may be used by the application 146 . The middleware 144 includes, for example, an application manager 201 , a window manager 203 , a multimedia manager 205 , a resource manager 207 , a power manager 209 , a database manager 211 , and a package manager 213 . ), a connectivity manager 215 , a notification manager 217 , a location manager 219 , a graphics manager 221 , a security manager 223 , a call manager 225 , or a voice recognition manager 227 . can

The application manager 201 may manage the life cycle of the application 146 , for example. The window manager 203 may manage one or more GUI resources used in the screen, for example. The multimedia manager 205, for example, identifies one or more formats required for playback of media files, and encodes or decodes a corresponding media file among the media files using a codec suitable for the selected format. can be done The resource manager 207 may manage the space of the source code of the application 146 or the memory of the memory 130 , for example. The power manager 209 may, for example, manage the capacity, temperature, or power of the battery 189 , and determine or provide related information necessary for the operation of the electronic device 101 by using the corresponding information. . According to an embodiment, the power manager 209 may interwork with a basic input/output system (BIOS) (not shown) of the electronic device 101 .

The database manager 211 may create, retrieve, or change a database to be used by the application 146 , for example. The package manager 213 may manage installation or update of an application distributed in the form of a package file, for example. The connectivity manager 215 may manage, for example, a wireless connection or a direct connection between the electronic device 101 and an external electronic device. The notification manager 217 may provide, for example, a function for notifying the user of the occurrence of a specified event (eg, an incoming call, a message, or an alarm). The location manager 219 may manage location information of the electronic device 101 , for example. The graphic manager 221 may manage, for example, one or more graphic effects to be provided to a user or a user interface related thereto.

Security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101 . The voice recognition manager 227, for example, transmits the user's voice data to the server 108, based at least in part on the voice data, a command corresponding to a function to be performed in the electronic device 101; Alternatively, the converted text data may be received from the server 108 based at least in part on the voice data. According to an embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to an embodiment, at least a portion of the middleware 144 may be included as a part of the operating system 142 or implemented as software separate from the operating system 142 .

Application 146 includes, for example, home 251 , dialer 253 , SMS/MMS 255 , instant message (IM) 257 , browser 259 , camera 261 , alarm 263 . , contacts 265, voice recognition 267, email 269, calendar 271, media player 273, album 275, watch 277, health 279 (such as exercise or blood sugar) measuring biometric information), or environmental information 281 (eg, measuring atmospheric pressure, humidity, or temperature information). According to an embodiment, the application 146 may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit specified information (eg, call, message, or alarm) to an external electronic device, or a device management application configured to manage the external electronic device. there is. The notification relay application, for example, transmits notification information corresponding to a specified event (eg, mail reception) generated in another application (eg, the email application 269 ) of the electronic device 101 to the external electronic device. can Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide it to the user of the electronic device 101 .

The device management application is, for example, a power supply (eg, turn-on or turn-off) of an external electronic device or some component thereof (eg, a display module or a camera module of the external electronic device) that communicates with the electronic device 101 . ) or a function (eg brightness, resolution, or focus). The device management application may additionally or alternatively support installation, deletion, or update of an application operating in an external electronic device.

Throughout this specification, a neural network, a neural network network, and a network function may be used interchangeably. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as “nodes”. These “nodes” may be referred to as “neurons”. A neural network is configured to include at least two or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

In a neural network, two or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may be in an input node relationship in a relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node sets values input to input nodes connected to the output node and links corresponding to the respective input nodes. An output node value may be determined based on the weight.

As described above, in a neural network, two or more nodes are interconnected through one or more links to form an input node and an output node relationship within the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between nodes and links, and the value of a weight assigned to each of the links. For example, when the same number of nodes and links exist and there are two neural network networks having different weight values between the links, the two neural network networks may be recognized as different from each other.

3 is a diagram for explaining an operation of a video streaming system according to an embodiment.

According to an embodiment, the video streaming system may include a plurality of terminals. For example, the video streaming system may include a first terminal 301 , a second terminal 303 , and a third terminal 305 . However, the number of terminals is an example, and four or more terminals may constitute the video streaming system.

The first terminal 301 may include the entire video to be streamed. The entire image may include video stream data and audio stream data. The entire image may further include metadata. The metadata may include, for example, subtitles. The video stream data may be an image to which various formats and codecs are applied. For example, the format of video stream data includes MPEG-4 (Moving Pictures Expert Group 4), WMV (Windows Media Video), AVI (Audio Video Interleave), MOV (Apple QuickTime Movie), MKV (MatrosKa Video), etc. can do. For example, the codec of the video stream data may include MPEG1, MPEG2, MP3, MPEG4, H.264, HEVC, etc. of the MPEG series.

Referring to FIG. 1 , any one of a plurality of terminals may generate a user input signal for requesting an image transmission. For example, the second terminal 303 may generate a user input signal in response to the user input. The second terminal 303 may transmit an image transmission request signal to the first terminal 301 including the entire image.

The second terminal 303 and the third terminal 305 may transmit an image transmission request signal to the first terminal 301 at the same time or at the same time. For example, the second terminal 303 may first transmit an image transmission request signal for the entire image to the first terminal 301 . While the first terminal 301 is transmitting the entire image to the second terminal 303 , the third terminal 305 may transmit an image transmission request signal for the same image to the first terminal 301 .

For example, each terminal may mark other terminals as pairable terminals. The second terminal 303 may display that the entire image is stored in the first terminal 301 . An image transmission request signal of the entire image is transmitted from the second terminal 303 to the first terminal 301 through a drag or touch input for the entire image stored in the user's first terminal 301. can However, this is only an example, and as an event for service initiation, various operations such as pressing a physical button on the terminal or drag and paste may be an event for service initiation. Also, a voice command operation may be an event for service initiation.

The first terminal 301 may perform pairing between the first terminal 301 and the second terminal 303 in response to the video transmission request signal received from the second terminal 303 . The first terminal 301 may set the first terminal 301 as the first host and the second terminal 303 as the first guest. The first terminal 301 and the second terminal 303 may exchange pairing information and terminal state information between the first terminal 301 and the second terminal 303 and perform pairing.

The pairing information may include information for mutual recognition and connection of two terminals on which pairing is performed. For example, ID information (identification information) and IP address information (connection information) of the two terminals may be included in the pairing information. The terminal status information means status information on the resources of the terminal capable of processing the received data packet, for example, the current buffer status, the total buffer size, the available buffer size, the maximum allowable buffer latency information, the buffer may include information related to the buffering processing capability of the buffer, such as overflow or underflow.

The first terminal 301 , the second terminal 303 , and the third terminal 305 may perform pairing independently of each other. The first terminal 301 , the second terminal 303 , and the third terminal 305 may transmit and receive images independently of each other after pairing is performed. Referring to FIG. 1 , a dotted line indicates pairing, and a solid line indicates image transmission.

When the second terminal 303 transmits an image transmission request signal to the first terminal 301 , the first terminal 301 and the second terminal 303 may perform pairing. The first terminal 301 may terminate pairing after or before providing the service to the second terminal 303 . When the third terminal 305 transmits a video transmission request signal to the first terminal 301, the first terminal ( 301) and the third terminal 305 may perform pairing and image transmission.

Since the first terminal 301 , the second terminal 303 and the third terminal 305 can execute pairing and service independently of each other, pairing and video between the first terminal 301 and the second terminal 303 . Pairing and service between the first terminal 301 and the third terminal 305 may be performed even while transmission is being performed. In addition, pairing between the second terminal 303 and the third terminal 305 and image transmission may also be performed regardless of establishing a communication session with the first terminal 301 . Terminals may perform pairing and image transmission between other terminals regardless of whether image transmission between each other is terminated.

Before starting image transmission, the first terminal 301 may terminate pairing between the first host and the first guest. For example, before termination of image transmission, the third terminal 305 may generate a user input signal in response to a user input for executing image transmission. The third terminal 305 may transmit an image transmission request signal to the first terminal 301 including the entire image.

For example, the third terminal 305 may display the first terminal 301 and the second terminal 303 as pairable terminals. The third terminal 305 may indicate that the entire image is stored in the first terminal 301 . An image transmission request signal may be transmitted from the third terminal 305 to the first terminal 301 through a user's drag or touch input for raw data.

Before termination of the service, the first terminal 301 may perform pairing between the first terminal 303 and the third terminal 305 in response to the video transmission request signal received from the third terminal 305 . The first terminal 301 may set the first terminal 301 as the second host and the third terminal 305 as the second guest in response to the video transmission request signal received from the third terminal 305 . there is. The first terminal 301 and the third terminal 305 may exchange pairing information and terminal state information between the first terminal 301 and the third terminal 305 and perform pairing.

With the implementation of strong 'social distancing' to prevent COVID-19, the number of office workers working remotely from home is increasing. The global expansion and prolongation of COVID-19 has led to a decrease in sales in most industries, but the solution market for online commerce education and remote work is expected to grow significantly. In particular, interest in remote work solution technology and market is increasing significantly after the COVID-19 crisis.

Existing video conferencing uses a method in which a long-distance participant connects each terminal through a server using remote communication. Although this method has the advantage of being able to transmit video data of a long-distance participant in real time, it requires a base station for remote communication and a server for image processing and transmission.

On the other hand, in the video streaming system according to an embodiment, when there is a request for transmission of the same video, the terminal storing the original video and the terminal having already received a part of the video may transmit the video to another terminal at the same time. Since images of different parts are simultaneously transmitted, other terminals can receive data faster.

A video streaming system may be used when a plurality of users have a non-face-to-face video conference in a narrow space such as an office or want to share a specific video in real time. The video streaming system may stream video faster with less resources by performing video streaming between a plurality of terminals using data communication not through a server, for example, a communication method such as Bluetooth.

Referring to FIG. 1 , a second terminal 303 may receive a portion of an image from a first terminal 301 . The first terminal 301 may receive an image transmission request signal from the third terminal 305 . The first terminal 301 may transmit a signal requesting image transmission to the third terminal 305 to the second terminal 303 . Here, the first terminal 301 may designate an image to be transmitted by the second terminal 303 .

The second terminal 303 may transmit the designated image to the third terminal 305 , and the first terminal 301 may transmit the remaining image to the third terminal 305 . The third terminal 305 may receive a portion of the entire image from the second terminal 303 and receive the remainder of the portion of the entire image from the first terminal 301 .

In order to perform video transmission in parallel, it is necessary to efficiently allocate a portion of the video transmitted by each terminal. The first terminal 301 may divide the part to be transmitted by the second terminal and the remaining part to be transmitted by the first terminal 301 using the image transmission prediction model. The third terminal 305 may receive the divided portion from the second terminal 303 based on the division result, and may receive the remaining portion from the first terminal 301 .

Here, the image transmission prediction model can efficiently divide an image to be transmitted by inputting state information of each terminal, network state information on a communication session formed between terminals, and information on an image to be transmitted. The network status information is status information on an established communication channel between terminals, and may include, for example, a bandwidth, a transmission delay difference between packets, a packet loss rate, a retransmission count for packet loss, and the like. Here, the bandwidth may be measured using a packet received by the terminal.

The training apparatus may pre-train the image transmission prediction model using a plurality of training data. The image transmission prediction model may include an input layer, one or more hidden layers, and an output layer. Training data consisting of a pair of a plurality of training images, terminal state information, network state information, and correct answer prediction information is input to the input layer of the image transmission prediction model and passes through one or more hidden layers and output layers to output an output vector. . The output vector may be input to a loss function layer connected to the output layer. The loss function layer may output a loss value using a loss function that compares the output vector with the correct vector for each training data. The parameters of the spatial information extraction model may be learned in a direction in which the loss value becomes smaller.

[Equation]

The loss function may follow Equation. In the formula, N is the number of a plurality of training data, n is a natural number identifying the training data, k is a natural number identifying the value of the n-th training data, nk is the k-th value of the n-th training data, t is It may mean correct answer data, y may mean an output vector, and E may mean a loss value.

For example, referring to FIG. 1 , the first terminal 301 includes terminal status information of each of the first terminal, the second terminal, and the third terminal, and network status information between the first terminal, the second terminal, and the third terminal. can be obtained.

The first terminal 301 uses a video transmission prediction model based on the quality information of the entire video, the status information of each terminal, and the network status information, the first transmission rate of the video transmission from the first terminal to the second terminal; A second transmission rate of video transmission from the first terminal to the third terminal and a third transmission rate of video transmission from the second terminal to the third terminal may be predicted.

The first terminal 301 selects a portion to be transmitted by the second terminal and the remaining portion to be transmitted by the first terminal based on the first transmission rate, the second transmission rate, the third transmission rate and the playback rate of the third terminal. can be divided For example, the first terminal 301 may determine the first division unit based on the first transmission rate, the second transmission rate, the third transmission rate, and the reproduction rate of the third terminal. The first terminal 301 may divide the entire image into a plurality of partial images including the first partial image, the second partial image, the third partial image, and the fourth partial image according to the first division unit. The first terminal 301 may allocate the first partial image and the third partial image to the first terminal, and allocate the second partial image and the fourth partial image to the second terminal.

Here, the first partial image, the second partial image, the third partial image, and the fourth partial image are examples, and the number of divided partial images is not limited. Each terminal participating in parallel transmission may be assigned a partial video according to the order of intersection. The segmentation of the image may be performed again according to the quality of the image included in the received image transmission request signal. Hereinafter, the first division unit and the second division unit may refer to division units applied to each division.

The first terminal 301 may compare the first transmission rate and the third transmission rate. When the first transmission rate is greater than the third transmission rate, the first terminal 301 may compare the third transmission rate and the reproduction rate. When the difference between the third transmission rate and the reproduction rate is equal to or greater than the first threshold, the first terminal 301 may set the first division unit as the first division length. When the difference between the third transmission rate and the reproduction rate is less than the first threshold value, the first terminal 301 may set the first division unit as the second division length. Here, the first divided length may be smaller than the second divided length. The playback speed may mean an image processing speed required to reproduce the received divided image. For example, the playback speed may refer to a speed until a finished image is output through image processing related to a video streaming format, an audio streaming format, a video codec, and metadata of a received divided image.

Conversely, when the third transmission rate is greater than the first transmission rate, the first terminal 301 may compare the first transmission rate and the reproduction rate, and the above process may be repeated. The first terminal 301 may repeat the above process by comparing the reproduction speed with the smallest transmission rate among all the transmission rates of the route composed of a plurality of routes.

According to another embodiment, while video streaming is performed in parallel through a plurality of terminals, the quality of the video may be changed. Referring to FIG. 1 , a first terminal 301 may receive an image quality change request signal from a third terminal 305 .

In response to the image quality change request signal, the first terminal 301 receives the first information from the first terminal using an image transmission prediction model based on the quality information to be changed of the entire image, the state information of each terminal, and the network state information. A fourth transmission rate of video transmission from the second terminal, a fifth transmission rate of video transmission from the first terminal to the third terminal, and a sixth transmission rate of video transmission from the second terminal to the third terminal may be predicted.

The first terminal 301 selects a portion to be transmitted by the second terminal and the remaining portion to be transmitted by the first terminal based on the fourth transmission rate, the fifth transmission rate, the sixth transmission rate, and the playback rate of the third terminal. can be divided For example, the first terminal 301 may determine the second division unit based on the fourth transmission rate, the fifth transmission rate, the sixth transmission rate, and the reproduction rate of the third terminal.

In order to determine the second division unit, the first terminal 301 may compare the first transmission rate and the third transmission rate. When the first transmission rate is greater than the third transmission rate, the first terminal 301 may compare the third transmission rate and the reproduction rate. When the difference between the third transmission rate and the reproduction rate is equal to or greater than the second threshold, the first terminal 301 may set the second division unit to the third division length. When the difference between the third transmission rate and the reproduction rate is less than the second threshold, the first terminal 301 may set the second division unit to the fourth division length. Here, the third division length may be smaller than the fourth division length.

The first terminal 301 divides the untransmitted image among the entire images according to the second division unit into a plurality of partial images including a fifth partial image, a sixth partial image, a seventh partial image, and an eighth partial image. can The first terminal 301 may allocate the fifth partial image and the seventh partial image to the first terminal, and allocate the sixth partial image and the eighth partial image to the second terminal.

According to another embodiment, after performing device virtualization, the image transmission system may perform parallel image transmission using the virtualization result. The video streaming system may determine whether to perform virtualization based on the type of video requested and the resource of each terminal. For example, the first terminal 301 may determine whether an image transmission request signal received from the second terminal 303 or the third terminal 305 satisfies a virtualization condition. The video streaming system may compare the resource required for the service with the resource of each terminal based on the type of video. The video streaming system may determine whether to perform virtualization based on the comparison result. For example, the virtual resource predicted through virtualization is higher than the required resource and the required resource is higher than that of the terminal, or x% of the required resource is higher than that of the terminal, or image transmission through simulation of the result of virtualization of the terminal When the time is significantly reduced, it may be determined that virtualization is performed.

For example, before video transmission to the second terminal 303 and the third terminal 305 is performed, or when video transmission to the second terminal 303 is in progress, the first terminal 301 requests It can be determined whether the service to be used satisfies the virtualization condition. The first terminal 301 may determine whether the virtualization condition is satisfied each time based on the type of image requested from each terminal. The first terminal 301 may determine whether the image transmission time is significantly reduced through the simulation of the virtualization performance result of the terminal.

According to an embodiment, when the virtualization condition is satisfied, the video streaming system may execute device virtualization using the first terminal, the second terminal, and the third terminal. Device virtualization may mean providing a system that more efficiently processes a given task by reconfiguring physically existing resources into virtual modules. The video streaming system may virtualize a plurality of terminals of a paired group. The video streaming system may generate a virtualization layer including a virtualization memory and a virtualization processor by using the resources of a plurality of terminals of a paired group.

The video streaming system may create a network layer by virtualizing network resources based on network state information. Network resources may include network nodes, gateways, traffic analysis tools, application support tools, and security tools. The network layer may provide a logical connection between the virtualization layer and the infrastructure layer. The network layer can efficiently configure network resources by considering the resources of each element of the actual hardware constituting the infrastructure layer and the communication state between each element.

The video streaming system may set one of a plurality of terminals as a control terminal for performing device virtualization. The control terminal may refer to a physical object that performs device virtualization using a plurality of terminals. For example, the video streaming system may set a terminal having the best resource among a plurality of terminals as the control terminal. The control terminal may be a subject performing device virtualization.

For example, when it is determined that the data processing efficiency is higher than the third threshold value, the first terminal 301 may perform device virtualization. The video streaming system may set the first terminal 301 having the best performance among the plurality of terminals as the control terminal. The first terminal 301 may set itself as the control terminal, or may set a terminal with better performance as the control terminal. That is, other terminals other than the terminal storing the entire image may be set as the control terminal, and device virtualization may proceed.

The first terminal 301 may obtain the first resource information of the first terminal 301 , the second resource information of the second terminal 303 , and the third resource information of the third terminal 305 . The first terminal 301 may acquire communication state information between the first terminal 301 , the second terminal 303 , and the third terminal 305 .

The first terminal 301 is based on the terminal status information, the first resource information, the second resource information, and the third resource information of the first terminal 301, the second terminal 303, and the third terminal 305, respectively. You can build an infrastructure layer. The infrastructure layer may refer to a layer including physical resources for device virtualization.

The first terminal 301 may build a network layer based on communication state information between the first terminal 301 , the second terminal 303 , and the third terminal 305 . The network layer provides a logical connection between the virtualization layer and the infrastructure layer, and may physically provide the ability to exchange data.

The first terminal 301 may build a virtualization layer. Here, the virtualization layer may include one or more virtualization processors and one or more virtualization memories. The virtualization layer may refer to a resource including a virtual module built through device virtualization.

The video streaming system may transmit video in parallel by using the result of device virtualization. The first terminal 301 may divide the entire image based on the result of device virtualization. The first terminal 301 may transmit the divided partial image to each of the second terminal 303 and the third terminal 305 . The second terminal 303 may transmit the partial image allocated to it to the third terminal 305 .

For example, the virtualization processor is based on the current buffer status of each terminal, the total buffer size, the available buffer size, the maximum allowable buffer latency information, and information related to the buffering capability of the buffer, such as overflow or underflow of the buffer. In this way, the entire video can be segmented and assigned to suit the transmission speed and processing capability of each terminal. The virtualization processor uses virtualized network nodes, gateways, traffic analysis tools, application support tools, security tools, etc. to access the first terminal, the second terminal, and the third terminal of the infrastructure layer, and allocates a segmented image to each terminal, , You can set a terminal to transmit the split image and a terminal to receive it.

4 is a flowchart illustrating an operation of an image transmission method according to an embodiment.

According to an embodiment, in operation 401, the first terminal may set the first terminal as the first host and set the second terminal as the first guest in response to the video transmission request signal received from the second terminal. there is.

According to an embodiment, in operation 403 , the first terminal and the second terminal may exchange pairing information and terminal state information between the first terminal and the second terminal and perform pairing.

According to an embodiment, in operation 405 , the first terminal may set information for transmitting the entire image in response to the image transmission request signal.

According to an embodiment, in operation 407 , the first terminal may terminate pairing between the first host and the first guest before transmitting the entire video.

According to an embodiment, in operation 409, the first terminal sets the first terminal as the second host in response to the video transmission request signal received from the third terminal before terminating the transmission of the entire video, and 3 The terminal can be set as a second guest.

According to an embodiment, in operation 411 , the first terminal and the third terminal may exchange pairing information and terminal state information between the first terminal and the third terminal and perform pairing.

According to an embodiment, in operation 413 , the third terminal may receive a portion of the entire image from the second terminal and receive the remainder of the portion of the entire image from the first terminal.

5 is a diagram illustrating a result of virtualization by a video streaming system according to an embodiment.

According to an embodiment, the data processing system may perform device virtualization using an existing physical resource. Here, the existing physical resources may include the first terminal 301 , the second terminal 303 , and the third terminal 305 . The data processing system can build a system that processes a given task more efficiently by reconfiguring physically existing resources into virtual modules. The data processing system may virtualize a plurality of terminals of a paired group.

The data processing system may generate the virtualization layer 510 including the virtualization memory 511 and the virtualization processor 513 by using the resources of the plurality of terminals of the paired group.

The data processing system may generate the network layer 520 by virtualizing network resources based on the network state information. The network status information is status information on an established communication channel between terminals, and may include, for example, a bandwidth, a transmission delay difference between packets, a packet loss rate, a retransmission count for packet loss, and the like. Here, the bandwidth may be measured using a packet received by the terminal.

The network resource may include a network node 521 , a gateway 523 , a traffic analysis tool 525 , an application support tool 527 , and a security tool 529 . The network layer 520 may provide a logical connection between the virtualization layer 510 and the infrastructure layer 530 . The network layer 520 may efficiently configure network resources in consideration of the resources of each element of the actual hardware constituting the infrastructure layer 530 and the communication state between each element.

The data processing system is an infrastructure layer ( 530) can be built. The infrastructure layer 530 may refer to a layer including physical resources for device virtualization.

6 is a flowchart illustrating an exemplary operation of an image transmission method according to an embodiment.

Referring to FIG. 6 , the image transmission system may include a first terminal 301 , a second terminal 303 , and a third terminal 305 .

In operation 601 , the second terminal 303 may transmit a service request signal to the first terminal 301 . Here, the service request signal may include a video streaming request signal.

In operation 602 , the first terminal 301 and the second terminal 303 may exchange pairing information and service execution information. The pairing information may include information for mutual recognition and connection of two terminals on which pairing is performed. For example, ID information (identification information) and IP address information (connection information) of the two terminals may be included in the pairing information. The information for service execution may include information on the type of image transmission being executed. For example, service type information is set according to the type of image, image quality, and transmission method, and the service counterpart terminal waits for service according to the set service type information.

In operation 603 , the first terminal 301 and the second terminal 303 may end pairing regardless of whether the service is terminated. The first terminal 301 and the second terminal 303 may end pairing before starting video transmission or end pairing during video transmission.

In operation 604 , the third terminal 305 may transmit a service request signal to the first terminal 301 . Here, the service request signal may include a video streaming request signal. The image requested by the second terminal 303 and the image requested by the third terminal 305 may be the same as that stored in the first terminal 301 .

In operation 605 , the first terminal 301 and the third terminal 305 may exchange pairing information and service execution information.

In operation 606 , the first terminal 301 and the third terminal 305 may end pairing regardless of whether the service is terminated. The first terminal 301 and the third terminal 305 may end pairing before starting video transmission or end pairing during video transmission.

In operation 607, the first terminal 301 may obtain network status information between each terminal. The first terminal 301 may obtain status information of each terminal.

In operation 608, the first terminal 301 may estimate a transmission rate between each terminal. The first terminal 301 may predict the transmission speed between each terminal by using the image transmission prediction model.

The first terminal 301 may determine whether to divide the image and transmit it in parallel. When it is decided to transmit the image in parallel, the first terminal 301 performs pairing between the second terminal 303 and the third terminal 305 to the second terminal 303 and the third terminal 305, respectively. A request signal can be transmitted. The second terminal 303 and the third terminal 305 may perform pairing.

In operation 609, the first terminal 301 may divide the entire image. The first terminal 301 may allocate the second terminal 303 and the images to be transmitted by the first terminal 301 to the first terminal 301 and the second terminal 303 based on the predicted transmission speed. For example, the first terminal 301 may divide the entire image into a first partial image, a second partial image, a third partial image, and a fourth partial image.

In operation 610 , the first terminal 301 may transmit the first partial image, the second partial image, the third partial image, and the fourth partial image to the second terminal 303 . The second terminal 303 is a terminal that first receives an image from the first terminal 301 , and may continuously receive images from the first terminal 301 .

In operation 611 , the second terminal 303 may transmit the second partial image and the fourth partial image to the third terminal 305 . In operation 612 , the first terminal 301 may transmit the first partial image and the third partial image to the third terminal 305 . The first terminal 301 and the second terminal 303 may transmit partial images to the third terminal 305 in parallel, thereby increasing the data throughput of the entire system.

In operation 613 , the third terminal 305 may transmit an image quality change request signal requesting to change the image quality to the first terminal 301 . The first terminal 301 may transmit to the second terminal 303 a signal for confirming whether to agree to the image quality change. When the first terminal 301 receives a signal agreeing to change the image quality from the second terminal 303 , the first terminal 301 may change the transmitted image to a quality corresponding to the request.

In operation 614 , the first terminal 301 may acquire network status information between each terminal. The first terminal 301 may obtain status information of each terminal. For example, the state of the terminal and the network state may be changed before and after the quality change. The first terminal 301 may newly acquire network state information and terminal state information in order to predict the transmission rate using the most recent state.

In operation 615 , the first terminal 301 may estimate a transmission rate between each terminal.

In operation 616 , the first terminal 301 may divide a non-transmitted portion of the entire image. For example, the first terminal 301 may divide an untransmitted portion of the entire image into a fifth partial image, a sixth partial image, a seventh partial image, and an eighth partial image.

In operation 617 , the first terminal 301 may transmit the fifth partial image, the sixth partial image, the seventh partial image, and the eighth partial image to the second terminal 303 .

In operation 618 , the first terminal 301 may directly transmit the fifth partial image and the seventh partial image to the third terminal 305 .

In operation 619 , the second terminal 303 may transmit the sixth partial image and the eighth partial image to the third terminal 305 .

7 is a diagram illustrating a configuration of an image transmission apparatus according to an exemplary embodiment.

According to an embodiment, each terminal constituting the video streaming system (eg, the first terminal 301, the second terminal 303, and the third terminal 305 of FIG. 3) is a processor 710, a memory, respectively. 720 and a communication unit 730 .

According to an embodiment, the processor 710 of the first terminal may set the first terminal as the first host and the second terminal as the first guest in response to a video transmission request signal received from the second terminal. .

According to an embodiment, the processor 710 of the first terminal and the processor of the second terminal may exchange pairing information and terminal state information between the first terminal and the second terminal and perform pairing.

According to an embodiment, in operation 405 , the processor 710 of the first terminal may set information for transmitting the entire image in response to the image transmission request signal.

According to an embodiment, in operation 407 , the processor 710 of the first terminal may terminate pairing between the first host and the first guest before transmitting the entire image.

According to an embodiment, in operation 409, the processor 710 of the first terminal sends the first terminal to the second host in response to the video transmission request signal received from the third terminal before terminating the transmission of the entire video. , and the third terminal may be set as the second guest.

According to an embodiment, in operation 411 , the processor 710 of the first terminal and the third terminal exchange pairing information and terminal state information between the first terminal and the third terminal and perform pairing.

According to an embodiment, in operation 413 , the third terminal may receive a portion of the entire image from the second terminal and receive the remainder of the portion of the entire image from the first terminal.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (3)

setting, by the first terminal, the first terminal as the first host and the second terminal as the first guest in response to the video transmission request signal received from the second terminal;
exchanging pairing information and terminal status information between the first terminal and the second terminal and performing pairing by the first terminal and the second terminal;
setting, by the first terminal, information for transmitting the entire video in response to the video transmission request signal;
terminating, by the first terminal, pairing between the first host and the first guest before transmitting the entire video;
Before terminating the transmission of the entire video by the first terminal, the first terminal is set as the second host in response to the video transmission request signal received from the third terminal, and the third terminal is set as the second guest action to set to;
exchanging pairing information and terminal status information between the first terminal and the third terminal and performing pairing by the first terminal and the third terminal; and
receiving, by the third terminal, a portion of the entire video from the second terminal, and receiving the remainder of the portion of the entire video from the first terminal
Including, video streaming method. The method of claim 1,
The operation of receiving a portion of the entire image from the second terminal and receiving the remainder of the portion of the entire image from the first terminal comprises:
dividing, by the first terminal, a part to be transmitted by the second terminal and a remaining part to be transmitted by the first terminal using an image transmission prediction model; and
receiving, by the third terminal, the divided portion from the second terminal based on the division result, and receiving the divided portion from the first terminal
Including, video streaming method. 3. The method of claim 2,
The image transmission prediction model includes an input layer, one or more hidden layers and an output layer,
Training data consisting of a pair of a plurality of training images, terminal state information, network state information, and correct answer prediction information is input to the input layer of the image transmission prediction model and passes through the one or more hidden layers and output layers to output an output vector and the output vector is input to a loss function layer connected to the output layer, and the loss function layer outputs a loss value using a loss function that compares the output vector with a correct vector for each training data, and The parameters of the image transmission prediction model are learned in a direction in which the loss value becomes smaller,
[Equation]

The loss function follows the above equation,
In the above formula, N is the number of the plurality of learning data, n is a natural number identifying the learning data, k is a natural number identifying the value of the n-th learning data, nk is the k-th value of the n-th learning data, t means the correct answer data, y means the output vector, E means the loss value,
How to stream video.

Images