KR102559721B1 - Control method of electronic apparatus for selectively restore images according to field of view of user - Google Patents

Abstract

A control method of an electronic device is disclosed. The control method includes: receiving streaming data including first image data corresponding to a preset visual range; identifying a visual field region of the user by tracking a direction of the user's eyeball; selecting target image data that matches at least a part of the identified visual field region among the first image data; inputting the target image data to an artificial intelligence model trained to perform high-resolution reconstruction to obtain reconstructed image data; substituting the target image data of the first image data with the reconstructed image data to obtain second image data; and outputting second image data. It includes steps to

Description

Method for controlling an electronic device that selectively restores an image according to a user's viewing area {CONTROL METHOD OF ELECTRONIC APPARATUS FOR SELECTIVELY RESTORE IMAGES ACCORDING TO FIELD OF VIEW OF USER}

The present disclosure relates to a control method of an electronic device that provides images, and more particularly, to a control method of an electronic device that provides high-resolution reconstructed images matching a user's viewing area.

Since the COVID-19 crisis, the realistic content market, such as VR (Virtual Reality) and AR (Augmented Reality), has begun to gain momentum, and the market size is increasing.

However, the mobile traffic cost of AR/VR content is burdening both operators and users, and as a result, the service and information gap in the 5G era is widening.

In particular, it is difficult to play ultra-high-definition realistic content in real time on commonly used consumer devices (PCs, smartphones), and due to the nature of VR content, pixel processing is required to process six times as many pixels as compared to general video. As a result, there are limitations in terms of edge device specifications (ex. expensive GPU equipment required) as well as communication environments.

Registered Patent Publication No. 10-1965746

The present disclosure provides a control method of an electronic device that receives low-resolution image data and provides a real-time image, and outputs an image by performing high-resolution reconstruction using an artificial intelligence model suitable for a user's field of view sensed in real time.

The objects of the present disclosure are not limited to the above-mentioned objects, and other objects and advantages of the present disclosure not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present disclosure. Further, it will be readily apparent that the objects and advantages of the present disclosure may be realized by means of the instrumentalities and combinations indicated in the claims.

A control method of an electronic device according to an embodiment of the present disclosure includes receiving streaming data including first image data corresponding to a preset viewing range, tracking a direction of the user's eyeballs and identifying a viewing area of the user within the preset viewing range, selecting target image data that matches at least a part of the identified viewing range from among the first image data, inputting the target image data to an artificial intelligence model trained to perform high-resolution reconstruction, and obtaining reconstructed image data, and the target image of the first image data. and replacing data with the reconstructed image data to obtain second image data, and outputting the second image data.

The receiving of the streaming data may include receiving a plurality of partial images in which the first image data is divided into regions, and receiving data for a plurality of neural network models trained to perform high-resolution reconstruction on each of the plurality of partial images. In the selecting of the target image data, at least one partial image included in the viewing area of the user may be selected from among the plurality of partial images. The acquiring of the restored image data may include inputting the selected at least one partial image to at least one neural network model that matches each of the at least one selected partial image.

The control method of the electronic device may include receiving, by the electronic device, streaming data including a plurality of image frames in which Virtual Reality (VR) content is divided by time from the server, and receiving, by the electronic device, data for a plurality of first neural network models trained to perform high-resolution reconstruction on each of the plurality of image frames from the server. Each of the plurality of first neural network models may include a plurality of second neural network models trained to perform high-resolution reconstruction on each of a plurality of partial images obtained by dividing each image frame into regions. The control method of the electronic device may further include selecting at least one partial image included in at least one image frame based on a time interval within the VR content reproduced by the electronic device and the user's viewing area sensed during the time interval, and performing high-resolution reconstruction of the selected at least one partial image through at least one second neural network model that matches each of the selected at least one partial image.

The control method of the electronic device may further include dividing the user's viewing area into a foveal area with a certain viewing angle including a central point toward which the user's gaze is directed, a blend area surrounding the foveal area and corresponding to a viewing angle greater than the viewing angle of the foveal area, and a peripheral area excluding the foveal area and the blend area. In this case, in the obtaining of the restored image data, first reconstructed image data may be obtained by inputting the first target image data matching the foveal region to the artificial intelligence model, and interpolation may be performed on the second target image data matching the blend region to obtain second restored image data.

Here, the control method of the electronic device may further include changing the viewing angle range of the foveal area based on at least one of a speed at which the electronic device receives streaming data and a speed at which the electronic device performs high-resolution reconstruction through the artificial intelligence model.

The control method of the electronic device may further include identifying a speed at which the user's viewing area changes, and changing a viewing angle range of the foveal area based on the identified speed.

According to an embodiment of the present disclosure, a control method of a system including a server and an electronic device includes transmitting, by the server, streaming data including first image data corresponding to a 360-degree field of view to the electronic device; identifying a user's field of view area by tracking the direction of the user's eyeballs by the electronic device; selecting, by the electronic device, target image data matching the identified field of view area from among the first image data; , acquiring reconstructed image data, replacing, by the electronic device, the target image data of the first image data with the reconstructed image data to obtain second image data, and outputting the second image data.

The method for controlling an electronic device according to the present disclosure receives and provides low-resolution image data in real time, and partially upscales only a certain line of sight in a user's viewing area, thereby reducing the load of the communication environment and the device environment. As a result, the method for controlling an electronic device according to the present disclosure enables a streaming service for high-definition video such as VR/AR.

1 is a block diagram for explaining a schematic operation of a server and an electronic device for providing a streaming service of image data according to an embodiment of the present disclosure;
2A is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure;
2B is a block diagram for explaining the configuration of a server according to an embodiment of the present disclosure;
3 is a flowchart for explaining an operation of an electronic device according to an embodiment of the present disclosure;
4 is a diagram for explaining an operation of an electronic device dividing a user's viewing area according to an embodiment of the present disclosure;
5 is a diagram for explaining an operation of selectively utilizing a plurality of artificial intelligence models trained for each region of image data by an electronic device according to an embodiment of the present disclosure;
6 is a diagram for explaining an operation of selectively utilizing a plurality of artificial intelligence models trained for each time interval of image data by an electronic device according to an embodiment of the present disclosure; and
7 is a block diagram for explaining the configuration of an electronic device according to various embodiments of the present disclosure.

Prior to a detailed description of the present disclosure, the method of describing the present specification and drawings will be described.

First, terms used in the present specification and claims are general terms in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention of a technician working in the art, legal or technical interpretation, and the emergence of new technologies. In addition, some terms are arbitrarily selected by the applicant. These terms may be interpreted as the meanings defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the art.

In addition, the same reference numerals or numerals in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of description and understanding, the same reference numerals or symbols are used in different embodiments. That is, even if all components having the same reference numerals are shown in a plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in the present specification and claims, terms including ordinal numbers such as “first” and “second” may be used to distinguish between elements. These ordinal numbers are used to distinguish the same or similar components from each other, and the meaning of the term should not be construed as being limited due to the use of these ordinal numbers. For example, the order of use or arrangement of elements associated with such ordinal numbers should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, but it should be understood that the presence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof is not excluded in advance.

In the embodiments of the present disclosure, terms such as “module,” “unit,” and “part” are terms used to refer to components that perform at least one function or operation, and these components may be implemented as hardware or software, or may be implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. may be integrated into at least one module or chip and implemented by at least one processor, except when each of them needs to be implemented with separate specific hardware.

Also, in an embodiment of the present disclosure, when a part is said to be connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that it may further include other components without excluding other components unless otherwise stated.

1 is a block diagram illustrating schematic operations of a server and an electronic device for providing a streaming service of image data according to an embodiment of the present disclosure.

The electronic device 100 may provide various images based on communication with the server 200 . The electronic device 100 may be implemented as a smartphone, a tablet PC, a TV, a VR device, an AR device, a wearable device (ex. glasses, a watch, a head mounted device (HMD), etc.), a console device, a set-top box, and other control devices.

The server 200 may provide images through the electronic device 100 by supporting a streaming service. The server 200 may be implemented as a system including one or more computers. The server 200 may transmit image data constituting various contents such as VR contents, AR contents, 2D contents, and 3D contents to the electronic device 100 in the form of streaming data. The server 200 may provide streaming services for content in various fields such as broadcasting, gaming, tourism, entertainment, IoT monitoring, military, medical, CCTV, and metaverse.

In this case, real-time video may be provided by the electronic device 100 outputting image data included in the streaming data. For example, when the electronic device 100 is implemented as a VR device (ex. HMD) including a display, the electronic device 100 may display a VR image of a certain viewing range (ex. 360 degrees) based on image data received from the server 200. Alternatively, when the electronic device 100 is implemented as a control device that controls the VR device, the electronic device 100 transmits image data received from the server 200 to the VR device and controls to provide VR images through the VR device.

2A is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2A , the electronic device 100 may include a memory 110, a processor 120, and a communication unit 130. Also, the electronic device 100 may further include at least one of the display 140 and the sensor unit 150.

The memory 110 is a configuration for storing an Operating System (OS) for controlling overall operations of components of the electronic device 100 and at least one instruction or data related to components of the electronic device 100 .

The memory 110 may include non-volatile memory such as ROM and flash memory, and may include volatile memory such as DRAM. Also, the memory 110 may include a hard disk, a solid state drive (SSD), and the like.

According to an embodiment, the memory 110 may include at least one artificial intelligence model trained to perform high-resolution reconstruction on at least one image.

As an example, this artificial intelligence model may correspond to a neural network model that is trained as weights between nodes belonging to different layers are updated, and may correspond to a convolutional neural network (CNN) model of a deep learning method including at least three or more convolutional layers, but is not limited thereto.

When at least one low-resolution image (eg, each pixel value) is input, the artificial intelligence model may output a high-resolution image composed of more pixels than the low-resolution image.

For example, the artificial intelligence model may include, but is not limited to, a feature extractor for extracting features from an image, a restorer for generating (upsampling) pixels and determining values of the generated pixels, and the like. Here, each of the feature extraction unit and restoration unit may include one or more layers.

The processor 120 is a component for controlling the electronic device 100 as a whole.

Specifically, the processor 120 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 110 while being connected to the memory 110 .

The processor 120 may include a general-purpose processor such as a CPU, an AP, or a digital signal processor (DSP), a graphics-only processor such as a GPU or a vision processing unit (VPU), or an artificial intelligence-only processor such as an NPU. An artificial intelligence-only processor may be designed as a hardware structure specialized for training or use of a specific artificial intelligence model.

The processor 120 may control the viewing area extraction module 121 , the high-resolution reconstruction module 122 , the display module 123 , and the like. These modules correspond to functionally defined modules, and each module may be implemented through software and/or hardware.

The viewing area extraction module 121 is a component for identifying the viewing area of the user by tracking the direction of the user's eyeballs. The viewing area extraction module 121 may extract the viewing area of the user based on the direction the user looks within a preset viewing range (eg, 360 degrees).

For example, when the electronic device 100 is a VR device in the form of an HMD, the viewing area extraction module 121 may track the pupil by photographing the user's eyeballs through at least one image sensor and infrared rays. In this case, the viewing area extraction module 121 may obtain the viewing area of the user by identifying a direction in which the user is looking based on the location of the pupil. As a specific example, the viewing area extraction module 121 may obtain a range of a preset viewing angle (eg, 30 degrees, 45 degrees, 60 degrees, etc.) centered on the user's viewing direction as the user's viewing area, but is not limited thereto.

As another example, the viewing area extraction module 121 may identify the posture (direction) of the HMD worn by the user based on at least one of a gravity sensor and a gyro sensor, and may also identify a direction the HMD is facing as a direction the user is looking at.

When the electronic device 100 is an HMD-type VR device, the viewing area extraction module 121 may extract the viewing area through at least one sensor (eg, an image sensor) provided in the sensor unit 150 of the electronic device 100. On the other hand, when the electronic device 100 is a control device that controls the VR device, the viewing area extraction module 121 communicates with the sensor unit (eg, image sensor) of the VR device through the communication unit 130 to extract the viewing area viewed by the user.

The high resolution restoration module 122 is a module for reconstructing at least a part of the image data received from the server 200 .

Specifically, for smooth streaming, the electronic device 100 may receive streaming data including low-quality image data from the server 200 .

In this case, the high-resolution reconstruction module 122 may select target image data that matches at least a part of the user's viewing area from among the low-quality image data, and perform high-resolution reconstruction on the target image data.

The high-resolution reconstruction module 122 may obtain high-resolution reconstruction image data for at least a part of the viewing area by performing high-resolution reconstruction through an artificial intelligence model stored in the memory 110 . As a result, final image data including high-resolution image data for a part of the viewing area and low-resolution image data for the remaining viewing area may be obtained.

The display module 123 is a component for visually outputting image data acquired according to streaming. The display module 123 may output an image through the display 140 of the electronic device 100 or support image output by controlling at least one display device connected to the electronic device 100 .

As an example, the display module 123 may visually output final image data obtained by performing high-resolution reconstruction on at least a part of the viewing area.

The communication unit 130 is a component for performing communication with at least one external device using various wired/wireless communication methods, and may include circuits, modules, chips, etc. matched to various communication methods.

The communication unit 130 may be connected to external devices through various networks.

The network may be a Personal Area Network (PAN), a Local Area Network (LAN), a Wide Area Network (WAN), or the like, depending on the area or scale, and may be an intranet, an extranet, or the Internet, etc., depending on the openness of the network.

The communication unit 130 is a long-term evolution (LTE), LTE Advance (LTE-A), 5th generation (5G) mobile communication, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), global system for mobile communications (GSM), time division multiple access (DMA), Wi-Fi (WiFi), WiFi Direct, Bluetooth, near field communication (NFC) It can be connected with external devices through various wireless communication methods such as , Zigbee.

In addition, the communication unit 130 may be connected to external devices through a wired communication method such as Ethernet, optical network, Universal Serial Bus (USB), or Thunderbolt.

In addition, the communication unit 130 may communicate with an external device through various commonly used communication methods.

The electronic device 100 may receive real-time streaming data from the server 200 through the communication unit 130 . Also, the electronic device 100 may receive data of at least one artificial intelligence model from the server 200 through the communication unit 130 .

Here, the data of the artificial intelligence model may include data on weights constituting the artificial intelligence model, but is not limited thereto.

The display 140 is a configuration for visually displaying various images, and may be implemented with a liquid crystal display (LCD), light emitting diodes (LED), organic light emitting diodes (OLED), transparent OLED (TOLED), or micro LED. In addition, the display 140 may be implemented as a curved display, a flexible display, a foldable display, or the like, in addition to a flat display. The display 140 may be driven/controlled by the above-described display module 123 to output various images.

Specifically, the display 140 may be implemented as a curved display for providing an image to a wide area around the user's eyes. In this case, the display 140 may include a separate display panel for providing images to each of the user's left and right eyes.

In addition, when the electronic device 100 corresponds to an AR device such as AR glasses, the display 140 may be composed of various optical systems for simultaneously showing a real surrounding environment and a virtually output image. In this case, at least a part of the display 140 may be implemented as a transparent display.

The sensor unit 150 may include various sensors for obtaining information related to a user using the electronic device 100 or surroundings. For example, the sensor unit 150 may include at least one image sensor (eg, camera) for photographing the user's eyeball or surroundings, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a gravity sensor, etc. In addition, the sensor unit 150 may include at least one contact sensor or proximity sensor for detecting whether or not the electronic device 100 implemented as a VR/AR device is worn.

Also, the sensor unit 150 may include a motion sensor for detecting motion of at least a part of the user's body. For example, when the electronic device 100 implemented as a VR device or an AR device includes a controller that can be operated by a user's hand, various sensors for detecting the movement speed, movement direction, posture, etc. of the controller may be provided.

On the other hand, Figure 2b is a block diagram for explaining the configuration of the server according to an embodiment of the present disclosure.

Referring to FIG. 2B , the server 200 may include a memory 210, a processor 220, a communication unit 230, and the like.

The memory 210 is a component for storing an Operating System (OS) for controlling overall operations of components of the electronic device 100 and at least one instruction or data related to components of the electronic device 100 .

The memory 210 may include non-volatile memory such as ROM and flash memory, and may include volatile memory composed of DRAM and the like. Also, the memory 210 may include a hard disk, a solid state drive (SSD), and the like.

The processor 220 is a component for controlling the electronic device 100 as a whole.

Specifically, the processor 220 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 210 while being connected to the memory 210 .

The processor 220 may include a general-purpose processor such as a CPU, an AP, or a digital signal processor (DSP), a graphics-only processor such as a GPU or a vision processing unit (VPU), or an artificial intelligence-only processor such as an NPU. An artificial intelligence-only processor may be designed as a hardware structure specialized for training or use of a specific artificial intelligence model.

Referring to FIG. 2B , the processor 220 may control an image processing module 221, a streaming module 222, an artificial intelligence training module 223, and the like. These modules correspond to functionally defined modules, and each module may be implemented through software and/or hardware.

The image processing module 221 is a module for encoding or decoding image data. Specifically, the image processing module 221 may acquire a low-resolution image by compressing a high-resolution image, or acquire a high-resolution image by restoring a low-resolution image.

As an example, the image processing module 221 compresses a high-resolution image to obtain a low-resolution image, thereby providing the image in the form of data that is easy to stream.

As a specific example, the image processing module 221 divides a high-resolution VR image corresponding to a 360-degree field of view into a plurality of divided images, converts each of the plurality of divided images into low-resolution images, and provides them in a form that is easy to stream.

The streaming module 222 is a component for transmitting real-time streaming data. The streaming module 222 may be connected to the electronic device 100 through the communication unit 230 and transmit streaming data including image data.

Specifically, the streaming module 222 may transmit streaming data including a plurality of low-resolution divided images to the electronic device 100 .

The artificial intelligence training module 223 is a module for training an artificial intelligence model for performing high-resolution reconstruction.

Here, the artificial intelligence model may correspond to a CNN model for outputting a high-resolution image when a low-resolution image is input. For example, a high-resolution target image defined as GT (ground truth) and a corresponding low-resolution version of an image (LR; Low Resolution Image) are input as a pair, and the low-resolution image (LR) is learned to restore the GT. It may correspond to the SR (Super Resolution) model.

For example, the artificial intelligence training module 223 may utilize a high-resolution version and a low-resolution version of VR images corresponding to a 360-degree viewing range, respectively.

At this time, the artificial intelligence training module 223 may divide the low-resolution VR image (: 360-degree field of view) according to the patch size suitable for the input of the artificial intelligence model. Similarly, the artificial intelligence training module 223 may divide a high-resolution VR image (360-degree field of view) according to a patch size suitable for the output of the artificial intelligence model. At this time, the artificial intelligence training module 223 may train the artificial intelligence model by utilizing each of the divided low-resolution-high-resolution image pairs. In this case, the artificial intelligence training module 223 may train a separate artificial intelligence model for each spatially separated segmented image.

In addition, even in the case of an image of the same field of view, in the case of a VR video, since the image may change over time, the artificial intelligence training module 223 distinguishes images over time and restores each image. Each AI model can be trained.

On the other hand, the artificial intelligence training module 223 can classify each image according to the std value (standard deviation) of the training images and use it for training of each artificial intelligence model matched to the std value.

The std value is a concept containing an approximate structure of each image, and the higher the std value, the larger the standard deviation between pixel values, so it can be interpreted as the greater the complexity of the image. Specifically, the artificial intelligence training module 223 may classify a plurality of training images into several groups according to the range of std values. For example, images having std values from 0 to 50 may be classified into a total of five groups, such as 0 to 10, 10 to 20, 20 to 30, 30 to 40, and 40 to 50. At this time, it can be used for training of separate artificial intelligence models for each group.

Specifically, since the std value is a concept that may vary depending on the field of the image (ex. x-ray image for bio, VR image for game, news image, etc.), the artificial intelligence training module 223 can use images for each group to train each of the artificial intelligence models matched to different fields. For example, the training images of the first group can be used to train an artificial intelligence model for high-resolution reconstruction of x-ray images, and the training images of the second group can be used to train an artificial intelligence model for high-resolution reconstruction of VR images.

In addition, the streaming module 222 may transmit data for at least one artificial intelligence model trained to perform high-resolution reconstruction to the electronic device 100 . Here, data of the artificial intelligence model may include weights between nodes of the artificial intelligence model.

The communication unit 230 is a component for communicating with at least one external device using various wired/wireless communication methods, and may include circuits, modules, chips, etc. matched to various communication methods. The communication unit 230 may be connected to external devices through various networks and may provide a streaming service through the electronic device 100 .

3 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure. FIG. 3 assumes a situation in which the electronic device 100 receives a VR video or AR video streaming service from the server 200 .

Referring to FIG. 3 , the electronic device 100 may receive streaming data including first image data corresponding to a preset viewing range (S310).

For example, the electronic device 100 may receive first image data corresponding to a VR image corresponding to a 360-degree field of view in real time. In this case, the first image data may correspond to image data of low resolution in a form in which original data (high resolution) is compressed.

In this case, the electronic device 100 may receive a plurality of partial images in which the first image data is divided into regions. Each region may correspond to a viewing region in which viewing angle ranges are distinguished from each other based on a central axis of 360 degrees.

In this case, the electronic device 100 may receive data on a plurality of neural network models trained to perform high-resolution reconstruction on each of a plurality of partial images from the server 200 .

In this way, when data for a plurality of neural network models for high-resolution reconstruction of each of the divided images is received, the capacity is relatively small and the transmission speed is fast. For example, in a state in which the layer structure of the neural network model is pre-stored in the electronic device 100, neural network model data (eg, weights between nodes) for performing high-resolution reconstruction on each divided image may be received.

In addition, the electronic device 100 may track the direction of the user's eyeballs and identify the user's viewing area within a preset viewing range (S320). In this case, the electronic device 100 may identify the user's viewing area through the viewing area extraction module 121 .

For example, the electronic device 100 may obtain an image of the user's eyes through an infrared camera, convert the image into a gray level, and detect a pupil area. In addition, the electronic device 100 may detect the gaze direction matching the eyeball direction by tracking the user's eyeball direction based on the center coordinates of the pupil. As a result, a viewing area of a certain viewing angle range centered on the viewing direction can be identified.

As a specific example, the viewing area may be set as an area within various viewing angle ranges such as 30 degrees, 45 degrees, and 60 degrees with the user's gaze direction as the center.

In this case, the electronic device 100 may divide the user's viewing area into a Foveal area, a Blend area, and a Peripheral area.

In relation to this, FIG. 4 is a diagram for explaining an operation of an electronic device according to an embodiment of the present disclosure to classify a user's viewing area.

Referring to FIG. 4, the Foveal area is an area with a certain viewing angle including the central point to which the user's gaze is directed, the Blend area is an area surrounding the Foveal area and corresponding to a viewing angle larger than the Foveal area's viewing angle, and the Peripheral area is the remaining area except for the Foveal area and the Blend area.

For example, the Foveal area may correspond to a viewing angle within 30 degrees, the Blend area from 30 degrees to 45 degrees, and the Peripheral area may correspond to a viewing angle range of 45 degrees or more, but is not limited thereto.

Then, the electronic device 100 may select target image data that matches at least a part of the identified viewing area from among the first image data (S330).

Specifically, the electronic device 100 may identify a foveal area among the user's viewing area, and may select target image data corresponding to the foveal area from among the first image data. That is, one or more divided images included in the foveal area may be selected from among the plurality of divided images.

Then, the electronic device 100 may obtain reconstructed image data by inputting target image data to an artificial intelligence model trained to perform high-resolution reconstruction (S340).

In this case, the electronic device 100 may acquire first reconstructed image data by inputting first target image data that matches the foveal area to the artificial intelligence model. Specifically, the electronic device 100 may respectively input the partial images to at least one neural network model that matches each of the at least one partial image included in the foveal area.

In addition, the electronic device 100 may obtain second target image data by performing interpolation correction on the second target image data that matches the blend region. At this time, interpolation correction may be performed in various conventionally known forms such as bicubic interpolation and cubic interpolation.

Then, the electronic device 100 may obtain second image data by replacing target image data with restored image data in the first image data, and output the second image data (S350). An image corresponding to the second image data may be output through the display 140 of the electronic device 100 or may be output through at least one display device connected to the electronic device 100 .

Here, the reconstructed image data corresponds to image data that has undergone at least one of high-resolution restoration and interpolation correction of the artificial intelligence model.

As a result, only a portion of the user's field of view can be selectively output in high resolution, and a high-quality content experience can be provided while minimizing the load required for streaming.

Meanwhile, FIG. 5 is a diagram for explaining an operation of selectively utilizing a plurality of artificial intelligence models trained for each region of image data by an electronic device according to an embodiment of the present disclosure. Although FIG. 5 does not exemplify an image for a 360-degree field of view, it corresponds to an example for easy explanation because the image has a common feature in area division.

Referring to FIG. 5 , an image 510 may be divided into a plurality of tiles. In this case, a separate artificial intelligence model may be trained for each tile (region). In this case, the server 200 may transmit a plurality of partial images corresponding to each of a plurality of tiles to the electronic device 100 . In addition, the server 200 may transmit data for each of a plurality of artificial intelligence models trained to reconstruct each of a plurality of partial images in high resolution to the electronic device 100 .

Thereafter, the electronic device 100 may selectively perform high-resolution reconstruction on the divided image selected according to the user's viewing area or a user command, and only the matched artificial intelligence model may be utilized.

Meanwhile, as an embodiment, the electronic device 100 may receive a plurality of image frames in which the first image data is divided by time, and may receive data for a plurality of first neural network models trained to perform high-resolution reconstruction on each of the plurality of image frames. That is, in addition to division according to space, division according to time may also be utilized.

In this case, each of the plurality of first neural network models may include a plurality of second neural network models trained to perform high-resolution reconstruction on each of a plurality of partial images obtained by dividing each image frame into regions. That is, in addition to division according to time, division according to space may also be applied.

In this regard, FIG. 6 is a diagram for explaining an operation of selectively utilizing a plurality of artificial intelligence models trained for each time interval of image data by an electronic device according to an embodiment of the present disclosure. 6 assumes a situation in which VR content sequentially including a plurality of image frames (image frames 1, 2, 3, 4, etc.) for a virtual experience of sequentially moving through various areas on a map is provided in real time through the electronic device 100.

At this time, the electronic device 100 may receive a plurality of image frames from the server 200, and may receive data for a first neural network model (models 1, 2, 3, 4, etc.) for performing high-resolution reconstruction on each image frame. In this case, a plurality of divided images in which each image frame is divided into regions may be received, and data for a plurality of second neural network models for high-resolution reconstruction of each of the divided images may be received.

Referring to FIG. 6 , the electronic device 100 may select at least one image frame from among the plurality of image frames described above by identifying a time interval in which VR content is reproduced. In addition, based on the user's field of view sensed in real time, at least one segmented image in the image frame is selected, and high-resolution reconstruction may be performed by inputting the selected segmented image to a matching second neural network model.

Meanwhile, the foveal area, the blend area, etc., in which the user's viewing area is divided, can be flexibly adjusted according to the system environment or communication environment.

As an embodiment, the electronic device 100 may change the viewing angle range of the foveal area based on at least one of a speed at which the electronic device 100 receives streaming data and a speed at which the electronic device 100 performs high-resolution reconstruction through an artificial intelligence model. The change in the viewing angle range means that the radius of the foveal area changes from the user's point of view looking at the image.

The speed at which streaming data is received may mean, for example, the capacity of data received per unit time or the number of image frames. The speed at which high-resolution reconstruction is performed may correspond to the number of pixels reconstructed per unit time, the number of image frames, or the capacity of reconstructed data.

For example, the viewing angle range of the foveal area may decrease as the streaming data reception speed decreases, and the viewing angle range of the foveal area may increase as the streaming data reception speed increases. In addition, the viewing angle range of the foveal region may decrease as the speed of performing the high-resolution reconstruction decreases, and the viewing angle range of the foveal region may decrease as the speed of performing the high-resolution reconstruction decreases.

As a specific example, when the speed at which high-resolution reconstruction is performed becomes less than the first threshold, the viewing angle range of the foveal area may be reduced according to a predetermined inclination until the speed again becomes less than the first threshold. At this time, the viewing angle range corresponding to the outer edge of the blend area is maintained, but as the foveal area decreases, the viewing angle range corresponding to the inner boundary of the blend area narrows, and the blend area may gradually increase. However, if the speed of performing high-resolution reconstruction reaches less than the second threshold, which is smaller than the first threshold, the foveal region disappears and is replaced with the blend region, and only interpolation correction for the entire blend region may be performed in real time.

As another example, in a state where the speed for performing high-resolution reconstruction is greater than or equal to the first threshold, the viewing angle range of the foveal area may gradually increase. At this time, the viewing angle range corresponding to the periphery of the blend area may also increase, or the blend area may gradually narrow as a result of maintaining the viewing angle range corresponding to the periphery of the blend area. In this case, as the foveal area widens, the speed at which high-resolution reconstruction is performed may gradually decrease, and when the speed reaches the first threshold value again, the viewing angle range of the foveal area may be maintained without increasing any more.

Also, as an example, the electronic device 100 may change the viewing angle range of the foveal area based on the speed at which the user's viewing area changes.

The speed at which the user's viewing area changes (hereinafter, referred to as "viewing area change speed") is proportional to the number of pixels that disappear from the user's viewing area and are newly created per unit time while the user's viewing direction is changed. The faster the eyeball movement or head turn of the user is, the faster the viewing area is changed.

Here, the electronic device 100 may set the viewing angle range of the real-time foveal area to be smaller as the user's view change speed is faster, requiring high-resolution reconstruction of the artificial intelligence model. As a result, a situation in which a VR experience delay occurs due to a load limit according to a user's rapid movement can be prevented.

In this case, the electronic device 100 may predict the user's view change speed in real time. For example, the electronic device 100 may obtain history data by monitoring a user's view change rate for a plurality of unit time intervals. In this case, the electronic device 100 may train an artificial intelligence model for prediction based on the history data so that the artificial intelligence model predicts the speed of change of the user's view within a next unit time interval.

Here, the artificial intelligence model for prediction is based on a Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM) for predicting the rate of change of the visual field based on the values of pixels included in the visual field area in each successive unit time interval. It may correspond to a model. Also, the artificial intelligence model for prediction may include CNN-based object recognition layers for recognizing at least one object included in the viewing area. For example, a feature vector or object recognition result related to object recognition may be input to an RNN or LSTM-based model. As a result, the artificial intelligence model for prediction is trained according to the relationship between the characteristics (ex. type, color, size, number of objects, etc.) of objects appearing in the field of view in real time and the rate of change in the field of view, and can predict the rate of change of the field of view for the next situation. For example, when a specific type of object appears, the artificial intelligence model for prediction can learn about a situation where the user's view change speed increases.

Thereafter, the electronic device 100 inputs the speed of change of the user's field of view in each of a plurality of unit time sections continuing from the past to the present into the artificial intelligence model, thereby predicting the speed of changing the user's field of view for the next future unit time section. The predicted view change speed per unit time section may be utilized/applied in real time to the real-time change setting of the above-described view angle range.

For example, the viewing angle range of the foveal area of each unit time interval may be set according to the user's visual field change rate predicted for each unit time interval, but is not limited thereto.

7 is a block diagram for explaining the configuration of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 7 , the electronic device 100 may further include a user input unit 160, an audio output unit 170, and the like in addition to a memory 110, a processor 120, a communication unit 130, a display 140, and a sensor unit 150.

The user input unit 160 is a component for receiving various commands or information from a user. The user input unit 160 may be implemented with at least one button, a touch pad, a touch screen, a microphone, a camera, a sensor, and the like. In addition, the electronic device 100 may be connected to various user input devices (eg, controllers) including at least one keypad, button, motion sensor, acceleration sensor, and gyro sensor.

The audio output unit 170 is a component for aurally outputting various information, and may include a speaker, an earphone/headphone terminal, and the like. For example, the electronic device 100 may receive streaming data including image data and audio data from the server 200, and output a reproduction section of audio data that matches the reproduction section of the image data through the audio output unit 170.

Meanwhile, the various embodiments described above may be implemented by combining two or more embodiments as long as they do not conflict or contradict each other.

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described in this disclosure may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

In some cases, the embodiments described herein may be implemented by a processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules described above may perform one or more functions and operations described herein.

Meanwhile, computer instructions or computer programs for performing processing operations in servers, electronic devices, etc. according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. Computer instructions or computer programs stored in such a non-transitory computer readable medium, when executed by a processor of a specific device, cause the above-described specific device to perform processing operations in a server, electronic device, etc. according to various embodiments described above.

A non-transitory computer readable medium is a medium that stores data semi-permanently and is readable by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of the non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art without departing from the gist of the present disclosure claimed in the claims.

100: electronic device 110: memory
120: processor 130: communication unit
140: display 150: sensor unit
200: server

Claims (7)

In the control method of an electronic device,
Receiving streaming data including first image data corresponding to a preset viewing range;
identifying a viewing area of the user within the preset viewing range by tracking a direction of the user's eyeballs;
selecting target image data that matches at least a part of the identified viewing area from among the first image data;
obtaining reconstructed image data by inputting the target image data to an artificial intelligence model trained to perform high-resolution reconstruction;
obtaining second image data by replacing the target image data of the first image data with the reconstructed image data, and outputting the second image data;
Dividing the user's viewing area into a foveal area with a certain viewing angle including the central point to which the user's gaze is directed, a blend area surrounding the foveal area and corresponding to a viewing angle greater than the viewing angle of the foveal area, and a peripheral area excluding the foveal area and the blend area;
changing a viewing angle range of the foveal area based on at least one of a speed at which the electronic device receives streaming data and a speed at which the electronic device performs high-resolution reconstruction through the artificial intelligence model;
identifying a rate at which the viewing area of the user changes; and
Based on the identified speed, changing the viewing angle range of the foveal area; includes,
Obtaining the restored image data,
obtaining first reconstructed image data by inputting first target image data matched to the foveal region to the artificial intelligence model;
A control method of an electronic device comprising obtaining second reconstructed image data by performing interpolation correction on second target image data that matches the blend region. According to claim 1,
Receiving the streaming data,
Receiving a plurality of partial images in which the first image data is divided into regions;
Receiving data for a plurality of neural network models trained to perform high-resolution reconstruction on each of the plurality of partial images;
The step of selecting the target image data,
Selecting at least one partial image included in the user's viewing area from among the plurality of partial images;
Obtaining the restored image data,
and inputting the selected at least one partial image to at least one neural network model that matches each of the selected at least one partial image. According to claim 1,
The control method of the electronic device,
Receiving, by the electronic device, streaming data including a plurality of image frames in which Virtual Reality (VR) content is divided by time; and
Receiving, by the electronic device, data for a plurality of first neural network models trained to perform high-resolution reconstruction on each of the plurality of image frames;
Each of the plurality of first neural network models,
Each image frame includes a plurality of second neural network models trained to perform high-resolution reconstruction on each of a plurality of partial images divided by region;
The control method of the electronic device,
selecting at least one partial image included in at least one image frame based on a time interval within the VR content reproduced by the electronic device and a field of view of the user detected during the time interval; and
The control method of an electronic device further comprising performing high-resolution reconstruction on the selected at least one partial image through at least one second neural network model that matches each of the selected at least one partial image. delete delete delete In the control method of a system including a server and an electronic device,
transmitting, by the server, streaming data including first image data corresponding to a viewing range of 360 degrees to the electronic device;
identifying, by the electronic device, a viewing area of the user by tracking the direction of the user's eyeballs;
selecting, by the electronic device, target image data matching the identified viewing area from among the first image data;
obtaining reconstructed image data by inputting the target image data to an artificial intelligence model trained to perform high-resolution reconstruction by the electronic device;
replacing, by the electronic device, the target image data of the first image data with the restored image data to obtain second image data, and outputting the second image data;
Dividing, by the electronic device, the user's viewing area into a foveal area with a certain viewing angle including the center point to which the user's gaze is directed, a blend area surrounding the foveal area and corresponding to a viewing angle greater than the viewing angle of the foveal area, and a peripheral area excluding the foveal area and the blend area;
changing, by the electronic device, a viewing angle range of the foveal area based on at least one of a speed at which the electronic device receives streaming data and a speed at which the electronic device performs high-resolution reconstruction through the artificial intelligence model;
identifying, by the electronic device, a rate at which the viewing area of the user is changed; and
Changing, by the electronic device, a viewing angle range of the foveal area based on the identified speed;
Obtaining the restored image data,
obtaining first reconstructed image data by inputting first target image data matched to the foveal region to the artificial intelligence model;
Characterized in that, interpolation correction is performed on the second target image data that matches the blend region to obtain second restored image data.

Images