KR102315899B1 - Electronic device, method, computer-readable storage medium and computer program for processing image - Google Patents

Abstract

According to various embodiments of the present disclosure, a method for processing an image includes: receiving a first image through a first bandwidth; receiving a second image through a second bandwidth; receiving, in a display unit including a first display area and a second display area corresponding to a part of the first display area, location information of the second display area on the first display area; and providing the first image to the first display area of the display unit and the second image to the second display area so that the second image is overlaid on the first image. have.

Description

ELECTRONIC DEVICE, METHOD, COMPUTER-READABLE STORAGE MEDIUM AND COMPUTER PROGRAM FOR PROCESSING IMAGE

The present invention relates to an electronic device, a method, a computer-readable recording medium, and a computer program for processing an image.

With the recent commercialization of 5th generation mobile telecommunication (5G), immersive media application services include real-time live broadcast services and high-quality video-on-demand systems that incorporate 360-degree cameras and virtual reality (VR) technologies. Technology development for providing content of video on demand (VOD) is being made.

Conventionally, a service for relaying game competitions or sports events in real time by applying virtual reality technology has been provided. For example, the service applies PIP image display technology so that the user can accurately view important scenes while feeling the atmosphere of the site where a game competition or sports event is held, and the entire site image obtained through a 360-degree camera An image in which a picture in picture (PIP) image, which is an important scene image, is overlaid on a 360-degree image is provided.

An embodiment of the present invention provides an electronic device, method, and computer readable electronic device for processing an image to provide a visually natural image by minimizing a sense of heterogeneity between images when providing a plurality of images using a PIP image display technology A recording medium and a computer program may be provided. For example, according to an embodiment of the present invention, when a PIP image is overlaid on a 360-degree image obtained through a 360-degree camera, a natural image can be provided by minimizing the sense of heterogeneity between the two images.

An embodiment of the present invention provides a new PIP image display technology, and an electronic device, method, computer readable recording medium, and computer program for processing an image that provides a high-quality image for an important image among a plurality of images can provide For example, according to an embodiment of the present invention, when receiving a 360-degree image and a PIP image acquired through a 360-degree camera in a streaming manner, the PIP image may be provided as a high-quality image.

According to an embodiment, an electronic device for processing an image, comprising: a communication module; a display unit including a first display area and a second display area corresponding to a portion of the first display area; and receiving a first image through a first bandwidth and a second image through a second bandwidth, using the communication module, on the first display area of the second display area using the communication module Receive location information, provide the first image to the first display area, and provide the second image to the second display area to control the display unit so that the second image is overlaid on the first image It may include a control unit that

According to an embodiment, a method for processing an image includes: receiving a first image through a first bandwidth; receiving a second image through a second bandwidth; receiving, in a display unit including a first display area and a second display area corresponding to a part of the first display area, location information of the second display area on the first display area; and providing the first image to the first display area of the display unit and the second image to the second display area so that the second image is overlaid on the first image. have.

An electronic device, a method, a computer-readable recording medium, and a computer program for processing an image according to an embodiment of the present invention minimize the sense of heterogeneity between images when providing a plurality of images using a PIP image display technology to provide a visual to provide a natural image. For example, according to an embodiment of the present invention, when a PIP image is overlaid on a 360-degree image obtained through a 360-degree camera, a natural image can be provided by minimizing the sense of heterogeneity between the two images.

An electronic device, a method, a computer-readable recording medium, and a computer program for processing an image according to an embodiment of the present invention provide a new PIP image display technology to provide a high-quality image for an important image among a plurality of images can make it happen For example, according to an embodiment of the present invention, when receiving a 360-degree image and a PIP image acquired through a 360-degree camera in a streaming manner, the PIP image may be provided as a high-quality image.

1A to 1C are block diagrams of a system for processing an image according to an embodiment of the present invention.
2 is a flowchart of an image processing operation according to an embodiment of the present invention.
3 is a flowchart of an image processing operation according to an embodiment of the present invention.
4 is a diagram illustrating an example of a PIP video providing screen using the prior art and a PIP video providing screen according to an embodiment of the present invention.
5 is a diagram for explaining an image processing operation according to an embodiment of the present invention.
6 is a flowchart of an image processing operation according to an embodiment of the present invention.
7 is a flowchart of an image processing operation according to an embodiment of the present invention.

First, the advantages and features of the present invention, and a method for achieving them will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. Here, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and are common in the technical field to which the present invention pertains. The technical scope of the present invention should be defined by the claims since it is provided by way of example so that those with knowledge can clearly understand the scope of the invention.

In addition, in the following description of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary depending on the intentions or customs of users, operators, etc., of course. Therefore, the definition should be made based on the technical idea described throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1A to 1C are block diagrams of a system for processing an image according to an embodiment of the present invention.

Referring to FIG. 1A , the system includes a camera device 110 , a stitching device 120 , a first device 130 , a content server 150 (online video platform (OVP) server), and a second device ( 170 ) and/or video server 190 .

The camera device 110 may capture (acquire) a still image and/or a moving image. For example, the camera device 110 may include one or more image sensors and one or more lenses. The lens may be a wide-angle lens or a telephoto lens. For example, the camera device 110 may be a 360 degree camera. The 360-degree camera may include a plurality of lenses to capture images (acquire image data) in a plurality of different directions.

The stitching device 120 confirms the boundary surface of each of the images for a plurality of different directions obtained from the camera apparatus 110, and interconnects (stitches) the adjacent boundary surfaces to obtain a 360-degree image (360-degree image data) A first image may be generated.

Referring to FIG. 1A , the first device 130 may include a controller 131 , a communication module 133 , and a memory 135 .

The controller 131 (also referred to as a processor, a control device, or a control circuit) includes at least one other component (eg, a hardware component (eg, a communication module 133 and a memory 135 ) of the connected first device 130 ). or software components), and perform various data processing and operations.

The communication module 133 is a wired or wireless connection between the first device 130 and an external device, for example, the stitching device 120 , the content server 150 , the second device 170 , and/or the video server 190 . It is possible to support establishment of a communication channel and performing communication through the established communication channel.

The memory 135 includes various data, for example, software (eg, a program) used by at least one component (control unit 131 , communication module 133 , and memory 135 ) of the first device 130 . ) and input data or output data for commands related thereto. The memory 135 may include a volatile memory or a non-volatile memory.

According to an embodiment, the controller 131 may receive the first image through the communication module 133 .

According to an embodiment, the controller 131 may receive an image (image data) from the video server 190 through the communication module 133 , adjust the size of the received image to a specified size, and You can create an image.

According to an embodiment, the memory 135 may store control data (control information) related to the first image and the second image, and the controller 131 may store the first image and the first image based on the control data. Image processing of the image and the second image may be performed.

For example, the control data may include region information (eg, region location information, region coordinate information) and/or the time stamp information of the first image on which the second image is to be overlaid on the first image. have. For example, the control data may further include information related to a specified distortion effect of the second image. For example, the specified distortion effect related information may be used to correct the second image, and may include a previously generated and stored distortion map. For example, when the second image is overlaid on a partial area of the first image, which is a 360-degree image, distortion may occur from the center to the edge of the second image. To solve this, the distortion map is corrected for distortion. can be used for

For example, based on the control data, the controller 131 combines the first image and the second image so that the second image is overlaid on a designated area of the first image (the first image is Insert a second image).

Referring to FIG. 1B , the control unit 131 of the first device 130 may include a time stamp generator 1311 , and the communication module 133 of the first device 130 provides a first It may include an encoder 1313 and a second encoder 1315 .

The time stamp generator 1311 may generate time stamp information (eg, presentation time stamp (PTS)) of the first image and the second image.

For example, the time stamp information may be generated whenever the first image and the second image are photographed (acquired).

For example, the first image and the second image may be images (or moving images) provided in a streaming manner, and the time stamp information includes synchronization time information (when playing) between the first image and the second image. time information for synchronization of For example, the time stamp information may include information for designating replay times of the first image and the second image. For example, the information may be a difference value from a program clock reference (PCR), which is reference time information used in the system.

The first encoder 1313 may encode the first image. For example, the first encoder 1313 may perform encoding to transmit the first image to the content server 150 or the second device 170 .

The second encoder 1315 may encode the second image. For example, the second encoder 1315 may perform encoding to transmit the second image to the content server 150 or the second device 170 .

Referring to FIG. 1A , the content server 150 may include a controller 151 , a communication module 153 , and/or a memory 155 .

The control unit 151 (also referred to as a processor, a control device, or a control circuit) includes at least one other component (eg, a hardware component (eg, a communication module 153 and a memory 155 ) of the connected first device 150 ). or software components), and perform various data processing and operations.

The communication module 153 establishes a wired or wireless communication channel between the content server 150 and an external device, for example, the first device 130 and/or the second device 170, and establishes a communication channel through the established communication channel. It can support performing communication.

The memory 155 includes various data, for example, software (eg, a program) used by at least one component of the content server 150 (control unit 151 , communication module 153 , and memory 155 ). And, it is possible to store input data or output data for a command related thereto. The memory 155 may include a volatile memory or a non-volatile memory.

According to an embodiment, the controller 151 may receive the encoded first image from the first device 130 through the communication module 153 . The controller 151 may receive the encoded second image from the first device 130 through the communication module 153 . The controller 151 may receive control data (control information) related to the encoded first image and the encoded second image from the first device 130 through the communication module 153 . For example, the control data may include region information (eg, region location information, region coordinate information) and/or the time stamp information of the first image on which the second image is to be overlaid on the first image. have. For example, the control data may further include information related to a specified distortion effect of the second image. For example, the specified distortion effect related information may be used to correct the second image, and may include a previously generated and stored distortion map. For example, when the second image is overlaid on a partial area of the first image, which is a 360-degree image, distortion may occur from the center to the edge of the second image. To solve this, the distortion map is corrected for distortion. can be used for

Referring to FIG. 1C , the communication module 153 of the content server 150 may include a first decoder 1511 and a second decoder 1513 , and the controller 151 of the content server 150 is an image adjuster. (1515).

The first decoder 1511 may decode the encoded first image received from the first device 130 .

The second decoder 1513 may decode the encoded second image received from the first device 130 .

The image controller 1515 may perform image processing of the decoded first image and the decoded second image based on the control data received from the first device 130 . For example, the image adjuster 1515 may be configured such that the decoded second image is overlaid on a designated area of the decoded first image based on the control data, so that the decoded first image and the decoded second image are overlaid. Images can be combined (inserted).

Referring to FIG. 1A , the second device 170 may include a control unit 171 , a communication module 173 , a display unit 175 , and/or a memory 177 .

The control unit 171 (also referred to as a processor, a control device, or a control circuit) includes at least one other component (eg, a hardware component (eg, a hardware component (eg, a communication module 173 ), a display unit 175 , and a connected second device 170 ). memory 177) or a software component), and may perform various data processing and operations.

The communication module 173 establishes a wired or wireless communication channel between the second device 170 and an external device, for example, the first device 130 and/or the content server 150, and establishes a communication channel through the established communication channel. It can support performing communication. For example, the communication module 173 may include a wireless communication module (eg, a cellular communication module, a Wi-Fi communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) and/or a wired communication module (eg, a LAN (local area network) including a communication module, or a power line communication module), and using a corresponding communication module among them, a first network (not shown) (eg, Bluetooth, WiFi direct, or short-range communication such as IrDA (infrared data association)) network) or a second network (not shown) (eg, a cellular network, the Internet, or a telecommunication network such as a computer network (eg, LAN or WAN)) via an external device, eg, the first device 130 and / or communicate with the content server 150 .

The display unit 175 is a device for visually providing information to the user of the second device 170, and includes, for example, a display (or touch screen), a hologram device, or a control circuit for controlling a projector and the device. may include

The memory 177 includes various data used by at least one component (control unit 171, communication module 173, display unit 175, and memory 177) of the second device 170, for example, It may store input data or output data for software (eg, a program) and instructions related thereto. The memory 177 may include a volatile memory or a non-volatile memory.

According to an embodiment, the controller 171 may receive the encoded first image from the first device 130 or the content server 150 through the communication module 173 . The controller 171 may receive the encoded second image from the first device 130 or the content server 150 through the communication module 173 . The control unit 171 may receive control data (control information) related to the encoded first image and the encoded second image from the first device 130 or the content server 150 through the communication module 173 . can

Referring to FIG. 1C , the communication module 173 of the second device 170 may include a first decoder 1711 and a second decoder 1713 , and the controller 171 of the second device 170 is An image adjuster 1715 may be included.

The first decoder 1711 may decode the encoded first image received from the first device 130 or the content server 150 .

The second decoder 1713 may decode the encoded second image received from the first device 130 or the content server 150 .

The image adjuster 1715 may perform image processing of the decoded first image and the decoded second image based on the control data. For example, the image adjuster 1715 may be configured to overlay the decoded second image on a designated area of the decoded first image based on the control data so that the decoded first image and the decoded second image are overlaid. Images may be combined (inserting the decoded second image into the decoded first image).

According to an embodiment, the controller 171 controls the display unit 175 to display the first image and the second image so that the decoded second image is overlaid on a designated area of the decoded first image. It may be displayed on the display unit 175 . For example, the controller 171 may control the display unit 175 to display the combined first image and the second image.

According to an embodiment, the controller 171 receives a first image through a first bandwidth and receives a second image through a second bandwidth, using the communication module 173 , and the display unit 175 The first image is provided in a first area (referred to as a first display area) of The display unit 175 may be controlled to provide an image.

For example, the control unit 171 may receive the location information of the second area using the communication module 173 and determine the location of the second area based on the location information.

For example, the second image may be a higher quality image than the first image.

For example, the control unit 171 checks a region corresponding to the user's gaze, and uses the communication module 173 to provide information on the region corresponding to the user's gaze, the first image and the The second image may be transmitted to the device that has transmitted it, for example, the first device 130 or the content server 150 .

For example, the first bandwidth and the second bandwidth may be determined based on information on an area corresponding to the user's gaze.

For example, the area corresponding to the user's gaze may include the second area. The first bandwidth and the second bandwidth may be determined by changing preset bandwidths, respectively, based on the area corresponding to the user's gaze including the second area.

According to the embodiment of FIG. 1A described above, the camera device 110, the stitching device 120, the first device 130, and the video server 190 are described as being configured (implemented) as respective devices. However, according to another embodiment, at least two of the camera device 110 , the stitching device 120 , the first device 130 , and the video server 190 may be integrated into one device to be configured. may be For example, the first device 130 may include the camera device 110 , the stitching device 120 , and the video server 190 .

In addition, although it has been described that the first image is a 360-degree image in the embodiment of FIG. 1A described above, according to another embodiment, the first image may be a two-dimensional image. For example, the first image may be a two-dimensional image acquired through the camera device 110 .

In addition, in the above-described embodiment of FIG. 1B , the control unit 131 of the first device 130 includes a component, that is, the time stamp generator 1311 , and the communication module 133 includes the components, that is, the first Although it has been described as including the encoder 1313 and the second encoder 1315, according to another embodiment, the control unit 131 performs the same or similar function performed by the time stamp generator 1311, and the communication module 133 may perform the same or similar functions performed by each of the first encoder 1313 and the second encoder 1315 . According to another embodiment, each of the components, that is, the time stamp generator 1311, the first encoder 1313, and the second encoder 1315, or at least two of the components are integrated into one entity, It may be configured as a module (hardware) separate from the control unit 131 and the communication module 133 .

In addition, in the above-described embodiment of FIG. 1C , the communication module 153 of the content server 150 includes components, that is, a first decoder 1511 and a second decoder 1513 , and the control unit 151 is configured Although it has been described as including the unit, that is, the image adjuster 1515, according to another embodiment, the communication module 153 performs the functions performed by each of the first decoder 1511 and the second decoder 1513 in the same or Similarly, the control unit 151 may perform the same or similar functions performed by the image adjuster 1515 . According to another embodiment, each of the constituent units, that is, the first decoder 1511 , the second decoder 1513 , and the image adjuster 1515 or at least two of the constituent units are integrated into one entity, and the It may be configured as a module (hardware) separate from the communication module 153 and the control unit 151 .

In addition, in the above-described embodiment of FIG. 1C , it has been described that the first decoder 1511 , the second decoder 1513 , and the image adjuster 1515 are included in the OVP device 150 . According to an example, a separate device including the first decoder 1511 , the second decoder 1513 , and the image adjuster 1515 may be communicatively connected to the second device 150 .

2 is a flowchart of an image processing operation according to an embodiment of the present invention.

Referring to FIG. 2 , the first device 130 separately (separates) each of a plurality of images, for example, a first image and a second image, to the content server 150 or the second device 170. can provide The content server 150 or the second device 170 that has received the first image and the second image may combine the first image and the second image.

For example, the first image and the second image may be images (or moving images) provided by a streaming method.

For example, the first image and the second image may be images used to provide a user with a real-time live relay service in which a 360-degree camera and virtual reality (VR) technology are grafted. For example, the first image may be a 360-degree image that is a background image, and the second image may be a PIP image that is an important image that is provided overlaid on a designated area of the background image. The first device 130 may provide each of the 360-degree video and the PIP video to the content server 150 or the second device 170 in a streaming manner, and the content server 150 or the second device ( 170) may combine the 360-degree image with the PIP image so that the PIP image is overlaid on a designated area of the 360-degree image.

In operation 201 , the first device 130 (or the controller 131 of the first device 130 ) may transmit the first image.

According to an embodiment, the first device 130 may transmit the first image through a communication module (eg, the communication module 133). For example, the first device 130 may transmit the first image through a first channel corresponding to a designated first bandwidth (first transmission bandwidth).

In operation 203, the content server 150 (or the controller 151 of the content server 150) or the second device 170 (or the controller 171 of the second device 170) may check the first image. have.

According to an embodiment, the content server 150 or the second device 170 receives the first image through a communication module (eg, the communication module 153 or the communication module 173), and the first image can confirm. For example, the content server 150 or the second device 170 may receive the first image through the first channel.

In operation 205 , the first device 130 may transmit the second image.

According to an embodiment, the first device 130 may transmit the second image through the communication module. For example, the first device 130 may transmit the second image through a second channel corresponding to a designated second bandwidth (second transmission bandwidth).

In operation 207 , the content server 150 or the second device 170 may check the second image.

According to an embodiment, the content server 150 or the second device 170 may receive the second image through the communication module and check the second image. For example, the content server 150 or the second device 170 may receive the second image through the second channel.

In operation 209 , the first device 130 may transmit control data.

According to an embodiment, the control data may include control data (control information) related to the first image and the second image, and information on a region of the first image on which the second image is to be overlaid on the first image ( Example: area location information, area coordinate information) and/or the time stamp information.

According to an embodiment, the first device 130 may transmit the control data through the communication module. For example, the content server 150 or the second device 170 may transmit the control data through the first channel, the second channel, or the third channel.

In operation 211 , the content server 150 or the second device 170 may check the control data.

According to an embodiment, the content server 150 or the second device 170 may receive the control data through the communication module and check the control data. For example, the content server 150 or the second device 170 may receive the control data through the first channel, the second channel, or the third channel.

In operation 213 , the content server 150 or the second device 170 may combine the first image and the second image based on the control data.

According to an embodiment, the content server 150 or the second device 170 may combine the first image and the second image so that the second image is overlaid on a designated area of the first image. For example, the content server 150 or the second device 170 may display the second image in a designated area of the first image based on information on an area on which the second image is to be overlaid on the first image. can be inserted. For example, time stamp information of the first image and the second image may correspond to each other.

3 is a flowchart of an image processing operation according to an embodiment of the present invention.

In operation 301 , the second device 170 (or the controller 171 of the second device 170 ) may check communication state information.

According to an embodiment, the communication state information may include the received electric field strength (strength) of the signal of the second device 170 . For example, the received electric field strength of the signal may include a strong electric field or a weak electric field.

In operation 303, the second device 170 may transmit the communication state information.

According to an embodiment, the second device 170 may transmit the communication state information using a communication module (eg, the communication module 173).

In operation 305, the first device 130 (or the control unit 131 of the first device 130) or the content server 150 (or the control unit 151 of the content server 150) transmits the communication state information to the Based on the transmission bandwidth, which is a bandwidth for transmitting the first image and the second image, the first bandwidth and the second bandwidth may be determined, respectively.

According to an embodiment, bandwidth information corresponding to each communication state information may be designated. The first device 130 or the content server 150 determines the bandwidth of the first image as the first bandwidth and determines the bandwidth of the second image as the second bandwidth, based on the bandwidth information corresponding to the designated communication state information. can

For example, the first device 130 or the content server 150 may store corresponding bandwidth information for each communication state information. The first device 130 or the content server 150 may determine the bandwidth of the first image as the first bandwidth and determine the bandwidth of the second image as the second bandwidth based on the stored bandwidth information for each communication state information. have.

For example, the bandwidth information corresponding to each communication state information may include bandwidth information corresponding to each received electric field strength (strength) of a signal of an external device.

In operation 307, the first device 130 or the content server 150 may transmit the first image based on the first bandwidth.

According to an embodiment, the first device 130 or the content server 150 uses a communication module (eg, the communication module 133 or the communication module 153 ) to configure the first image based on the first bandwidth. can be transmitted. For example, the first device 130 or the content server 150 may transmit the first image through a first channel corresponding to the first bandwidth.

According to an embodiment, the first image may be an image (or a moving image) provided by a streaming method. For example, the first image may be a 360-degree image.

In operation 309, the second device 170 may check the first image.

According to an embodiment, the second device 170 may receive the first image using a communication module, and may check the received first image.

In operation 311 , the first device 130 or the content server 150 may transmit a second image based on the second bandwidth.

According to an embodiment, the first device 130 or the content server 150 may transmit the second image based on the second bandwidth using a communication module. For example, the first device 130 or the content server 150 may transmit the second image through a second channel corresponding to the second bandwidth.

According to an embodiment, the second image may be an image (or a moving image) provided by a streaming method. For example, the second image may be a PIP image.

In operation 313 , the second device 170 may check the second image.

According to an embodiment, the second device 170 may receive the second image using a communication module, and may check the received second image.

In operation 315 , the first device 130 or the content server 150 may transmit control data.

According to an embodiment, the first device 130 or the content server 150 may transmit the control data using a communication module.

For example, the control data may be control data (control information) related to the first image and the second image, and may include area information of the first image on which the second image is to be overlaid on the first image (eg: area location information, area coordinate information) and/or the time stamp information.

In operation 317 , the second device 170 may check the control data.

According to an embodiment, the second device 170 may receive the control data using a communication module and may check the received control data.

In operation 319, the second device 170 may provide (display) the first image and the second image based on the control data.

According to an embodiment, the second device 170 may provide the first image and the second image to the user so that the second image is overlaid on a designated area of the first image. For example, the second device 170 may combine the first image and the second image so that the second image is overlaid on a designated area of the first image, and the combined first image and the second image video can be provided.

In order to provide a real-time live relay service in which a 360-degree camera and virtual reality technology are grafted in the embodiment of FIG. 3, the first image is a 360-degree image that is a background image, and the second image is a designated image of the background image. When used as a PIP image, which is an important image that is overlaid on the area, it is possible to solve the problem of deterioration in image quality of the conventional PIP image provided to users.

4 is a diagram illustrating an example of a PIP video providing screen using the prior art and a PIP video providing screen according to an embodiment of the present invention.

Conventionally, the 360-degree image and the PIP image are provided between the content server 150 and the second device 170 in various bandwidths according to network traffic so that the user can use the 360-degree virtual reality content. However, in the related art, the content server 150 transmits the 360-degree image and the PIP image to the second device 170 in a streaming manner through one channel. When the PIP image is received at a bit rate and the PIP image is decoded, there has been a problem in that the quality of the PIP image deteriorates.

For example, in the prior art, when the content server 150 transmits the 360-degree image and the PIP image to the second device 170 at 8 Mbps, referring to FIG. 4A , the second device 170 When the PIP area 403 in which the PIP image is to be displayed is 1/8 of the total display area 401 in which the 360-degree image is displayed, the data transmission amount of the PIP image is 1 Mbps on average.

On the other hand, according to the embodiment of FIG. 3 described above, when the content server 150 transmits the 360-degree image and the PIP image to the second device 170 at 8 Mbps, respectively, referring to FIG. 4(b), In the second device 170, even if the PIP area 403 where the PIP image is to be displayed is 1/8 of the total display area 401, the PIP image is transmitted at 3 Mbps and the 360-degree image is transmitted at 5 Mbps. can do. For example, according to the above-described embodiment of FIG. 3 , in the case of an important area where the user's attention is concentrated, such as the PIP area 403 , the bandwidth is increased compared to the background area such as the entire display area 401 . It can be implemented in high quality (high resolution).

5 is a diagram for explaining an image processing operation according to an embodiment of the present invention.

Referring to FIG. 5 , the second device 170 tracks the user's gaze and/or the movement of the user's head, and provides the tracking-related information to the first device 130 or the content server 150, The first device 130 or the content server 150 may variably set a bandwidth for each of the images to be provided to the second device 170 . Accordingly, the second device 170 may receive images transmitted by the first device 130 or the content server 150 with different bandwidths, respectively. For example, the second device 170 may receive each of the images through different bandwidths.

According to an embodiment, the second device 170 may be a device in the form of a head mounted display (HMD) and may be worn on the user's head. The second device 170 may receive a plurality of images, for example, a first image and a second image, and display it on a display unit (eg, the display unit 175 ), and the first image and the second image A three-dimensional coordinate space (virtual space) may be generated and provided based on the image. For example, the first image may be provided to a first area 501 (also referred to as a first display area) that is a background area in a 3D coordinate space, and the second image is an interest in a 3D coordinate space. It may be provided in a second area 503 (also referred to as a second display area) that is an area (important area). For example, the second area 503 may be overlaid on the first area 501 . For example, the first image may be a 360 degree image, and the second image may be a 2D image. For example, the first image and the second image may be images provided in a streaming method.

According to an embodiment, the second device 170 may track the user's gaze and/or the movement of the user's head using eye tracking and/or head tracking technology. have. For example, the second device 170 may track the position and/or movement of the user's gaze using the gaze tracking technology. For example, the second device 170 uses the head tracking technology to place the center of the user's head at the origin of the three-dimensional coordinate system, and moves the user's head forward, backward, left, right, up, or down. can be tracked.

According to an embodiment, the second device 170 may identify a 3D coordinate region to which the user's gaze is directed in the 3D coordinate space based on the tracking of the user's gaze and/or the movement of the user's head. . For example, the three-dimensional coordinate region to which the user's gaze is directed may be a region that the user is interested in viewing. Accordingly, an image provided to the three-dimensional coordinate region toward which the user's gaze is directed is provided as a high-quality image. may be required to be In the case of an image provided in a 3D coordinate area other than the 3D coordinate area to which the user's gaze is directed on the 3D coordinate space, it may be provided as a high-quality image, but it may be irrelevant even if it is not provided as a high-quality image have.

According to an embodiment, in order to provide a high-quality image in the image provided to the 3D coordinate region toward which the user's gaze is directed, the second device 170 may obtain region information corresponding to the identified 3D coordinate region. , to a device providing the first image and the second image, for example, the first device 130 or the content server 150 . The first device 130 or the content server 150 that has received the region information corresponding to the 3D coordinate region sets a bandwidth so that a high-quality image can be provided for the image to be provided in the checked 3D coordinate region. (or reset) and/or for an image to be provided in a region other than the identified 3D coordinate region, a bandwidth may be set (or reset) so that an image of normal or low quality may be provided. For example, in order to set (or reset) the bandwidth, the first device 130 or the content server 150 that has received region information corresponding to the three-dimensional coordinate region provides the identified three-dimensional coordinate region. By changing the frame rate of the image to be provided and/or the frame rate of the image to be provided in an area other than the identified 3D coordinate area, the image to be provided in the identified 3D coordinate area and provided to an area other than the identified 3D coordinate area It is possible to change the bit rate of the image to be made.

For example, when the 3D coordinate area to which the user's gaze is directed in the 3D coordinate space includes the second area 503 , the second device 170 determines that the user's gaze is directed in the 3D coordinate space. Information indicating that the three-dimensional coordinate region facing the second region 503 is transmitted to a device providing the first image and the second image, for example, the first device 130 or the content server 150 can be transmitted The first device 130 or the content server 150, which has received the information that the 3D coordinate region to which the user's gaze is directed includes the second region 503 on the 3D coordinate space, the second region By maintaining the frame rate of the image to be provided to the 503 and variably dropping the frame rate of the image to be provided to the first area 501, the transmission amount of the image (image data) to be provided to the first area is reduced, resulting in limited communication A high-definition image may be implemented for the second image, which is an important image in a network situation. For example, the first device 130 or the content server 150, which has received the information that the 3D coordinate area to which the user's gaze is directed on the 3D coordinate space includes the second area 503, The frame rate of the image to be provided to the first region 501 may be set (determined) using Equation 1 below.

[Equation 1]

Frame rate of the image to be provided to the first area = coefficient * Frame rate of the image to be provided to the second area

For example, when the 3D coordinate area to which the user's gaze is directed in the 3D coordinate space includes a part of the first area 501 that does not include the second area 503 , the second device 170 . provides information that the 3D coordinate area to which the user's gaze is directed in the 3D coordinate space includes a part of the first area 501 that does not include the second area 503, the first image and the second The image may be transmitted to a device providing an image, for example, the first device 130 or the content server 150 . The first device 130 that receives the information that the 3D coordinate area to which the user's gaze is directed in the 3D coordinate space includes a part of the first area 601 that does not include the second area 503 Alternatively, the content server 150 may maintain the frame rate of the image to be provided to the first area 501 .

For example, the frame rate of the first image to be provided to the first region 501 may be 10 Hz to 60 Hz, and a predetermined frame rate may be 60 Hz. For example, a frame rate of the second image to be provided to the second region 503 may be 30 Hz to 60 Hz, and a predetermined frame rate may be 60 Hz. For example, the first device 130 or the content server 150, which has received the information that the 3D coordinate area to which the user's gaze is directed on the 3D coordinate space includes the second area 503, The frame rate of the second image to be displayed on the second region 503 may be maintained at 60 Hz, and the frame rate of the first image to be displayed on the first region 501 may be variably dropped from 10 Hz to 30 Hz. For example, the first device that receives the information that the 3D coordinate area to which the user's gaze is directed in the 3D coordinate space includes a part of the first area 501 that does not include the second area 503 , 130 or the content server 150 may maintain the frame rate of the first region 501 at 60 Hz.

6 is a flowchart of an image processing operation according to an embodiment of the present invention.

In operation 601 , the second device 170 (or the controller 171 of the second device 170 ) may track the user's gaze or the head in the 3D coordinate space.

According to an embodiment, the second device 170 may be a device in the form of a head worn display, and may be worn on the user's head.

According to an embodiment, the second device 170 receives a plurality of images, for example, a first image and a second image, and displays the received first image and the second image on a 3D coordinate space. may be provided in For example, at least a portion of the first image and/or at least a portion of the second image may be displayed on the display unit of the second device 170 . For example, the second device 170 may generate the three-dimensional coordinate space by generating a three-dimensional space model based on the first image and the second image using a conventional three-dimensional coordinate space generation technique. can

According to an embodiment, the second device 170 may track the user's gaze on the 3D coordinate space by using the gaze tracking technology.

According to an embodiment, the second device 170 may track the movement of the user's head in the three-dimensional coordinate space by using a head tracking technology, and based on the movement of the user's head, Thus, it is possible to determine (estimate) the direction in which the user's gaze is directed in the three-dimensional coordinate space.

In operation 603 , the second device 170 may identify an area corresponding to the user's gaze on the 3D coordinate space based on the gaze tracking or the head tracking.

According to an embodiment, the second device 170, based on the eye tracking or the head tracking, is an area corresponding to the user's gaze on the 3D coordinate space, the area in the direction of the user's gaze (user area) can be checked.

For example, the 3D coordinate space may be divided into a plurality of 3D coordinate regions, for example, the first region 501 and the second region 503 of FIG. 5 .

For example, the second device 170 may include, based on the eye tracking or the head tracking, at least a part of the area viewed by the user, among a plurality of 3D coordinate areas on the 3D coordinate space. You can check the dimensional coordinate area. For example, the second device 170 determines whether the area viewed by the user includes the first area 501 or a part of the second area 503 that does not include the first area 501 . You can check whether the area includes the area.

In operation 605, the second device 170 may transmit area information of the identified area.

According to an embodiment, the second device 170 may transmit area information of the identified area using a communication module (eg, the communication module 173).

For example, the area information of the identified area may include location information (coordinate information) and size information of the area in the 3D coordinate space.

For example, the region information of the identified region may include information indicating a designated region of interest (important region) or information indicating a region other than the designated region of interest. For example, the designated ROI may include information called the first region 501 of FIG. 5 . For example, the information that the region is other than the designated region of interest may include information that the region is a part of the first region 501 that does not include the second region 503 of FIG. 5 .

In operation 607, the first device 130 or the content server 150 may change or maintain a predetermined bandwidth of the first image and a predetermined bandwidth of the second image based on the region information.

According to an embodiment, when the region information includes the information about the designated region of interest, the bandwidth of the image to be transmitted to the specified region of interest is widened and the bandwidth of the image to be transmitted to the region other than the specified region of interest is reduced. can

Referring to FIG. 5 , for example, the first image may be provided to a first region 503 that is a region other than a designated ROI, and the second image is a second region 503 that is the designated ROI. can be provided on

For example, when the first device 130 or the content server 150 includes information that the region information is the second region 503 that is a designated region of interest, the first device to be provided to the second region 503 is In order to increase the bit rate of the second image, the frame rate of the first image to be provided to the first region 501 may be changed to be lower according to a specified criterion.

According to an embodiment, when the region information includes information that is a region other than the designated region of interest, the bandwidth of the first image and the bandwidth of the second image may maintain the preset bandwidth.

Referring to FIG. 5 , for example, when the region information includes the information that the region information is the first region 501, the first region 501 and the content server 150 The bit rates of the first image and the second image to be provided to the second region 503 may be maintained.

In operation 609, the first device 130 or the content server 150 may transmit the first image based on the changed or maintained bandwidth of the first image.

According to an embodiment, the first device 130 or the content server 150 uses a communication module (eg, the communication module 133 or the communication module 153), and the bandwidth of the changed or maintained first image The first image may be transmitted based on

For example, the first image may be an image (or a moving image) provided by a streaming method. For example, the first image may be a 360-degree image.

In operation 611 , the second device 170 may check the first image.

According to an embodiment, the second device 170 may receive the first image using a communication module, and may check the received first image.

In operation 613 , the first device 130 or the content server 150 may transmit the second image based on the changed or maintained bandwidth.

According to an embodiment, the first device 130 or the content server 150 may transmit the second image based on the changed or maintained bandwidth using a communication module.

For example, the second image may be an image (or a moving image) provided by a streaming method. For example, the second image may be a PIP image.

In operation 615 , the second device 170 may check the second image.

According to an embodiment, the second device 170 may receive the second image using a communication module, and may check the received second image.

In operation 617 , the second device 170 may provide the first image and the second image.

According to an embodiment, at least a portion of the first image and/or at least a portion of the second image may be displayed on the display unit of the second device 170 .

7 is a flowchart of an image processing operation according to an embodiment of the present invention.

In operation 710 , the second device 170 (also referred to as an electronic device) may receive the first image through the first bandwidth.

In operation 720, the second device 170 may receive the second image through the second bandwidth.

In operation 730, the second device 170 provides the first image to a first area (also referred to as a first display area) of the display unit, and a second area (overlayed) on at least a partial area of the first area. The second image may be provided to a second display area).

For example, the second device 170 may receive location information of the second area and determine the location of the second area based on the location information.

For example, the second image may be a higher quality image than the first image.

For example, the second device 170 identifies a region corresponding to the user's gaze, and transmits information on the region corresponding to the user's gaze, the first image and the second image; For example, it may be transmitted to the first device 130 or the content server 150 .

For example, the first bandwidth and the second bandwidth may be determined based on information on an area corresponding to the user's gaze.

For example, the area corresponding to the user's gaze may include the second area. The first bandwidth and the second bandwidth are respectively changed to preset bandwidths based on the area corresponding to the user's gaze including the second area under the control of the second device 170 . may have been decided.

The operations in the above-described embodiments of FIGS. 1A to 7 may be sequentially, parallelly, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Various embodiments of the present document include instructions stored in a machine-readable storage media (eg, memory 113 (internal memory or external memory)) readable by a machine (eg, a computer). It may be implemented in software (eg, a program). The device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device 100 ) according to the disclosed embodiments. When the command is executed by a control unit (eg, the control unit 101 ) (or a processor), the control unit may directly or use other components under the control of the control unit to perform a function corresponding to the command. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

According to an example, the method according to various embodiments disclosed in the present document may be included and provided in a computer program product.

According to an embodiment, there is provided a computer-readable recording medium storing a computer program, comprising: an operation of receiving a first image through a first bandwidth, an operation of receiving a second image through a second bandwidth, a first display area; receiving location information on the first display area of the second display area in a display unit including a second display area corresponding to a part of the first display area; and instructions for causing the processor to perform a method including providing a first image and providing the second image in the second display area so that the second image is overlaid on the first image. can

According to an embodiment, as a computer program stored in a computer-readable recording medium, an operation of receiving a first image through a first bandwidth, an operation of receiving a second image through a second bandwidth, a first display area; receiving location information on the first display area of the second display area in a display unit including a second display area corresponding to a part of the first display area; and instructions for causing the processor to perform a method including providing a first image and providing the second image in the second display area so that the second image is overlaid on the first image. can

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains may make various substitutions, modifications, and changes within the scope not departing from the essential characteristics of the present invention. It will be easy to see that this is possible. That is, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments.

Accordingly, the protection scope of the present invention should be interpreted by the claims described below, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

According to an embodiment of the present invention, when providing content using virtual reality technology, an image may be provided in a virtual space, and accordingly, a technology that allows a user to experience viewing the image in a virtual space Available.

Claims (8)

A method for processing an image, comprising:
receiving a first image through a first bandwidth;
receiving a second image through a second bandwidth;
receiving, in a display unit including a first display area and a second display area corresponding to a part of the first display area, location information of the second display area on the first display area;
providing the first image to the first display area of the display unit and the second image to the second display area so that the second image is overlaid on the first image;
checking an area corresponding to the user's gaze; and
The method further includes transmitting information on an area corresponding to the user's gaze to a device that has transmitted the first image and the second image,
adjusting a frame rate for receiving the first image based on the area corresponding to the user's gaze including the second display area;
A method for processing an image. The method of claim 1 , wherein the second image is an image having a higher quality than that of the first image. delete The method of claim 1, wherein the first bandwidth and the second bandwidth are:
A method for processing an image, which is determined based on information on an area corresponding to the user's gaze. The method of claim 1,
The area corresponding to the user's gaze includes the second display area,
The first bandwidth and the second bandwidth are
A method for processing an image, wherein preset bandwidths are respectively changed and determined based on the area corresponding to the user's gaze including the second display area. As a computer-readable recording medium storing a computer program,
The computer program is
A computer-readable recording medium comprising instructions for causing a processor to perform the method according to any one of claims 1, 2, 4, and 5. As a computer program stored in a computer-readable recording medium,
The computer program is
A computer program comprising instructions for causing a processor to perform the method according to claim 1 , 2 , 4 or 5 . An electronic device for processing an image, comprising:
communication module;
a display unit including a first display area and a second display area corresponding to a portion of the first display area; and
using the communication module to receive a first image through a first bandwidth and receive a second image through a second bandwidth;
receiving location information on the first display area of the second display area by using the communication module;
a controller configured to control the display unit to provide the first image to the first display area and to provide the second image to the second display area so that the second image is overlaid on the first image,
the control unit
The frame rate of the first image is adjusted based on region information corresponding to the user's gaze and the region corresponding to the user's gaze including the second display region.
An electronic device for processing an image.

Images