KR102497227B1 - Apparatus and method for seamless video streaming playback - Google Patents

Abstract

The present invention discloses a seamless video streaming playback device and an operating method thereof. According to one embodiment of the present invention, the seamless video streaming playback device includes a processor and a memory connected to the processor. The processor may be configured to: determine the number of blocks to divide a playback area based on a display resolution, a video resolution, and a playback resolution; divide the playback area into one or more blocks according to the determined number of blocks; detect a change in capacitance input through a touch input tool from at least one touch sensor; obtain eye tracking information of a user, input through a camera; set one or more blocks divided based on the change in capacitance and the eye tracking information as a first region of interest; adjust the resolution of the first region of interest according to the Internet download speed; and reset the first region of interest according to a frequency at which the change in capacitance and the eye tracking information are detected in one or more divided blocks. According to the present invention, a video is provided by adjusting information about images transmitted to a terminal used by a user according to an environment for playback of the video in the terminal, and thus, a video streaming environment can be provided.

Description

Seamless video streaming playback device and method {APPARATUS AND METHOD FOR SEAMLESS VIDEO STREAMING PLAYBACK}

The present invention relates to a seamless video streaming playback device and method, and more particularly, to a device and method for providing seamless video streaming by adjusting a playback environment based on a user's communication environment and playback information of a video being played. It is about.

A video streaming service provided through wired and wireless internet requires an internet speed of at least 500 Kbps or higher to enjoy smooth video, and an internet line providing a bandwidth of several Mbps or more per second is required to enjoy higher quality.

In particular, in the case of the wireless Internet, conditions for using a frequency of a terminal vary according to surrounding environments, such as a bandwidth of a frequency being used and Internet usage of users distributed in a corresponding cell. Accordingly, even when a user watches the same video in the same environment, the quality of a video that can be streamed is determined according to the surrounding Internet usage environment, and sometimes it is impossible to watch the video smoothly due to severe loading and video interruption.

In order to solve this problem, hardware and software decoders are continuously being developed worldwide to increase the compression rate of video and reduce the latency of video reception and decoding.

In addition to this, 5G communication has now been commercialized through the development of communication technology and improvement of communication infrastructure, and research is continuing ahead of commercialization of 6G communication technology.

However, in spite of such studies on external factors, it is difficult to satisfy all playback environments suitable for each user and the situation of each user terminal because the video playback environment of each user and user terminal is different for each user.

Accordingly, it is necessary to study for seamless playback of video according to the user and the user terminal's situation.

Korean Patent Publication No. 10-2274466

An object of the present invention to solve the above problems is to provide an apparatus and method capable of replaying a video seamlessly according to a user's video playback environment during video streaming.

One aspect of the present invention for achieving the above object provides a seamless video streaming device. The seamless streaming device includes a processor and a memory connected to the processor, and the processor determines the number of blocks to partition a playback area based on a display resolution, a video resolution, and a playback resolution, and determines the number of blocks according to the determined number of blocks. According to this, the play area is divided into one or more blocks, a change in capacitance input from at least one touch sensor through a touch input tool is detected, user eye tracking information input through a camera is acquired, and change in capacitance is obtained. and setting one or more blocks partitioned based on the gaze tracking information as a first region of interest, adjusting a resolution of the first region of interest according to an Internet download speed, and changing capacitance and gaze tracking information in one or more partitioned blocks. The first ROI may be reset according to the detection frequency.

In this case, the processor may recognize an object in the streamed video based on the quality information of the video, and set one or more partitioned blocks including the recognized object as the second region of interest.

In addition, the processor may detect color information of a video streamed in the play area in units of predetermined frames, and reset the second region of interest when specific color information in each unit frame changes by exceeding a predetermined rate.

Here, if the processor detects a decrease in the Internet download speed, the processor changes the playback resolution to the lowest level possible for streaming within the reduced Internet download speed, and increases the resolution of the first region of interest with priority according to the increase in the Internet download speed. can be adjusted

In addition, when there is one or more first ROIs, the processor classifies the first ROI into an interest group, and the center first interest corresponding to the center of the interest group based on coordinate information of the first ROI included in the interest group. The coordinates of the regions may be calculated, and reproduction resolution may be adjusted starting from an interest group including a central first region of interest close to the center of the reproduction region according to a distance between the center of the first region of interest and the center of the reproduction region.

Meanwhile, the processor may classify interest groups so that an average value of a distance between the coordinates of the central first ROI and each of the first ROI included in the interest group has the smallest value.

In the case of using the seamless video streaming device and method according to the present invention as described above, the video is provided by adjusting the information about the video transmitted to the terminal according to the user or the environment for video playback of the terminal used by the user. It is possible to provide a streaming environment for videos without video.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

Also other aspects as described above, features and benefits of certain preferred embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.
1 is an exemplary diagram illustrating a seamless video streaming device according to an embodiment of the present invention.
2A and 2B are exemplary diagrams illustrating division of a video play area according to an exemplary embodiment.
3 is an exemplary diagram illustrating an example of detecting a change in capacitance through a touch sensor.
4 is an exemplary diagram illustrating an example of obtaining user eye tracking information input through a camera.
5 is an exemplary diagram for explaining setting a first region of interest.
6 is an exemplary diagram for explaining calculation of a distance between a central first region of interest and a reproduction region in an interest group.
7A and 7B are exemplary diagrams for explaining an example of setting a second region of interest based on video quality information.
8A and 8B are exemplary diagrams for explaining resetting of a second region of interest based on video quality information.
9 is an exemplary hardware configuration diagram of the seamless video streaming device according to FIG. 1 .
It should be noted that throughout the drawings, like reference numbers are used to show the same or similar elements, features and structures.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring it by omitting unnecessary description.

For the same reason, in the accompanying drawings, some components are exaggerated, omitted, or schematically illustrated. Also, the size of each component does not entirely reflect the actual size. In each figure, the same reference number is assigned to the same or corresponding component.

Advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

At this time, it will be understood that each block of the process flow chart diagrams and combinations of the flow chart diagrams can be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, so that the instructions executed by the processor of the computer or other programmable data processing equipment are described in the flowchart block(s). It creates means to perform functions. These computer program instructions may also be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular way, such that the computer usable or computer readable memory The instructions stored in are also capable of producing an article of manufacture containing instruction means that perform the functions described in the flowchart block(s). The computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to generate computer or other programmable data processing equipment. Instructions for performing processing equipment may also provide steps for performing the functions described in the flowchart block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative implementations it is possible for the functions mentioned in the blocks to occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on their function.

At this time, the term '~unit' used in this embodiment means software or a hardware component such as a field-programmable gate array (FPGA) or application specific integrated circuit (ASIC), and what role does '~unit' have? perform them However, '~ part' is not limited to software or hardware. '~bu' may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In describing the embodiments of the present invention in detail, an example of a specific system will be the main target, but the main subject matter to be claimed in this specification extends the scope disclosed herein to other communication systems and services having a similar technical background. It can be applied within a range that does not deviate greatly, and this will be possible with the judgment of those skilled in the art.

1 is an exemplary diagram illustrating a seamless video streaming device according to an embodiment of the present invention.

Referring to FIG. 1 , a seamless video streaming device (hereinafter referred to as 'device') 100 includes a processor 110 and a memory 120 . In this case, the processor 110 may include at least one software or hardware module for calculation, and may directly include at least one memory therein.

The memory 120 may store information related to driving of the seamless video streaming device or a program in which a method for driving is implemented. The memory 120 may be a volatile memory or a non-volatile memory, and may be a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (e.g., , SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), PROM (Programmable Read Only Memory) ), at least one type of storage medium such as a magnetic memory, a magnetic disk, and an optical disk.

The processor 110 may execute a program and control the device 100 . Program codes executed by the processor 110 may be stored in the memory 120 . The device 100 may be connected to an external device (eg, a personal computer or network) through an input/output device (not shown) and exchange data.

In addition, the device 100 can communicate with a desktop computer equipped with a display, a laptop computer, a notebook, a smart phone, a tablet PC, a mobile phone ( mobile phone), smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game device, navigation device, digital camera, DMB (digital multimedia) broadcasting) player, digital video recorder, digital video player, personal digital assistant (PDA), and the like.

However, for convenience, below, the device 100 will be described based on including a display capable of touch input, but is not limited thereto, and the device 100 can interact with a user through various input tools such as a mouse cursor. there is.

2A and 2B are exemplary diagrams illustrating division of a video play area according to an exemplary embodiment.

Referring to FIGS. 1 and 2A and 2B , the device 100 may include a touchable display 210 on which a video can be played, and a play area 211 and 212 on which the video is played on the display 210. may have different sizes and positions depending on the video playback mode.

For example, when the video playback mode is the full screen mode, the video may be played on the entire area 211 of the display 210, and when the video playback mode is the normal playback mode, the video may be played on a partial area of the display 210. (212).

Meanwhile, the device 100 may partition the playback areas 211 and 212 into one or more blocks according to the playback resolution of the video.

For example, the apparatus 100 may partition the play area into one or more blocks according to the display resolution (P_disp), the video resolution (P_max), and the playback resolution (P_play), and divide the play area into squares or lozenges. The playback area can be partitioned into one or more blocks in various forms.

Here, the display resolution (P_disp) means the resolution based on the physical pixels of the display 210 and means the maximum resolution that the device 100 can support, and the video resolution (P_max) is the resolution of the original video to be played and is a streaming target. It means the maximum resolution of the video, and the playback resolution (P_play) means the resolution of the video being played in the device 100 regardless of the resolution of the video according to the computing power of the device 100, the communication environment, and the user's settings. .

For example, in a device with a display resolution (P_disp) of 720p (eg, 1280*720), a video having a video resolution (P_max) of 720p and a reproduction resolution (P_play) of 360p may be streamed. In this case, the apparatus 100 may divide the playback areas 211 and 212 into 20 pixels according to the playback resolution (P_play) of 360p to generate a total of 576 blocks having 18 horizontal areas and 32 vertical areas.

More specifically, the device 100 may determine the number of blocks (Num_block) to be partitioned based on the display resolution (P_disp), the video resolution (P_max), and the playback resolution (P_play), which is determined through the following equation. do.

Meanwhile, it is assumed that the playback areas 211 and 212 are configured with a resolution higher than the lowest resolution among the display resolution (P_disp), the video resolution (P_max), and the playback resolution (P_play).

[Equation 1]

Here, V_Prow is the number of pixels in the row of the playback resolution (P_play), V_Pcol is the number of pixels in the column of the playback resolution (P_play), and D_row and D_col are the columns and rows of the playback resolution (P_play), respectively. It is a determining factor for determining how many blocks the reproduction area is divided into by multiplication, and is determined through Equations 2 and 3 below.

[Equation 2]

[Equation 3]

Here, w_row and w_col are performance coefficients set by the user or determined according to the computing capability of the user terminal, and are positive real numbers and have a value less than 1. As the values of w_row and w_col decrease, the value of the determinant decreases, and as a result, the number of blocks can be made small, and as the values increase, the number of blocks can increase.

That is, if the device 100 has sufficient computing power, it is possible to set w_row and w_col to large values to set a more detailed block and finely adjust the resolution.

In addition, P_Rmax is the number of pixels in a column of video resolution, P_Rdisp is the number of pixels in a column of display resolution, P_Rplay is the number of pixels in a column of playback resolution, P_Cmax is the number of pixels in a row of video resolution, and P_Cdisp is the number of pixels in a row of display resolution. number, P_Cplay means the number of pixels in a row of the playback resolution.

According to Equations 2 and 3, the determinants D_row and D_col are determined by multiplying the number of pixels of the video resolution by the number of pixels of the display resolution and multiplying the value obtained by dividing by the square of the pixels of the reproduction resolution by the coefficient of performance, respectively.

That is, the determinants D_row and D_col have a value greater than 1 when the video resolution (P_max) is greater than the playback resolution (P_play), and have a value greater than 1 when the display resolution (P_disp) is greater than the playback resolution (P_play). As a result, more dense blocks are partitioned in the reproduction area.

For example, when the display resolution (P_disp) and the video resolution (P_max) are each 720p with 1280*720 pixels, the playback resolution (P_play) is 360p with 640*360 pixels, and the performance coefficients w_row and w_col are set to 0.1. Assume

At this time, the determining factors D_row and D_col are calculated as in Equation 4 below.

[Equation 4]

On the other hand, since pixels have physically indivisible properties, the determinants D_row and D_col may need to be adjusted as follows. In other words, the reciprocal of the decision factors D_row and D_col becomes a factor determining how many pixels one block consists of.

For example, when the inverses of the determinants D_row and D_col are not integers, the adjusted determinants D'_row and D'_col are calculated by rounding up the values at one decimal place.

For example, in the above operation, since the reciprocal of the determinants D_row and D_col are each 2.5, 3 can be calculated as the adjusted determinant D'_row by rounding up, and the adjusted determinant D'_col is calculated as 3 in the same way. It can be done, which is shown in Equation 5 below.

[Equation 5]

At this time, the total number of blocks is determined by Equation 6 below.

[Equation 6]

That is, when the playback resolution (P_play) is 360p, 640 pixels horizontally and 360 pixels vertically, the device 100 divides the horizontal 640 pixels into 3 pixel units and divides the vertical 360 pixels into 3 pixel units, so that a total of 25,600 pixels can be displayed. Blocks can be partitioned.

By determining the resolution of the block in this way, the higher the computing power of the device 100, the higher the resolution that the device 100 can provide, and the better the playback Internet environment, the higher the resolution. As described above, the playback resolution (P_play) can be adjusted in detail by dynamically adjusting the resolution of each block.

3 is an exemplary diagram illustrating an example of detecting a change in capacitance through a touch sensor.

The device 100 may detect a change in capacitance input from at least one touch sensor through a touch input tool. The display may include one or more touch sensors and may be composed of a touch screen panel 310 (TSP, Touch Screen Panel).

The device 100 may detect a change 330 in capacitance input to the touch sensor of the display through a human hand or a touch input tool 320 including a dielectric having the same effect. At this time, in addition to the case where the user directly contacts the display including the touch sensor, the device 100 changes the capacitance according to the change in the electric field generated by the touch sensor when the touch input tool 320 approaches the display within a certain distance. By detecting , it is possible to detect which area of the display the user has contacted or approached with the touch input tool 320 .

The device 100 can detect which part of the image the user is directly interested in according to the change in capacitance according to the proximity of the touch input tool 320 as described above.

More specifically, the capacitive touch sensor forms an electric field between the electrodes by flowing a minute current between the electrodes, and detects the presence or absence of a touch through a change in the capacitance that stores charge by the electric field. Direct or indirect contact that can affect the charge charged in an electrode such as a person's hand or dielectric can cause a change in capacitance, and the device 100 prevents the user from directly touching the display or pointing to a specific area through this. can be detected.

4 is an exemplary diagram illustrating an example of obtaining user eye tracking information input through a camera.

When the user looks at the screen displayed on the display 410, the device 100 may track the user's gaze and determine which part of the screen displayed on the display 410 is focused on.

For example, gaze tracking may be implemented by detecting a point at which the user's gaze gazes in an image shown to the user, the angle of the user's head, and the angle of the eyeball through the camera 420 mounted in the device 100. It can be implemented by specifying movement and tracking the gaze point of the subject.

5 is an exemplary diagram for explaining setting a first region of interest.

The apparatus 100 may set one or more blocks partitioned based on the change in capacitance and eye-tracking information as the first ROI.

1 and 5 , after dividing one or more blocks, the apparatus 100 determines a first ROI 520-1 based on the change in capacitance and gaze tracking information within the play area 510. can be set

Here, the first region of interest 520 - 1 may mean a region that the user is carefully observing within the corresponding reproduction region 510 .

For example, when the playback resolution (P_play) is 360p, 640 pixels wide and 360 pixels long, and the inverses of the determining factors D_row and D_col are 3, respectively, the device 100 can partition 25,600 blocks. At this time, if the user touches a block corresponding to the 50th horizontally and 50th vertically, and as a result of eye tracking, gaze tracking information is acquired by looking at the same block, the device 100 selects a block corresponding to the 50th horizontally and vertically. It can be set as the first ROI 520-1.

On the other hand, if the device 100 obtains eye-tracking information by touching a block corresponding to the 50th horizontally and 50th vertically, and looking at a block corresponding to the 50th horizontally and 40th vertically as a result of the eye tracking, the horizontal An area corresponding to the 50th and 40th vertical positions may be set as the first ROI 520 - 2 .

As such, the first ROI may be independently generated based on capacitance change and gaze tracking information, and the same block may be set as the ROI if the user is touching and looking at the same block.

The apparatus 100 may adjust the resolution of the first ROI of the video streamed in the play area according to the Internet download speed. One of the factors that determines whether or not there is a break in video streaming is the speed of the Internet, and in particular, whether or not there is a break in video streaming and the upper limit of the resolution that can be viewed can be determined according to the download speed.

For example, SD-level videos of 360p and 480p require a sustained download speed of at least 1 Mbps, and HD-level or higher videos such as 720p and 1080p require a continuous download speed of 2.5 Mbps or higher for smooth streaming. can

At this time, if the download speed is 1 Mbps when the display resolution (P_disp) and the video resolution (P_max) are at the level of 1080p, even if the user sets the playback resolution (P_play) to 1080p through the device 100, the video is normally played. It cannot be streamed, and can be played by automatically lowering the playback resolution (P_play) according to the performance of the device 100.

The device 100 increases the resolution from the first region of interest of the video streamed in the play area according to the Internet download speed, and sequentially increases the resolution of the other parts, so that the playback set by the user or determined by the device 100 according to the environment is performed. In spite of the resolution (P_play), it is possible to realize a seamless streaming environment while enjoying a video in high resolution only for the region of interest.

For example, if a drop in Internet download speed is detected while a video with a video resolution (P_max) of 720p is streaming, and the playback resolution (P_play) cannot be maintained at the preset level of 720p, the playback resolution (P_play) is set to 720p. It is changed to the lowest level that can be streamed within the download speed of , (eg, 144p), and as the download speed of the Internet increases, the resolution of the first region of interest can be first adjusted and streamed.

Through this, users can experience rapid picture quality improvement in the area they are watching in the playback area and receive an uninterrupted streaming environment despite changes in the surrounding environment, such as changes in Internet speed.

Also, when there is more than one first ROI, the device 100 classifies the first ROI into one or more interest groups, and based on coordinate information of the first ROI included in the classified interest groups, the central first ROI. The coordinates of the region may be calculated, and the resolution of the reproduced image may be adjusted according to the distance between the center of the first region of interest and the center of the reproduction region. Meanwhile, the center of the play area may mean the coordinates of the center of an area where a video is streamed.

For example, the central first region of interest may be calculated through the centroid according to the Euclidean distance or the coordinates of the center of gravity between several coordinates, and includes the central first region of interest close to the center of the reproduction region according to the speed of the Internet. Image quality can be improved starting from interest groups.

6 is an exemplary diagram for explaining calculation of a distance between a central first region of interest and a reproduction region in an interest group.

For example, when there is one or more first ROIs, the apparatus 100 may configure at least one or more interest groups such that an average value of a distance between the center of the first ROI and each first ROI is the smallest. can be classified as In addition, after classifying the first ROI into one or more interest groups, the apparatus 100 may obtain centers between the first ROI included in the interest groups.

Hereinafter, for convenience of description, coordinates for distinguishing blocks will be treated as the same as coordinates on the x-y plane.

Referring to FIG. 6 , there are three first regions of interest in the interest group 630, the coordinates of the first region of interest 620-1 are (50, 49), and the first region of interest 620-2 The coordinates of are (49,51), and the coordinates of the first region of interest 620-3 are (51,51).

In this case, the coordinates of the center of the three coordinates of the first region of interest may be calculated through the formula below, and the calculated center coordinate value may be rounded to integer.

[Equation 7]

The calculated coordinates of the center of the interest group are (50, 50.3), and the device 100 rounds them off to determine a block corresponding to the position of (50, 50) as the center first region of interest 640 .

Meanwhile, the center of the first ROI 640 and the center of the reproduction area 650 are calculated through the following formula.

[Equation 8]

In this way, the apparatus 100 calculates the distance between the center 650 of the playback area and the center first ROI 640, and selects the center first ROI 640 closest to the center 650 of the playback area. Image quality can be improved starting from the interest group that is included.

Meanwhile, the apparatus 100 may reset one or more blocks partitioned according to the change in capacitance and the frequency at which gaze tracking information is detected as the first ROI.

For example, the user may touch or look at various parts of the play area during video streaming, and the device 100 continuously adjusts the resolution of the video displayed in the play area 510 according to the user's video viewing pattern so that the user By displaying a high-resolution video in the area that the user is interested in and playing the video in a relatively low resolution in the other areas, the user can experience seamless streaming of the high-resolution video at all times.

The apparatus 100 may recognize an object of a streamed image based on the quality information of the video and set one or more blocks including the object as the second region of interest.

7A and 7B are exemplary diagrams for explaining an example of setting a second region of interest based on video quality information.

Referring to FIG. 7A , when a user streams a video, at least one distinguishable object 720 - 1 or 720 - 2 may be displayed in the play area 710 of the device 100 .

For example, the device 100 may recognize an object using deep learning based on video quality information. For example, the device 100 may classify an object in an image based on a CNN (Convolution Neural Network) based object recognition learning model. In this case, the learning model may recognize an object in an image by analyzing characteristics of various objects based on a previously input learning image, and for example, the learning model may perform object recognition using image quality information of the image.

For example, the quality of a video being streamed may include quality information such as color balance, color reproducibility, contrast, and sharpness, and a region in which the quality information of the video has a rapid change is detected and at least one of the partitioned blocks is removed. 2 can be set as a region of interest.

Referring to FIG. 7A , various objects 720-1 and 720-2 may be displayed in the play area 710, and image quality information may rapidly change at the boundary of the objects. For example, if the person 720-1 is black, the dog 720-2 is gray, and all other areas of the play area 710 are white, image quality information, that is, color information, color information, is based on the boundary between the two objects. Balance and contrast can change drastically. For example, the RGB of the pixels constituting the human 720-1 may be '000000', the RGB of gray constituting the dog 720-1 may be '999999', and the RGB of the pixels constituting the background may be '' FFFFFF'.

In this case, the apparatus 100 may recognize the corresponding region as the boundary of the object when the RGB values of adjacent pixels change by 15% or more, and set a block in which the object recognized through the boundary exists is present as the second region of interest. .

After the second region of interest is set, the device 100 may adjust the resolution of an image reproduced in the second region of interest of the reproduction region according to the Internet download speed.

The apparatus 100 adjusts the resolution of the image reproduced in the second region of interest of the reproduction region according to the Internet download speed, thereby increasing the resolution of the region where the object detected in the image is located, and sequentially increasing the resolution of the region where the object is not detected in the image, thereby improving the environment. In spite of the playback resolution (P_play) set by the user or determined by the device 100 according to , a seamless streaming environment can be implemented while enjoying a video at a high resolution only in the region of interest.

For example, if a video with a video resolution (P_max) of 720p cannot be maintained at a level of 720p because the download speed of the Internet drops below 1Mpbs while streaming a video, blocks other than the second region of interest are playable. It is reproduced with the lowest quality (eg, 144p), and as the download speed of the Internet increases, the quality of the second region of interest can be improved first and then reproduced through streaming. Through this, the user can quickly improve the quality of the area watched in the playback area and watch the video without interruption despite changes in the surrounding environment such as changes in Internet speed.

In addition, when there is more than one second ROI, the device 100 classifies the second ROI into one or more interest groups, and coordinates of the second ROI at the center among the second ROIs included in the classified interest groups. , and the resolution of the reproduced image may be adjusted according to the distance between the central second region of interest and the center of the reproduction region.

For example, the central second region of interest may be calculated through the centroid according to the Euclidean distance or the coordinates of the center of gravity between several coordinates, and includes the central second region of interest close to the center of the reproduction region according to the speed of the Internet. Image quality can be improved starting from interest groups.

Meanwhile, the apparatus 100 may set the first ROI and the second ROI independently and then classify interest groups including the first ROI and the second ROI. In this case, the apparatus 100 classifies the first ROI and the second ROI into one or more interest groups, and determines the coordinates of the central ROI among the first and second ROI included in the classified interest groups. It is possible to calculate the resolution of the reproduced image according to the distance between the central region of interest and the center of the reproduction region.

For example, the central region of interest can be calculated through the centroid according to Euclidean distance or the coordinates of the center of gravity between several coordinates, and according to the speed of the Internet, the interest group including the central region of interest close to the center of the reproduction area can be calculated. can improve

The calculation of the second center ROI, the calculation of the distance between the center of the second ROI from the center of the playback area, and the calculation of the center ROI and the calculation of the distance between the center of the ROI from the center of the playback area are the first ROI described above with reference to FIG. 6 . Since the method of calculating the center of gravity of the region of interest and classifying the interest group including the first region of interest is the same, duplicate descriptions will be omitted.

Meanwhile, the device 100 may reset the second region of interest based on the quality information of the video. 8A and 8B are exemplary diagrams for explaining resetting of a second region of interest based on video quality information.

Referring to FIGS. 8A and 8B , the device 100 detects color information of a video being played in the play area 810 in units of predetermined frames, and detects specific color information in unit frames of the video being streamed in the play area 810. The second region of interest may be reset when changes in excess of a predetermined ratio.

For example, the RGB value '000000' occupies 10% of the playback area 810 due to the object 820 displayed in the nth frame of the streaming video, and the object 830 displayed in the n+mth frame If the RGB value of '000000' occupies 20% in the playback area 810, the device 100 resets the second region of interest and streams the video so that objects appearing in the video can be reproduced at a higher resolution. there is.

9 is an exemplary hardware configuration diagram of the seamless video streaming device according to FIG. 1 .

Referring to FIG. 9 , a seamless video streaming device 900 includes at least one processor 910; and a memory 920 storing instructions instructing the at least one processor 910 to perform at least one step.

Here, the at least one processor 910 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor for performing methods according to embodiments of the present invention. can Each of the memory 920 and the storage device 960 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 920 may include at least one of a read only memory (ROM) and a random access memory (RAM).

In addition, the seamless video streaming device 900 may include a transceiver 930 that communicates through a wireless network. In addition, the seamless video streaming device 900 may further include an input interface device 940 , an output interface device 950 , a storage device 960 , and the like. Each component included in the seamless video streaming device 900 is connected by a bus 970 to communicate with each other.

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention or may be known and usable to those skilled in computer software.

Examples of computer readable media may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter and the like. The hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or device may be implemented by combining all or some of its components or functions, or may be implemented separately.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

Claims (5)

In the seamless video streaming playback device,
processor; and
Including a memory (memory) connected to the processor,
The processor:
Determine the number of blocks to divide the play area based on the display resolution, the video resolution, and the playback resolution;
Partitioning the play area into one or more blocks according to the determined number of blocks to be partitioned;
detecting a change in capacitance input from at least one touch sensor through a touch input tool;
Acquire eye tracking information of a user input through a camera;
Setting one or more partitioned blocks as a first region of interest based on the change in capacitance and gaze tracking information;
adjusting the resolution of the first region of interest according to an Internet download speed;
resetting a first region of interest according to a change in capacitance and a frequency at which gaze tracking information is detected in the one or more partitioned blocks;
The processor:
When there is one or more first regions of interest, classifying the first regions of interest as interest groups;
calculating coordinates of a central first ROI corresponding to a center of the interest group based on coordinate information of first ROIs included in the interest group;
characterized in that, according to the distance between the central first ROI and the center of the playback area, a resolution is adjusted from an interest group including the central first ROI close to the center of the playback area, characterized in that, seamless video streaming playback Device. In claim 1,
the processor,
Recognize an object in a video streamed based on the quality information of the video,
Characterized in that, the one or more partitioned blocks including the recognized object are set as a second region of interest, seamless video streaming playback device. In claim 2,
the processor,
Seamless video streaming that detects color information of a video streamed in the play area in units of predetermined frames, and resets the second region of interest when specific color information in unit frames of the play area changes by exceeding a predetermined rate. playback device. In claim 1,
the processor,
When a decrease in the Internet download speed is detected, the playback resolution is changed to the lowest level possible for streaming within the reduced Internet download speed;
The seamless video streaming reproducing apparatus characterized in that the resolution of the first region of interest is firstly increased according to the increase in the Internet download speed. delete

Images