KR102596308B1 - Electronic apparatus and method for intelligent video conversion - Google Patents

Abstract

The present invention relates to an electronic device and method for intelligent image conversion. An electronic device according to an embodiment of the present invention is an electronic device that converts and reproduces an image, comprising: a sensor unit that detects a sensor value related to a holding state of the electronic device; and a control unit that performs conversion on the image according to the grip state according to the sensor value and the image ratio of the image, wherein the control unit includes: a first conversion that improves image quality while maintaining the content of the image; At least one of a second transformation that adds new video content to the video content and a third transformation that improves image quality while enlarging at least a portion of the video content is performed.

Description

Electronic device and method for intelligent video conversion {ELECTRONIC APPARATUS AND METHOD FOR INTELLIGENT VIDEO CONVERSION}

The present invention relates to an electronic device and method related to video playback, and more specifically, to an electronic device and method for adaptively converting video according to the holding state of a terminal and the video ratio of the video (hereinafter referred to as “video”). It's about.

According to the conventional aspect ratio conversion technology (hereinafter referred to as “the prior art”), when playing a video on a terminal, the screen ratio (i.e., the terminal Play the video at the maximum size according to the size of the screen (ratio between width and height).

Figure 1 shows examples of playing a 16:9 video ratio video in terminals with various screen ratios using the prior art.

Accordingly, when the prior art is applied, an empty space is created on the top and bottom or left and right sides of the terminal screen, and the empty space is colored black. That is, referring to Figure 1, when a video with a 16:9 aspect ratio is played on a terminal with a 4:3 screen ratio, a black image called a letter box is inserted into the empty space created at the top and bottom of the terminal screen. Additionally, when a 16:9 aspect ratio video is played on a 21:9 screen ratio terminal, a black image called a pillar box is inserted into the empty space on the left and right sides of the terminal screen.

In other words, the prior art is simply a technology that matches the screen of the terminal while maintaining the video ratio, so when the screen ratio of the terminal does not match the video ratio, the user's visual satisfaction is low due to the use of excessive letter boxes or fillers. There is a problem with losing. In particular, when the screen state of a terminal to which the prior art is applied is portrait (the screen ratio of the terminal is longer than width), and a horizontal video (an image with an aspect ratio that is longer than vertical) is played, the corresponding screen As horizontal images are scaled down and played to match the ratio, these problems are bound to become more prominent.

KR10- 2007-0039790 A

In order to solve the problems of the prior art as described above, the purpose of the present invention is to provide a technology for adaptively converting images according to the holding state of the terminal and the aspect ratio of the image.

However, the problem to be solved by the present invention is not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An electronic device according to an embodiment of the present invention for solving the above problems is an electronic device that converts and reproduces images, comprising: a sensor unit that detects a sensor value related to a holding state of the electronic device; and a control unit that performs conversion on the image according to the grip state according to the sensor value and the image ratio of the image.

The control unit performs a first transformation to improve picture quality while maintaining the content of the video, a second transformation to add and synthesize new video content to the content of the video, and a video that enlarges at least a portion of the content of the video but has a longer length. At least one of the third transformations may be performed to improve image quality while enlarging the vertical image at a certain ratio.

The control unit may perform the first conversion or the second conversion on a horizontal image with a longer horizontal aspect ratio when the display is in a horizontal holding state.

The control unit may perform the third transformation on the horizontal image when the screen is in a vertical position with a longer screen ratio.

The control unit may perform the first transformation or the second transformation on a vertical image with a longer vertical aspect ratio when the image is in the horizontal holding state or the vertical holding state.

The control unit performs the second conversion on the horizontal image when the horizontal image must be played by filling the area of a pillar box or letter box in the horizontal holding state, and performs the second transformation on the horizontal image, and fills the horizontal image with a pillar box ( When the pillar box or letter box area is unnecessary, the first transformation can be performed on the horizontal image.

When performing the first conversion, the control unit generates an enlarged image by applying a previously learned machine learning model to the image to enlarge the image to an image larger than the screen size of the grasped state while improving image quality, and then generates an enlarged image. It can be converted to the screen ratio of the grip state by performing size interpolation.

When performing the second conversion, the control unit may synthesize new image content included in the previous or next frame image into the current frame image based on corresponding point matching using the previous and next frame images.

When performing the second transformation, the control unit performs boundary expansion based on a generative adversarial network (GAN) on the current frame image if the synthesized current frame image is less than the resolution of the target image. New video content can be synthesized.

When performing the second conversion, the control unit generates an enlarged image by applying a previously learned machine learning model to the synthesized current frame image to enlarge the image into an image larger than the screen size of the grasped state while improving image quality. Afterwards, size interpolation can be performed on the enlarged image to convert it to the screen ratio of the holding state.

When performing the third conversion, the control unit analyzes the contents of the frame image for each frame image of the horizontal image to calculate a playback area corresponding to a portion of the frame image, and divides the horizontal image into a plurality of small units. A process of separating and selecting an optimal AI model applied to each subunit according to the content of the horizontal image within the subunit from among a plurality of artificial intelligence (AI) models previously learned for each type of content of the image, and selecting the optimal AI model applied to each subunit based on the playback area of the horizontal image. A process of extracting a vertical image with a longer aspect ratio for each frame image and a process of enlarging and converting the extracted vertical image by applying the selected optimal AI model to each subunit can be performed.

In the calculation process, the control unit detects an area for an object and a face in each frame image, calculates a maximum playback area including the detected area, and performs a cropping process on the calculated maximum playback area to view horizontally. At least one playback area having a second vertically longer video ratio can be calculated.

The maximum playback area may be an area that includes all of the detected areas when there are multiple detected areas.

Each of the AI models may be a model learned to generate enlarged images with improved quality from low-quality images of different content according to machine learning techniques.

The optimal AI model can improve image quality by at least one of increasing resolution, removing noise, and increasing dynamic range.

The control unit can control the image enlarged according to the third transformation to be played back in all pixels of the display.

A method according to an embodiment of the present invention is a method for converting and playing back an image in an electronic device, the method comprising: detecting a sensor value related to a holding state of the electronic device; and performing conversion on the image according to the grip state according to the sensor value and the image ratio of the image.

The step of performing the transformation includes performing a first transformation to improve image quality while maintaining the content of the image, and performing a second transformation to add new image content to the content of the image. At least one step may be included among the steps of performing a third transformation to improve image quality while enlarging at least a portion of the video content.

The step of performing the conversion is to perform the first conversion or the second conversion on the horizontal image with the longer horizontal aspect ratio when the horizontal image is in a horizontal holding state, and the second conversion is performed on the horizontal image with the longer horizontal aspect ratio. In the case of the vertical holding state, the third transformation is performed on the horizontal image with a longer horizontal aspect ratio, and in the case of the horizontal holding state or the vertical holding state, the third transformation is performed on the vertical image with the longer vertical aspect ratio. It may include performing the first transformation or the second transformation.

The step of performing the first conversion includes generating an enlarged image by applying a previously learned machine learning model to the image to enlarge the image into an image larger than the screen size of the grasped state while improving image quality, and then generating an enlarged image. It may include performing size interpolation on and converting to the screen ratio of the grip state.

The step of performing the second transformation may include synthesizing new image content included in the previous or next frame image into the current frame image based on corresponding point matching using the previous and next frame images.

In the step of performing the second transformation, when the synthesized current frame image is less than the resolution of the target image, a generative adversarial network (GAN)-based boundary expansion is performed on the current frame image to create a new image. Content may include steps.

The step of performing the second conversion includes generating an enlarged image by applying a previously learned machine learning model to the synthesized current frame image to enlarge the image into an image larger than the screen size of the grip state while improving image quality, It may include performing size interpolation on the enlarged image and converting it to the screen ratio of the holding state.

The step of performing the conversion may include performing the third conversion on a horizontal image with a longer horizontal aspect ratio when the image is in a vertical holding state with a longer aspect ratio.

The step of performing the third transformation includes calculating a playback area corresponding to a portion of the frame image by analyzing the content of the frame image for each frame image of the horizontal image; Separating the horizontal image into a plurality of subunits and selecting an optimal AI model applied to each subunit according to the content of the horizontal image within the subunit from among a plurality of artificial intelligence (AI) models already learned for each type of content of the image; extracting a vertical image with a longer vertical aspect ratio for each frame image from the horizontal image based on the playback area; and enlarging and converting the extracted vertical image by applying the selected optimal AI model for each subunit.

The present invention, constructed as described above, does not insert a black image (pillar box or letter box) when the aspect ratio of the image and the screen ratio of the electronic device that reproduces it are different, but processes it so that the aspect ratio of the image is adjusted. There is an advantage in that the video can be appropriately adjusted to the screen ratio of the electronic device being played.

In addition, the present invention not only effectively performs conversion on the aspect ratio of an image by adaptively performing at least one of the first to third conversions according to the holding state of the electronic device and the aspect ratio of the image. There is an advantage in achieving higher resolution of the converted video.

In addition, the present invention not only minimizes letter boxes or pillar boxes when playing video, but also enlarges the video while including the main objects, and reproduces the video in high definition by improving the low-quality problem that occurs during enlargement. , it has the advantage of increasing the user's visual satisfaction.

In addition, the present invention allows playback using the screen of the electronic device as much as possible, which not only increases viewing immersion, but also has a large exposure effect, so when the played video is an advertisement, the advertisement effect is great.

In addition, since the present invention can be applied to various image quality improvement techniques, it has the advantage of being applicable not only to video on demand (VOD) but also to real-time streaming.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

Figure 1 shows examples of playing a 16:9 video ratio video in terminals with various screen ratios using the prior art.
Figure 2 shows a block diagram of an electronic device 100 according to an embodiment of the present invention.
Figure 3 shows a flowchart of a method according to one embodiment of the present invention.
Figure 4 shows a flow chart for the first conversion performed by the control unit 160.
Figure 5 shows examples of various enlargements of an image.
Figure 6 shows a flow chart for the second conversion performed by the control unit 160.
Figure 7 shows an example of the second transformation.
Figure 8 shows a flowchart for the third conversion performed by the control unit 160.
Figure 9 shows a more detailed flow chart for S310 in the third transformation.
Figure 10 shows an example of selecting an optimal AI model in S320.
Figure 11 shows a comparison example of playing a horizontal image in an electronic device held vertically.
FIG. 12 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is held horizontally in step S20 of the method according to an embodiment of the present invention.
FIG. 13 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is held vertically in step S20 of the method according to an embodiment of the present invention.

The above purpose and means of the present invention and the resulting effects will become clearer through the following detailed description in conjunction with the accompanying drawings, and thus the technical idea of the present invention will be easily understood by those skilled in the art. It will be possible to implement it. Additionally, in describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

The terminology used herein is for describing embodiments and is not intended to limit the invention. In this specification, singular forms also include plural forms, as appropriate, unless specifically stated otherwise in the context. In this specification, terms such as “comprise,” “provide,” “provide,” or “have” do not exclude the presence or addition of one or more other components other than the mentioned components.

In this specification, terms such as “or” and “at least one” may represent one of words listed together, or a combination of two or more. For example, “or B” “and at least one of B” may include only A or B, or both A and B.

In this specification, descriptions under “for example” and the like may not exactly match the information presented, such as cited characteristics, variables, or values, and may be subject to tolerances, measurement errors, limits of measurement accuracy and other commonly known factors. Effects such as modifications, including, should not limit the embodiments of the invention according to various embodiments of the present invention.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in between. It must be understood that it may be possible. On the other hand, when a component is mentioned as being 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

In this specification, when a component is described as being ‘on’ or ‘in contact with’ another component, it may be in direct contact with or connected to the other component, but there may be another component in between. It must be understood that it can be done. On the other hand, if a component is described as being 'right above' or 'in direct contact' with another component, it can be understood that there is no other component in the middle. Other expressions that describe the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. Additionally, the above term should not be interpreted as limiting the order of each component, but may be used for the purpose of distinguishing one component from another component. For example, a 'first component' may be named a 'second component', and similarly, a 'second component' may also be named a 'first component'.

Unless otherwise defined, all terms used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

Figure 2 shows a block diagram of an electronic device 100 according to an embodiment of the present invention.

The electronic device 100 according to an embodiment of the present invention is a device that converts and reproduces images. At this time, the video may refer to a video, and may be a pre-stored video or a video transmitted from another device (server). For example, the service provided by the electronic device 100 may be a service that plays pre-stored images, video on demand (VOD), or real-time streaming, but is not limited thereto.

In particular, the video is either a video with content with an aspect ratio that is longer than the width (hereinafter referred to as “horizontal video”), or a video with content with an aspect ratio that is longer than width (hereinafter referred to as “vertical video”). may refer to it). In addition, the holding state is a state in which the electronic device 100 is given by the user, and the display 130 is in a state where the screen ratio is longer than the vertical (hereinafter referred to as “horizontal holding state”), or the display 130 is in a state where the screen ratio is longer than the width. It may be in a state of a longer screen ratio (hereinafter referred to as “vertical grip state”).

In other words, the electronic device 100 can perform various transformations on the image and then play it, depending on the current grip state and the image ratio of the image (i.e., whether it is a horizontal image or a vertical image).

At this time, the electronic device 100 may be a terminal capable of computing. For example, the electronic device 100 may be a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, or a personal digital assistant (PDA). It may be a digital assistant, a smart phone, a smart pad, or a mobile phone, but is not limited thereto.

As shown in FIG. 2 , the electronic device 100 may include an input unit 110, a communication unit 120, a display 130, a memory 140, and a control unit 160.

The input unit 110 generates input data in response to various user inputs and may include various input means.

For example, the input unit 110 includes a keyboard, key pad, dome switch, touch panel, touch key, touch pad, and mouse. It may include (mouse), menu button, etc., but is not limited thereto.

The communication unit 120 is a component that performs communication with other devices such as servers, and can receive information about bitstreams for images, information about pre-learned models (AI models, machine learning models, GAN models, etc.) from other devices. You can.

For example, the communication unit 120 supports 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication may be performed, or wired communication such as cable communication may be performed, but is not limited thereto.

The display 130 displays various image data on a screen and may be composed of a non-emissive panel or an emissive panel. Additionally, the display 230 can display the converted image according to the grip state and the aspect ratio of the image.

For example, the display 130 may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a micro electromechanical system (MEMS). It may include, but is not limited to, a mechanical systems display, an electronic paper display, etc. Additionally, the display 130 may be combined with the input units 120 and 220 and implemented as a touch screen or the like.

The memory 140 stores various information necessary for the operation of the electronic device 100. Information stored in the memory 140 may include, but is not limited to, images, models, converted images, and program information related to methods to be described later. In particular, a plurality of AI models may be stored and may be stored in compressed form, but are not limited thereto.

For example, the memory 140 is classified into hard disk type, magnetic media type, CD-ROM (compact disc read only memory), and optical media type depending on its type. ), magneto-optical media type, multimedia card micro type, flash memory type, ROM type (read only memory type), or random RAM type. access memory type), etc., but is not limited thereto. Additionally, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The sensor unit 150 detects status information of the electronic device 100 or its surroundings. This sensor unit 150 may include various sensors such as an acceleration sensor, a gyro sensor, a proximity sensor, an RGB sensor, a brightness sensor, a hall sensor, a motion sensor, a temperature/humidity sensor, a barometer, and a geomagnetic sensor.

In particular, the electronic device 100 may be in various holding states, and the sensor unit 150 may detect these. That is, the sensor unit 150 may include a sensor that detects sensor values related to the gripping state of the electronic device 100, such as a gyro sensor or a motion sensor.

The control unit 160 can perform various control operations of the electronic device 100. That is, the control unit 160 can control the execution of a method to be described later, and the remaining components of the electronic device 100, namely, the input unit 110, the communication unit 120, the display 130, the memory 140, and the sensor unit. Operations such as (150) can be controlled.

For example, the control unit 160 may include a hardware processor or a software process executed on the processor, but is not limited thereto.

In particular, the control unit 160 can determine the gripping state of the electronic device 100 using the sensor value of the gripping state-related sensor of the sensor unit 150, and depending on the grasping state and the image ratio of the image, the image You can control the conversion performed.

At this time, the transformation for the image may be at least one of the first to third transformations. In other words, the first transformation is a transformation that enlarges the image while maintaining its content, but also improves the image quality. The second transformation is a transformation that expands by adding new video content to the video content. Of course, the second transformation may be a transformation in which the first transformation is additionally performed after new image content is additionally synthesized. Additionally, the third transformation is a transformation that enlarges at least a portion of the video content into a vertical image while also improving the image quality.

Table 1 below shows the types of conversion performed by the control unit 160 depending on the grip state and the aspect ratio of the image.

horizontal grip state Vertical grip state horizontal video First Transformation or Second Transformation third transformation vertical video First Transformation or Second Transformation First Transformation or Second Transformation

That is, when the electronic device 100 is held in a horizontal position and the image is a horizontal image, the control unit 160 may control the horizontal image to perform first or second conversion. At this time, the control unit 160 may perform the first transformation or the second transformation depending on whether the area of the pillar box is needed. In other words, when the horizontal image is held horizontally and the horizontal image needs to be played by filling the pillar box area (for example, when the resolution of the original horizontal image is insufficient and the converted horizontal image requires the pillar box area) ), the second transformation can be performed on the horizontal image. On the other hand, when the horizontal image is in a horizontal grip state and there is no need to fill the pillar box area in the horizontal image (for example, when the resolution of the original horizontal image is sufficient and the pillar box area is unnecessary in the converted horizontal image), the horizontal image A first transformation may be performed. Additionally, the control unit 160 may control the electronic device 100 to perform a third transformation on a horizontal image when the electronic device 100 is held in a vertical position.

Additionally, the control unit 160 may control the electronic device 100 to perform first or second transformation on a vertical image when the electronic device 100 is in a horizontal or vertical holding state. At this time, the control unit 160 may perform the first transformation or the second transformation depending on whether the area of the pillar box is needed. In other words, when a vertical video needs to be played by filling the pillar box area (for example, when the resolution of the original vertical video is insufficient and the converted horizontal or vertical video needs the pillar box area), the vertical video The second conversion may be performed. On the other hand, if there is no need to fill the pillar box area in the vertical image (for example, if the resolution of the original vertical image is sufficient and the pillar box area is unnecessary in the converted horizontal or vertical image), the first conversion is performed on the vertical image. It can be done.

Figure 3 shows a flowchart of a method according to one embodiment of the present invention.

The method according to an embodiment of the present invention is a method for converting and playing back an image in the electronic device 100, and includes steps S10 to S30, as shown in FIG. 3. At this time, the performance of S10 to S30 can be controlled through various hardware configurations of the control unit 160 or software processes.

First, the control unit 160 detects sensor values related to the holding state of the electronic device 100 and determines the holding state (S10). At this time, the control unit 160 may determine the gripping state of the electronic device 100 using the sensor value detected from the gripping state-related sensor of the sensor unit 150. In other words, it is possible to determine whether the electronic device 100 is held horizontally or vertically.

Thereafter, the control unit 160 adaptively performs conversion on the image according to the identified grip state and the image ratio of the image (S20). That is, the control unit 160 may perform at least one of the first to third transformations depending on the holding state (i.e., horizontal holding state/vertical holding state) and the image ratio (i.e., horizontal image/vertical image). there is. Of course, before performing S20, the control unit 160 determines whether the image pre-stored in the memory 140 or received through the communication unit 120 is a horizontal image or a vertical image, and then performs S20. .

Thereafter, the control unit 160 controls to reproduce the image on which at least one of the first to third transformations has been performed and display it on the display 130 (S30).

That is, in the case of a horizontal grip state and a horizontal image, the converted horizontal image in which the first or second conversion has been performed on the horizontal image may be displayed on the display 130. Additionally, in the case of a vertical grip state and a horizontal image, a high-definition vertical image obtained by performing a third transformation on the horizontal image may be displayed on the display 130. At this time, the enlarged and converted high-definition vertical image can be played in all pixels of the display 140 of the electronic device 100, as shown in FIG. 11, but is not limited to this. Additionally, in the case of a vertical image regardless of the grip state, a converted horizontal or vertical image obtained by performing the first or second conversion on the vertical image may be displayed on the display 130.

Of course, in S30, when playing an image on which at least one of the first to third conversions has been performed, the controller 160 may synchronize the audio for the image and control it to be output (played) from the electronic device 100. there is.

<First transformation>

When performing the first conversion, the control unit 160 generates an enlarged image by applying a previously learned first machine learning model to improve the image quality and enlarge the image to an image larger than the screen size in the grasped state, and then generates an enlarged image. By performing size interpolation on the enlarged image, it can be converted to fit the screen ratio of the grip state.

FIG. 4 shows a flow chart for the first transformation performed by the control unit 160, and FIG. 5 shows examples of various enlargements of an image.

Specifically, referring to FIG. 4 , the control unit 160 may perform steps S101 to S104 when performing the first conversion.

The control unit 160 calculates the aspect ratio between the image currently being converted (input image) and the image after conversion (target image) (S101). At this time, the target image is an image according to the current grip state of the electronic device 100.

That is, in S101, the control unit 160 can determine the number of horizontal and vertical pixels of the input image and the number of horizontal and vertical pixels of the target image, and calculate the ratio between each identified number of pixels. Of course, the control unit 160 may calculate the horizontal ratio (i.e., the ratio of the number of horizontal pixels between the input image and the target image) and the horizontal ratio (i.e., the ratio of the number of vertical pixels between the input image and the target image). For example, if the input image is 320×240 and the target image is 720×480, the control unit 160 may calculate the width/height ratio as 2.25/2, respectively.

Afterwards, the control unit 160 sets the horizontal/vertical enlargement ratios respectively (S102). At this time, the horizontal enlargement ratio is a ratio for enlarging the input image in the horizontal direction, and the vertical enlargement ratio is a ratio for enlarging the input image in the vertical direction. This horizontal/height enlargement ratio may be different from the horizontal/height ratio calculated in S101, and in particular, it may be set larger than the horizontal/height ratio calculated in S101. For example, if the horizontal/height ratios are calculated to be 2.25/2 in S101, the control unit 160 may set the horizontal enlargement ratio to be greater than 2.25 and the vertical enlargement ratio to be greater than 2.

Thereafter, the control unit 160 inputs the input image and the horizontal/vertical magnification ratio to the previously learned first machine learning model, thereby improving the image quality of the input image and generating an image enlarged at the horizontal/vertical magnification ratio. . At this time, the generated enlarged image is an image with a higher resolution than the target image (i.e., an image having the screen size of the current grip state).

If the conventional technology is applied, the image is simply enlarged and converted at a certain rate, so image quality deteriorates, such as low resolution, and the user's visual satisfaction is inevitably lowered. To solve this problem, the present invention uses a first machine learning model to enlarge and convert the input image and improve its image quality. In other words, the first machine learning model is a model learned according to machine learning techniques, and is a model learned to generate images with improved quality from low-quality images and to generate images enlarged according to the input horizontal/vertical magnification ratio. am.

Specifically, the first machine learning model is a model learned according to the machine learning technique of supervised learning through training data of input data and output data pairs (dataset). That is, the first machine learning model can be learned using training data including input data of a low-quality image and a horizontal/vertical enlargement ratio, and output data of an image enlarged according to the horizontal/vertical enlargement ratio but with improved image quality. Accordingly, the first machine learning model has a function for the relationship between the low-quality image and horizontal/vertical enlargement ratio as input data and the enlarged image with improved image quality as output data, and expresses this using various parameters.

For example, the first machine learning model can use parameters of weights and biases to express the relationship between a low-quality image and an image enlarged at the horizontal/vertical magnification ratio with improved image quality. Accordingly, when a low-quality input image and input data with a horizontal/vertical magnification ratio are input to the learned first machine learning model, the output data of the enlarged image with improved image quality is enlarged at the horizontal/vertical magnification ratio according to the function. can be printed.

At this time, the type of image quality improvement may be at least one of increasing resolution, removing noise, and increasing dynamic range compared to low-quality images. That is, when a low-quality image is input, the first machine learning model can output an image with improved image quality in any one of increased resolution, removal of noise, and increased dynamic range. However, since the input image must be enlarged and converted, it may be desirable to include an increase in resolution. For example, when a low-quality image and a horizontal/vertical magnification ratio are input, the first machine learning model outputs an image with improved image quality by increasing the resolution at the horizontal/vertical magnification ratio, or increases resolution at the horizontal/vertical magnification ratio and removes noise. You can output an image with improved image quality, or an image with improved image quality with increased resolution and dynamic range in the horizontal/vertical magnification ratio.

Thereafter, the control unit 160 may perform size interpolation by applying a resolution transformation technique to the enlarged image generated in S103, thereby converting it into an image that matches the screen ratio of the current grip state of the electronic device 100. For example, the resolution modification technique may be bilinear, bicubic interpolation, down-sampling, etc., but is not limited thereto.

In particular, referring to Figure 5, in the case of the conventional deep learning-based screen enlargement technique, there is a limitation that enlargement can only be done at a fixed ratio for width and height. In order to improve this limitation, the present invention sets the horizontal/vertical magnification ratio to be larger than the resolution of the target image, generates an image larger than the target image through a first machine learning model, and then uses traditional resolution transformation techniques (Bilinear, bicubic By additionally applying size interpolation (interpolation, down-sampling, etc.), it is possible to finely correct the screen ratio of the generated enlarged image.

<Second Transformation>

Meanwhile, when performing the second conversion, the control unit 160 can enlarge and convert the current frame image by combining new image content with the current frame image based on corresponding point matching using the previous and next frame images. Of course, if the current frame image enlarged and converted in this way is less than the resolution of the target image, the control unit 160 uses a border extension technique (border extension) based on a generative adversarial network (GAN) for the current frame image. ), new video content can be synthesized into the current frame video.

FIG. 6 shows a flow chart for the second conversion performed by the control unit 160, and FIG. 7 shows an example of the second conversion. In Figure 7, F(N) is the current frame image, F(N-1) is the previous frame image, and F(N+1) is the next frame image.

Specifically, referring to FIG. 6 , the control unit 160 may perform steps S201 to S202 when performing the second conversion.

That is, the control unit 160 uses F(N-1), F(N), and F(N+1) to change the new image content included in F(N-1) or F(N+1) to F. F'(N) can be synthesized by adding to (N) (S201). At this time, the control unit 160 finds the corresponding points of F(N-1) and F(N+1) that match in F(N), and a part of F(N-1) or F(N+1) according to the corresponding points. You can create F(N)' by adding an image to F(N). In other words, geometric synthesis can be performed by adding some images from the remaining portions of F(N-1) and F(N+1), excluding the portion overlapping with F(N), to F(N).

Afterwards, if F'(N), which has a resolution greater than F(N), falls short of the resolution of the target image, the control unit 160 performs GAN-based boundary expansion on F'(N) to create new image contents. F''(N) can be synthesized by adding to F'(N) (S202). That is, the control unit 160 can generate F''(N) that further expands the image of the edge portion of F'(N) by inputting F'(N) into the GAN model previously learned according to the GAN technique. there is.

This GAN technique uses two predefined network models, a generator (G) and a classifier (D). In other words, it is a method of learning the classifier (D) first, then the generator (G), and learning repeatedly by exchanging results with each other. This is a method in which the generator (G) and classifier (D) compete with each other and learn little by little. . At this time, in the case of the classifier (D), it receives the actual input image (real data) and learns to classify the input image as real (real), and then, on the contrary, the synthetic input image (fake data) generated by the generator (G) ) can be input and learned to classify the input image as synthetic (fake).

In particular, in the case of the generator (G), it can be trained to receive an input image and generate an image that expands the edges of the input image. At this time, fake data generated by the generator (G) can be input into the discriminator (D), and the generator (G) can be trained to produce data similar to real data enough to classify the fake data as real. In this way, F''(N) can be generated by using the sufficiently learned generator (G) as a GAN model and inputting F'(N).

For example, in addition to general GAN, GAN techniques include DCGAN (Deep Convolutional GAN), LSGAN (Least Squares GAN), SGAN (Semi-Supervised GAN), ACGAN (Auxiliary Classifier GAN), WGAN (Wasserstein Generative Adversarial Networks), and ConGAN (Continuous GAN), cGAN (Conditional GAN), SNcGAN (Spectral Normalization Conditional GAN), starGAN, etc., but are not limited thereto.

<The Third Transformation>

Next, when performing the third transformation, the control unit 160 performs various image processing on the horizontal image so that it can be played in a vertical holding state. That is, the control unit 160 can convert the horizontal image to enlarge at least a portion of the content of the horizontal image to a vertical image while improving image quality. For example, the control unit 160 can change the horizontal image into a vertical image by cutting out the remaining main content portion, and enlarging and converting the changed vertical image to fit the screen size in the vertically held state.

Figure 8 shows a flowchart for the third conversion performed by the control unit 160.

Specifically, referring to FIG. 8, the control unit 160 may perform steps S310 to S340 when performing the third conversion. Of course, the order of S310 and S320 may be changed or performed in parallel at the same time.

First, the control unit 160 analyzes the content of the frame image for each frame image of the horizontal image and calculates a playback area corresponding to a portion of the frame image (S310). That is, the control unit 160 can calculate the playback area by performing image content analysis on the horizontal image.

At this time, the playback area is calculated for each frame image for the horizontal image, and is a partial area in the corresponding frame image, and is an area corresponding to the main content of the corresponding frame image. For example, when the electronic device 100 cuts out part of the horizontal image and changes it to a vertical image for playback in a vertical holding state, the corresponding playback area in the horizontal image is the main content, so it is not cut out and only the remaining portion is cut out. That is, the playback area may be referred to as an area that must be included in the vertical image when changing a horizontal image to a vertical image in the electronic device 100.

For example, if a horizontal image has 1000 frames, information on 1000 playback areas, one for each 1000 frames, can be calculated.

Figure 9 shows a more detailed flow chart for S310 in the third transformation.

Specifically, referring to FIG. 9, the control unit 160 detects areas for objects and faces in each frame image of the horizontal image (S311). At this time, the control unit 160 may detect an area for an object and an area for a face in each frame image using an object detector and a face detector. That is, the object detector can detect key objects, and the face detector can detect the faces of key characters. At this time, each detector detects each area by applying various algorithms related to object detection, and may be stored in the memory 140.

For example, each detector may be a detector using Canny Edger, Harris corner, Haar-like feature, HOG (Histogram of Oriented Gradient), SIFT (Scale Invariant Feature Transform), or machine learning model, but is limited thereto. no.

Afterwards, the control unit 160 calculates an area (hereinafter referred to as “maximum playback area”) including each detected area (S312). That is, since each area detected in S311 corresponds to a candidate area that can be a playback area, the maximum playback area including all of them is calculated. For example, if there are multiple areas detected in S311, each detected area may be included in the maximum playback area.

In particular, by using an object detector and a face detector instead of a single detector in S311 and calculating the maximum playback area to include all detection areas in S312, the present invention further improves the accuracy of video content analysis for each frame image. You can.

Thereafter, the control unit 160 calculates a playback area having the aspect ratio of the vertical image in each frame through a cropping process for the calculated maximum playback area (S313). That is, considering the type of screen ratio (1:1, 4:5, 9:16, 10:21, etc.) in the vertical holding state that the electronic device 100 can have, the maximum playback area is set to fit the screen ratio. Perform the cut process for . Of course, the screen ratio that the electronic device 100 can have in a vertically held state may be a screen ratio where the height is longer than the width, and the screen ratio where the height and width are the same.

For example, the cutting process may be performed to cut out a certain area of the maximum playback area (e.g., an area for a specific type of object or face) and other areas, but is limited to this. That is not the case.

Afterwards, the control unit 160 divides the horizontal image into a plurality of subunits and selects the optimal model applied to each subunit according to the content of the horizontal image within the subunit among the plurality of AI models pre-learned for each content type of the image and stored in the memory 140. Select an AI model (S320).

At this time, the control unit 160 may separate the horizontal image into multiple small units using various scene change detection algorithms. For example, the control unit 160 may calculate the difference value between neighboring frames and determine that a shot change has occurred when the calculated difference value is greater than a specific reference value, thereby distinguishing each subunit.

Of course, the number of frames included in each separated subunit may not be constant. That is, the number of frames of the first subunit and the number of frames of the second subunit may be the same or different. For example, subunits may be divided according to shots, scenes, or sequences, but are not limited to this. However, it may be desirable for the small unit to be a unit larger than the frame. That is, each subunit may include a plurality of frames.

Meanwhile, the AI model is a model applied when enlarging and converting a changed vertical image to fit the screen size of the electronic device 100. If the conventional technology is applied, the changed vertical image is simply enlarged and converted at a certain rate, so image quality deteriorates, such as low resolution, and the user's visual satisfaction is inevitably lowered. To solve this problem, in the present invention, by using an AI model, it is possible to enlarge and convert a vertical image and improve the image quality. In other words, the AI model is a machine learning model learned according to machine learning techniques, and is a model learned to generate enlarged images with improved quality from low-quality images.

Specifically, the AI model is a machine learning model learned according to the machine learning technique of supervised learning through training data of input data and output data pairs (datasets). In other words, the AI model can be learned using training data including input data of low-quality images and output data of enlarged images with improved quality. Accordingly, the AI model has a function for the relationship between the low-quality image as input data and the enlarged image with improved quality as output data, and expresses this using various parameters.

For example, an AI model can express the relationship between low-quality images and enlarged images with improved quality using parameters of weights and biases. Accordingly, when input data of a low-quality image (e.g., a changed vertical image) is input to the learned AI model, an enlarged image with improved image quality according to the function (e.g., enlarged to the screen size of the electronic device 100 but in high definition) Output data of the converted vertical video may be output.

At this time, the type of image quality improvement may be at least one of increasing resolution, removing noise, and increasing dynamic range compared to low-quality images. In other words, when a low-quality image is input, the AI model can output an image with improved image quality in any one of increased resolution, removal of noise, and increased dynamic range. However, since the vertical image must be enlarged and converted, it may be desirable to include an increase in resolution. For example, when a low-quality image is input, the AI model can output an image with improved image quality with increased resolution, an image with improved image quality with increased resolution and noise removal, or an image with improved image quality with increased resolution and increased dynamic range.

Meanwhile, a plurality of AI models may be stored in the memory 140 or another device connected to the electronic device 100. At this time, each AI model may be a model learned based on the type of vertical video content. For example, the content type of the video may be sports, drama, game, news, education, entertainment, etc., but is not limited thereto.

In other words, each AI model can be learned based on images with different types of content. In this way, by providing a number of AI models that are applied in various ways depending on the type of video content, there is an advantage in that the efficiency of image quality improvement can be further improved when enlarging and converting a changed vertical video.

For example, the control unit 160 can separate the horizontal image into shots, receive information about consecutive frame images included in the separated shots as input, and select the optimal AI model for each shot. there is.

Figure 10 shows an example of selecting an optimal AI model in S320.

Referring to FIG. 10, the control unit 160 can select the optimal AI model using a classifier. In other words, the classifier is a second machine learning model learned according to the machine learning technique of supervised learning through the training data of the input data and output data pair (dataset).

At this time, the classifier can be learned using learning data including input data of continuous frame images and output data for the content type of these frame images (e.g., sports, drama, game, news, education, entertainment, etc.). there is. Accordingly, the classifier has a function for the relationship between the continuous frame image as input data and the content type as output data, and expresses this using various parameters.

For example, a classifier can express the relationship between consecutive frame images and the content types of these frame images using parameters of weights and biases. Accordingly, as shown in Figure 10, when input data of consecutive frame images (F(t-1), F(t), F(t+1)) within a certain subunit is input to the learned classifier, Output data for the content type of the corresponding subunit images (F(t-1), F(t), and F(t+1)) according to the function may be output.

For example, machine learning techniques applied to the AI model, first machine learning model, and second machine learning model (classifier) include Artificial neural network, Boosting, Bayesian statistics, Decision tree, Gaussian process regression, Nearest neighbor algorithm, Support vector It may include, but is not limited to, machine, random forests, symbolic machine learning, ensembles of classifiers, or deep learning.

In particular, when the AI model, the first machine learning model, and the second machine learning model are deep learning models learned using deep learning techniques, the relationship between input data and output data is expressed as multiple layers. These multiple representation layers are also referred to as “neural networks.” Such deep learning models can have encouraging performance in image processing fields such as the present invention.

For example, deep learning techniques include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, etc. It can be done, but it is not limited to this.

In particular, when trying to create a new AI model for an image within a small unit (for example, a shot, etc.), the learning process takes a long time, so it is not suitable for real-time transmission. Accordingly, in the present invention, a number of AI models (i.e., AI model DB) learned in advance according to the type of video content are stored in the memory 140 or a separate database device, and the control unit 160 controls the current small unit. The optimal suitable AI model is searched and used in the AI model DB. That is, the control unit 160 searches the pre-stored AI model DB for an AI model that matches the type of content output by the classifier for the input of a continuous frame image of a certain subunit, and uses the searched AI model as the optimal AI model applied to the subunit. You can select . As a result, the present invention has the advantage of being more suitable for real-time playback of video.

Additionally, since the AI model is learned according to the type of video content, it may be more effective to divide it into shot units or scene units, which are units for a series of scenes of the same content type.

Thereafter, the control unit 160 extracts the vertical image for each frame image from the horizontal image based on the playback area calculated in S310 (S330). In other words, the vertical image of the frame can be extracted by removing the remainder from each frame of the horizontal image, leaving a portion corresponding to the playback area of the frame.

The control unit 160 can extract the vertical image of the frame by removing the remainder from each frame of the horizontal image, leaving a portion corresponding to the playback area of the frame.

Thereafter, the control unit 160 applies the AI model according to the optimal AI model information selected in S320 for each subunit and enlarges and converts the vertical image extracted in S330 (S340). That is, the vertical image, which is a low-quality image separated from the control unit 160 S330, is input to the AI model. As a result, the AI model can output an enlarged image with improved image quality according to the built-in function, that is, a vertical image enlarged to the size of the display 140 of the electronic device 100 but with high definition.

Of course, if necessary, the control unit 160 may perform image processing by cutting a portion of the high-definition vertical image output from the AI model and enlarged and converted, or may perform size interpolation according to a resolution transformation technique. This can be performed when the enlarged high-definition vertical image is larger than the display 140 of the electronic device 100. For example, the resolution modification technique may be bilinear, bicubic interpolation, down-sampling, etc., but is not limited thereto.

Figure 11 shows a comparison example of playing a horizontal image in an electronic device held vertically. That is, the left side is an image played according to the prior art, and the right side is an image played according to the present invention.

Referring to FIG. 11, when playing a horizontal image on a terminal in a vertical position, the present invention, unlike the prior art, not only minimizes letter boxes or fillers, but also enlarges the image while including the main object. It is possible to play a high-definition vertical video that improves the low-definition problem that occurs while playing. As a result, the present invention has the advantage of increasing the user's visual satisfaction.

In other words, while the conventional technology (left side of Figure 11) occupies 25% of the screen, in the case of the present invention (right side of Figure 11), the entire screen can be used, so visual satisfaction is high and the exposure effect is large, so the playback video can be used as an advertisement. In some cases, the advertising effect has a great advantage.

FIG. 12 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is held horizontally in step S20 of the method according to an embodiment of the present invention.

Meanwhile, when performing image processing on a horizontal image when the electronic device 100 is in a horizontal grip state, as shown in FIG. 12, the control unit 160 may perform steps S211 to S214 in S20.

That is, the control unit 160 determines whether a black image is needed, that is, whether a pillar box or letter box area is needed (S211). That is, by comparing the screen ratio of the display 130 and the aspect ratio of the horizontal image according to the holding state of the electronic device 100, it is possible to determine whether a pillar box or letter box area is needed.

If, as a result of S211, a black image area is required (i.e., when the horizontal image needs to be played on the display 130 by filling in the area of the pillar box or letter box in the horizontal holding state), the control unit 160 controls the horizontal image The second conversion described above is performed (S212).

On the other hand, as a result of S211, when the black image area is unnecessary (i.e., when the horizontal image can be played in all pixels of the display 130 without the need to fill the area of the pillar box or letter box in the horizontal image in the horizontal holding state), The control unit 160 performs the above-described first transformation on the horizontal image (S214).

Meanwhile, after performing S212, the control unit 160 compares the resolution of the horizontal image on which the second conversion was performed with the screen resolution of the horizontally held state of the display 130 to determine whether the resolution of the horizontal image is insufficient. Do it (S213).

If, as a result of S213, the screen resolution of the horizontally held state of the display 130 is larger than the resolution of the horizontal image on which the second conversion has been performed and the resolution of the horizontal image is insufficient, the control unit 160 determines whether the second conversion has been performed. The above-described first transformation is additionally performed on the horizontal image to achieve high resolution of the horizontal image. As a result, the control unit 160 can reproduce the horizontal image on which the second transformation and the first transformation were sequentially performed on the display 130 at high resolution.

On the other hand, as a result of S213, if the resolution of the horizontal image on which the second conversion was performed is larger than the screen resolution of the horizontally held state of the display 130, and the resolution of the horizontal image is not insufficient, the control unit 160 performs the second conversion. The performed horizontal video can be played in high resolution.

Of course, the above-described S211 to S214 can be equally performed in S20 even when image processing is performed on a vertical image when the electronic device 100 is held vertically. However, in this case, in the contents of S211 to S214 described above, the horizontal image may be replaced with a vertical image, and the horizontal holding state may be replaced with a vertical holding state.

FIG. 13 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is held vertically in step S20 of the method according to an embodiment of the present invention.

Additionally, when performing image processing on a horizontal image when the electronic device 100 is held vertically, as shown in FIG. 13, the control unit 160 may perform steps S221 to S223 in S20.

That is, the control unit 160 performs the above-described third transformation on the horizontal image (S221).

Thereafter, the control unit 160 compares the resolution of the vertical image on which the third conversion was performed with the screen resolution in the vertically held state of the display 130 to determine whether the resolution of the vertical image is insufficient (S222).

If, as a result of S222, the screen resolution in the vertically held state of the display 130 is larger than the resolution of the vertical image on which the third conversion has been performed and the resolution of the vertical image is insufficient, the control unit 160 determines whether the third conversion has been performed. The above-described first transformation is additionally performed on the vertical image to achieve high resolution of the vertical image. As a result, the control unit 160 can reproduce the vertical image on which the third transformation and the first transformation were sequentially performed on the display 130 at high resolution.

On the other hand, as a result of S222, if the resolution of the vertical image on which the third conversion has been performed is larger than the screen resolution in the vertically held state of the display 130, and the resolution of the vertical image is not insufficient, the control unit 160 performs the third conversion. The captured vertical video can be played in high resolution.

The present invention, constructed as described above, does not insert a black image (pillar box or letter box) when the aspect ratio of the image and the screen ratio of the electronic device that reproduces it are different, but processes the image so that the aspect ratio of the image is adjusted. There is an advantage in that the video can be appropriately adjusted to the screen ratio of the electronic device that plays it. In addition, the present invention not only effectively performs conversion on the aspect ratio of an image by adaptively performing at least one of the first to third conversions according to the holding state of the electronic device and the aspect ratio of the image. There is an advantage in that high resolution of the converted image can be achieved by additionally performing the first conversion even after performing the second or third conversion. In addition, the present invention not only minimizes letter boxes or pillar boxes when playing video, but also enlarges the video while including the main objects, and reproduces the video in high definition by improving the low-quality problem that occurs during enlargement. , it has the advantage of increasing the user's visual satisfaction. In addition, the present invention allows playback using the screen of the electronic device as much as possible, which not only increases viewing immersion, but also has a large exposure effect, so when the played video is an advertisement, the advertisement effect is great. In addition, since the present invention can be applied to various image quality improvement techniques, it has the advantage of being applicable not only to video on demand (VOD) but also to real-time streaming.

In the detailed description of the present invention, specific embodiments have been described, but of course, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, but should be defined by the claims described below and equivalents to these claims.

100: electronic device 110: input unit
120: Communication unit 130: Display
140: memory 150: sensor unit
160: control unit

Claims (20)

Image conversion to a horizontal image of the first aspect ratio with a longer width or a vertical image of a second aspect ratio with a longer height depending on the horizontal holding state of the longer horizontal aspect ratio and the vertical holding state of the longer vertical aspect ratio. As an electronic device that reproduces,
a sensor unit that detects a sensor value related to a holding state of the electronic device; and
A control unit that adaptively performs at least one of first to third transformations on the image according to the grip state according to the sensor value and the image ratio of the image,
The first transformation is a transformation that improves image quality while maintaining the content of the image,
The second transformation is a transformation that adds new video content to the video content,
The third transformation is a transformation that improves image quality by enlarging at least a portion of the image content of the horizontal image in the case of the vertical holding state and enlarging it to the vertical image,
When performing the third conversion, the control unit
A process of analyzing the content of each frame of the horizontal image and calculating at least one playback area corresponding to a partial image of the frame and having the second video ratio for each frame;
The horizontal image is divided into a plurality of subunits corresponding to image units larger than the frame, and among a plurality of AI (artificial intelligence) models previously learned for each content type of the image, a subunit is applied according to the content type of the horizontal image within the subunit. The process of selecting the optimal AI model,
A process of extracting a vertical image with a longer vertical aspect ratio for each frame image from the horizontal image based on the playback area;
The process of enlarging and converting the extracted vertical images is performed by applying the selected optimal AI model to each subunit.
The number of frames included in at least two different subunits is different,
Each of the AI models is a model learned to generate an enlarged image of a second aspect ratio with improved quality from a low-quality second aspect ratio image of different content according to a machine learning technique,
When performing the process of selecting the optimal AI model, the control unit calculates a difference value between neighboring frames in the horizontal image, distinguishes the subunit if the calculated difference value is greater than a reference value, and determines the subunit. When frame input data is input, the optimal AI model for each subunit is selected using a classifier, which is a machine learning model that has been previously learned to output the content type for the continuous frame image, and the optimal AI model output from the classifier is selected. An electronic device that selects an AI model already learned for a content type corresponding to the content type as the optimal AI model for the corresponding subunit.
According to paragraph 1,
The control unit,
When in the horizontal holding state, perform the first transformation or the second transformation on the horizontal image,
An electronic device that performs the first transformation or the second transformation on the vertical image when it is in the horizontal holding state or the vertical holding state.
According to paragraph 2,
The control unit,
In the horizontal holding state, when the horizontal image needs to be played by filling the area of a pillar box or letter box, the second transformation is performed on the horizontal image, and a pillar box is formed. Or, an electronic device that performs the first transformation on the horizontal image when a letter box area is unnecessary.
According to paragraph 1,
When performing the first conversion, the control unit
For the image, an enlarged image is generated by applying a pre-learned machine learning model to enlarge the image to a larger image than the screen size of the grasp state while improving image quality, and then size interpolation is performed on the enlarged image to determine the grasp state. An electronic device that converts to an aspect ratio of .
According to paragraph 1,
When performing the second conversion, the control unit
An electronic device that synthesizes new image content included in the previous or next frame image into the current frame image based on matching points using the previous and next frame images.
According to clause 5,
When performing the second conversion, the control unit
An electronic device that synthesizes new image content by performing generative adversarial network (GAN)-based boundary expansion on the current frame image when the synthesized current frame image is less than the resolution of the target image.
According to claim 5 or 6,
When performing the first conversion, the control unit
For the synthesized current frame image, an enlarged image is generated by applying a previously learned machine learning model to enlarge the image to a larger image than the screen size of the grip state while improving image quality, and then perform size interpolation on the enlarged image. An electronic device that converts the screen ratio of the holding state.
delete According to paragraph 1,
When performing the calculation process, the control unit detects areas for objects and faces in each frame image, calculates a maximum playback area including the detected areas, and performs a cropping process on the calculated maximum playback area. An electronic device that calculates at least one said playback area.
According to clause 9,
The maximum playback area is an area that includes all of the detected areas when there are multiple detected areas.
delete According to paragraph 1,
The optimal AI model is an electronic device that improves image quality including increasing resolution and removing noise, or improves image quality including increasing resolution and increasing dynamic range.
According to paragraph 1,
The control unit controls the reproduction of the image enlarged according to the third conversion in all pixels of the display.
Image conversion to a horizontal image of the first aspect ratio with a longer width or a vertical image of a second aspect ratio with a longer height depending on the horizontal holding state of the longer horizontal aspect ratio and the vertical holding state of the longer vertical aspect ratio. A method performed in an electronic device to reproduce
detecting a sensor value related to a holding state of the electronic device; and
Adaptively performing at least one of first to third transformations on the image according to the holding state according to the sensor value and the aspect ratio of the image,
The first transformation is a transformation that improves image quality while maintaining the content of the image.
The second transformation is a second transformation that adds new video content to the video content,
The third transformation is a transformation that improves image quality by enlarging at least a portion of the image content of the horizontal image in the case of the vertical holding state and enlarging it to the vertical image,
The steps performed when performing the third transformation are:
Analyzing the content of each frame of the horizontal image and calculating at least one playback area corresponding to a partial image of the frame and having the second aspect ratio for each frame;
The horizontal image is divided into a plurality of subunits corresponding to image units larger than the frame, and among a plurality of AI (artificial intelligence) models previously learned for each content type of the image, a subunit is applied according to the content type of the horizontal image within the subunit. Selecting an optimal AI model;
extracting a vertical image with a longer vertical aspect ratio for each frame image from the horizontal image based on the playback area; and
A step of enlarging and converting the extracted vertical image by applying the selected optimal AI model for each subunit;
The number of frames included in at least two different subunits is different,
Each of the AI models is a model learned to generate an enlarged image of a second aspect ratio with improved quality from a low-quality second aspect ratio image of different content according to a machine learning technique,
In the step of selecting the optimal AI model, the difference value between neighboring frames in the horizontal image is calculated, and if the calculated difference value is greater than the reference value, the subunit is distinguished, and the input data of consecutive frames within the subunit is When input, the optimal AI model for each subunit is selected using a classifier, which is a machine learning model that has been previously learned to output the content type for the continuous frame image, and the optimal AI model corresponding to the content type output from the classifier is selected. A method of selecting an AI model already learned based on the content type as the optimal AI model for the corresponding subunit.
According to clause 14,
In the above performing steps,
In the case of the horizontal holding state, the first transformation or the second transformation is performed on the horizontal image,
A method in which the first transformation or the second transformation is performed on the vertical image when it is in the horizontal holding state or the vertical holding state.
According to clause 14,
When performing the first conversion,
For the image, an enlarged image is generated by applying a pre-learned machine learning model to enlarge the image to a larger image than the screen size of the grasp state while improving image quality, and then size interpolation is performed on the enlarged image to determine the grasp state. How to convert to aspect ratio.
According to clause 14,
When performing the second conversion,
A method of synthesizing new image content included in the previous or next frame image into the current frame image based on corresponding point matching using the previous and next frame images.
According to clause 14,
When performing the second conversion,
A method of synthesizing new image content by performing generative adversarial network (GAN)-based boundary expansion on the current frame image when the synthesized current frame image is less than the resolution of the target image.
According to claim 17 or 18,
When performing the first conversion,
For the synthesized current frame image, an enlarged image is generated by applying a previously learned machine learning model to enlarge the image to a larger image than the screen size of the grip state while improving image quality, and then perform size interpolation on the enlarged image. A method of converting the screen ratio of the holding state. delete