KR20230007718A - Electronic apparatus and method for intelligent video conversion - Google Patents

Abstract

The present invention relates to an electronic device and method for intelligent video conversion. The electronic device according to one embodiment of the present invention converts and plays a video and comprises: a sensor unit that detects a sensor value related to a holding state of the electronic device; and a control unit that performs conversion on the video depending on the holding state according to the sensor value and the aspect ratio of the video. The control unit performs at least one of a first conversion that improves video quality while maintaining the content of the video, a second conversion that adds new video content to the content of the video, and a third conversion that improves video quality while enlarging at least a portion of the content of the video. Accordingly, the present invention can increase the visual satisfaction of a user.

Description

Electronic device and method for intelligent video conversion {ELECTRONIC APPARATUS AND METHOD FOR INTELLIGENT VIDEO CONVERSION}

The present invention relates to an electronic device and method related to video playback, and more particularly, to an electronic device and method for adaptively converting an image according to a holding state of a terminal and an aspect ratio of a video (hereinafter referred to as “video”). It is about.

According to the conventional aspect ratio conversion technology (hereinafter referred to as the “prior art”), when playing an image on a terminal, the aspect ratio (ie, the aspect ratio of the terminal) while maintaining the aspect ratio (ie, the ratio between the width and height of the video source) The video is played back at the maximum size according to the size of the screen's width-to-height ratio).

1 shows examples of playing an image of 16:9 aspect ratio in terminals having various aspect ratios to which the prior art is applied.

Accordingly, when the prior art is applied, an empty space is created on the top, bottom or left and right sides of the terminal screen, and the empty space is treated as black. That is, referring to FIG. 1 , when a video having an aspect ratio of 16:9 is reproduced in a terminal having an aspect ratio of 4:3, a black image called a letter box is inserted into an empty space at the top and bottom of the screen of the terminal. In addition, when a video having an aspect ratio of 16:9 is reproduced in a terminal having an aspect ratio of 21:9, a black image called a pillar box is inserted into an empty space formed on the left and right sides of the screen of the terminal.

That is, since the prior art is merely a technology of reproducing according to the screen of the terminal while maintaining the aspect ratio, when the aspect ratio of the terminal does not match the aspect ratio, the user's visual satisfaction is low due to excessive use of letter boxes or fillers. There is a losing problem. In particular, when the screen state of a terminal to which the prior art is applied is in a vertical state (a state in which the aspect ratio of the terminal is longer than the horizontal ratio), when a horizontal video (an image having an aspect ratio longer than vertical) is played, the corresponding screen As the horizontal video is scaled down and reproduced according to the ratio, this problem is inevitably highlighted.

KR10- 2007-0039790 A

In order to solve the problems of the prior art as described above, an object of the present invention is to provide a technique for adaptively converting an image according to a holding state of a terminal and an aspect ratio of an image.

However, the problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An electronic device according to an embodiment of the present invention for solving the above problems is an electronic device that converts and reproduces an image, and includes a sensor unit that detects a sensor value related to a holding state of the electronic device; and a control unit that converts the image according to a holding state according to the sensor value and an aspect ratio of the image.

The control unit performs a first transformation for improving the image quality while maintaining the contents of the video, a second transformation for adding and synthesizing new video contents to the contents of the video, and a longer image while enlarging at least a part of the contents of the video. At least one of the third transformations for improving picture quality while enlarging the vertical image at a ratio may be performed.

The controller may perform the first conversion or the second conversion on a horizontal image having a longer horizontal aspect ratio when the image is held horizontally with a longer horizontal aspect ratio.

The control unit may perform the third conversion on the horizontal image when the vertical image is held in a vertical holding state of a longer vertical aspect ratio.

The controller may perform the first conversion or the second conversion on a vertical image having a longer vertical aspect ratio in the horizontal holding state or the vertical holding state.

The control unit performs the second conversion on the horizontal image when the horizontal image needs to be reproduced by filling the area of a pillar box or letter box in the horizontal holding state, and the pillar box ( When a pillar box or letter box area is unnecessary, the first transformation may be performed on the horizontal image.

When the first conversion is performed, the control unit generates an enlarged image by applying a pre-learned machine learning model to enlarge the image to an image larger than the screen size in the holding state while improving the image quality, and then generates the enlarged image It can be converted into the aspect ratio of the holding state by performing size interpolation on .

When the second transformation is performed, the control unit may synthesize new image contents included in previous or subsequent frame images with the current frame image based on corresponding point matching using previous and subsequent frame images.

When the second transformation is performed, the control unit performs boundary expansion based on a generative adversarial network (GAN) on the current frame image when the synthesized current frame image does not reach the resolution of the target image, New video contents can be synthesized.

When the second transformation is performed, the control unit generates an enlarged image by applying a pre-learned machine learning model to enlarge the synthesized current frame image to an image larger than the screen size of the holding state while improving the image quality. Afterwards, size interpolation is performed on the enlarged image to convert it to the aspect ratio of the holding state.

When the third conversion is performed, the control unit analyzes the content of the corresponding frame image for each frame image of the horizontal image and calculates a playback area corresponding to a part of the corresponding frame image, and divides the horizontal image into a plurality of small units. A process of selecting an optimal AI model applied to each subunit according to the content of the horizontal video within the corresponding subunit from among a plurality of AI (artificial intelligence) models previously learned for each content type of the video; In , a process of extracting a vertical image having a longer vertical aspect ratio for each frame image and a process of enlarging and converting the extracted vertical image by applying the selected optimal AI model to each subunit may be performed, respectively.

In the calculation process, the control unit detects areas for objects and faces in each frame image, calculates a maximum reproduction area including the detected areas, and cuts the calculated maximum reproduction area to obtain a horizontally wider area. At least one play area having a longer second aspect ratio may be calculated.

The maximum reproduction area may be an area including all of the detected areas when there are a plurality of the detected areas.

Each AI model may be a model learned to generate an enlarged image with improved quality from low-quality images of different contents according to a machine learning technique.

The optimal AI model may improve at least one image quality among resolution increase, noise removal, and dynamic range increase.

The control unit may control an image enlarged according to the third transformation to be reproduced in all pixels of the display.

A method according to an embodiment of the present invention is a method for converting and reproducing an image in an electronic device, comprising: sensing a sensor value related to a holding state of the electronic device; and performing conversion on the image according to a holding state according to the sensor value and an aspect ratio of the image.

The performing of the conversion may include: performing a first conversion for improving image quality while maintaining the content of the video; performing a second conversion for adding and synthesizing new video content to the content of the video; At least one of the steps of performing a third transformation for improving image quality while enlarging at least a part of the content of the image may be included.

The performing of the conversion may include performing the first conversion or the second conversion on a horizontal image having a longer horizontal aspect ratio when the horizontal holding state has a longer horizontal aspect ratio, and the longer vertical aspect ratio. The third conversion is performed on a horizontal image having a longer horizontal aspect ratio in the case of a vertical holding state, and the third transformation is performed on a vertical image having a longer aspect ratio in the horizontal holding state or the vertical holding state. It may include performing the first transformation or the second transformation.

In the step of performing the first conversion, an enlarged image is generated by applying a pre-learned machine learning model to enlarge the image to an image larger than the screen size of the holding state while improving the image quality, and then the enlarged image It may include performing size interpolation on and converting to an aspect ratio of the holding state.

The performing of the second transformation may include synthesizing new image contents included in previous or subsequent frame images with the current frame image based on corresponding point matching using previous and subsequent frame images.

In the step of performing the second transformation, when the synthesized current frame image does not meet the resolution of the target image, boundary expansion based on a generative adversarial network (GAN) is performed on the corresponding current frame image to generate a new image. Content can include steps.

The step of performing the second transformation generates an enlarged image by applying a pre-learned machine learning model to enlarge the synthesized current frame image to an image larger than the screen size of the holding state while improving the image quality, It may include performing size interpolation on the enlarged image and converting it into an aspect ratio of the holding state.

The performing of the conversion may include performing the third conversion on a horizontal image having a longer horizontal aspect ratio when the vertical holding state has a longer vertical aspect ratio.

The performing of the third transformation may include analyzing content of the corresponding frame image for each frame image of the horizontal image and calculating a reproduction area corresponding to a part of the corresponding frame image; Separating the horizontal image into a plurality of sub-units and selecting an optimal AI model applied for each sub-unit according to the content of the horizontal video within the sub-unit from among a plurality of AI (artificial intelligence) models pre-learned for each content type of the image; extracting, for each frame image, a vertical image having a longer vertical aspect ratio in the horizontal image based on the play area; and enlarging and converting the extracted vertical image by applying the selected optimal AI model for each subunit.

The present invention configured as described above does not insert a black image (pillar box or letter box) when the aspect ratio of the image and the aspect ratio of the electronic device that reproduces the image are different, but adjusts the aspect ratio of the image. There is an advantage in that a corresponding image can be appropriately matched to an aspect ratio of an electronic device that reproduces the image.

In addition, the present invention not only effectively converts the aspect ratio of an image by adaptively performing at least one of the first to third conversions according to the holding state of the electronic device and the aspect ratio of the image, There is an advantage that high resolution of the converted image can be promoted.

In addition, the present invention can not only minimize the letter box or pillar box when reproducing an image, but also enlarge it while including the main object of the image, but reproduce the low-quality problem that occurs while enlarging it as a high-definition image. , there is an advantage of increasing the user's visual satisfaction.

In addition, since the screen of the electronic device can be reproduced using the screen of the electronic device as much as possible, the viewing immersion is increased, and the exposure effect is also great, so that the advertisement effect is great when the reproduced image is an advertisement.

In addition, since the present invention can be applied to various picture quality improvement techniques, there is an advantage that it can be applied not only to video on demand (VOD) but also to real-time streaming.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows examples of playing an image of 16:9 aspect ratio in terminals having various aspect ratios to which the prior art is applied.
2 shows a block configuration diagram of an electronic device 100 according to an embodiment of the present invention.
3 shows a flow chart of a method according to one embodiment of the present invention.
4 shows a flowchart of the first conversion performed by the control unit 160 .
5 shows examples of various magnifications of an image.
6 shows a flowchart of the second conversion performed by the control unit 160 .
7 shows an example for the second transformation.
8 shows a flowchart of the third conversion performed by the control unit 160 .
9 shows a more detailed flow chart for S310 in the third transformation.
10 shows an example of selecting an optimal AI model in S320.
11 compares and shows an example of reproducing a horizontal image in an electronic device in a vertical holding state.
FIG. 12 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is in a horizontal grip state in S20 of the method according to an embodiment of the present invention.
13 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is in a vertical holding state in S20 of the method according to an embodiment of the present invention.

The above objects and means of the present invention and the effects thereof will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily understand the technical idea of the present invention. will be able to carry out. In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms in some cases unless otherwise specified in the text. In this specification, terms such as "comprise", "have", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may represent one of the words listed together, or a combination of two or more. For example, "or B" and "at least one of B" may include only one of A or B, or may include both A and B.

In this specification, descriptions following "for example" may not exactly match the information presented, such as cited characteristics, variables, or values, and tolerances, measurement errors, limits of measurement accuracy and other commonly known factors It should not be limited to the embodiments of the present invention according to various embodiments of the present invention with effects such as modifications including.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In the present specification, when an element is described as being 'on' or 'in contact with' another element, it may be in direct contact with or connected to the other element, but another element may be present in the middle. It should be understood that On the other hand, if an element is described as being 'directly on' or 'directly in contact with' another element, it may be understood that another element in the middle does not exist. Other expressions describing the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. In addition, the above terms should not be interpreted as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'.

Unless otherwise defined, all terms used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

2 shows a block configuration diagram of an electronic device 100 according to an embodiment of the present invention.

The electronic device 100 according to an embodiment of the present invention is a device that converts and reproduces an image. In this case, the image may refer to a video, and may be a pre-stored image or an image transmitted from another device (server). For example, the service provided by the electronic device 100 may be a service of playing a pre-stored video, a video on demand (VOD) service, or a service such as real-time streaming, but is not limited thereto.

In particular, the video is an image having content with an aspect ratio longer than the vertical (hereinafter referred to as “horizontal video”) or an image having content with an aspect ratio longer than horizontal (hereinafter referred to as “vertical video”). referred to). In addition, the holding state is a state given by the user of the electronic device 100, and the display 130 is in a state in which the aspect ratio of the display 130 is longer than the vertical (hereinafter, referred to as “horizontal holding state”), or the display 130 is in a vertical rather than horizontal state. It may be a state of longer aspect ratio (hereinafter referred to as “vertical holding state”)

That is, the electronic device 100 may reproduce a corresponding image after performing various transformations according to the current holding state and the aspect ratio of the image (that is, whether the image is horizontal or vertical).

In this case, the electronic device 100 may be a terminal capable of computing. For example, the electronic device 100 includes a desktop personal computer (PC), a laptop personal computer (laptop PC), a tablet personal computer (tablet PC), a netbook computer, a workstation, and a personal PDA (personal PDA). It may be a digital assistant, a smart phone, a smart pad, or a mobile phone, but is not limited thereto.

As shown in FIG. 2 , the electronic device 100 may include an input unit 110, a communication unit 120, a display 130, a memory 140, and a control unit 160.

The input unit 110 generates input data in response to various user inputs and may include various input means.

For example, the input unit 110 includes a keyboard, a key pad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), menu button (menu button), etc. may be included, but is not limited thereto.

The communication unit 120 is a component that communicates with other devices such as a server, and receives information about a bitstream for an image and a pre-learned model (AI model, machine learning model, GAN model, etc.) from another device. can

For example, the communication unit 120 may perform 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication or wired communication such as cable communication may be performed, but is not limited thereto.

The display 130 displays various image data on a screen, and may be composed of a non-emissive panel or a light-emitting panel. Also, the display 230 may display the converted image according to the holding state and the aspect ratio of the image.

For example, the display 130 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or electronic paper (electronic paper) display, etc. may be included, but is not limited thereto. Also, the display 130 may be combined with the input units 120 and 220 and implemented as a touch screen or the like.

The memory 140 stores various types of information necessary for the operation of the electronic device 100 . The storage information of the memory 140 may include, but is not limited to, images, models, converted images, program information related to a method to be described later, and the like. In particular, a plurality of AI models may be stored, and may be stored in a compressed form, but is not limited thereto.

For example, the memory 140 may be of a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), or an optical media type according to its type. ), magneto-optical media type, multimedia card micro type, flash memory type, read only memory type, or RAM type (random access memory type), etc., but is not limited thereto. In addition, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The sensor unit 150 detects state information of the electronic device 100 or its surroundings. The sensor unit 150 may include various sensors such as an acceleration sensor, a gyro sensor, a proximity sensor, an RGB sensor, a brightness sensor, a Hall sensor, a motion sensor, a temperature/humidity sensor, a barometer, and a geomagnetic sensor.

In particular, the electronic device 100 may be in various holding states, and the sensor unit 150 may detect them. That is, the sensor unit 150 may include a sensor that detects a sensor value related to a holding state of the electronic device 100, such as a gyro sensor or a motion sensor.

The controller 160 may perform various control operations of the electronic device 100 . That is, the controller 160 may control the execution of a method to be described later, and the rest of the components of the electronic device 100, that is, the input unit 110, the communication unit 120, the display 130, the memory 140, and the sensor unit. (150) and the like can be controlled.

For example, the control unit 160 may include, but is not limited to, a processor that is hardware or a process that is software that is executed in the corresponding processor.

In particular, the control unit 160 can determine the holding state of the electronic device 100 using the sensor value of the sensor unit 150 related to the holding state, and the image is displayed according to the grasped holding state and the aspect ratio of the image. You can control the execution of conversions.

In this case, the transformation of the image may be at least one of the first to third transformations. That is, the first transformation is a transformation that enlarges the image while maintaining the content of the image, but also improves the image quality. The second transformation is a transformation in which new video content is added and synthesized while enlarging the video content. Of course, the second transformation may be a transformation in which the first transformation is additionally performed after new video content is additionally synthesized. In addition, the third transformation is a transformation in which at least a part of the content of the video is enlarged to a vertical video while improving the image quality.

Table 1 below shows the type of conversion performed by the controller 160 according to the holding state and the aspect ratio of the image.

horizontal gripping state vertical gripping state horizontal video First Transformation or Second Transformation Third Transformation vertical video First Transformation or Second Transformation First Transformation or Second Transformation

That is, when the electronic device 100 is in a horizontal holding state and is in a horizontal image, the controller 160 may control the first transform or the second transform to be performed on the horizontal image. In this case, the controller 160 may perform a first transformation or a second transformation depending on whether a pillar box area is required. That is, when the horizontal holding state and the horizontal image are reproduced by filling the area of the pillar box in the horizontal image (for example, when the converted horizontal image requires the area of the pillar box because the resolution of the original horizontal image is insufficient) ), the second transformation may be performed on the horizontal image. On the other hand, when the horizontal holding state and the horizontal image do not need to fill the pillar box area in the horizontal image (for example, when the original horizontal image has sufficient resolution and the converted horizontal image does not require the pillar box area), the horizontal image The first conversion may be performed on the image. In addition, the controller 160 may control the horizontal image to perform the third conversion when the electronic device 100 is in a vertical holding state.

Also, when the electronic device 100 is in a horizontal holding state or a vertical holding state, the controller 160 may control the vertical image to perform a first conversion or a second conversion. In this case, the controller 160 may perform a first transformation or a second transformation depending on whether a pillar box area is required. That is, when a vertical image needs to be reproduced by filling a pillar box area (for example, when a horizontal or vertical image to be converted requires a pillar box area because the resolution of the original vertical image is insufficient), for a vertical image The second transformation may be performed. On the other hand, when it is not necessary to fill the pillar box area in the vertical image (for example, when the original vertical image has sufficient resolution and the converted horizontal or vertical image does not need the pillar box area), a first transformation is performed on the vertical image. can be done

3 shows a flow chart of a method according to one embodiment of the present invention.

A method according to an embodiment of the present invention is a method for converting and reproducing an image in the electronic device 100, and includes S10 to S30 as shown in FIG. 3 . At this time, the execution of S100 to S300 can be controlled through various hardware configurations or software processes of the control unit 160 .

First, the control unit 160 detects a sensor value related to the holding state of the electronic device 100 to determine the holding state (S10). At this time, the control unit 160 may determine the gripping state of the electronic device 100 by using the sensor value detected from the gripping state related sensor of the sensor unit 150 . That is, it is possible to determine whether the electronic device 100 is in a horizontal gripping state or a vertical gripping state.

Thereafter, the controller 160 adaptively converts the image according to the grasped holding state and the aspect ratio of the image (S20). That is, the controller 160 may perform at least one of the first to third conversions according to the holding state (ie, horizontal holding state/vertical holding state) and aspect ratio (ie, horizontal image/vertical image). there is. Of course, before performing S20, the controller 160 determines whether the image previously stored in the memory 140 or received through the communication unit 120 is a horizontal image or a vertical image, and then performs S20. .

Thereafter, the controller 160 reproduces the image on which at least one of the first to third conversions has been performed and controls the image to be displayed on the display 130 (S30).

That is, in the case of a horizontal holding state and a horizontal image, a converted horizontal image obtained by performing a first conversion or a second conversion on the horizontal image may be displayed on the display 130 . Also, in the case of a vertical holding state and a horizontal image, a high-quality vertical image obtained by performing a third conversion on the horizontal image may be displayed on the display 130 . In this case, the enlarged and converted high-quality vertical image may be reproduced in all pixels of the display 140 of the electronic device 100 as shown in FIG. 11 , but is not limited thereto. In addition, regardless of the gripping state, in the case of a vertical image, a converted horizontal or vertical image obtained by performing a first conversion or a second conversion on the vertical image may be displayed on the display 130 .

Of course, in S30, the control unit 160 may control the electronic device 100 to output (reproduce) the audio of the corresponding video by synchronizing the audio of the video when at least one of the first to third conversions is performed. there is.

<First Transformation>

When performing the first conversion, the control unit 160 generates an enlarged image by applying a first machine learning model that has been previously learned to enlarge the image to a larger image than the screen size of the holding state while improving the image quality, and then generates It can be converted to fit the aspect ratio of the holding state by performing size interpolation on the enlarged image.

FIG. 4 shows a flowchart of the first transformation performed by the controller 160, and FIG. 5 shows examples of various enlargements of an image.

Specifically, referring to FIG. 4 , the controller 160 may perform S101 to S104 when performing the first conversion.

The controller 160 calculates an aspect ratio between an image to be converted (input image) and an image after conversion (target image) (S101). At this time, the target image is an image according to the current holding state of the electronic device 100 .

That is, in S101 , the controller 160 may determine the number of horizontal and vertical pixels of the input image and the number of horizontal and vertical pixels of the target image, and calculate a ratio between the determined number of pixels. Of course, the controller 160 may calculate the horizontal ratio (ie, the ratio of the number of horizontal pixels between the input image and the target image) and the horizontal ratio (ie, the ratio of the number of vertical pixels between the input image and the target image). For example, when the input image is 320×240 and the target image is 720×480, the controller 160 may calculate the horizontal/vertical ratio as 2.25/2, respectively.

Thereafter, the controller 160 sets each horizontal/vertical magnification ratio (S102). In this case, the horizontal expansion ratio is a ratio for horizontally expanding the input image, and the vertical expansion ratio is a ratio for vertically expanding the input image. This horizontal/vertical expansion ratio may be different from the horizontal/vertical ratio calculated in S101, and may be set larger than the horizontal/vertical ratio calculated in S101. For example, when the horizontal/vertical ratio is calculated to be 2.25/2 in S101, the controller 160 may set the horizontal expansion ratio to be greater than 2.25 and the vertical expansion ratio to be greater than 2.

Thereafter, the controller 160 may generate an image enlarged at the horizontal/vertical magnification ratio while improving the image quality of the input image by inputting the input image and the horizontal/vertical magnification ratio to the pre-learned first machine learning model. . At this time, the generated enlarged image is an image having a higher resolution than the target image (ie, an image having the screen size of the current holding state).

If the prior art is applied, since the image is simply enlarged and converted at a certain ratio, image quality degradation such as low resolution occurs, and thus the user's visual satisfaction is inevitably lowered. In order to solve this problem, in the present invention, by using the first machine learning model, the input image can be enlarged and converted, but the image quality can also be improved. That is, the first machine learning model is a model learned according to a machine learning technique, and generates an image with improved quality from a low-quality image, but is trained to generate an enlarged image according to the input horizontal / vertical magnification ratio to be.

Specifically, the first machine learning model is a model learned according to a machine learning technique of supervised learning through training data of input data and output data pairs (datasets). That is, the first machine learning model may be learned using training data including input data of a low-quality image and horizontal/vertical magnification, and output data of an image enlarged according to the horizontal/vertical magnification but with improved quality. Accordingly, the first machine learning model has a function for a relationship between input data, such as a low-quality image and horizontal/vertical magnification ratio, and output data, an enlarged image with improved quality, and expresses the function using various parameters.

For example, the first machine learning model may express a relationship between a low-quality image and an image enlarged at a horizontal/vertical magnification ratio with improved image quality using parameters of weights and biases. Accordingly, when a low-quality input image and input data of horizontal/vertical magnification are input to the first machine learning model learned, the output data of the enlarged image with improved quality is enlarged at the horizontal/vertical magnification according to the corresponding function. can be output.

In this case, the quality improvement type may be at least one of resolution increase, noise removal, and dynamic range increase compared to the low-quality image. That is, when a low-quality image is input, the first machine learning model may output one of image quality improvement among resolution increase, noise removal, and dynamic range increase. However, since the input image needs to be enlarged and converted, it may be desirable to necessarily include an increase in resolution. For example, when a low-quality image and horizontal/vertical expansion ratio are input, the first machine learning model outputs a quality-enhanced image of increasing the resolution in the horizontal/vertical expansion ratio, or increasing the resolution in the horizontal/vertical expansion ratio and removing noise. A quality-enhanced image may be output, or a quality-enhanced image with an increase in resolution and an increase in dynamic range may be output.

Then, the controller 160 may convert the enlarged image generated in step S103 into an image suitable for the aspect ratio of the current holding state of the electronic device 100 by performing size interpolation by applying a resolution modification technique. For example, the resolution transformation technique may be bilinear, bicubic interpolation, down-sampling, etc., but is not limited thereto.

In particular, referring to FIG. 5 , in the case of the conventional deep learning-based screen magnification technique, there is a limitation that can only be enlarged at a fixed ratio for width/length. In order to improve this limitation, the present invention sets the horizontal / vertical magnification ratio to be larger than the resolution of the target image, generates an image larger than the target image through the first machine learning model, and then uses traditional resolution transformation techniques (Bilinear, bicubic interpolation, down-sampling, etc.) is additionally applied to perform size interpolation, so that it is possible to fine-tune the aspect ratio of the generated enlarged image.

<Second Transformation>

Meanwhile, when the second transformation is performed, the controller 160 may magnify and transform the current frame image by combining new video content with the current frame image based on correspondence point matching using previous and subsequent frame images. Of course, when the resolution of the current frame image enlarged and converted in this way is less than the resolution of the target image, the controller 160 performs a border extension technique based on a generative adversarial network (GAN) for the corresponding current frame image. ), it is possible to synthesize new image contents to the corresponding current frame image.

6 shows a flowchart of the second transformation performed by the control unit 160, and FIG. 7 shows an example of the second transformation. In FIG. 7 , F(N) is a current frame image, F(N−1) is a previous frame image, and F(N+1) is a subsequent frame image.

Specifically, referring to FIG. 6 , the controller 160 may perform S201 to S202 when performing the second conversion.

That is, the controller 160 uses F(N-1), F(N), and F(N+1) to convert new image contents included in F(N-1) or F(N+1) to F By adding to (N), F'(N) can be synthesized (S201). At this time, the control unit 160 finds a corresponding point of F(N-1) and F(N+1) that matches in F(N), and F(N-1) or a part of F(N+1) according to the corresponding corresponding point An image can be added to F(N) to create F(N)′. That is, geometric synthesis may be performed by adding some images among the remaining portions of F(N−1) and F(N+1) to F(N) except for the portion overlapping with F(N).

Thereafter, if F′(N) having a higher resolution than F(N) does not reach the resolution of the target image, the controller 160 performs GAN-based boundary expansion on F′(N) to create new image content. By adding to F'(N), F''(N) can be synthesized (S202). That is, the controller 160 may generate F'(N) that further expands the image of the edge of F'(N) by inputting F'(N) to the GAN model previously learned according to the GAN technique. there is.

This GAN technique is a method using two predefined network models, a generator (G) and a classifier (D). That is, the classifier (D) is trained first, then the generator (G) is trained, and the generator (G) and the classifier (D) compete with each other to learn little by little. . At this time, in the case of the classifier (D), after receiving the actual input image (real data) and learning to classify the input image as real (real), on the contrary, the synthetic input image (fake data) generated by the generator (G) ) and can be learned to classify the corresponding input image as synthesized (fake).

In particular, in the case of the generator (G), it may be learned to receive an input image and generate an image by extending the edge of the input image. At this time, the fake data generated by the generator (G) is input to the discriminator (D), and the generator (G) can be trained to produce data similar to real data enough to classify the fake data as genuine. F''(N) can be generated by inputting F'(N) using the sufficiently learned generator G as a GAN model.

For example, in addition to general GAN, GAN techniques include DCGAN (Deep Convolutional GAN), LSGAN (Least Squares GAN), SGAN (Semi-Supervised GAN), ACGAN (Auxiliary Classifier GAN), WGAN (Wasserstein Generative Adversarial Networks, ConGAN (Continuous Continuous GAN) GAN), conditional GAN (cGAN), spectral normalization conditional GAN (SNcGAN), starGAN, etc., but is not limited thereto.

<Third Transformation>

Next, when performing the third conversion, the controller 160 performs various image processing on the horizontal image so that it can be reproduced in a vertical holding state. That is, the controller 160 may convert a horizontal image to enlarge a horizontal image while improving image quality of at least a part of the contents of the horizontal image, but also to enlarge the vertical image. For example, the controller 160 may leave a main content portion of a horizontal image and cut out the remaining portion, change the horizontal image to a vertical image, and enlarge and convert the changed vertical image to fit the screen size in a vertical holding state.

8 shows a flowchart of the third conversion performed by the control unit 160 .

Specifically, referring to FIG. 8 , the controller 160 may perform S310 to S340 when performing the third transformation. Of course, the order of S310 and S320 may be changed or performed in parallel at the same time.

First, the controller 160 analyzes the contents of the corresponding frame image for each frame image of the horizontal image and calculates a reproduction area corresponding to a part of the corresponding frame image (S310). That is, the controller 160 may calculate the play area by performing image content analysis on the horizontal image.

At this time, the reproduction area is calculated for each frame image of the horizontal image, and is a partial area of the corresponding frame image, and is an area corresponding to the main content of the corresponding frame image. For example, when the electronic device 100 cuts a part of a horizontal image and changes it to a vertical image for playback in a vertical holding state, the corresponding play area in the horizontal image is not cut because it is the main content, and only the remaining portion is cut. That is, when the electronic device 100 changes a horizontal image to a vertical image, the play area may be referred to as an area to be included in the vertical image.

For example, if a horizontal image has 1000 frames, information on 1000 play areas applied to each of the 1000 frames may be calculated.

9 shows a more detailed flow chart for S310 in the third transformation.

Specifically, referring to FIG. 9 , the controller 160 detects the object and face regions in each frame image of the horizontal image (S311). In this case, the controller 160 may detect an object area and a face area in each frame image using an object detector and a face detector. That is, the object detector may detect a main object, and the face detector may detect a face of a main character. In this case, each detector detects each region by applying various algorithms related to object detection, and may be stored in the memory 140 .

For example, each detector may be a detector using a Canny Edger, Harris corner, Haar-like feature, Histogram of Oriented Gradient (HOG), Scale Invariant Feature Transform (SIFT), or a machine learning model. not.

Then, the controller 160 calculates an area (hereinafter, referred to as a "maximum reproduction area") including each detected area (S312). That is, since each region detected in S311 corresponds to a candidate region that can become a reproduction region, the maximum reproduction region including all of them is calculated. For example, if there are a plurality of regions detected in S311, each detected region may be included in the maximum reproduction region.

In particular, by using both an object detector and a face detector instead of one detector in S311 and calculating the maximum reproduction area to include all detection areas in S312, the present invention can further improve the accuracy of image content analysis for each frame image. can

Thereafter, the controller 160 calculates a playback area having an aspect ratio of the vertical image in each frame through a cropping process for the calculated maximum playback area (S313). That is, considering the type of aspect ratio (1:1, 4:5, 9:16, 10:21, etc.) of the vertical holding state that the electronic device 100 can have, the maximum playback area to fit the corresponding aspect ratio Perform the cut-off process on . Of course, the aspect ratio of the vertical holding state that the electronic device 100 can have may be an aspect ratio in which the vertical and horizontal dimensions are the same, in addition to an aspect ratio in which the vertical axis is longer than the horizontal axis.

For example, the cutting process may be performed to cut out from other areas around a certain area of the maximum reproduction area (eg, centered on a specific type of object or area for a face), but is limited thereto. It is not.

Thereafter, the control unit 160 divides the horizontal image into a plurality of sub-units, and among the plurality of AI models pre-learned for each type of content of the image and pre-stored in the memory 140, the optimum applied for each sub-unit according to the content of the horizontal image within the sub-unit. Select an AI model (S320).

In this case, the controller 160 may divide the horizontal image into a plurality of small units using various scene change detection algorithms. For example, the controller 160 may calculate a difference value between neighboring frames, determine that a shot change has been made when the calculated difference value is greater than a specific reference value, and classify each subunit.

Of course, the number of frames included in each divided subunit may not be constant. That is, the number of frames of the first sub-unit and the number of frames of the second sub-unit may be the same or different. For example, a small unit may be divided according to a shot, a scene, or a sequence, but is not limited thereto. However, it may be preferable that the small unit is a unit larger than the frame. That is, each subunit may include a plurality of frames.

Meanwhile, the AI model is a model applied when enlarging and converting the changed vertical image to fit the screen size of the electronic device 100 . If the prior art is applied, since the changed vertical image is simply enlarged and converted at a certain ratio, image quality deterioration such as low resolution occurs, and thus the user's visual satisfaction is inevitably lowered. In order to solve this problem, in the present invention, by using an AI model, it is possible to enlarge and convert a vertical image, but also improve the image quality. That is, the AI model is a machine learning model learned according to a machine learning technique, and is a model learned to generate an enlarged image with improved quality from a low-quality image.

Specifically, the AI model is a machine learning model learned according to a machine learning technique of supervised learning through training data of input data and output data pairs (datasets). That is, the AI model may be learned using training data including input data of a low-quality image and output data of an enlarged image with improved quality. Accordingly, the AI model has a function for the relationship between a low-quality image, which is input data, and an enlarged image with improved quality, which is output data, and is expressed using various parameters.

For example, the AI model may express a relationship between a low-quality image and an enlarged image with improved quality using parameters of weights and biases. Accordingly, when input data of a low-quality image (eg, a changed vertical image) is input to the learned AI model, an enlarged image with improved quality according to the function (eg, the screen size of the electronic device 100 is enlarged to a high-definition image). Output data of the converted vertical image) may be output.

In this case, the quality improvement type may be at least one of resolution increase, noise removal, and dynamic range increase compared to the low-quality image. That is, when a low-quality image is input, the AI model may output one of image quality improvement among resolution increase, noise removal, and dynamic range increase. However, since the vertical image needs to be enlarged and converted, it may be desirable to necessarily include an increase in resolution. For example, when a low-quality image is input, the AI model may output a quality-improved image of increased resolution, an image of improved image quality of increased resolution and noise removal, or an image of improved image quality of increased resolution and increased dynamic range.

Meanwhile, a plurality of AI models may be stored in the memory 140 or another device to which the electronic device 100 accesses. In this case, each AI model may be a model in which vertical images are learned according to the content type. For example, the content type of video may be sports, drama, game, news, education, entertainment, etc., but is not limited thereto.

That is, each AI model can be learned based on images having different types of content. In this way, by providing a plurality of AI models that are applied in various ways according to the type of image content, there is an advantage in that the efficiency of image quality improvement can be further improved when the changed vertical image is enlarged and converted.

For example, the control unit 160 divides horizontal images into shot units, receives information on continuous frame images included in the separated shots as an input, and selects an optimal AI model for each shot. there is.

10 shows an example of selecting an optimal AI model in S320.

Referring to FIG. 10 , the controller 160 may select an optimal AI model using a classifier. That is, the classifier is a second machine learning model learned according to a machine learning technique of supervised learning through learning data of input data and output data pairs (datasets).

At this time, the classifier may be learned using learning data including input data of consecutive frame images and output data for the content type (eg, sports, drama, game, news, education, entertainment, etc.) of these frame images. there is. Accordingly, the classifier has a function for a relationship between a continuous frame image as input data and a content type as output data, and expresses this function using various parameters.

For example, the classifier may express a relationship between continuous frame images and content types of these frame images using parameters of weights and biases. Accordingly, as shown in FIG. 10, when input data of consecutive frame images (F(t-1), F(t), F(t+1)) within a certain subunit is input to the learned classifier, Output data for the content type of the sub-unit images (F(t-1), F(t), F(t+1)) according to the corresponding function may be output.

For example, machine learning techniques applied to the AI model, the first machine learning model and the second machine learning model (classifier) include Artificial neural network, Boosting, Bayesian statistics, Decision tree, Gaussian process regression, Nearest neighbor algorithm, Support vector machine, random forests, symbolic machine learning, ensembles of classifiers, or deep learning, but is not limited thereto.

In particular, when the AI model, the first machine learning model, and the second machine learning model are deep learning models learned by deep learning techniques, the relationship between input data and output data is expressed in multiple layers (layers) These multiple expression layers are also referred to as “neural networks”. Such a deep learning model may have impressive performance in the field of image processing such as the present invention.

For example, deep learning techniques include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, etc. It can be done, but is not limited thereto.

In particular, when trying to create a new AI model for an image in a small unit (eg, a shot, etc.), the learning process takes a long time, so it is not suitable for real-time transmission. Accordingly, in the present invention, a plurality of AI models (ie, AI model DB) learned in advance according to the type of content of the image are previously stored in the memory 140 or a separate database device, and the control unit 160 is currently Search for and use the optimal AI model from the AI model DB. That is, the control unit 160 searches the pre-stored AI model DB for an AI model that matches the type of content output by the classifier for the input of a continuous frame image of a certain sub-unit, and the searched AI model is an optimal AI model applied to the sub-unit. can be selected with As a result, the present invention has the advantage of being more suitable for real-time reproduction of images.

In addition, since the AI model is learned according to the type of content of the video, it may be more effective to divide the AI model by shot unit or scene unit, which is a unit for a series of scenes of the same content type.

Thereafter, the controller 160 extracts a vertical image for each frame image from the horizontal image based on the reproduction area calculated in S310 (S330). That is, a vertical image of a corresponding frame may be extracted by leaving a portion corresponding to the reproduction area of the corresponding frame in each frame image of the horizontal image and removing the remainder.

The controller 160 may extract a vertical image of a corresponding frame by leaving a portion corresponding to the reproduction area of the corresponding frame in each frame of the horizontal image and removing the remainder.

Thereafter, the controller 160 enlarges and transforms the vertical image extracted in S330 by applying the AI model according to the optimal AI model information selected in S320 for each subunit (S340). That is, the vertical image, which is a low-quality image separated in S330 of the control unit 160, is input to the AI model. As a result, the AI model can output an enlarged image with improved quality according to the built-in function, that is, a high-quality vertical image enlarged by the size of the display 140 of the electronic device 100.

Of course, if necessary, the controller 160 may perform image processing in the process of cropping a part of the high-definition vertical image that has been enlarged and converted while being output from the AI model, or may perform size interpolation according to a resolution transformation technique. This may be performed when the enlarged and converted high-quality vertical image is larger than the display 140 of the electronic device 100 . For example, the resolution transformation technique may be bilinear, bicubic interpolation, down-sampling, etc., but is not limited thereto.

11 compares and shows an example of reproducing a horizontal image in an electronic device in a vertical holding state. That is, the left side is an image reproduced according to the prior art, and the right side is an image reproduced according to the present invention.

Referring to FIG. 11, when a horizontal image is played on a terminal in a vertical holding state, the present invention, unlike the prior art, can not only minimize the letter box or pillar, but also enlarge it while including the main object of the image, It is possible to reproduce a high-definition vertical video that has improved the low-quality problem that occurs while playing. As a result, the present invention has the advantage of increasing the user's visual satisfaction.

That is, while the prior art (left side of FIG. 11) occupies 25% of the screen, in the case of the present invention (right side of FIG. 11), the entire screen can be utilized, so the visual satisfaction is high and the exposure effect is large, so that the playback video is advertised. In one case, the advertising effect has a great advantage.

FIG. 12 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is in a horizontal grip state in S20 of the method according to an embodiment of the present invention.

Meanwhile, when image processing is performed on a horizontal image when the electronic device 100 is in a horizontal holding state, the controller 160 may perform S211 to S214 in S20 as shown in FIG. 12 .

That is, the controller 160 determines whether a black image is needed, that is, whether a pillar box or letter box area is needed (S211). That is, it is possible to determine whether a pillar box or letter box area is needed by comparing the aspect ratio of the display 130 according to the holding state of the electronic device 100 and the aspect ratio of the horizontal image.

If, as a result of S211, if a black image area is required (that is, a case in which a pillar box or letter box area of a horizontal image needs to be filled in and reproduced on the display 130 in a horizontal gripping state), the controller 160 controls the horizontal image The above-described second conversion is performed on (S212).

On the other hand, as a result of S211, when the black image area is unnecessary (ie, when the horizontal image can be reproduced in all pixels of the display 130 without filling the pillar box or letter box area of the horizontal image in the horizontal gripping state), The controller 160 performs the above-described first transformation on the horizontal image (S214).

Meanwhile, after performing S212, the controller 160 compares the resolution of the horizontal image on which the second conversion has been performed with the screen resolution of the display 130 in a horizontal holding state, and determines whether the resolution of the corresponding horizontal image is insufficient. Do (S213).

If, as a result of S213, the screen resolution of the display 130 in the horizontal holding state is greater than the resolution of the horizontal image on which the second conversion is performed, if the resolution of the corresponding horizontal image is insufficient, the controller 160 determines that the second conversion is performed. The above-described first transformation is additionally performed on the horizontal image to achieve high resolution of the horizontal image. As a result, the controller 160 can reproduce the horizontal image on which the second transformation and the first transformation are sequentially performed on the display 130 in high resolution.

On the other hand, as a result of S213, if the resolution of the horizontal image on which the second conversion is performed is greater than the resolution of the screen in the horizontal holding state of the display 130, and the resolution of the corresponding horizontal image is not insufficient, the controller 160 determines that the second conversion The performed horizontal image can be reproduced in high resolution.

Of course, the above-described S211 to S214 may be equally performed in S20 even when image processing is performed on a vertical image when the electronic device 100 is in a vertical holding state. However, in this case, in the above-described contents of S211 to S214, the horizontal image may be replaced with the vertical image, and the horizontal holding state may be replaced with the vertical holding state.

13 shows a more detailed flowchart of image processing for a horizontal image when the electronic device 100 is in a vertical holding state in S20 of the method according to an embodiment of the present invention.

In addition, when image processing is performed on a horizontal image when the electronic device 100 is in a vertical holding state, the controller 160 may perform S221 to S223 in S20 as shown in FIG. 13 .

That is, the control unit 160 performs the above-described third transformation on the horizontal image (S221).

Thereafter, the controller 160 compares the resolution of the vertical image for which the third conversion has been performed with the screen resolution of the display 130 in a vertically held state, and determines whether or not the resolution of the vertical image is insufficient (S222).

If, as a result of S222, if the screen resolution of the vertical holding state of the display 130 is greater than the resolution of the vertical image to which the third conversion has been performed, and the resolution of the corresponding vertical image is insufficient, the controller 160 determines that the third conversion is performed. The above-described first transformation is additionally performed on the vertical image to achieve high resolution of the vertical image. As a result, the controller 160 can reproduce the vertical image on which the third conversion and the first conversion are sequentially performed on the display 130 at high resolution.

On the other hand, as a result of S222, if the resolution of the vertical image on which the third conversion has been performed is greater than the resolution of the vertical image in the vertical holding state of the display 130, the controller 160 determines that the third conversion is performed. The performed vertical image can be reproduced in high resolution.

The present invention configured as described above does not insert a black image (pillar box or letter box) when the aspect ratio of the image and the aspect ratio of the electronic device that reproduces the image are different, but adjusts the aspect ratio of the image. There is an advantage in that a corresponding image can be appropriately matched to the aspect ratio of an electronic device that reproduces the image. In addition, the present invention not only effectively converts the aspect ratio of an image by adaptively performing at least one of the first to third conversions according to the holding state of the electronic device and the aspect ratio of the image, By additionally performing the first transformation after performing the second or third transformation, there is an advantage in that the converted image can be improved in resolution. In addition, the present invention can not only minimize the letter box or pillar box when reproducing an image, but also enlarge it while including the main object of the image, but reproduce the low-quality problem that occurs while enlarging it as a high-definition image. , there is an advantage of increasing the user's visual satisfaction. In addition, since the screen of the electronic device can be reproduced by maximally utilizing the screen of the electronic device, the viewing immersion is increased, and the exposure effect is great, so that the advertisement effect is great when the reproduced image is an advertisement. In addition, since the present invention can be applied to various picture quality improvement techniques, there is an advantage that it can be applied to real-time streaming as well as video on demand (VOD).

In the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and equivalents thereof.

100: electronic device 110: input unit
120: communication unit 130: display
140: memory 150: sensor unit
160: control unit

Claims (20)

An electronic device that converts and reproduces an image,
a sensor unit sensing a sensor value related to a holding state of the electronic device; and
A control unit that converts the image according to a holding state according to the sensor value and an aspect ratio of the image;
The control unit,
A first transformation that improves the image quality while maintaining the content of the video, a second transformation that adds and synthesizes new video content to the content of the video, and enlarges at least a part of the content of the video, but has a longer aspect ratio. An electronic device that performs at least one of the third transformations for improving image quality while enlarging an image.
According to claim 1,
The control unit,
Performing the first conversion or the second conversion on a horizontal image having a longer horizontal aspect ratio in a horizontal holding state of a longer horizontal aspect ratio;
Performing the third conversion on a horizontal image in the case of a vertical holding state of a longer vertical aspect ratio;
An electronic device that performs the first conversion or the second conversion on a vertical image having a longer vertical aspect ratio in the horizontal holding state or the vertical holding state.
According to claim 2,
The control unit,
In the horizontal holding state, when the horizontal image needs to be reproduced by filling a pillar box or letter box area, the second conversion is performed on the horizontal image, and the pillar box Alternatively, the electronic device that performs the first transformation on the horizontal image when a letter box area is unnecessary.
According to claim 1,
When the control unit performs the first conversion,
After generating an enlarged image by applying a pre-learned machine learning model to enlarge the image to an image larger than the screen size of the gripping state while improving the image quality, size interpolation is performed on the enlarged image to obtain the gripping state An electronic device that converts to an aspect ratio of .
According to claim 1,
When the controller performs the second conversion,
An electronic device that synthesizes new image contents included in previous or subsequent frame images with a current frame image based on correspondence point matching using previous and subsequent frame images.
According to claim 5,
When the controller performs the second conversion,
An electronic device for synthesizing new image content by performing boundary expansion based on a generative adversarial network (GAN) on the current frame image when the synthesized current frame image does not meet the resolution of the target image.
According to claim 5 or 6,
When the controller performs the second conversion,
To the synthesized current frame image, an enlarged image is generated by applying a pre-learned machine learning model to enlarge an image larger than the screen size of the holding state while improving image quality, and then size interpolation is performed on the enlarged image An electronic device that converts the aspect ratio of the holding state by using
According to claim 1,
The control unit performs the third conversion on a horizontal image having a longer horizontal aspect ratio when the vertical holding state has a longer vertical aspect ratio, and when performing the third conversion,
analyzing the content of the corresponding frame image for each frame image of the horizontal image and calculating a playback area corresponding to a part of the corresponding frame image;
A process of dividing the horizontal image into a plurality of sub-units and selecting an optimal AI model applied to each sub-unit according to the content of the horizontal video in the sub-unit from among a plurality of AI (artificial intelligence) models previously learned for each content type of the image;
extracting, for each frame image, a vertical image having a longer vertical aspect ratio in the horizontal image based on the play area;
An electronic device that each performs a process of enlarging and converting the extracted vertical image by applying the selected optimal AI model for each subunit.
According to claim 8,
In the calculation process, the control unit detects areas for objects and faces in each frame image, calculates a maximum reproduction area including the detected areas, and cuts the calculated maximum reproduction area to obtain a horizontally wider area. An electronic device that calculates at least one play area having a second aspect ratio that is longer in height.
According to claim 9,
The maximum reproduction area is an area including all of the detected areas when there are a plurality of the detected areas.
According to claim 8,
Each of the AI models is a model learned to generate an enlarged image with improved quality from low-quality images of different contents according to a machine learning technique.
According to claim 11,
The optimal AI model is an electronic device that improves at least one of image quality among resolution increase, noise removal, and dynamic range increase.
According to claim 8,
The control unit controls to reproduce an image enlarged according to the third conversion in all pixels of a display.
As a method for converting and reproducing an image in an electronic device,
sensing a sensor value related to a holding state of the electronic device; and
Including; performing conversion on the image according to the gripping state according to the sensor value and the aspect ratio of the image;
The step of performing the conversion is,
performing a first transformation to improve picture quality while maintaining the contents of the video;
performing a second transformation to add and synthesize new video contents to the contents of the video; and
performing a third transformation to improve picture quality while enlarging at least a portion of the contents of the video;
A method comprising at least one step among
According to claim 14,
The step of performing the conversion is,
Performing the first conversion or the second conversion on a horizontal image having a longer horizontal aspect ratio in a horizontal holding state of a longer horizontal aspect ratio;
Performing the third conversion on a horizontal image of a longer horizontal aspect ratio in a vertical holding state of a longer aspect ratio;
and performing the first transformation or the second transformation on a vertical image having a longer vertical aspect ratio in the horizontal holding state or the vertical holding state.
According to claim 14,
Performing the first transformation,
After generating an enlarged image by applying a pre-learned machine learning model to enlarge the image to an image larger than the screen size of the gripping state while improving the image quality, size interpolation is performed on the enlarged image to obtain the gripping state A method comprising converting to an aspect ratio of
According to claim 14,
Performing the second transformation,
A method comprising synthesizing new image contents included in previous or subsequent frame images with a current frame image based on correspondence point matching using previous and subsequent frame images.
According to claim 14,
Performing the second transformation,
and performing boundary expansion based on a generative adversarial network (GAN) on the corresponding current frame image when the synthesized current frame image does not meet the resolution of the target image to obtain new image content.
The method of claim 17 or 18,
Performing the second transformation,
To the synthesized current frame image, an enlarged image is generated by applying a pre-learned machine learning model to enlarge an image larger than the screen size of the holding state while improving image quality, and then size interpolation is performed on the enlarged image and converting it into an aspect ratio of the holding state.
According to claim 14,
The performing of the conversion includes performing the third conversion on a horizontal image having a longer horizontal aspect ratio when the vertical holding state has a longer vertical aspect ratio,
Performing the third transformation,
calculating a playback area corresponding to a part of the corresponding frame image by analyzing contents of the corresponding frame image for each frame image of the horizontal image;
Separating the horizontal image into a plurality of sub-units and selecting an optimal AI model applied for each sub-unit according to the content of the horizontal video within the sub-unit from among a plurality of AI (artificial intelligence) models pre-learned for each content type of the image;
extracting, for each frame image, a vertical image having a longer vertical aspect ratio in the horizontal image based on the play area; and
Enlarging and converting the extracted vertical image by applying the selected optimal AI model for each subunit;
How to include.

Images