KR20230009806A - An image processing apparatus and a method thereof - Google Patents

Abstract

Disclosed is an image processing method that includes the steps of: obtaining object information about an important object included in a first image from the first image; obtaining control information for picture quality processing; and obtaining a second image by performing image quality processing on the important object from the first image based on the object information and the user control information.

Description

An image processing apparatus and a method thereof

Various disclosed embodiments relate to an image processing device and an operating method thereof, and more particularly, to an image processing device rendering an image suitable for user characteristics and an operating method thereof.

It is difficult for the visually impaired to accurately recognize objects included in the image when viewing the image. Therefore, visually impaired people often use the magnification function to enjoy images. However, when the magnification function is used, the resolution of the image is also lowered, so there is a problem that the viewing desire of the visually impaired cannot be satisfied.

In addition, the visually impaired tend to focus on recognizing the overall outline of an object or the components of an object in an image without focusing on the details of the entire image.

Accordingly, there is a need for a technology for enabling visually impaired people to enjoy content by providing an image in which objects included in the image can be more accurately recognized according to the characteristics of the visually impaired person.

Various embodiments are intended to provide an image processing apparatus and an operating method for performing image quality processing on an important object included in an image.

Various embodiments are intended to provide an image processing apparatus and an operation method for performing image quality processing of an important object included in an image according to a user's preference.

An image processing device according to an embodiment includes a memory that stores one or more instructions and a processor that executes the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to obtain the first image from a first image. Acquire object information about an important object included in the image, obtain control information for image quality processing, and perform image quality processing on the important object based on the object information and the control information to obtain a second image. can do.

In an embodiment, the object information may include information on at least one of the type, location, and size of the important object.

In an embodiment, the processor detects a plurality of objects from the first image by executing the one or more instructions, outputs object identification information indicating the plurality of objects, and objects selected by the user in accordance with the output of the object identification information. can be identified as the important object.

In an embodiment, the control information may include control information on at least one of whether or not the object is enlarged, the degree of enlargement of the object, contour processing, and flattening processing.

In an embodiment, the processor may perform at least one of upscaling of the object, processing an outline around the object, and flattening the inside of the object according to the control information by executing the one or more instructions.

In an embodiment, the control information includes at least one of reasoning control information and real-time user control information, the reasoning control information is obtained from previous control history information of the user for a previous image, and the real-time user control information is the Real-time control information of the user for the first image may be included.

In an embodiment, the processor obtains the second image from the first image using a neural network by executing the one or more instructions, and the neural network determines an input image and a user's interest in the input image. It may be a neural network that learns, as a learning data set, an object region and a ground truth image obtained by image quality processing of the object region of interest to the user.

In an embodiment, the neural network may obtain a second image in which the object is quality-processed from at least one of the first image, the control information, and the object information.

In an embodiment, the neural network obtains a flattening parameter and a contour parameter for image quality processing of the object from at least one of the first image, the user control information, and the object information, and the processor receives a user control signal. According to this, a second image quality-processed may be obtained by adjusting the degree of blending between the image quality-processed according to the flattening parameter and the image quality-processed according to the contour parameter.

In an embodiment, the image quality processing includes at least one of processing the outline of the object, flattening the inside of the object, and upscaling the object, and the processing of the outline of the object includes detail, intensity of the outline of the object , color, and the flattening process inside the object includes a process for adjusting the degree of flattening inside the object, and upscaling the object includes processing of the object while maintaining the resolution of the object. It may include a process of enlarging the size.

According to an embodiment, an image processing method performed by an image processing device includes obtaining object information about an important object included in a first image from a first image, obtaining control information for image quality processing, and the object. and acquiring a second image by performing image quality processing on the important object from the first image based on information and the user control information.

A computer-readable recording medium according to an embodiment includes the steps of obtaining object information about an important object included in a first image from a first image, obtaining control information for image quality processing, and the object information and the A computer-readable record in which a program for implementing an image processing method is recorded, comprising obtaining a second image by performing image quality processing on the important object from the first image based on user control information. can be a medium.

An image processing device and method of operating the same according to an embodiment may perform image quality processing on an important object included in an image.

An image processing device and method of operating the same according to an embodiment may perform image quality processing on an important object included in an image according to a user's preference.

1 is a diagram for explaining that an image processing device outputs a quality-processed image according to an exemplary embodiment.
2 is an internal block diagram of an image processing device according to an embodiment.
3 is a diagram for explaining how a user inputs information on a visual impairment by using an image processing device according to an exemplary embodiment.
4 is an internal block diagram of the processor of FIG. 2 according to an embodiment.
5 is a diagram for explaining that an object information obtaining unit obtains object information from an input image according to an embodiment.
6 is a diagram for explaining that a control information acquisition unit obtains inference control information according to an embodiment.
7 is a diagram for explaining how a quality processing unit obtains an output image subjected to quality processing from an input image according to an exemplary embodiment.
8 is a diagram for explaining that an image processing device receives selection of an important object from a user according to an embodiment.
9 is an internal block diagram of an image processing device according to an embodiment.
10 is a diagram for describing image quality processing performed by an image processing device on an input image according to an exemplary embodiment.
11 is a diagram illustrating a user interface screen including a quality processing function according to an exemplary embodiment.
12 is a diagram illustrating a resultant image obtained by performing image quality processing on an input image by an image processing device according to an exemplary embodiment.
13 is a flowchart illustrating a method of acquiring a second image in which an important object is quality-processed from a first image according to an embodiment.
14 is a flowchart illustrating that image quality processing is performed according to user control information according to an embodiment.

In this disclosure, the expression “at least one of a, b, or c” means “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b” and c”, or variations thereof.

Hereinafter, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present disclosure with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

The terminology used in the present disclosure has been described as a general term currently used in consideration of the functions mentioned in the present disclosure, but it may mean various other terms depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. can Therefore, the terms used in the present disclosure should not be interpreted only as the names of the terms, but should be interpreted based on the meanings of the terms and the contents throughout the present disclosure.

Also, terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. .

As used in this specification, particularly in the claims, “above” and similar designations may refer to both the singular and plural. Further, unless there is a description that explicitly specifies the order of steps in describing a method according to the present disclosure, the recited steps may be performed in any suitable order. The present disclosure is not limited by the order of description of the described steps.

The appearances of phrases such as “in some embodiments” or “in one embodiment” in various places in this specification are not necessarily all referring to the same embodiment.

Some embodiments of the present disclosure may be represented as functional block structures and various processing steps. Some or all of these functional blocks may be implemented as a varying number of hardware and/or software components that perform specific functions. For example, functional blocks of the present disclosure may be implemented by one or more microprocessors or circuit configurations for a predetermined function. Also, for example, the functional blocks of this disclosure may be implemented in various programming or scripting languages. Functional blocks may be implemented as an algorithm running on one or more processors. In addition, the present disclosure may employ prior art for electronic environment setting, signal processing, and/or data processing. Terms such as “mechanism”, “element”, “means” and “composition” may be used broadly and are not limited to mechanical and physical components.

In addition, connecting lines or connecting members between components shown in the drawings are only examples of functional connections and/or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that can be replaced or added.

In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Also, in the specification, the term “user” means a person who uses an image processing device, and may include a consumer, an evaluator, a viewer, a manager, or an installer. Here, the consumer is a person who watches an image using an image processing device, and may include a visually impaired person.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining that the image processing device 100 outputs a quality-processed image according to an embodiment.

Referring to FIG. 1 , the image processing device 100 may be an electronic device capable of processing and outputting an image. According to an example, the image processing device 100 may be implemented as various types of electronic devices including displays. The image processing device 100 may be a fixed type or a mobile type, and may be a digital TV capable of receiving digital broadcasting, but is not limited thereto. The image processing device 100 may output video.

In an embodiment, the image processing device 100 may include an image quality processing module. The image quality processing module may provide a viewing assistance function. The viewing assistance function may refer to processing the quality of video and/or frames so that a user with a visual impairment can better recognize content.

The image quality processing module may be manufactured in the form of at least one hardware chip and mounted in the image processing device 100, or may be included in the image processing device 100 in the form of a chip or device. Alternatively, the image quality processing module may be implemented as a software module in the image processing device 100 .

According to an embodiment, the image processing device 100 may perform picture quality processing using a picture quality processing module. The image processing device 100 may first perform image quality processing on the first image 110 using the image quality processing module before outputting the first image 110 included in the video through the display. The image processing device 100 may perform image quality processing on each of a plurality of frames included in a video.

In an embodiment, the image processing device 100 includes a memory for storing one or more instructions and a processor for executing the one or more instructions stored in the memory, and the processor executes the one or more instructions so that based on object information and control information , image quality processing can be performed on important objects.

To this end, the image processing device 100 may obtain object information about an important object included in the first image 110 from the first image 110 .

In an embodiment, an important object is an object of interest that attracts a user's attention, and may refer to an object to be subjected to image quality processing.

In an embodiment, object information may include information on at least one of the type, location, and size of an important object.

When a plurality of objects are included in an image, the image processing device 100 may identify important objects in the image using various methods. For example, the image processing device 100 may identify an object that appears the most in a video including a corresponding image among a plurality of objects as an important object. Alternatively, the image processing device 100 may identify a currently speaking object as an important object. Alternatively, the image processing device 100 may identify an object located in the center of the image and having a size greater than a reference value as an important object.

In an embodiment, the image processing device 100 may identify an object selected by the user as an important object. For example, when detecting a plurality of objects in an image, the image processing device 100 may output object identification information indicating each detected object around the objects. The user may select one of a plurality of pieces of object identification information corresponding to the object identification information displayed on the screen. The image processing device 100 may identify an object corresponding to the object identification information selected by the user as an important object.

In an embodiment, the image processing device 100 may obtain control information for image quality processing. In an embodiment, the control information may be information representing a user control command for image quality processing of an important object. The control information may include control information on at least one of whether or not the important object is enlarged and how much, outline processing, and flattening processing.

In an embodiment, the image processing device 100 may perform image quality processing on important objects according to control information.

In an embodiment, the image processing device 100 may perform image quality processing on important objects so that visually impaired users can better recognize content.

In an embodiment, image quality processing may perform at least one of object upscaling, object contour processing, and object internal flattening processing.

In an embodiment, the control information may include at least one of reasoning control information and real-time user control information.

In an embodiment, the inferred control information may refer to control information inferred by considering the tendency of people with visual impairments or users.

In an embodiment, reasoning control information may be acquired based on general cognitive characteristics or preferences of visually impaired people. Although it may vary depending on the degree of visual impairment, in general, visually impaired people tend to want to see important objects in a larger size, prefer to omit detailed expressions inside objects, and prefer images with more clearly emphasized outlines of objects. there is Therefore, the image processing device 100 determines whether to enlarge an important object included in the image currently output on the screen, that is, the first image 110 of FIG. , control information on how to process outline processing, detail processing, etc. can be inferred.

In some cases, users may have tastes or preferences that are partially different from those of general visually impaired people. Accordingly, in an embodiment, inference control information may be obtained based only on past control records of users who use the image processing apparatus 100, not those with visual impairments. The image processing device 100 may accumulate information about what control command the user issued when a certain image was output, and infer the user's preference or taste from this.

Alternatively, in an embodiment, the reasoning control information may be obtained based on user preference information previously input by the user using the image processing device 100 .

In an embodiment, the real-time user control information may refer to user control information for an image currently displayed on a screen. Real-time user control information can be distinguished from inference control information in that it represents real-time control information of a user for the first image 110 currently displayed on the screen. There may be cases in which the user wants to control an important object included in the currently output first image 110 differently than in the past. For example, the user may want to magnify and view the important object included in the first image 110 larger than before. In this case, the user may input user control information to the image processing device 100 in real time using a remote control or the like. The image processing device 100 may receive real-time user control information from a user, and accordingly up-scale important objects to a larger size and output them.

In an embodiment, the image processing device 100 may obtain a second image 120 in which an important object is quality-processed from the first image 110 .

In an embodiment, the image quality processing may include at least one of processing the outline of the important object, flattening the inside of the important object, and upscaling the important object.

In an embodiment, the image processing device 100 may acquire the second image 120 from the first image 110 using artificial intelligence (AI). AI technology can be composed of machine learning (deep learning) and element technologies using machine learning. AI technology can be implemented by utilizing algorithms. Here, an algorithm or a set of algorithms for implementing AI technology is called a neural network. The neural network may receive input data, perform calculations for analysis and classification, and output result data.

In an embodiment, the image processing device 100 may perform image quality processing on an important object included in the first image 110 using a neural network.

In an embodiment, the neural network generates an input image, an object region in which the user is interested in the input image, user control information for image quality processing, and a ground truth image obtained by image quality processing of the object region in which the user is interested. It may be a neural network learned with a training data set.

In an embodiment, the neural network may obtain the second image 120 in which the important object is quality-processed from at least one of the first image 110, control information, and object information.

In an embodiment, the neural network performs at least one of upscaling an important object, displaying a thicker object outline, processing the color of an object outline with a specific color, or performing a flattening process inside an object. A second image 120 may be acquired.

In an embodiment, when the entire first image 110 is selected as an important object and image quality is processed, the second image 120 may be an image obtained by image quality processing of the entire first image 110 . Alternatively, when only the important object region is image quality-processed instead of the entire first image 110, the second image 120 may be an image including only the image quality-processed important object region. Alternatively, the second image 120 may include only the important object region in the first image 110 by performing image quality processing, and other regions other than the important object region may be the same image as the first image 110 .

In an embodiment, the neural network may obtain a flattening parameter and a contour parameter for image quality processing of an object from at least one of the first image 110 , user control information, and object information. The image processing apparatus 100 receives a control signal for blending between parameters selected by a user, and accordingly adjusts the degree of blending of the image processed according to the flattening parameter and the image processed according to the contour parameter to produce a quality-processed second image. can be obtained

Alternatively, in another embodiment, the neural network obtains a flattening parameter and a contour parameter for image quality processing of an object, and then automatically adjusts a degree of blending of the flattening parameter and the contour parameter to obtain a quality-processed second image as a result. may be

In this way, according to the embodiment, the image processing device 100 may process the quality of important objects according to user characteristics. The image processing device 100 may render an important object included in an image based on object information about the important object and control information for image quality processing. Accordingly, the user can watch the image in which the quality of the important object region of interest has been processed, and thus the satisfaction with viewing the image can be improved.

However, as an example, the image quality processing module may not be included in the image processing device 100 and may be implemented as a separate device from the image processing device 100 . That is, the image processing device 100 may communicate with an external device or server including an image quality processing module through a communication network (not shown). In this case, the image processing device 100 may transmit video to an external device or server through a communication network. Also, the image processing device 100 may transmit a control command to an external device or server. An external device or server may receive a video including a plurality of frames from the image processing device 100 and detect an important object region from the image using a picture quality processing module. The image quality processing module may perform image quality processing on the important object region according to a control command. The image quality processing module may perform image quality processing on an important object included in a frame using a neural network. The external device or server may transmit the quality-processed image to the image processing device 100 again through a communication network. The image processing device 100 may output a quality-processed image through a screen.

In this way, according to an embodiment, the image processing device 100 may output an image in which the important object region in the video has been quality-processed to the user.

2 is an internal block diagram of an image processing device 100 according to an embodiment.

The image processing device 100 shown in FIG. 2 may be the image processing device 100 of FIG. 1 .

In an embodiment, the image processing device 100 is a desktop computer, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a laptop PC ( laptop personal computer), netbook computer, digital camera, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), camcorder, navigation, wearable device, smart watch, home network system , a security system, and a medical device.

The image processing device 100 may be implemented as a curved display device, which is a screen having a curvature, or a flexible display device capable of adjusting the curvature, as well as a flat display device. The output resolution of the image processing device 100 may have various resolutions, such as, for example, High Definition (HD), Full HD, Ultra HD, or a resolution sharper than Ultra HD.

The image processing device 100 may output video. A video may consist of a plurality of frames. Videos may include items such as television programs provided by content providers or various movies or dramas through a VOD service. A content provider may mean a terrestrial broadcasting station or a cable broadcasting station that provides various contents including video to consumers, an OTT service provider, or an IPTV service provider.

Referring to FIG. 2 , the image processing device 100 may include a processor 210 and a memory 120 .

The memory 120 according to an embodiment may store at least one instruction. The memory 120 may store at least one program executed by the processor 110 . At least one neural network and/or predefined operating rules or AI models may be stored in the memory 120 . Also, the memory 120 may store data input to or output from the image processing device 100 .

The memory 120 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium.

In an embodiment, one or more instructions for performing image processing may be stored in the memory 120 .

In an embodiment, one or more instructions for obtaining object information about an important object included in an image may be stored in the memory 120 .

In an embodiment, one or more instructions for obtaining control information for picture quality processing may be stored in the memory 120 .

In an embodiment, one or more instructions for performing image quality processing on an important object may be stored in the memory 120 based on object information and control information.

In an embodiment, at least one neural network and/or a predefined operating rule or AI model may be stored in the memory 120 .

In an embodiment, the neural network stored in the memory 120 outputs an input image, an object region of interest to the user in the input image, and a ground truth image obtained by quality-processing the object region of interest to the user as training data. It may be a neural network trained with three.

In an embodiment, the neural network stored in the memory 120 may obtain a second image by performing image quality processing on the important object region from the first image.

The processor 110 controls overall operations of the image processing device 100 . The processor 110 may control the image processing device 100 to function by executing one or more instructions stored in the memory 120 .

In an embodiment, the processor 110 may obtain object information about an important object included in the image from the image.

In an embodiment, if there are a plurality of objects included in the image, the processor 110 may identify, as an important object, an object that appears most often in the video, an object currently uttered, or an object located in the region of interest among the plurality of objects. can

Alternatively, in an embodiment, the processor 110 may display object identification information indicating a plurality of objects on a screen, and may identify an object corresponding to the object identification information selected by the user as an important object.

In an embodiment, the processor 110 may obtain control information for image quality processing.

In an embodiment, the processor 110 may acquire a quality-processed image by performing quality processing on an important object based on object information and control information.

In an embodiment, the processor 110 may process the quality of the important object from at least one of the first image, control information, and object information using a neural network.

In an embodiment, the neural network may perform at least one of upscaling the important object, processing an outline around the important object, and flattening the inside of the important object.

3 is a diagram for explaining how a user inputs information on a visual impairment using the image processing device 100 according to an embodiment.

In an embodiment, the image processing device 100 may determine the degree of visual impairment of the user in advance in order to provide a viewing assistance function. When the image processing device 100 obtains reasoning control information using the neural network, the neural network takes time to determine the user's interaction history. That is, reliable control information cannot be inferred until the neural network becomes an optimized model through learning. Therefore, the image processing device 100 can quickly acquire more reliable inference control information by quickly learning a learning model based on the visual impairment information input from the user.

In an embodiment, the image processing device 100 may allow a visually impaired user to directly input information about the visual impairment in order to determine the degree of the user's visual impairment.

Alternatively, in an embodiment, the image processing device 100 may determine the user's level of visual impairment by providing an example screen to the user and allowing the user to select a preferred example.

In an embodiment, when the user selects activating the viewing assistance function, the image processing device 100 may show various images to the user in order to determine the level of the user's visual impairment. A user may provide user interaction while viewing an effect image to which various functions provided by the image processing device 100 are applied. For example, the user may input information about image quality processing preferred by the user for object size, contour processing, flattening processing, and the like.

FIG. 3A shows the image processing device 100 outputting a first base image 300 and a plurality of images 301, 303, and 305 generated by processing the outline of an object included in the first base image 300. show what

In an embodiment, the image processing device 100 may determine whether or not to preserve the contour of the object included in the first basic image 300, process the texture inside the object, remove noise, and strength of the contour. , the contour of the object included in the basic image 300 may be modified in various ways according to the degree of detail of the contour. The intensity of the outline may mean the degree of thickness or depth of the outline.

The user may view the plurality of images 301, 303, and 305 shown in FIG. 3A and select an image having an outline that is easy to recognize or the user prefers.

The image processing device 100 may store information about contour processing performed on an image selected by a user, and use the information as inference control information to process contours of objects included in an image in the future.

Users with visual impairments tend to prefer preservation of only the contours of an object or the shape of an object rather than minute details included in an image. In an embodiment, the image processing device 100 may receive a selection from the user of the degree of removal of detail inside the object, that is, the degree of flatness. Flattening the inside of an object may mean removing a texture or detailed expression inside the object so that the inside of the object is blurred or crushed. A greater degree of flattening can mean that more textures or details are removed, resulting in more crushing of the interior of the object.

The image processing device 100 may perform a flattening process on the second basic image 310 according to the degree of flattening selected by the user, and output the resultant image to the user.

3B , the image processing device 100 outputs a second basic image 310 and an image 313 on which flattening is performed according to the degree of flattening selected by the user for the second basic image 310 and the object included in the second basic image 310. show what

Referring to FIG. 3B , unlike the second basic image 310 , it can be seen that the flattened image 313 has the texture or pattern inside the petal removed.

The user may view the flattened image 313, confirm the flattening level, or select the flattening level again. When the user selects the level of flattening again, the image processing device 100 may perform the flattening process again on the second base image 310 according to the level of flattening selected by the user and output the resulting image to the user.

The image processing device 100 may store information about the level of flatness selected by the user and use it as reasoning control information. When image quality is processed in the future, the image processing device 100 may perform flattening on an object included in the image by using information about the degree of flattening selected by the user.

Although not shown in FIG. 3 , in an embodiment, the image processing device 100 may be selected by the user to what size to enlarge an important object included in the basic image. The image processing device 100 stores information about the degree of magnification selected by the user and uses this as reasoning control information to upscale the important object to the size selected by the user when processing the quality of the important object included in the image in the future. can

In this way, according to an embodiment, the image processing device 100 may receive a degree of visual impairment or preference from a user and generate control information based on the information input from the user.

4 is an internal block diagram of the processor 110 of FIG. 2 according to an embodiment.

Referring to FIG. 4 , the processor 110 may include an object information acquisition unit 111 , a control information acquisition unit 113 , and an image quality processing unit 115 .

In an embodiment, the object information acquisition unit 111, the control information acquisition unit 113, and the image quality processing unit 115 may be included in the processor 710 in a module form. A module may mean a functional and structural combination of hardware for implementing the technical idea of the present disclosure and software for driving the hardware. For example, a module may refer to a predetermined code and a logical unit of hardware resources for executing the predetermined code, and is not necessarily a physically connected code or limited to one type of hardware.

In an embodiment, the object information acquisition unit 111 may analyze an input image to detect an object and obtain object information about the object.

In an embodiment, object information may include information on at least one of the type, location, and size of an important object.

Since the important object is an object of interest, it may be located in a region of interest that mainly attracts the user's attention. The region of interest may vary according to user's visual characteristics and the like. For example, since people generally tend to view the central portion of the screen more than the edge portion, the central portion of the screen may be a region of interest. In addition, people generally tend to give more importance to objects located in the foreground than objects located in the background. Also, people tend to focus more on moving objects than on still objects.

In an embodiment, the object information acquisition unit 111 may identify an object located in the middle area of the input image as an important object.

In an embodiment, the object information obtaining unit 111 may identify a foreground object as a more important object than a background object.

In an embodiment, the object information acquisition unit 111 may consider whether the object is moving based on a plurality of frames and identify an object having greater motion than a stationary object as an important object.

When only one object is included in the input image, the object information obtaining unit 111 may identify one object as an important object.

When a plurality of objects are included in the input image, the object information obtaining unit 111 may identify an important object from among the plurality of objects. The important object may be a part of a plurality of objects included in the input image. The important object may be one, but is not limited thereto, and may be plural depending on the type or characteristic of an image, user's selection, and the like.

The region of interest may vary according to the type of image. In an embodiment, the object information acquisition unit 111 may identify the genre of the input image by analyzing the input image. In an embodiment, the object information acquisition unit 111 may differently identify the region of interest according to the genre of the input image.

For example, if the video is of a genre such as a drama or a movie, since the user tends to look at the main character or the current speaker with interest, the main character or the speaker may be an important object. Also, when a person appears in the image, the face region may be identified as the region of interest because the human has a strong desire to recognize the face shape of the person included in the image.

In an embodiment, when the input image is a movie or drama, the object information acquisition unit 111 may identify a person appearing most often in the input image as an important object. Alternatively, in an embodiment, the object information acquisition unit 111 may identify a speaker from the movement of an object included in an image, for example, a movement of lips included in a face, and identify the speaker as an important object.

When the input image is news, the user usually tends to pay close attention to subtitles displayed at the bottom of the news screen. Accordingly, in an embodiment, when the genre of the input video is news or the input video includes a subtitle, the object information acquisition unit 111 identifies the lower area of the screen or the area including the subtitle as an area of interest, A character included in the region of interest may be identified as an important object.

In an embodiment, the object information acquisition unit 111 may acquire information about the type of important object or the class to which the object belongs as object information.

In an embodiment, the object information may include information about the location of an important object. For example, the object information may include information about whether an important object is located in the center of the screen, at the bottom of the screen, or at the top right of the screen. The position of the important object may be acquired as a coordinate value in the screen.

In an embodiment, the object information may include information about the size of an important object. The size of an important object may mean a weight occupied by an important object in an image, a horizontal or vertical length of an important object, or a length of a diagonal or a diameter of an important object.

In an embodiment, the object information acquisition unit 111 may transmit object information to the control information acquisition unit 113 and the image quality processing unit 115 .

In an embodiment, the control information acquisition unit 113 may obtain control information.

In an embodiment, the control information may include at least one of reasoning control information and real-time user control information.

In an embodiment, the inferred control information may refer to control information inferred by considering the tendency of people with visual impairments or users. The inference control information may be control information obtained by inferring how to perform image quality processing on an important object included in an image to be currently displayed on a screen according to a tendency of people with visual impairments and/or users.

In an embodiment, the control information acquisition unit 113 may obtain inferential control information based on cognitive characteristics or preferences of visually impaired people. For example, the control information acquisition unit 113 may store information on cognitive characteristics of visually impaired people in advance. The control information acquisition unit 113 may infer control information for an input image to be displayed on the current screen based on pre-stored information on the cognitive characteristics of the visually impaired person.

Blind people may have visual characteristics unique to the blind, in addition to general visual characteristics possessed by people without visual impairment. For example, people with visual impairment tend to focus on recognizing the overall outline of an object or components within an object within an image without focusing on details of the entire image. In addition, people with visual impairment tend to want to see only a specific object in a larger size than to see the entire image together. In addition, visually impaired people tend to view objects with high color contrast with more interest. For example, when there is an image of a red maple leaf falling on a green lawn, visually impaired people have a great desire to see an object with high color contrast in more detail. Accordingly, the control information acquisition unit 113 may obtain control information preferred by visually impaired people as inferential control information.

In an embodiment, the control information acquisition unit 113 may obtain inference control information from a user's interaction history. Users may have tastes or preferences that are partially different from those of people with general visual impairments. In general, a user with a visual impairment often uses the image processing device 100 alone. That is, users with visual disabilities often use the image processing device 100 as a personalized device. Accordingly, the image processing device 100 may infer the user's preference or taste based on the control command history input by the user to process the image quality of the object.

The control information acquisition unit 113 may link the user's control history with the image, accumulate information on what control command the user gave when a certain image was output, and infer the user's preference or taste from this. . The control information acquisition unit 113 identifies patterns of user control commands by time-sequentially accumulating history of past control commands input by a user for a plurality of frames or a plurality of videos, and infers the user's preference or taste therefrom. can

The control information acquisition unit 113 may infer an image quality processing command for an important object by considering the user's control history as well as the cognitive characteristics and preferences of people with normal visual impairments. Alternatively, the control information acquisition unit 113 may infer an image quality processing command for an important object by excluding the cognitive characteristics of people with normal visual impairments and considering only the user's control history.

Alternatively, in an embodiment, the control information acquisition unit 113 may acquire inference control information based on user preference information previously input by the user using the image processing device 100 . The user may use the image processing device 100 to input information about the degree of visual impairment of the user or the degree of image quality processing preferred by the user in advance. The image processing device 100 may obtain inferred control information by inferring control information to be applied to the current image from user preference information previously input by the user.

In an embodiment, the control information obtaining unit 113 may receive real-time user control information from a user. The real-time user control information may mean real-time control information of a user for an input image currently output on a screen. The control information acquisition unit 113 may update pre-stored control information using real-time user control information input from the user. In an embodiment, the control information acquisition unit 113 may transmit updated control information to the quality processing unit 115 .

In an embodiment, the quality processing unit 115 may receive object information from the object information acquisition unit 111 and control information from the control information acquisition unit 113 .

In an embodiment, the image quality processing unit 115 may obtain a second image by performing image quality processing on an important object based on object information and control information.

In an embodiment, the image quality processing unit 115 may perform contour processing of important objects. Processing the outline of the important object may include processing at least one of the detail of the outline, the intensity of the outline, the color of the outline, and the degree of contrast between the outline and other regions.

In an embodiment, the image quality processing unit 115 may perform a flattening process inside the important object. The flattening process may include a process of adjusting a degree of flattening of an inner region of the important object except for the outline of the important object.

In an embodiment, the picture quality processor 115 may upscale important objects. The image quality processing unit 115 may increase the size of the important object and also perform sharpness processing on the important object. That is, the image quality processing unit 115 may increase the size of the important object and simultaneously improve the resolution through upscaling of the important object.

In an embodiment, the image quality processing unit 115 identifies an object to be subjected to image quality processing based on information about the genre, location, and size of an important object obtained from object information, matches control information to the identified region, An area identified according to the control information may be image quality processed.

For example, the image quality processing unit 115 may obtain information about a subtitle as an important object, a position of the subtitle at the bottom of the screen, and a size of the subtitle occupying an image. The picture quality processing unit 115 may identify a caption area and perform picture quality processing on the caption area based on a control command for picture quality processing obtained from the control information. For example, the picture quality processing unit 115 may locate a subtitle, crop an area including the subtitle, and upscale the cropped area to a predetermined size. The image quality processing unit 115 may perform image quality processing on an important object by performing a process of displaying the edge of the subtitle in purple, which is a color preferred by the user, thick and dark over a certain intensity.

5 is a diagram for explaining that the object information acquisition unit 111 obtains object information from an input image according to an embodiment.

In an embodiment, the object information acquisition unit 111 may include appropriate logic, circuits, interfaces, and/or codes capable of performing a function of acquiring the object information 520 from the input image 510 . In an embodiment, the object information acquisition unit 111 may acquire object information 520 for an object included in the input image 510 by analyzing the input image 510 based on a rule. When the data processing performance of the image processing device 100, operating system, CPU processing speed, etc. is difficult to perform a large amount of calculation in a short time, the object information acquisition unit 111 obtains the input image 510 based on rules It may be designed to acquire object information from the input image 510 by analyzing it.

Alternatively, in an embodiment, the object information acquisition unit 111 may obtain object information 520 from the input image 510 using a neural network. In an embodiment, the neural network included in the object information obtaining unit 111 will be referred to as a first neural network 500.

In an embodiment, the first neural network 500 may be a deep neural network (DNN) including two or more hidden layers. The first neural network 500 may include a structure in which input data is received, the input data is processed through hidden layers, and the processed data is output.

In FIG. 5 , a case in which the hidden layer of the first neural network 500 is a deep neural network (DNN) having two depths is shown as an example.

The object information acquisition unit 111 may analyze the input image 510 by performing an operation through the first neural network 500 . The first neural network 500 may perform learning through learning data. Here, the first neural network 500 can be designed in a variety of ways according to a model implementation method, result accuracy, result reliability, and processing speed and capacity of a processor.

The first neural network 500 includes an input layer 501, a hidden layer 502, and an output layer 503, and may perform calculations for genre determination and object detection. The first neural network 500 includes a first layer (Layer 1) 504 formed between the input layer 501 and the first hidden layer (HIDDEN LAYER1), the first hidden layer (HIDDEN LAYER1) and the second hidden layer ( A second layer (Layer 2) 505 formed between the HIDDEN LAYER2 and a third layer (Layer 3) 506 formed between the second hidden layer (HIDDEN LAYER2) and the output layer (OUTPUT LAYER) 503. can be formed

Each of the plurality of layers forming the first neural network 500 may include one or more nodes. For example, the input layer 501 may include one or more nodes 530 that receive data. In FIG. 5, a case in which the input layer 501 includes a plurality of nodes is shown as an example. Also, the input image 510 may be input to the plurality of nodes 530 . Here, two adjacent layers are connected by a plurality of edges (eg, 540) as shown. Since each node has a corresponding weight value, the first neural network 500 may obtain output data based on a value obtained by calculating, for example, multiplying the input signal and the weight value.

The first neural network 500 may be built as a model that is learned based on a plurality of learning data and infers the object information 520 from the input image 510 .

In an embodiment, the first neural network 500 may be a neural network trained by learning a predetermined dataset using a supervised learning method. The neural network can be trained to detect object information regions that are the same as or similar to the learned dataset.

In an embodiment, the first neural network 500 may be an algorithm that extracts features by analyzing and classifying input data by taking a plurality of learning data as input values. The first neural network 500 may be a model learned to extract object information from input data. To this end, the first neural network 500 may learn a method of detecting an important object in an image. The first neural network 500 can be trained using images of various genres, labels of each image, and types, locations, and sizes of important objects included in each image as a learning data set.

In an embodiment, the first neural network 500 may analyze an object that people consider important in an image and analyze the type of the object. More specifically, the first neural network 500 may train information about an important object and the type of the object in an image for which visually impaired people use an emphasis function as learning data.

In an embodiment, the first neural network 500 may learn a saliency dataset to find a region of interest. A saliency data set can also be called a saliency map. The saliency map may refer to a map that distinguishes a saliency area that attracts people's attention from other areas and expresses it. In an embodiment, the first neural network 500 may learn information about a saliency area in which visually impaired people are mainly interested. The saliency area may represent an area that attracts the attention of visually impaired people in the video frame, that is, an area with high visual concentration. For example, the first neural network 500 may be a model obtained by pre-learning a saliency area obtained by tracking the eyes of visually impaired people. The first neural network 500 calculates color change or distribution, edges, spatial frequency, structure, distribution, histogram, It may be learned to acquire a saliency map for an input video frame in consideration of a texture or the like. In an embodiment, the first neural network 500 may acquire object information about an important object by assigning a high weight to the saliency map area.

In an embodiment, the first neural network 500 may learn a segmentation dataset as training data in addition to the saliency data set. The first neural network 500 may identify the type of important object included in the image by considering semantic information of the object. The first neural network 500 may acquire object information on an important object by assigning different weights according to the type of important object.

Also, the first neural network 500 may be trained using an image understanding dataset as training data. The first neural network 500 may learn what an object is and what a spatial relationship between objects is by interpreting an object or a region of the object. The first neural network 500 may obtain object information about an important object according to an image understanding data set.

In an embodiment, the first neural network 500 includes images and labels of each image, information on whether visually impaired people are images using image quality processing, e.g., an emphasis function, image quality processing, e.g., enhanced images. Information about objects can be learned. In an embodiment, the first neural network 500 is trained to learn the relationship between an image using an emphasis function for visually impaired people and object information about an object whose quality has been processed in the image, and to obtain object information from the image. can

In an embodiment, the first neural network 500 may be trained using, as a set of correct answers, types of objects for which visual impairments have performed image quality processing or regions including such objects. The first neural network 500 infers/predicts the type, location, or size of an object from an input image, and can be repeatedly trained so that the predicted result is equal to three correct answers.

Specifically, in order to increase the accuracy of the result output through the first neural network 500, training is repeatedly performed from the output layer 503 toward the input layer 501 based on a plurality of training data, Weight values can be modified to increase the accuracy of the output result.

And, the first neural network 500 having finally modified weight values may use object information as a model. Specifically, the first neural network 500 may analyze information included in the input image 510 as input data and output object information 520 as a result.

The trained first neural network 500 may be installed in the object information acquisition unit 111 and used to acquire object information 520 from the input image 510 . When an input image 510 is input, the first neural network 500 detects a label of the input image 510 and an important object included in the input image 510, and at least among the type, location, and size of the important object. Object information including information about one can be obtained.

In an embodiment, the object information acquisition unit 111 may transmit the object information obtained using the first neural network 500 to the control information acquisition unit 113 and the image quality processing unit 115 .

In an embodiment, object information transmitted from the object information acquisition unit 111 to the control information acquisition unit 113 and object information transmitted to the quality processing unit 115 may be different from each other. If the object information transmitted from the object information acquisition unit 111 to the control information acquisition unit 113 is first object information, the first object information may be information obtained from the input image 510, that is, one frame. there is. Since the control information acquisition unit 113 obtains control information from the correlation between the genre of the video and the user control command, the object information acquisition unit 111 determines the genre of the frame obtained from each frame by the control information acquisition unit 113. information can only be transmitted. For example, the first object information may include only information about the genre of the input image 510, eg, whether the input image 510 is news or drama.

In an embodiment, if the object information transmitted from the object information obtaining unit 111 to the image quality processing unit 115 is second object information, the second object information includes not only the input image 510 but also the input image 510. It may include information obtained by analyzing multiple frames or videos. The second information may include information about the genre of the important object in the region of interest identified according to the genre of the input image 510 and the location and size of the important object. Since the image quality processing unit 115 performs image quality processing on important objects included in the input image 510, information such as the type, location, and size of important objects is required. Accordingly, in an embodiment, the object information obtaining unit 111 may analyze the video including the input image 510 to obtain object information about an important object therefrom, and transmit the obtained object information to the quality processing unit 115 . For example, when the genre of the input video 510 is news, the second object information is information that the genre of the important object is a subtitle, information about where the subtitle is located in the input video 510, and what size the subtitle is. may include information about

However, this is an example, and the object information transmitted from the object information acquisition unit 111 to the control information acquisition unit 113 and the object information transmitted to the quality processing unit 115 may be the same information. For example, the object information acquisition unit 111 may transmit genre and object information of a video, that is, information about the type, location, and size of an object, to both the control information acquisition unit 113 and the quality processing unit 115.

6 is a diagram for explaining that the control information acquisition unit 113 acquires inference control information according to an embodiment.

In an embodiment, the control information acquisition unit 113 may include appropriate logic, circuitry, interfaces, and/or codes capable of performing a function of acquiring inference control information from input data.

In an embodiment, the control information acquisition unit 113 may infer control information about an input image from user control information and object information about an input image using various algorithms.

In an embodiment, the control information obtaining unit 113 may infer control information based on a rule such as a program or an instruction, or obtain inferred control information using a neural network.

In an embodiment, FIG. 6 illustrates a case in which the control information acquisition unit 113 acquires inference control information using a neural network. If the neural network included in the control information acquisition unit 113 is referred to as the second neural network 600, in an embodiment, the second neural network 600 may learn a user's interaction history for important objects.

In an embodiment, the second neural network 600 may be implemented as an RNN.

In an embodiment, the second neural network 600 may learn information about image quality processing mainly used by visually impaired people for a specific image. The second neural network 600 may be trained using, as learning data, the type, degree of control, control frequency, etc. of control commands input by people with visual impairments in relation to image quality processing. In a state where the second neural network 600 has learned data of the general blind, the type, degree of control, and control frequency of a control command input by the user of the image processing device 100 in relation to image quality processing for an image In the case of additional learning, etc., the second neural network 600 may be a semi-supervised learning algorithm.

Alternatively, in another embodiment, the second neural network 600 may learn only the type, control degree, control frequency, etc. of a control command input by the user of the image processing device 100 in relation to image quality processing. . In this case, the second neural network 600 may be an unsupervised learning algorithm that learns a user's pattern with a user's own interaction history.

In an embodiment, the user's interaction history may include a control information history for at least one of contour processing, flattening processing, and upscaling processing for important objects. User interaction may be input through a user interface such as a remote controller or a keypad provided in the image processing device 100 .

In an embodiment, the second neural network 600 may accumulate a user interaction history and learn a user's preference, habit, usage pattern, etc. from this. The user interaction history is the type of image for which the user has selected image quality processing, the type of object that is subject to image quality processing in the image, the size of the object, the position of the object, the type or degree of image processing selected by the user, and the quality processing function used. It may include a history of at least one of the information about the viewpoint.

The type or degree of image quality processing selected by the user may include information about the thickness or detail of the outline of the object requested by the user, the color of the outline, the degree of flattening inside the object, and information about the size of the object to which it has been enlarged. there is. Information on the timing of using the quality processing function is such as whether the user used the quality processing function before playing the video, or whether the user used the quality processing function after the video was played, after a certain point in time, or only when a specific video was output. information may be included.

In an embodiment, the second neural network 600 may additionally learn the user's control command when the user newly inputs the control command.

In an embodiment, the second neural network 600 may receive object information about an input image from the object information obtainer 111 . Object information on the input image may include information on the type of the input image.

In an embodiment, the second neural network 600 may receive user control information for a corresponding image.

In an embodiment, the second neural network 600 determines the type of the input image from object information on the input image, and uses the user control information on the input image to allow the user to perform a quality processing function on the input image. It is possible to learn information about whether or not a control command was used and what control command the user gave to the input image.

In an embodiment, the second neural network 600 may output inference control information from a usage pattern obtained by statistically learning a user interaction history. The inference control information may be control information that infers what kind of image quality processing is to be performed for a current video based on a user's interaction history. Control information obtained from the second neural network 600 may be transmitted to the image quality processing unit 115 .

7 is a diagram for explaining how the quality processing unit 115 obtains an output image subjected to quality processing from an input image according to an exemplary embodiment.

Referring to FIG. 7 , the quality processing unit 115 may obtain output data from input data using a neural network.

If the neural network used by the image quality processing unit 115 is the third neural network 700, in an embodiment, the third neural network 700 is a Convolution Neural Network (CNN), Deep Convolution Neural Network (DCNN), or CAPSnet ( Capsnet) based neural network.

In an embodiment, the third neural network 700 receives various data, analyzes the input data, classifies the input data, and/or extracts features necessary for generating result data from the input data. It can be trained to discover or learn how to do it yourself. The third neural network 700 can be made into an artificial intelligence model with desired characteristics by applying a learning algorithm to a plurality of learning data. Such learning may be performed in the image processing device 100 itself or through a separate server/system. Here, the learning algorithm is a method of training a predetermined target device (eg, a robot) using a plurality of learning data so that the predetermined target device can make a decision or make a prediction by itself.

In an embodiment, the third neural network 700 may be trained as a data reasoning model through supervised learning using learning data as an input value.

Referring to FIG. 7 , a third neural network 700 may include an input layer 710, a hidden layer 720, and an output layer 730. In an embodiment, the hidden layer 720 may include a plurality of hidden layers. The third neural network 700 may include one or more hidden layers. For example, the third neural network 700 may be a deep third neural network 700 (DNN) including two or more hidden layers. A deep neural network (DNN) is a neural network that performs calculations through a plurality of layers, and the depth of the network may increase according to the number of internal layers that perform calculations. Deep neural network (DNN) calculations may include convolutional neural network (CNN) calculations and the like.

The third neural network 700 may include a plurality of hidden layers 720 between the input layer 710 and the output layer 730 . The depth or shape of the layer of the third neural network 700 can be designed in various ways in consideration of the accuracy of the result, the reliability of the result, the processing speed and capacity of the processor, and the like.

Each of the plurality of layers forming the third neural network 700 may include one or more nodes. For example, the input layer 710 may include one or more nodes receiving data. Here, the number of nodes included in the input layer 710 of the third neural network 700 may be the same as the number of nodes included in the output layer 730 . 7 shows a case in which the number of nodes of the first hidden layer included in the third neural network 700 is 50, the number of nodes of the second hidden layer is 100, and the number of nodes of the third hidden layer is 50. However, this is an example, and the number of nodes of the third neural network 700 may be designed in various ways.

In an embodiment, input data may be input to a plurality of nodes included in the input layer 710 .

People with visual impairments do not need such images because they do not perceive high frequency textures. Since visually impaired people place more importance on recognizing large outlines of large objects, it is required to generate an image in which the size of an important object is large, the inside of the important object is flat, and the outline of the important object is emphasized.

Accordingly, in an embodiment, the third neural network 700 may use images of various genres, object information about areas or objects that visually impaired people are interested in in each image, and controls input by visually impaired people for image quality processing. A set of correct answers reflecting information and visual characteristics of the visually impaired can be used as learning data.

In an embodiment, the third neural network 700 may receive object information from the object information acquisition unit 111 and control information from the control information acquisition unit 113 . The third neural network 700 may receive, as input data, images of various genres, object information acquired by the object information acquisition unit 111 and control information obtained by the control information acquisition unit 113 for each image. there is.

Nodes of two adjacent layers may be connected by a plurality of edges. Each edge has a corresponding weight value and operation information such as multiplication or addition. The third neural network 700 may perform an operation by multiplying or adding the weight value of the edge to the input data, and output the resultant value as a node value of the next layer connected to the edge. In an embodiment, the layers included in the third neural network 700 may be formed as a fully connected layer in which all nodes of a previous layer are connected to all nodes of a next layer.

The third neural network 700 passes values input to nodes through a function and then transfers them to the next layer. At this time, a function that determines an output of the next layer is called an activation function. The activation function may be a function that determines how to transfer input data to the next layer. In an embodiment, the third neural network 700 may use Rectified Linear Unit (ReLU) as an activation function used in the hidden layer. ReLU is one of the nonlinear activation functions, and has the advantage of being fast to learn and simple to implement. However, it is not limited thereto, and the third neural network 700 may use other nonlinear activation functions such as a sigmoid or hyperbolic tangent/tang function. Alternatively, the third neural network 700 may use a binary activation function or a linear activation function instead of a nonlinear function as an activation function.

In an embodiment, the third neural network 700 receives input data from nodes included in the input layer 710, performs an operation between each layer on the input data, and obtains the resultant value as output data. can That is, the third neural network 700 analyzes and classifies input data, extracts and processes features necessary for image quality processing, and obtains image quality processed result data as output data.

In an embodiment, the third neural network 700 may identify an important object region of the input image based on object information. In an embodiment, the third neural network 700 performs at least one of contour processing on the important object, flattening process inside the important object, and upscaling the important object based on the user control information, so that the important object Image quality processing may be performed on the region.

In an embodiment, the third neural network 700 may obtain a parameter for image quality processing of an object from at least one of images having various genres, user control information, and object information. Parameters for image quality processing may include at least one of a flattening parameter, a contour parameter, and an upscaling parameter.

In an embodiment, the third neural network 700 may adjust a parameter value using user control information, and accordingly, image quality of an important object may be processed.

In an embodiment, the third neural network 700 may perform contour processing of important objects according to contour parameters. The third neural network 700 determines how much detail of the outline of an important object to be maintained, how strong the outline is, that is, how thick or dark it is, and what color the outline is, based on user control information. , the level of color contrast between the background area and the color of the outline can be determined.

In an embodiment, the third neural network 700 may flatten the inside of the important object according to the flattening parameter. The third neural network 700 may perform a process of adjusting the degree of flattening of the inner region of the important object, excluding the contour of the important object, in a method preferred by the user according to the user control information.

In an embodiment, the third neural network 700 may upscale the important object according to the upscaling parameter. The third neural network 700 may enlarge the size of the important object to a size preferred by the user according to user control information. At the same time, the third neural network 700 may perform sharpness processing on an important object whose size has increased.

In an embodiment, the third neural network 700 may upscale the size of an important object by creating a new pixel between pixels instead of simply copying the size of a pixel to increase the number of pixels. The third neural network 700 may generate new pixels in various ways. The third neural network 700 may generate a new pixel between pixels using various upsampling methods such as a nearest method, a bilinear method, a joint bilateral method, and the like. That is, the third neural network 700 simultaneously increases the size of the important object while adjusting the resolution of the important object, so that the size of the important object increases and the resolution is also improved.

In an embodiment, the third neural network 700 may learn a degree of blending between a result of applying a flattening parameter and a result of applying a contour parameter based on user control information. The third neural network 700 learns the blending degree preferred by the user by learning the control information, and accordingly blends the results to which the flattening parameter and the contour parameter are applied, so that the inside of the important object is flattened and the contour of the important object is contoured. can obtain the final result performed.

In another embodiment, the third neural network 700 may output the flattening parameter and the contour parameter of the important object as output data instead of acquiring the image quality processing result of the important object as output data. In this case, the image quality processing unit 115 may additionally receive blending information about a result of applying the flattening parameter and a result of applying the contour parameter from the user. When the user feels that the flattening degree is too strong and the contour processing is too weak, the user may adjust blending information of the flattening parameter and the contour parameter using the user interface. The image quality processing unit 115 may adjust the degree of blending of parameters according to a control signal from a user to finally obtain a quality-processed image.

In an embodiment, the third neural network 700 repeatedly performs training in the direction from the output layer 730 to the input layer 710 based on a plurality of training data to increase the accuracy of the result, resulting in an output result. Weight values can be modified to increase the accuracy of .

In an embodiment, the third neural network 700 may obtain a difference between ground truths of quality-processed result data output from the output layer 730 as a loss function. The ground truth may be data quality-processed according to a quality-processing method preferred by the user for an important object of interest to the user in the image. The third neural network 700 may receive the loss function again and continuously modify weight values of edges included in the hidden layer 720 so that the loss function becomes a minimum. The weight values of the edges may be optimized through iterative learning, and may be repeatedly modified until the accuracy of the result satisfies a predetermined degree of reliability. The third neural network 700 may be formed by weight values of edges finally set.

According to the embodiment, an operation of learning how to perform image quality processing on an important object included in an image using the third neural network 700 may be performed in advance before being installed in the image processing device 100. . When some of the plurality of learning data is changed, the learning model may also be updated. If new training data is used or added at predetermined intervals, the third neural network 700 may learn how to perform image quality processing from the new training data again, and the learning model may be updated accordingly. .

In an embodiment, an operation of learning how to perform image quality processing using the third neural network 700 may be performed in an external computing device (not shown). An operation of learning how to perform image quality processing on an object included in an image using the third neural network 700 may require a relatively complex amount of computation. Accordingly, the computing device may perform a learning operation, and the image processing device 100 may receive a learning model from the computing device through a communication network. Alternatively, a manufacturer of the image processing device 100 may install the third neural network 700 trained by the computing device into the image processing device 100 so that the learning model is an image quality-processed by the image processing device 100. can be used to obtain

Alternatively, in another embodiment of the present disclosure, the image processing device 100, not the computing device, may directly perform the learning operation through the third neural network 700. In this case, the image processing device 100 may obtain learning data, determine a learning model by learning the third neural network 700 with the learning data, and acquire a quality-processed image through the determined learning model. .

In an embodiment, the trained third neural network 700 may receive real-time video, object information, and control information as input data. The third neural network 700 may receive object information from the object information acquisition unit 111 for an input image input in real time. The third neural network 700 may identify the type, location, or size of an important object in a real-time image based on object information. The third neural network 700 receives control information to be applied to the real-time video generated based on the user's interaction history from the control information acquisition unit 113, and selects an important object included in the real-time video according to the control information. By processing the image quality, the important object can acquire the image image processed for image quality as output data.

8 is a diagram for explaining how the image processing device 100 receives selection of an important object from a user according to an embodiment.

In an embodiment, the image processing device 100 may detect an object from an input image. When a plurality of objects are included in the input image, the image processing device 100 may identify an important object from among the plurality of objects. The important object may be a part of a plurality of objects included in the input image.

In an embodiment, the image processing device 100 may directly receive a user's selection of an object to be processed as an important object from among a plurality of objects. To this end, the image processing device 100 may output object identification information for identifying each of the plurality of objects around the plurality of objects.

FIG. 8 illustrates that, when the image processing device 100 detects a plurality of objects in an image, object identification information is displayed on a screen to receive a user's selection of which object to determine as an important object among the plurality of objects.

8A shows that object identification information is output on the screen when the image output on the current screen is a still image. When a user tries to watch a video, the image processing device 100 may output a still image including a plurality of people appearing in the corresponding video and output object identification information to the still image. The image processing device 100 may output object identification information 811 , 812 , 8123 , and 814 for each of a plurality of objects around the object. A user may select one of a plurality of pieces of object identification information using a user interface such as a remote controller. For example, when the user selects the object identification information 812, the image processing device 100 displays the color, thickness, transparency, etc. of the object identification information 812 selected by the user differently from other unselected object identification information to the user. You can notify that a specific object is selected.

8B and 8C are diagrams illustrating that the image processing apparatus 100 identifies a speaker while playing a video, and outputs object identification information corresponding to the speaker to surroundings of the speaker. The image processing device 100 may analyze a currently output image to identify a speaker in the image. For example, when an object included in a currently playing video is a person, the image processing device 100 may analyze a person's face and identify a subject whose lips are moving as a speaker. Alternatively, the image processing device 100 may receive video frames and audio frames together, analyze and classify characteristics of the input video frames and audio frames, and identify the location of the current speaker, thereby detecting the speaker.

FIG. 8A illustrates that the image processing device 100 outputs object identification information 821 for indicating the speaker to the surroundings of the speaker at a specific time point, for example, time t0. The user can select the object identification information 821 around the speaker using a remote control, so that the speaker is identified as an important object.

When the user does not select the output object identification information 821 at time t0, the image processing device 100 displays, at time t1, object identification information 831 for displaying a speaker different from the speaker at time t0 on the screen. can be output to 8C shows that the image processing device 100 outputs object identification information 831 for indicating a speaker at time t1 to the screen. The user may select the object corresponding to the object identification information 831 output at time t1 as an important object by selecting the object identification information 831 output on the screen at time t1.

As such, according to the embodiment, the image processing device 100 may directly select an important object to perform image quality processing from among a plurality of objects from the user. Accordingly, the image processing device 100 may allow the user to better recognize an object desired by the user by performing image quality processing on the important object selected by the user.

9 is an internal block diagram of an image processing device 100a according to an embodiment.

The image processing device 100a of FIG. 9 may be an example of the image processing device 100 of FIG. 2 . Hereinafter, descriptions overlapping with those described in FIG. 2 will be omitted.

Referring to FIG. 9 , the image processing device 100a includes a tuner unit 910, a communication unit 920, a sensing unit 930, an input/output unit 940, a video, in addition to a processor 210 and a memory 220. A processing unit 950, a display unit 960, an audio processing unit 970, an audio output unit 980, and a user interface 990 may be further included.

The tuner unit 910 determines the frequency of a channel desired to be received by the image processing device 100a among many radio wave components through amplification, mixing, resonance, etc. You can select only by tuning. The content received through the tuner unit 910 is decoded and separated into audio, video and/or additional information. The separated audio, video and/or additional information may be stored in the memory 220 under the control of the processor 210 .

The communication unit 920 may connect the image processing device 100a to a peripheral device, an external device, a server, or a mobile terminal under the control of the processor 210 . The communication unit 920 may include at least one communication module capable of performing wireless communication. The communication unit 920 may include at least one of a wireless LAN module 921, a Bluetooth module 922, and a wired Ethernet 923 corresponding to the performance and structure of the image processing device 100a.

The Bluetooth module 922 may receive a Bluetooth signal transmitted from a peripheral device according to the Bluetooth communication standard. The Bluetooth module 922 may be a Bluetooth Low Energy (BLE) communication module and may receive a BLE signal. The Bluetooth module 922 may continuously or temporarily scan a BLE signal to detect whether a BLE signal is received. The wireless LAN module 921 may transmit and receive Wi-Fi signals with neighboring devices according to Wi-Fi communication standards.

The sensing unit 930 detects a user's voice, a user's image, or a user's interaction, and may include a microphone 931, a camera unit 932, a light receiving unit 933, and a sensing unit 934. The microphone 931 may receive an audio signal including a user's utterance or noise, convert the received audio signal into an electrical signal, and output the converted electrical signal to the processor 210 .

The camera unit 932 may include a sensor (not shown) and a lens (not shown), take and capture an image formed on the screen, and transmit it to the processor 210 .

The light receiving unit 933 may receive light signals (including control signals). The light receiving unit 933 may receive an optical signal corresponding to a user input (eg, touch, pressure, touch gesture, voice, or motion) from a control device such as a remote controller or a mobile phone.

The input/output unit 940 receives video (eg, a dynamic image signal or still image signal), audio (eg, audio) from an external device of the image processing device 100a under the control of the processor 210 . signals, music signals, etc.) and additional information may be received.

The input/output unit 940 is one of a High-Definition Multimedia Interface port (941), a component jack (942), a PC port (943), and a USB port (944). can include The input/output unit 940 may include a combination of an HDMI port 941 , a component jack 942 , a PC port 943 , and a USB port 944 .

The video processing unit 950 processes image data to be displayed by the display unit 960, and performs various image processing operations such as decoding, rendering, scaling, noise filtering, frame rate conversion, and resolution conversion for image data. can do.

In an embodiment, the video processing unit 950 may perform image quality processing on important objects according to control information. The video processing unit 950 may perform quality processing such as increasing the size of an important object, converting the resolution, flattening the inside of the important object, and processing the contour of the important object.

The display unit 960 may display content received from a broadcasting station, an external server, or an external storage medium on a screen. The content is a media signal and may include a video signal, an image, a text signal, and the like.

In an embodiment, the display unit 960 may output an image of an important object quality-processed by the video processing unit 950 on a screen.

In an embodiment, the display unit 960 may output a screen including object identification information around the object.

In an embodiment, the display unit 960 may output an interface screen for receiving a user's selection of a viewing assistance function.

The audio processor 970 processes audio data. The audio processing unit 970 may perform various processes such as decoding or amplifying audio data and filtering noise.

The audio output unit 980 controls audio included in the content received through the tuner unit 910 under the control of the processor 210, audio input through the communication unit 920 or the input/output unit 940, and memory ( 220) can output audio stored in it. The audio output unit 980 may include at least one of a speaker 981, headphones 982, and a Sony/Philips Digital Interface (S/PDIF) 983.

The user interface 990 may receive a user input for controlling the image processing device 100a. The user interface 990 includes a touch panel that detects a user's touch, a button that receives a user's push manipulation, a wheel that receives a user's rotation manipulation, a keyboard, and a dome switch, and voice recognition. It may include various types of user input devices including a microphone for sensing motion, a motion sensor for sensing motion, and the like, but is not limited thereto. When a remote control or other mobile terminal controls the image processing device 100a, the user interface 990 may receive a control signal received from the mobile terminal.

In an embodiment, the user interface 990 may select whether or not to use the image quality processing function from the user. In an embodiment, the user interface 990 may be selected to use which function among image quality processing functions and to what extent.

In an embodiment, the user interface 990 may receive object identification information selected from a screen by a user.

10 is a diagram for describing image quality processing performed by the image processing device 100 on an input image according to an exemplary embodiment.

Referring to FIG. 10 , the image processing device 100 may receive an input image 1010 and process it, respectively, to obtain a first processed image 1021 and a second processed image 1025 .

In an embodiment, the image processing device 100 may adjust the degree of flattening of the inside of the object included in the input image 1010 . The image processing device 100 may determine how much detail inside an object included in the input image 1010 should be deleted according to a user's interaction history or real-time control information. The image processing apparatus 100 may obtain a flattening parameter for image quality processing of an important object from at least one of an image, user control information, and object information by using the third neural network 700 . The flattening parameter may be a parameter representing a degree of flattening.

In an embodiment, the image processing device 100 may flatten the inside of an object included in the input image 1010 using various filters. For example, the image processing device 100 may adjust the degree of blurring of internal details of objects included in the input image 1010 by using an edge preserving smoothing filter.

In an embodiment, the image processing device 100 may adjust the degree of flattening of an important object by adjusting a flattening parameter of a filter based on user control information.

In an embodiment, the first processed image 1021 shows an image on which flattening has been performed on an object included in the input image 1010 . Looking at the first processed image 1021, it can be seen that all detailed expressions such as patterns inside the petals, stamens, and pollen have disappeared and are flattened.

In an embodiment, the image processing device 100 may process the outline of an object included in the input image 1010 . In an embodiment, the image processing device 100 may obtain a contour parameter for image quality processing of an important object from at least one of image, user control information, and object information using the third neural network 700. . The contour parameter may be a parameter for the degree of preservation of the contour line, the removal of surrounding noise, the thickness and thickness of the contour line, whether to remove details of the contour line, and the like.

In an embodiment, the image processing device 100 may process the contour of the important object by adjusting the contour parameter based on user control information.

In an embodiment, the second processed image 1025 shows an image in which the outline of an object included in the input image 1010 has been processed. Looking at the second processed image 1025, it can be seen that only the shape of the petal is included as a thick outline, and all other detailed expressions around it have been removed. Also, it can be seen that the color of the outline is different from the background or inside the object.

In an embodiment, the third neural network 700 included in the image processing device 100 may learn the blending degree of the flattened image and the contoured image based on user control information. The third neural network 700 learns user control information to learn the degree of blending preferred by the user, and accordingly blends the image to which the flattening parameter is applied and the image to which the contour parameter is applied, so that the inside of the important object is flattened and the image of the important object is flattened. Contours may obtain a final result of contour processing. That is, in FIG. 10 , the third neural network 700 may automatically adjust the degree of blending between the first processed image 1021 and the second processed image 1025, and obtain a final image accordingly.

In another embodiment, the image processing device 100 may receive a control command for a blending degree of an image to which a flattening parameter is applied and an image to which a contour parameter is applied from a user.

In an embodiment, the image processing device 100 adjusts the degree of blending between the flattening parameter and the contour parameter-applied image according to the user's control command, and adjusts the intensity of the flattening degree and the degree of contour processing to obtain a final result. can be obtained

In this case, the third neural network 700 only performs contour processing and the first processed image 1021 acquired by performing only flattening on the important object, instead of outputting the result of image quality processing on the important object as output data. The acquired second processed image 1025 may be output as output data.

The image processing device 100 may additionally receive blending information between the image on which only the flattening process has been performed and the image on which only the outline process has been performed, from the user. When the user feels that the flattening degree is too strong and the outline processing is too weak, the user can adjust the blending degree between the flattened image and the outline processed image using the user interface. The image quality processing unit 115 may adjust the degree of blending between images according to a control signal from a user to finally obtain a quality-processed image.

11 is a diagram illustrating a user interface screen including a quality processing function according to an exemplary embodiment.

Before viewing an image, the user may select a viewing assistance function in advance to select whether or not to process image quality or the degree of image quality processing. Alternatively, when the user wants to perform image quality processing on a specific object while watching a video, the user may select the viewing assist function to be activated so that the image quality processing is performed on the specific object in real time.

The image processing device 100 may output an interface screen for providing a viewing assistance function. In an embodiment, the image processing device 100 may output a screen setting interface screen for setting various functions of the screen among various setting functions.

Referring to FIG. 11(a) , the screen setting interface screen 1110 may include a menu 1115 for selecting a viewing assistance function. The menu 1115 for selecting the viewing assistive function may include a bar-shaped menu bar for adjusting whether to apply the viewing assistive function or the degree of application of the viewing assistive function. However, this is an example, and the screen setting interface screen 1110 or the menu 1115 for selecting a viewing assistance function may have various structures, arrangements, and shapes.

11( b ) shows another example of a screen setting interface screen 1120 . Another screen setting interface screen 1120 may include a menu 1121 for contour processing, a menu 1123 for size adjustment, and a menu 1125 for flattening. The user may view another screen setting interface screen 1120 and select the type of image quality processing to be performed on the important object.

Also, although not shown in FIG. 11 , when a user selects a quality processing type, the image processing device 100 may output a user interface screen for inputting information on how much the selected type of image quality processing is to be performed. there is. The user can select and input the outline processing, leveling degree, and magnification degree.

12 is a diagram illustrating a resultant image obtained by performing image quality processing on an input image by the image processing device 100 according to an exemplary embodiment.

Referring to FIG. 12 , the image processing device 100 may analyze an input image 1210 and detect an important object included in the input image 1210 . The image processing device 100 may analyze the important object and identify that the important object is a person.

In an embodiment, the image processing device 100 may perform image quality processing on all people. The image processing device 100 may perform only flattening and contour processing of an object based on user control information, that is, according to the user's past history or taste, while maintaining the size of the object without enlarging the object. For example, based on the user interaction history, the image processing device 100 identifies that the user did not use the magnification function when the size of an important object is greater than a certain size, and determines the size of the object included in the input image 1210. Depending on this, it is possible to determine whether or not to use the enlargement function. When the size of the object is larger than a predetermined size, the image processing device 100 may perform only other picture quality processing without enlarging the object included in the input image 1210 .

In an embodiment, the image processing device 100 may obtain a first resultant image 1220 by flattening the entire object and processing the contour. It can be seen that the object included in the first result image 1220 has the same size as the object in the input image 1210 . In addition, it can be seen that the object included in the first result image 1220 has been subjected to contouring and flattening of the object included in the input image 1210 .

Alternatively, in another embodiment, when the object included in the input image 1210 is a person, the image processing device 100 may identify only the person's face as the important object. For example, based on the user interaction history, the image processing device 100 identifies only the human face region 1211 of the input image 1210 as an important object region when the user prefers to enlarge and watch only the human face. can do. The image processing device 100 may crop a face region that is an important object region and perform image quality processing on the cropped important object region. The image processing device 100 may enlarge the cropped important object region by the size of the input image 1210 and simultaneously adjust the resolution of the important object region. Also, the image processing device 100 may perform flattening and contour processing preferred by the user on the important object region.

In an embodiment, the image processing device 100 may acquire a second resultant image 1230 by performing image quality processing only on important objects. It can be seen that the second result image 1230 includes only the face region among the objects included in the input image 1210 . In addition, it can be seen that the outline of the face of the object included in the input image 1210 is processed, and the remaining regions of the face except for the eyes, nose, and mouth are flattened.

13 is a flowchart illustrating a method of acquiring a second image in which an important object is quality-processed from a first image according to an embodiment.

Referring to FIG. 13 , the image processing device 100 may receive a first image and obtain object information about an important object included in the first image (operation 1310).

In an embodiment, the image processing device 100 may analyze the first image and detect an important object from the first image. The image processing device 100 may obtain information about at least one of the type, location, and size of an important object as object information.

In an embodiment, the image processing device 100 may obtain user control information for picture quality processing (operation 1320).

In an embodiment, the user control information may include control information on at least one of whether the object is enlarged, the degree of enlargement of the object, contour processing, and flattening processing.

In an embodiment, the image processing device 100 may obtain a second image by performing image quality processing on an important object based on object information and control information (operation 1330).

In an embodiment, the image processing device 100 may perform image quality processing on an object included in the first image by referring to user control information for the location, size, type, etc. of the acquired object based on the object information. there is.

14 is a flowchart illustrating that image quality processing is performed according to user control information according to an embodiment.

In an embodiment, the user control information may include at least one of reasoning control information and real-time user control information.

In an embodiment, the image processing device 100 may obtain reasoning control information based on a user's interaction history pre-stored in the image processing device 100 .

In an embodiment, the image processing device 100 may perform image quality processing on an important object according to inference control information (operation 1410).

In an embodiment, the image processing device 100 may identify whether real-time control information of the user has been received (operation 1420).

In an embodiment, when user's real-time control information is input, the image processing device 100 may additionally perform image quality processing on the object according to the real-time control information (operation 1430). When the user's real-time control information conflicts with the reasoning control information, the image processing device 100 may perform image quality processing on the object according to the user's real-time control information. The image processing device 100 may output a quality-processed image.

A method and apparatus for operating an image processing device according to some embodiments may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media.

In addition, the above-described image processing apparatus and method of operation according to an embodiment of the present disclosure include acquiring object information about an important object included in the first image from a first image, and acquiring user control information for image quality processing. A program for implementing an image processing method comprising the step of obtaining a second image by performing image quality processing on the important object from the first image based on the object information and the user control information. may be implemented as a computer program product including a computer-readable recording medium/storage medium.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is semi-permanently stored in the storage medium and temporary It does not discriminate if it is saved as . For example, the 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store or between two user devices (eg smartphones). It can be distributed (e.g., downloaded or uploaded) directly or online. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored in the memory of the manufacturer's server 150, the application store's server 150, or the relay server 150. At least temporarily stored in a storage medium readable by a device, or may be temporarily created.

The above description is for illustrative purposes, and those skilled in the art will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

Claims (20)

In the image processing device,
a memory that stores one or more instructions; and
a processor to execute the one or more instructions stored in the memory;
By executing the one or more instructions, the processor:
Obtain object information about an important object included in the first image from the first image;
Acquiring control information for image quality processing;
Based on the object information and the control information, a second image is acquired by performing image quality processing on the important object. The image processing apparatus according to claim 1, wherein the object information includes information on at least one of a type, location, and size of the important object. The method of claim 1 , wherein the processor executes the one or more instructions to:
Detecting a plurality of objects from the first image;
outputting object identification information indicating the plurality of objects;
An image processing device that identifies an object selected by a user as the important object in accordance with the output of the object identification information. The image processing apparatus according to claim 1, wherein the control information includes control information on at least one of whether an object is enlarged, a degree of enlargement of the object, an outline process, and a flatten process. 5. The method of claim 4, wherein the processor by executing the one or more instructions:
According to the control information, at least one of upscaling the object, processing an outline around the object, and flattening the inside of the object is performed. The method of claim 1, wherein the control information includes at least one of reasoning control information and real-time user control information,
The reasoning control information is obtained from the user's previous control history information for a previous image,
The real-time user control information includes real-time control information of the user for the first image. The method of claim 1 , wherein the processor acquires the second image from the first image using a neural network by executing the one or more instructions;
The neural network is a neural network that learns an input image, an object region of interest to the user in the input image, and a ground truth image obtained by image quality processing of the object region of interest to the user as a learning data set. , image processing device. The image processing apparatus according to claim 7 , wherein the neural network obtains a second image in which the object is quality-processed from at least one of the first image, the control information, and the object information. The method of claim 7, wherein the neural network obtains a flattening parameter and a contour parameter for image quality processing of the object from at least one of the first image, the user control information, and the object information,
wherein the processor adjusts a blending degree of an image quality-processed according to the flattening parameter and an image quality-processed according to the contour parameter according to a user control signal to obtain a quality-processed second image. The method of claim 8 or 9, wherein the image quality processing includes at least one of processing an outline of the object, flattening processing inside the object, and upscaling the object,
The processing of the outline of the object includes processing of at least one of detail, strength, and color of the outline of the object;
The flattening process inside the object includes a process for adjusting the degree of flattening inside the object,
Upscaling the object includes a process of enlarging a size of the object while maintaining a resolution of the object. In the image processing method performed by the image processing device,
Obtaining object information about an important object included in the first image from a first image;
obtaining control information for picture quality processing; and
and obtaining a second image by performing image quality processing on the important object from the first image based on the object information and the user control information. The image processing method of claim 11 , wherein the object information includes information on at least one of a type, location, and size of the important object. The method of claim 11 , further comprising: detecting a plurality of objects from the first image;
outputting object identification information indicating the plurality of objects; and
The image processing method further comprising the step of identifying an object selected by a user as the important object in correspondence with the output of the object identification information. The image processing method of claim 11 , wherein the control information includes control information on at least one of an object magnification, an object magnification degree, an outline process, and a flattening process. 15. The method of claim 14, wherein the performing of image quality processing on the important object comprises performing at least one of upscaling of the object, processing an outline around the object, and flattening the inside of the object according to the control information. Including, image processing method. 12. The method of claim 11, wherein the control information includes at least one of reasoning control information and real-time user control information,
The reasoning control information is obtained from the user's previous control history information for a previous image,
The real-time user control information includes real-time control information of the user for the first image. 12. The method of claim 11, wherein the obtaining of the second image from the first image is performed using a neural network,
The neural network is a neural network that learns an input image, an object region of interest to the user in the input image, and a ground truth image obtained by image quality processing of the object region of interest to the user as a learning data set. , image processing method. 18 . The image processing method of claim 17 , wherein the neural network obtains a second image in which the object is quality-processed from at least one of the first image, the control information, and the object information. The method of claim 17, wherein the neural network obtains a flattening parameter and a contour parameter for image quality processing of the object from at least one of the first image, the user control information, and the object information,
The acquiring of the second image from the first image may include adjusting a blending degree between an image quality-processed according to the flattening parameter and an image quality-processed according to the contour parameter under user control to obtain a quality-processed second image. A method of processing an image, comprising obtaining an image. Obtaining object information about an important object included in the first image from a first image;
obtaining control information for picture quality processing; and
A computer recorded with a program for implementing an image processing method, comprising obtaining a second image by performing image quality processing on the important object from the first image based on the object information and the user control information. readable recording medium.

Images