WO2020036468A1 - Procédé d'application d'effet bokeh sur une image et support d'enregistrement - Google Patents

Procédé d'application d'effet bokeh sur une image et support d'enregistrement Download PDF

Info

Publication number
WO2020036468A1
WO2020036468A1 PCT/KR2019/010449 KR2019010449W WO2020036468A1 WO 2020036468 A1 WO2020036468 A1 WO 2020036468A1 KR 2019010449 W KR2019010449 W KR 2019010449W WO 2020036468 A1 WO2020036468 A1 WO 2020036468A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
bokeh effect
depth
segmentation mask
neural network
Prior art date
Application number
PCT/KR2019/010449
Other languages
English (en)
Korean (ko)
Inventor
이용수
Original Assignee
주식회사 날비컴퍼니
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 날비컴퍼니 filed Critical 주식회사 날비컴퍼니
Priority claimed from KR1020190100550A external-priority patent/KR102192899B1/ko
Publication of WO2020036468A1 publication Critical patent/WO2020036468A1/fr
Priority to US17/101,320 priority Critical patent/US20210073953A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to a recording medium and a method for providing bokeh effect to an image using computer vision technology.
  • the portable terminal has been rapidly developed and widely used, and it has become common to take an image with a camera device or the like provided in the portable terminal device. This replaces the need to carry a separate camera device in order to shoot a conventional image. Furthermore, in recent years, there has been a great deal of interest in obtaining images or photos to which high quality images or advanced image processing technology provided by high-end camera equipment are applied, not just by capturing images from a smartphone. .
  • the Bokeh effect refers to the aesthetical appearance in which the out-of-focus part of the photographed image is blurred. Although the focal plane is clear, the front or rear of the focal plane is blurred to emphasize the focal plane. In the broad sense, the bokeh effect refers not only to outfocusing out of focus (blurring or bokeh) but also to in-focusing or highlighting out of focus.
  • the portable terminal has a difficulty in implementing a bokeh effect such as a DSLR due to a structural problem.
  • the bokeh effect provided by DSLR cameras can be created by default due to the specific shape of the aperture mounted on the camera lens (eg, the shape of one or more aperture blades).
  • cameras in portable terminals use lenses without aperture blades due to the manufacturing cost and / or size of portable terminals, making it difficult to implement bokeh effects.
  • the conventional portable terminal camera uses a method of configuring two or more RGB cameras or measuring a distance using an infrared distance sensor at the time of image capturing in order to realize the bokeh effect.
  • the present disclosure aims to disclose an apparatus and method for implementing an outfocusing and / or infocusing effect, i.e. bokeh effect, that can be implemented in a high quality camera on an image photographed from a smartphone camera or the like through computer vision technology. .
  • a method of applying a bokeh effect to an image in a user terminal includes receiving an image and inputting the received image as an input layer of a first artificial neural network model to obtain depth information about pixels in the image. Generating a representative depth map and applying a bokeh effect to the pixels in the image based on the depth map representing the depth information for the pixels in the image, wherein the first artificial neural network model includes: It may be generated by receiving a plurality of reference images as an input layer and performing machine learning to infer depth information contained in the plurality of reference images.
  • the method of applying the bokeh effect further includes generating a segmentation mask for an object included in the received image, and generating the depth map comprises generating the segmentation mask. Compensating the depth map using the.
  • applying the bokeh effect comprises: determining a reference depth corresponding to the segmentation mask, calculating a difference between the reference depth and the depth of other pixels in an area other than the segmentation mask in the image; And applying a bokeh effect to the image based on the calculated difference.
  • a second artificial neural network model configured to receive a plurality of reference images as an input layer and infer segmentation masks in the plurality of reference images is generated through machine learning, and a segmentation mask
  • the generating may include inputting the received image to the input layer of the second artificial neural network model to generate a segmentation mask for an object included in the received image.
  • the method of applying the bokeh effect further includes generating a detection area that detects an object included in the received image, and generating the segmentation mask to the object in the generated detection area. And generating a segmentation mask for the segmentation mask.
  • the method of applying the bokeh effect may further include receiving setting information on applying the bokeh effect, wherein the received image includes a plurality of objects, and generating the detection area may include: Generating a plurality of detection areas for detecting each of the plurality of objects included in the received image, wherein generating the segmentation mask comprises: generating a plurality of detection areas for each of the plurality of objects within each of the plurality of detection areas. Generating a segmentation mask of the method, and applying the bokeh effect, when the setting information indicates a selection for at least one segmentation mask of the plurality of segmentation masks, And out-of-focus the area.
  • a third artificial neural network model configured to receive a plurality of reference segmentation masks as an input layer and infer depth information of the plurality of reference segmentation masks is generated through machine learning
  • Generating the depth map may include inputting a segmentation mask to an input layer of the third artificial neural network model to determine depth information indicated by the segmentation mask, and applying the bokeh effect may be based on depth information of the segmentation mask. To apply the bokeh effect to the segmentation mask.
  • the generating of the depth map may include performing an image preprocessing to generate data required for an input layer of the first artificial neural network model.
  • generating the depth map includes determining at least one object in the image through the first neural network model, and applying the bokeh effect corresponds to the determined at least one object. Determining a reference depth, calculating a difference between the reference depth and each depth of the other pixels in the image, and applying a bokeh effect to the image based on the calculated difference.
  • a computer-readable recording medium in which a computer program for executing a method of applying a bokeh effect to an image in a user terminal described above is recorded on a computer.
  • a depth image requiring expensive equipment by using depth information of a depth map generated using a learned neural network model in applying a bokeh effect to an image is required. Even without infrared sensors, dramatic bokeh can be applied to images taken from entry-level equipment, such as smartphone cameras. In addition, even if no bokeh effect is given at the time of shooting, it is possible to apply the bokeh effect to a stored image file, for example, an image file of a single RGB or YUV format.
  • a desired bokeh effect may be obtained by more clearly distinguishing a subject and a background by compensating for an error or error of a generated depth map.
  • a problem that some parts are blurred due to depth difference can be applied even inside a subject, which is a single object, to apply a more improved bokeh effect.
  • a specialized bokeh effect may be applied to the specific object. For example, by using an artificial neural network model trained separately for a person, in a portrait, a more detailed depth map may be obtained in the person area, and a more dramatic bokeh effect may be applied.
  • a user experience that provides an easy and effective bokeh effect to a user is provided.
  • FIG. 1 is an exemplary diagram illustrating a process of generating a depth map from an image and applying a bokeh effect based on the bokeh effect applying apparatus according to an embodiment of the present disclosure.
  • FIG. 2 is a block diagram illustrating a configuration of an apparatus for applying bokeh effect according to an embodiment of the present disclosure.
  • FIG. 3 is a schematic diagram illustrating a method of training an artificial neural network model according to an embodiment of the present disclosure.
  • FIG. 4 is a flowchart illustrating a method of correcting a depth map based on a segmentation mask generated from an image and applying a bokeh effect by using the corrected depth map, according to an embodiment of the present disclosure.
  • FIG. 5 is a schematic diagram illustrating a process of generating a segmentation mask for a person included in an image and applying a bokeh effect to an image based on the corrected depth map by the apparatus for applying a bokeh effect according to an embodiment of the present disclosure.
  • FIG. 6 is a comparison diagram showing a device contrasted with a depth map generated from an image and a depth map corrected based on a segmentation mask corresponding to the image, according to an exemplary embodiment.
  • FIG. 7 illustrates an example in which the apparatus for applying a bokeh effect according to an embodiment of the present disclosure determines a reference depth corresponding to a selected object in an image, calculates a difference in depth between the reference depth and other pixels, and applies the bokeh effect to the image based on the reference depth. It is also.
  • FIG. 8 illustrates a process in which the apparatus for applying a bokeh effect according to an embodiment of the present disclosure generates a depth map from an image, determines an object in the image using the learned artificial neural network model, and applies the bokeh effect based on the image.
  • FIG. 9 illustrates a mask input as an input layer of an artificial neural network model that is separately learned in the process of generating a segmentation mask for an object included in an image and applying the bokeh effect by the apparatus for applying a bokeh effect according to an embodiment of the present disclosure.
  • a flowchart illustrating a process of obtaining depth information of and applying a bokeh effect to a mask based on the depth information.
  • FIG. 10 is an exemplary diagram illustrating a process of generating a segmentation mask for a plurality of objects included in an image by the apparatus for applying bokeh effect according to an embodiment of the present disclosure and applying a bokeh effect based on the selected segmentation mask. .
  • FIG. 11 is an exemplary diagram illustrating a process of changing a bokeh effect according to setting information for applying a bokeh effect received to an apparatus for applying a bokeh effect according to an embodiment of the present disclosure.
  • FIG. 12 is an exemplary diagram illustrating a process of implementing a telephoto lens zoom effect by extracting a narrower area from a background in an image as the apparatus for applying a bokeh effect according to an embodiment of the present disclosure becomes stronger.
  • FIG. 13 is a flowchart illustrating a method of applying a bokeh effect to an image in a user terminal according to an exemplary embodiment of the present disclosure.
  • FIG. 14 is a block diagram of a bokeh effect application system according to an exemplary embodiment of the present disclosure.
  • a “part” or “module” refers to software or hardware components, and “unit” or “module” plays certain roles. However, “part” or “module” is not meant to be limited to software or hardware.
  • the “unit” or “module” may be configured to be in an addressable storage medium and may be configured to play one or more processors.
  • a “part” or “module” may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, It may include at least one of procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Components and “parts” or “modules” may be combined with a smaller number of components and “parts” or “modules” or with additional components and “parts” or “modules”. Can be further separated.
  • the “unit” or “module” may be implemented as a processor and a memory.
  • the term “processor” should be interpreted broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like.
  • a “processor” may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like.
  • ASIC application specific semiconductor
  • PLD programmable logic device
  • FPGA field programmable gate array
  • processor refers to a combination of processing devices such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configuration. May be referred to.
  • memory should be interpreted broadly to include any electronic component capable of storing electronic information.
  • the term “memory” refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM) May refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. If the processor can read information from and / or write information to the memory, the memory is said to be in electronic communication with the processor.
  • the memory integrated in the processor is in electronic communication with the processor.
  • a 'user terminal' includes a communication module, which is accessible to a server or a system through a network connection, and is any electronic device capable of displaying or displaying an image or a video (for example, a smartphone, a PC, Tablet PC), and the like.
  • a user may input any command for image processing such as a bokeh effect on an image through an interface of the user terminal (for example, a touch display, a keyboard, a mouse, a touch pen or a stylus, a microphone, and a motion sensor).
  • 'system' may refer to at least one device of a server device and a cloud server device, but is not limited thereto.
  • image refers to an image including one or more pixels, and when the entire image is divided into a plurality of local patches, it may refer to one or more divided local patches.
  • 'image' may refer to one or more images or images.
  • 'receiving an image' may include receiving an image captured and acquired from an image sensor attached to the same device.
  • "receiving an image” may include receiving an image from an external device or transmitting from a storage device via a wired or wireless communication device.
  • a 'depth map' refers to a set of numbers or numbers that represent or characterize the depth of pixels in an image, for example, a depth map is a matrix or vector of multiple numbers representing depth. It can be represented in the form of.
  • the term “bokeh effect” may refer to any aesthetic or aesthetic effect applied to at least a portion of an image.
  • a bokeh effect may refer to an effect generated by outfocusing a portion that is not in focus and / or an effect generated by highlighting, highlighting, or infocusing a portion in focus.
  • the term 'bokeh effect' may refer to a filter effect or any effect that can be applied to an image.
  • Computer vision technology is a technology that performs the same form of the human eye through a computing device.
  • the computing device analyzes an image input from an image sensor and provides useful information such as objects and / or environmental features in the image. It can represent the technology to generate.
  • Machine learning using artificial neural networks can be performed through any computing system implemented by focusing on the neural networks of the human or animal brain, and is one of the detailed methodologies of machine learning. It can refer to machine learning using the form of dog connected network.
  • the object in the image may be compensated for by compensating for an error or error that may occur in a result output through the trained neural network model.
  • a more effective bokeh effect can be obtained by more clearly classifying a subject, a background, and the like.
  • the bokeh effect is applied based on the depth difference inside the subject as a single object, it is possible to apply the bokeh effect within the subject as a single object.
  • FIG. 1 is an exemplary diagram illustrating a process in which a user terminal generates a depth map from an image and applies a bokeh effect based on the image according to an embodiment of the present disclosure.
  • the user terminal may generate an image 130 to which the bokeh effect is applied from the original image 110.
  • the user terminal may apply the bokeh effect by receiving the original image 110.
  • the user terminal receives an image 110 including a plurality of objects, and focuses on a specific object (for example, a person).
  • the image 130 to which the bokeh effect is applied may be generated by applying the out focusing effect to the objects other than the person (here, the background).
  • the out-of-focus effect may refer to blurring an area or processing some pixels with light bokeh, but is not limited thereto.
  • the original image 110 may comprise an image file composed of pixels, each of which has information.
  • image 110 may be a single RGB image.
  • the "RGB image” is an image composed of numerical values of red (R), green (G), and blue (B), for example, a value between 0 and 255 for each pixel.
  • a “single” RGB image is distinguished from an RGB image obtained from an image sensor having two or more lenses, and may refer to an image photographed from one image sensor.
  • the image 110 has been described as an RGB image, but is not limited thereto and may represent images of various known formats.
  • a depth map may be used in applying the bokeh effect to the image.
  • a bokeh effect may be applied by leaving a low-depth portion of an image as it is or applying a highlight effect, and blurring a high-depth portion.
  • the depth of a specific pixel or region as the reference depth and determining the relative depth between other pixels or regions, the height of the depth between the pixels or regions in the image can be determined.
  • the depth map may be a kind of image file. Depth may be indicative of depth within the image, for example, the distance from the lens of the image sensor to the object represented by each pixel.
  • the most common method for obtaining a depth map is to use a depth camera, but since the depth camera itself is expensive and few cases are applied to a portable terminal, there is a limitation in applying a bokeh effect using a depth map on a portable terminal. there was.
  • the depth map in the method of generating the depth map 120, may be generated by inputting the image 110 as an input variable to the trained neural network model.
  • the depth map 120 may be generated using the neural network model from the image 110, and the image 130 having the bokeh effect applied thereto may be generated based on the depth map 120.
  • the depth, or depth, of the objects in the image from the image can be obtained through the learned neural network model.
  • the bokeh effect using the depth map it may be applied according to a predetermined rule, or may be applied according to information received from a user.
  • the depth map 120 is represented as a grayscale image.
  • the depth map 120 is an example for showing the difference between depths of pixels.
  • the depth map is a set of numbers or numbers representing or characterizing depths of pixels in an image. Can be represented.
  • a low-end device such as a smartphone camera without using a depth camera or an infrared sensor that requires expensive equipment.
  • Dramatic bokeh effects can be applied to images captured from.
  • the bokeh effect can be applied to the stored image file, for example, an RGB image file.
  • the user terminal 200 may include a depth map generation module 210, a bokeh effect application module 220, a segmentation mask generation module 230, a detection area generation module 240, and an I / O device 260.
  • the user terminal 200 is configured to communicate with the bokeh effect application system 205, the first artificial neural network model to be described below to be learned in advance through the machine learning module 250 of the bokeh effect application system 205 It may be provided with a trained artificial neural network model including a second artificial neural network model, a third artificial neural network model.
  • the machine learning module 250 is illustrated as being included in the bokeh effect application system 205, but is not limited thereto.
  • the machine learning module 250 may be included in the user terminal.
  • the depth map generation module 210 may be configured to receive a captured image from an image sensor and generate a depth map based on the image. According to one embodiment, such an image may be provided to the depth map generation module 210 immediately after being imaged from the image sensor. According to another exemplary embodiment, the image captured by the image sensor may be stored in a storage medium included in or accessible to the user terminal 200, and the user terminal 200 may access the storage medium to store the image stored when generating the depth map. Can be received.
  • the depth map generation module 210 may be configured to generate a depth map by inputting the received image as an input variable to the trained first artificial neural network model.
  • the first artificial neural network model may be learned through the machine learning module 250. For example, it may be learned to receive a plurality of reference images as input variables and infer depths for each pixel or for each pixel group including a plurality of pixels. In this process, by using a reference image including depth map information corresponding to the reference image measured by a separate device (eg, a depth camera), the error of the depth map output through the first neural network model is reduced. Can be learned to be reduced.
  • the depth map generation module 210 may obtain depth information included in the image from the image 110 through the trained first artificial neural network model. According to one embodiment, the depth information may be given for every pixel in the image, may be given for every adjacent pixel, or may be given the same value for several adjacent pixels.
  • the depth map generation module 210 may be configured to generate a depth map corresponding to the image in real time.
  • the depth map generation module 210 may correct the depth map in real time using the segmentation mask generated by the segmentation mask generation module 230. Even if the depth map is not implemented in real time, the depth map generation module 210 may generate a plurality of blur images in which the bokeh blur is applied at different intensities (for example, kernel size). For example, the depth map generation module 210 renormalizes the pre-generated depth map and interpolates the pre-generated blur images blurred at different intensities according to the value of the renormalized depth map to real-time. The effect of varying the bokeh intensity can be realized.
  • the effect of correcting the depth map or changing the bokeh intensity in real time can be applied to the image.
  • the depth map generation module 210 receives the RGB image captured by the RGB camera and the depth image photographed from the depth camera, and matches the depth image to the RGB image using a given camera variable, and the like, and the depth image aligned with the RGB image. Can be generated. Then, the depth map generation module 210 may derive the areas of the points where the reliability is lower than the preset value and the points where holes are generated in the generated depth image. In addition, the depth map generation module 210 may estimate the depth image from the RGB image using an artificial neural network model (eg, a first artificial neural network model) trained to derive a predicted depth map from the RGB image. Can be derived.
  • an artificial neural network model eg, a first artificial neural network model
  • depth information may be estimated for points where reliability is lower than a preset value and holes where holes are generated, and estimated depth information may be input to the depth image to derive a completed depth image.
  • depth information on points where the reliability in the image is lower than a predetermined value and holes generated may be estimated using bilinear interpolation, histogram matching, and pretrained artificial neural network model.
  • depth information may be estimated by using a median of the values obtained using this method or a value derived from a weighted arithmetic mean applied to a preset ratio. If the estimated depth image is smaller than the required height and width, it may be upscaled to the required size using a pre-trained neural network model.
  • the bokeh effect application module 220 may be configured to apply the bokeh effect to pixels in the image based on depth information about pixels in the image represented by the depth map.
  • the intensity of the bokeh effect to be applied may be designated as a predetermined function using the depth as a variable.
  • the predetermined function may be to vary the degree and shape of the bokeh effect by using the depth value as a variable.
  • the interval of depth may be divided to provide a bokeh effect discontinuously.
  • the following effects or one or more combinations of the following effects may be applied according to the depth information of the extracted depth map.
  • depth information can be set to 0 for the closest object and 100 for the farthest object.
  • a section of 0 to 20 applies a photo filter effect
  • a section of 20 to 40 applies an out focusing effect.
  • the above section may be configured to replace the background.
  • a stronger outfocusing effect eg, a gradation effect
  • various bokeh effects may be applied according to setting information for applying the bokeh effect input from the user.
  • the bokeh effect application module 220 may generate a bokeh effect by applying a filter that is already selected to the input image using depth information in the depth map.
  • the predetermined filter may be applied to the reduced input image. For example, for images and a depth map to which a filter is applied, the symbol or image corresponding to a pixel of the input image is calculated by using bilinear interpolation, and the filter or image corresponding to the area of the calculated value is applied.
  • the pixel values can be computed using bilinear interpolation as well.
  • Bokeh effect is applied to a specific area for object domains in the image, after changing the depth value of that area so that the depth estimate in the depth map calculated for the object segmentation mask area for that area is within the given numerical range, The image is reduced and pixel values can be calculated using bilinear interpolation.
  • the segmentation mask generation module 230 may generate a segmentation mask for the object in the image, that is, a segmented image area.
  • the segmentation mask may be generated by dividing the pixels corresponding to the objects in the image.
  • image segmentation may refer to a process of dividing a received image into a plurality of pixel sets.
  • Image segmentation is to simplify or transform the representation of an image into something more meaningful and easier to interpret, and is used to find objects, boundaries (lines, curves) corresponding to objects in an image, for example.
  • One or more segmentation masks may be generated in the image.
  • semantic segmentation is a technique of extracting a boundary of a specific object, a person, etc.
  • segmentation mask generation module 230 may use any technique known in the art of segmentation techniques, for example thresholding methods, argmax methods, histogram-based methods region growing methods, split-and-merge. Segmentation masks for one or more objects in an image may be generated using mapping algorithms such as methods and graph partitioning methods and / or trained neural network models, but are not limited thereto.
  • the neural network model learned here may be a second artificial neural network model, and may be learned by the machine learning module 250. The learning process of the second artificial neural network model will be described in detail with reference to FIG. 3.
  • the depth map generation module 210 may be further configured to correct the depth map using the generated segmentation mask.
  • the user terminal 200 may correct an incorrect depth map or set a reference depth to give the bokeh effect.
  • a segmentation mask may be input to a trained neural network model to generate a precise depth map and apply a specialized bokeh effect.
  • the learned artificial neural network model may be a third artificial neural network model, it may be learned by the machine learning module 250. The learning process of the third artificial neural network model will be described in detail with reference to FIG. 3.
  • the detection area generation module 240 may be configured to detect an object in the image and generate a specific area for the detected object.
  • the detection zone generation module 240 may identify the object in the image to generate the zone schematically. For example, a person in the image 110 may be detected and the area may be divided into a rectangular shape.
  • the generated detection area may be one or more according to the number of objects in the image area.
  • Methods for detecting an object from an image may include, but are not limited to, RapidCheck, Histogram of Oriented Gradient (HOG), Cascade HoG, ChnFtrs, part-based model, and / or learned neural network model.
  • the object to extract the boundary becomes limited and clear, so that the load of the computing device extracting the boundary may be reduced.
  • the mask generation time is shortened and a more detailed segmentation mask can be obtained. For example, it may be more effective to limit the area of the person and extract the mask for the area of the person than to instruct to extract the mask for the area of the person from the entire image.
  • the detection region generation module 240 may be configured to detect an object in the input image by using an object detection artificial neural network that has been previously trained on the input image.
  • An object segmented neural network learned in advance for the detected object region may be used to segment the object within the detected object region.
  • the detection area generation module 240 may derive the smallest area including the divided object partition mask as the detection area. For example, the smallest area including the divided object partition mask may be derived as a rectangular area. The region thus derived may be output in the input image.
  • the I / O device 260 may be configured to receive setting information regarding the bokeh effect to be applied from the device user or to output or display an original image and / or an image processed image.
  • the I / O device 260 may be a touch screen, a mouse, a keyboard, a display, or the like, but is not limited thereto.
  • information for selecting a mask to which the highlight is applied may be received from the plurality of segmentation masks.
  • a touch gesture may be received through a touch screen as an input device, and a gradual and various bokeh effect may be given according to the information.
  • the touch gesture may refer to any touch operation of a user's finger on the touch screen as an input device.
  • the touch gesture may include a long touch, a screen push, and a gesture of spreading or reducing the touch of a plurality of fingers. May be referred to.
  • the received information on applying the bokeh effect may be configured to be set from the user to be given a bokeh effect, it may be configured to be stored in the module, for example in the bokeh effect application module 220.
  • the I / O device 260 may include any display device that outputs an original image or displays an image on which image processing such as a bokeh effect has been performed.
  • any display device may include a touch-panel display capable of touch input.
  • the I / O device 260 is illustrated as being included in the user terminal 200.
  • the present invention is not limited thereto, and the user terminal 200 receives setting information on a bokeh effect to be applied through a separate input device.
  • the Bokeh effect image can be output through a separate output device.
  • the user terminal 200 may be configured to correct distortion of an object in the image.
  • barrel distortion which may be caused by a parabolic surface of a lens egg having a curvature
  • the nose looks relatively larger than other parts, and the center of the lens is distorted like a convex lens to correct the image of the human face differently from the actual one.
  • the user terminal 200 may recognize an object (eg, a human face) in the image in three dimensions and correct the image to include an object that is the same as or similar to the real object.
  • the ear region of the face of the invisible person may be generated using a generative model such as deep learning GAN.
  • a generative model such as deep learning GAN.
  • not only deep learning techniques, but also any technique that can naturally attach invisible parts to objects can be adopted.
  • the user terminal 200 may be configured to blend the hair or the color of the hair contained in any object in the image.
  • the segmentation mask generation module 230 may generate a segmentation mask corresponding to the hair region in the input image using an artificial neural network trained to derive the hair region from the input image including the human, the animal, and the like.
  • the bokeh effect application module 220 may change the color space of the area corresponding to the segmentation mask to black and white and generate a histogram of the brightness of the changed black and white area.
  • a sample hair color to be changed having various brightnesses may be prepared and stored in advance.
  • the bokeh effect application module 220 may change the color space for the sample hair color to black and white and generate a histogram for the brightness of the changed black and white area. In this case, histogram matching may be performed such that a similar color may be selected or applied to a portion having the same brightness. The bokeh effect application module 220 may substitute the matched color into an area corresponding to the segmentation mask.
  • the artificial neural network model 300 is a machine learning (Machine Learning) technology and cognitive science, a statistical learning algorithm implemented based on the structure of a biological neural network or a structure for executing the algorithm.
  • the artificial neural network model 300 is a node that is artificial neurons that form a network by synaptic coupling as in the biological neural network (Node) iteratively adjust the weight of the synapse, so that the correct response to a specific input
  • the neural network model 300 may include any probabilistic model, a neural network model, or the like used in artificial intelligence learning methods such as machine learning and deep learning.
  • artificial neural network model 300 may refer to any artificial neural network model or neural network described herein, including a first artificial neural network model, a second artificial neural network model, and / or a third artificial neural network model.
  • the neural network model 300 is implemented with a multilayer perceptron (MLP) composed of multiple nodes and a connection therebetween.
  • the neural network model 300 may be implemented using one of various artificial neural network model structures including an MLP.
  • the artificial neural network model 300 includes an input layer 320 that receives an input signal or data 310 from an external source, and an output layer that outputs an output signal or data 350 corresponding to the input data.
  • 340 which is located between the input layer 320 and the output layer 340, receives n signals from the input layer 320, extracts the characteristics, and transfers the characteristics to the output layer 340, where n is a positive integer. Consists of hidden layers 330_1 to 330_n.
  • the output layer 340 receives a signal from the hidden layers 330_1 to 330_n and outputs the signal to the outside.
  • the learning method of the artificial neural network model 300 includes a supervised learning method for learning to be optimized for solving a problem by inputting a teacher signal (correct answer), and unsupervised learning that does not require the teacher signal. There is a way.
  • the machine learning module 250 analyzes the input image by using supervised learning to provide depth information of objects such as a subject, a background, and the like in the received image, and provides depth information corresponding to the image.
  • the artificial neural network model 300 that is, the first artificial neural network model may be trained to be extracted.
  • the learned neural network model 300 may generate a depth map containing depth information in response to the received image and provide the depth map to the depth map generation module 210, and the image to which the bokeh effect application module 220 is received Can provide a basis for applying bokeh effects to
  • the input variable of the artificial neural network model 300 that can extract depth information may be an image.
  • the input variable input to the input layer 320 of the artificial neural network model 300 may be an image vector 310 composed of one vector data element.
  • the output variable output from the artificial neural network model 300 may be a vector representing a depth map.
  • the output variable may be composed of the depth map vector 350.
  • the depth map vector 350 may include depth information of pixels of an image as a data element.
  • the output variables of the artificial neural network model 300 are not limited to the types described above, and may be represented in various forms related to the depth map.
  • the input layer 320 and the output layer 340 of the neural network model 300 are matched with a plurality of output variables corresponding to the plurality of input variables, respectively, and thus the input layer 320, the hidden layers 330_1 to 330_n and the output layer
  • the synaptic value between the nodes included in 340 it is possible to learn to extract the correct output corresponding to a specific input.
  • the learned neural network model 300 that is, the first neural network model
  • the depth map 350 in the received image may be generated in response to the input image.
  • the machine learning module 250 receives the plurality of reference images as input variables of the artificial neural network model 300, that is, the input layer 310 of the second artificial neural network model,
  • the output variable output from the output layer output from the output layer 340 may be learned to be a vector representing a segmentation mask for objects included in the plurality of images.
  • the learned second neural network model may be provided to the segmentation mask generation module 230.
  • the machine learning module 250 may include a part of a plurality of reference images, for example, a plurality of reference segmentation masks of the artificial neural network model 300, that is, the input layer 310 of the third artificial neural network model. Can be received as an input variable.
  • the input variable of the third artificial neural network model may be a segmentation mask vector in which each of the plurality of reference segmentation masks is composed of one vector data element.
  • the machine learning module 250 may train the third artificial neural network model so that the output variable output from the output layer 340 of the third artificial neural network model may be a vector representing precise depth information of the segmentation mask.
  • the learned third neural network model may be provided to the bokeh effect application module 220 to be used to apply more precise bokeh effects to specific objects in the image.
  • the range [0, 1] used in the existing neural network model could be calculated by dividing by 255.
  • the artificial neural network model of the present disclosure may include a range of [0, 255/256] calculated by dividing by 256. Only the artificial neural model can be trained by applying this.
  • a method of dividing by a power of two may be used. According to this technique, since the neural network learning uses a power of 2, the computational amount of the computer architecture can be optimized and the computation can be accelerated during multiplication / division.
  • FIG. 4 is a flowchart illustrating a method in which the user terminal 200 corrects a depth map based on a segmentation mask generated from an image, and applies a bokeh effect using the corrected depth map, according to an exemplary embodiment.
  • the method 400 of applying the bokeh effect may include an operation S410 of receiving the original image by the depth map generation module 210.
  • the user terminal 200 may be configured to receive the captured image from the image sensor.
  • the image sensor may be mounted on a device included in or accessible to the user terminal 200, and the captured image may be provided to the user terminal 200 or stored in a storage device.
  • the user terminal 200 may be configured to access the storage device and receive the image.
  • the storage device may be included in one device with the user terminal 200 or may be connected to the user terminal 200 via a wired or wireless connection as a separate device.
  • the segmentation mask generation module 220 may generate a segmentation mask for the object in the image (S420).
  • a 2D map having a probability value for each class is obtained as a result of the artificial neural network model, and thresholding or argmax is applied.
  • the segmentation mask map By generating the segmentation mask map, the segmentation mask can be generated.
  • the artificial neural network model can be trained to provide segmentation masks of objects included in each image by providing various images as input variables of the neural network learning model, and received through the learned artificial neural network model.
  • the segmentation mask of the object in the image can be extracted.
  • the segmentation mask generation module 220 may be configured to generate the segmentation mask by calculating segmentation prior information of the image through the trained neural network model.
  • the input image may be preprocessed to satisfy a data feature required in a given artificial neural network model before being input into the artificial neural network model.
  • the data feature may be a minimum value, a maximum value, an average value, a variance value, a standard deviation value, a histogram, or the like of a specific data in the image, and the channel of the input data (for example, an RGB channel or a YUV channel) may be used as needed. It can be processed together or separately.
  • the division dictionary information may refer to information indicating numerically whether each pixel in the image is an object to be divided, that is, a semantic object (eg, a person, an object, etc.).
  • the division dictionary information may represent a numerical value corresponding to the dictionary information of each pixel as a value between 0 and 1 through quantization.
  • it may be determined that a value closer to 0 corresponds to a background, and a value closer to 1 corresponds to an object to be divided.
  • the segmentation mask generation module 220 may set the final segmentation dictionary information to 0 (background) or 1 (object) for each pixel or for a group including a plurality of pixels by using a predetermined threshold value. .
  • the segmentation mask generation module 220 considers the distribution level of the partition dictionary information of each pixel group including each pixel or a plurality of pixels in consideration of the distribution and the numerical value of the partition dictionary information corresponding to the pixels in the image. ) And the dividing dictionary information and the reliability for each pixel or each pixel group can be used together when setting the final dividing dictionary information. Then, the segmentation mask generation module 220 may generate a separation mask, that is, a segmentation mask, of pixels having a value of 1. For example, when there are a plurality of meaningful objects in the image, segmentation mask 1 may represent object 1, ..., and segmentation mask n may represent object n (where n is a positive number of 2 or more).
  • a segmentation mask region corresponding to meaningful objects in the image may have a value of 1 and an outer region of the mask may have a value of 0.
  • the segmentation mask generation module 220 may calculate and generate a product of the received image and the generated segmentation mask map to generate an object-specific image or a background image.
  • the segmentation mask generation module 230 may be configured to generate a detection area for detecting the object included in the image.
  • the detection region generation module 240 may identify an object in the image 110 and roughly generate an area of the object.
  • the segmentation mask generation module 230 may then be configured to generate a segmentation mask for the object within the generated detection area. For example, a person can be detected within an image to separate the area into rectangular shapes. Then, the segmentation mask generation module 230 may extract an area corresponding to the person in the detection area.
  • the objects and regions corresponding to the objects are limited, thereby increasing the speed of generating segmentation masks corresponding to the objects, increasing the accuracy, and / or loading the computing device. Can be lowered.
  • the depth map generation module 210 may generate a depth map of an image by using an already learned artificial neural network model (S430).
  • the artificial neural network model may be trained to infer a depth for each pixel or a pixel group including a plurality of pixels by receiving a plurality of reference images as input variables as mentioned in FIG. 3.
  • the depth map generation module 210 inputs an image to an artificial neural network model using an input variable, and generates a depth map having depth information on each pixel or a group of pixels including a plurality of pixels in the image. can do.
  • the resolution of the depth map 120 may be the same as or lower than the image 110. When the resolution is lower than the image 110, the depth of several pixels of the image 110 is changed to one pixel.
  • the depth map 120 may be configured to provide one depth per four pixels of the image 110.
  • the generating of the segmentation mask (S420) and the generating of the depth map (S430) may be performed independently.
  • the depth map generation module 210 may receive the segmentation mask generated from the segmentation mask generation module 230 and correct the generated depth map by using the segmentation mask (S440). According to an embodiment, the depth map generation module 210 may determine pixels in the depth map corresponding to the segmentation mask as one object and correct depths of the pixels corresponding thereto. For example, if the deviation of the depths of the pixels of the portion determined to be one object in the image is large, the deviation may be corrected to reduce the depth of the pixels in the object. As another example, if a person stands at an angle, the bokeh effect may be applied to a part of the person in different depths within the same person, so that the bokeh effect is not applied to the pixels corresponding to the segmentation mask for the person. Depth information for may be corrected. When the depth map is corrected, there is an effect of reducing an error such as an out of focus effect applied to an unwanted portion, and a bokeh effect may be applied to a more accurate target.
  • the bokeh effect application module 220 may generate an image to which the bokeh effect is applied to the received image (S450).
  • the bokeh effect may be applied to the various bokeh effects described in the bokeh effect application module 220 of FIG. 2.
  • the Bokeh effect may be applied to a pixel or a group of pixels corresponding to the depth based on the depth in the image, where the outer area of the mask gives a strong focus effect, and the mask area is relatively weak compared to the outside. It can be given or no bokeh effect.
  • FIG. 5 illustrates a process in which the user terminal 200 generates a segmentation mask 530 for a person included in an image and applies a bokeh effect to the image based on the corrected depth map according to an embodiment of the present disclosure.
  • the image 510 may be an image of a person standing on a background of an indoor corridor.
  • the detection region generation module 240 may receive the image 510 and detect the person 512 from the received image 510. For example, as shown, the detection area generation module 240 may generate the detection area 520 having a rectangular shape, including the area of the person 512.
  • the segmentation mask generation module 220 may generate the segmentation mask 530 for the person 512 from the detection area 520 in the detection area.
  • the segmentation mask 530 is shown as a virtual region as white in FIG. 5, but is not limited to this, and any indication or numerical values representing an area corresponding to the segmentation mask 530 on the image 510 may be used. Can be represented as a set.
  • the segmentation mask 530 may include an area within an object on the image 510.
  • the depth map generation module 210 may generate a depth map 540 of an image representing depth information from the received image.
  • the depth map generation module 210 may generate the depth map 540 using the trained neural network model.
  • the depth information may be expressed such that the near part is closer to black and the far part is closer to white.
  • the depth information may be expressed as a numerical value, and may be expressed within an upper limit and a lower limit (eg, the closest portion is 0 and the farthest portion is 100) of the depth value.
  • the depth map generation module 210 may correct the depth map 540 based on the segmentation mask 530. For example, a certain depth may be given to a person in the depth map 540 of FIG. 5.
  • the constant depth may be an average value, a median value, a mode value, a minimum value, or a maximum value of the depth in the mask, or may be expressed as a depth of a specific area, for example, the tip of the nose.
  • the bokeh effect application module 220 may apply a bokeh effect to a region other than a person based on the depth map 540 and / or a corrected depth map (not shown). As illustrated in FIG. 5, the bokeh effect application module 220 may apply an blur effect to an area other than a person in an image to give an out of focus effect. Alternatively, the area corresponding to the person may not be given any effect or the emphasis effect may be applied.
  • FIG. 6 illustrates a depth map 630 corrected based on a depth map 620 generated from an image 610 and a segmentation mask corresponding to the image 610, according to an embodiment of the present disclosure.
  • This is a comparison chart.
  • the image 610 may be an image captured by a plurality of people near an external parking lot.
  • the depth map 620 when the depth map 620 is generated from the image 610, even the same object may exhibit a significant depth deviation depending on the position or posture. For example, a depth corresponding to a shoulder of a person standing at an angle may have a large difference from an average value of depths corresponding to a human object. As shown in FIG. 6, the depth map 620 generated from the image 610 is lighter because the depth corresponding to the shoulder of the right person is relatively larger than other depth values in the right person. When the bokeh effect is applied based on the depth map 620, even if the right person in the image 610 is selected as the object to be focused, a part of the right person, for example, the right shoulder part may be out of focus. .
  • depth information of the depth map 620 may be corrected using a segmentation mask corresponding to an object in the image.
  • the correction may be, for example, an average value, a median value, a mode value, a minimum value or a maximum value, or a correction to a depth value of a specific portion.
  • the depth map generation module 210 may correct the depth corresponding to the object that the user wants to apply the bokeh effect to using the generated segmentation mask, so that the bokeh effect may be applied to the user's intention.
  • the depth map generation module 210 may not correctly recognize a part of the object.
  • the segmentation mask may be improved by using a segmentation mask. For example, depth information may not be correctly grasped for a handle of a cup missed at an angle, and the depth map generation module 210 may use a segmentation mask to determine that the handle is part of a cup, and to obtain correct depth information. You can correct it.
  • FIG. 7 illustrates a reference depth corresponding to a selected object in the image 700 by the user terminal 200 according to an exemplary embodiment of the present disclosure, and calculates a difference between a reference depth and depths of other pixels based on the image 700.
  • the bokeh effect application module 220 may determine a reference depth corresponding to the selected object, calculate a difference between the reference depth and the depths of other pixels in the image, and apply the difference to the image based on the calculated difference. It can be further configured to apply the bokeh effect.
  • the reference depth may be an average value, a median value, a mode value, a minimum value or a maximum value of the depth values of pixel values corresponding to the object, or may be expressed as a depth of a specific part, for example, the tip of a nose.
  • the bokeh effect application module 220 determines a reference depth for each of the depths corresponding to three people 710, 720, and 730, and applies the bokeh effect based on the determined reference depth. It can be configured to. As shown, if the person 730 located in the center is selected to be focused, the out focusing effect may be applied to the other people 710 and 720.
  • the bokeh effect application module 220 may be configured to apply different bokeh effects according to the relative depth difference between the reference depth of the selected object in the image and other pixels. For example, if the centered person 730 is selected to be focused, then the person 710 closest to it is relative to the person 730 farthest relative to the person 730 located in the center. Because it is far away, as shown, the bokeh effect application module 220 applies an outfocusing effect to the farthest person 720 that applies to the person 710 closest in the image 700. It can be processed more strongly.
  • the depth map generation module 210 may be configured to determine at least one object 830 in the image 810 through the first artificial neural network model.
  • the bokeh effect application module 220 determines a reference depth corresponding to the determined at least one object 830, calculates a difference between the reference depth and each depth of each of the other pixels in the image, and based on the calculated difference It can be further configured to apply the bokeh effect to the image.
  • the input variable of the neural network model 300 to extract the depth information may be an image 810, the output variable output from the output layer 340 of the artificial neural network model 300, It may be a vector representing the depth map 820 and the determined at least one object 830.
  • the obtained object may be given a uniform depth.
  • the uniform depth may be expressed as an average value of depths of pixels of the acquired object. In this case, an effect similar to that used by generating a mask may be obtained without a separate process of generating a segmentation mask.
  • the depth map may be corrected to obtain a depth map more suitable for applying a bokeh effect.
  • the procedure for generating the segmentation mask may be omitted and the similar bokeh effect may be applied, thereby improving the speed of the entire mechanism and reducing the load of the apparatus.
  • FIG. 9 is a diagram illustrating input of a separately trained artificial neural network model using a segmentation mask generated in the process of generating a segmentation mask for an object included in an image and applying a bokeh effect according to an exemplary embodiment of the present disclosure.
  • a flow diagram illustrating a process of inputting a variable and obtaining depth information of a segmentation mask and applying a bokeh effect to an image corresponding to the segmentation mask based on the information.
  • Steps S910, S920, S930, and S940 included in the flowchart of FIG. 9 may include operations identical or similar to those of steps S410, S420, S430, and S440 included in the flowchart of FIG. 4.
  • contents overlapping with those described in the flowchart of FIG. 4 are omitted.
  • the depth map generation module 210 inputs the segmentation mask generated from the original image into an input trained neural neural network model (for example, the third neural network model) as an input variable to determine precise depth information of the segmentation mask. It may be further configured (S950). According to an embodiment of the present disclosure, the depth map generation module 210 may use an artificial neural network model specialized for a specific target in addition to artificial neural networks that will be commonly used for general images. For example, the third artificial neural network model may be trained to receive an image including a person and infer a depth map of the person or the person's face. In this process, the third artificial neural network model may be supervised using a plurality of reference images including a person having a predetermined depth to infer a more precise depth map.
  • an input trained neural neural network model for example, the third neural network model
  • the depth map generation module 210 may use an artificial neural network model specialized for a specific target in addition to artificial neural networks that will be commonly used for general images.
  • the third artificial neural network model may be trained to receive an image
  • the depth map generation module 210 may use the following method to obtain precise depth information about an object corresponding to the segmentation mask.
  • precise depth information of an object corresponding to the segmentation mask may be generated using a depth camera such as a time of flight (TOF) and a structured light.
  • depth information inside an object (eg, a person) corresponding to the segmentation mask may be generated using computer vision technology such as feature matching.
  • the bokeh effect application module 220 may be further configured to apply the bokeh effect in the original image based on the corrected depth map of the generated segment map and the corrected depth map corresponding to the original image generated in steps S930 and S940. (S960).
  • S960 the corrected depth map of the generated segment map
  • the segmentation mask area may be provided with the bokeh effect using depth information of the segmentation mask, and the area other than the segmentation mask may be provided with the bokeh effect using the depth map.
  • a specific segmentation mask area for example, a face of a person in which precise depth information is generated, may be applied with a very fine bokeh effect, and the remaining segmentation areas and areas other than the mask may be hybridized to give a less detailed bokeh effect.
  • the bokeh effect can be applied. Through this configuration, a high quality result can be obtained while minimizing the computing load of the user terminal.
  • the bokeh effect is applied using the depth maps generated in steps S930 and S940 together with the mask depth information generated in step S950.
  • the present invention is not limited thereto and is generated in S930 without going through step S940.
  • the bokeh effect may be applied in operation S960 using the depth map and the mask depth information generated in S950.
  • the image 1010 may include a plurality of objects.
  • the detection region generation module 240 may be further configured to generate the plurality of detection regions 1020_1 and 1020_2 that detect each of the plurality of objects included in the received image 1010. For example, as shown, the areas of the left person and the right person were detected in their respective squares.
  • the segmentation mask generation module 230 may be configured to generate a segmentation mask 1030 for the object.
  • the segmentation mask generation module 230 may include a plurality of segmentation masks for each of the plurality of objects (left person and right person) within each of the plurality of detection areas, as shown in FIG. 10. 1033_1, 1033_2 may be further configured.
  • the depth map generation module 210 may correct the depth map 1040 generated from the image 1010 through the generated segmentation mask. In this process, at least one mask selected without correcting using the entire segmentation mask may be used. Can only be corrected. For example, as illustrated in FIG. 10, when the mask 1033_2 of the right person is selected, the corrected depth map 1050 may be obtained using the mask. Alternatively, the depth map may be corrected using both the mask 1033_1 of the left person and the mask 1033_2 of the right person.
  • the selected mask may apply the highlighting effect, or the remaining mask may apply the defocus effect.
  • the mask to which the focus is given may be different. In this process, regions of the mask that are not selected may be treated similarly to regions other than the mask.
  • a mask 1033_2 of the right person is selected to represent an image to which the out focusing effect is applied except for the selected segmentation mask 1036.
  • the mask 1033_1 of the left person is detected and the corresponding object is extracted, but the out focusing effect is given.
  • an out focusing effect may be applied to the mask area of the right person.
  • FIG. 11 is a schematic diagram illustrating a process of changing a bokeh effect according to setting information for applying a bokeh effect received to a user terminal 200 according to an exemplary embodiment.
  • the input device 260 may include a touch screen, and setting information about applying the bokeh effect may be determined based on a touch input of the touch screen.
  • the input device 260 of the user terminal 200 may be configured to receive information for setting a bokeh effect to be applied.
  • the bokeh effect application module 220 may apply the bokeh effect to at least a portion of the image according to the received information.
  • the pattern for applying the bokeh effect for example, the intensity or the shape of the bokeh may be changed or the filter may be variously applied. For example, as shown in FIG. 10, dragging the touch screen to the left generates an image 1120 to which the filter effect specified 1 is applied to the image 1110, and dragging to the right produces 2 2 assigned to the image 1110.
  • An image 1130 to which the second filter is applied may be generated, and a stronger bokeh effect may be given as the dragging degree increases.
  • the focus effect when dragging left and right, the focus effect may be diversified to areas other than the mask, and when dragging up and down, the infocus effect may be diversified to the mask area.
  • the diversification includes various modifications of the visual effect, such as varying the filter or varying the shape of the bokeh, and are not limited to the described embodiments.
  • What bokeh effect is given according to a touch gesture such as dragging or zooming may be configured to be set by a user, and may be configured to be stored in the bokeh effect application module 220.
  • the user terminal 200 may be configured to generate a segmentation mask for an object included in the received image.
  • the user terminal 200 may display an object (eg, a person) included in the background and the generated segmentation mask in the image.
  • a touch input for example, a touch
  • the graphic element may be replaced with another graphic element. have.
  • the filter of the background or the background part in the image may be changed.
  • the filter of the human part in the image may be replaced.
  • the result of the filter change according to the left and right swipes and the upper and lower swipes may be interchanged.
  • the background in the image may automatically change continuously.
  • the acceleration of the background substitution may be increased.
  • the background substitution in the image may be stopped. For example, when there are two or more graphic elements in the image, only one graphic element may be changed according to a touch input in the image, that is, one gesture.
  • the user focused in the image may be changed.
  • the person focused by swiping left and right may change.
  • the segmented person is tapped
  • the person focused can be changed.
  • the user terminal 200 receives a touch input corresponding to a tap for any part of the image
  • the focusing person in the image may be changed in order.
  • the area in the image may be calculated using face segmentation and instant segmentation in the image. Also, how far apart the instant segmentation may be based on the person who focused. Based on this calculated value, out-focusing of different intensity may be applied to each person.
  • the user terminal 200 since the user terminal 200 generates the segmentation mask corresponding to the object area, the user terminal 200 knows where the area corresponding to the object in the image is. Therefore, the user touches a part corresponding to the object area in the image. Without touching any part of the image it is possible to change the focusing of the object area.
  • the user terminal 200 may calculate an area of an object (eg, a person) in the image by using a segmentation mask generated for one or more objects in the image.
  • the number of people in the image can be calculated using instance segmentation techniques.
  • the optimal filter can be applied through the calculated area of the person and the number of people.
  • the optimal filter may include a graphic element to be a background replacement and a color filter to change the mood of the image, but is not limited thereto. According to such a filter application, a user may apply a photo filter effect to an image smartly.
  • the user terminal 200 may display the location of an object (eg, a person) in the image using a segmentation mask corresponding to one or more objects in the image. Accordingly, the user terminal 200 may display a Graphic User Interface (GUI) in which a computer graphic function is available at a portion other than a position corresponding to the displayed object in the image. If the image is an image, the GUI may be displayed so as not to obstruct the person by tracking the position of the person in the frames in the image. For example, a subtitle may be displayed as a GUI in an area other than a person in the image.
  • GUI Graphic User Interface
  • the user terminal 200 may detect a user's touch input in the image through an input device such as a touch screen, and the contacted portion in the image may be focused and the non-contacted portion may be out of focus. Can be.
  • the user terminal 200 may be configured to detect contact of two fingers of the user.
  • the user terminal 200 may detect the zoom in and / or zoom out motion of two fingers in the image, and thus adjust the bokeh intensity in the image.
  • the user terminal 200 may use zoom in and / or zoom out motion as a way of adjusting the out focusing intensity, and the object to be out of focus in the image is one in the image.
  • the segmentation mask corresponding to the above object may be extracted.
  • the user terminal 200 may be configured to separate the hair object from the human object using a segmentation mask corresponding to one or more objects in the image. Then, the user terminal 200 may provide a list of dyes to the user, and may receive one or more dyes from the user and apply a new color to the separated hair area. For example, a user can swipe a portion of a person in an image to cause a new color to be applied to the human hair area. As another example, the user terminal 200 may receive a swipe input with respect to an upper portion of the hair region in the image, and thus a color to be applied to the hair region may be selected. In addition, a swipe input may be received for the underside of the hair area in the image, thereby selecting a color to be applied to the hair area.
  • the user terminal 200 may select two colors according to a swipe input input above and below the hair region, and may combine gradients of the selected two colors to apply the gradation dye to the Merakilac region.
  • the dyeing color may be selected and applied according to the current hair color displayed on the human area in the image.
  • the hair region displayed on the human region in the image may be various dyed hairs such as bleached hair, healthy hair, and dyed hair, and different dyed colors may be applied according to the hair form or color.
  • the user terminal 200 may be configured to separate the background and the human area from the received image.
  • the segmentation mask can be used to separate the background and human regions within the image. First, it may be out of focus in the background area within the image. Then, the background can be replaced with another image, and various filter effects can be applied. For example, when a swipe input is detected in a background area in an image, another background may be applied to the background area.
  • lighting effects of different environments for each background may be applied to the human area and the hair area in the image. For example, different background lighting effects may be applied to each background to see which colors can be seen in each area when viewed in different lights.
  • the image to which the color, bokeh, or filter effect is applied may be output.
  • the user can select a dyeing color in a beauty salon or a cosmetics shop, and experience augmented reality (AR) by virtually applying a dye on his hair in advance using segmentation technology.
  • AR augmented reality
  • the background and the person in the image is separated and various effects described above may be applied.
  • the user terminal 200 may be configured to automatically track and focus an object in an image touched by the user.
  • An image can be separated into a plurality of graphical elements.
  • an algorithm, segmentation, and / or detection technique using an artificial neural network model may be used as the method of separating into graphic elements, but is not limited thereto.
  • the touched graphic element may be tracked and automatically focused.
  • the unfocused graphic elements of the image can be subjected to an out focusing effect.
  • other image transformation functions such as filter application and background substitution may be applied.
  • the focusing in the image can be changed with the touched graphic element.
  • the user terminal 200 may generate a segmentation mask for each part of the person area in the input image by using an artificial neural network trained to derive the person area from the input image including the person.
  • the portrait part may include, but is not limited to, various parts such as hair, face, skin, eyes, nose, mouth, ears, clothes, left arm, upper, lower arm, upper right arm, lower right arm, upper, lower and shoes.
  • the method of dividing may be applied to any algorithm or technique already known in the art of segmentation. Then, various effects such as color change, filter application, and background substitution may be applied to each divided part.
  • a method of naturally changing color includes changing a color space to black and white for an area corresponding to a segmentation mask, generating a histogram of the brightness of the converted black and white area, and having various brightnesses. Preparing a sample color; Generating a histogram for brightness by applying the sample to be changed and matching the histograms derived using a histogram matching technique to derive the color to be applied to each black and white region. For example, histogram matching may be performed such that similar colors are applied to portions having the same brightness. The matched color may be applied to an area corresponding to the segmentation mask.
  • the user terminal 200 may be configured to change the clothes of the people around to highlight a particular person in the image. Since the most diverse and complex area in the image containing the person is the clothes part, it can be implemented to correct the clothes so that the people around them are less noticeable so that a specific person is emphasized more.
  • the artificial neural network model trained to derive the person area from the person image may be derived by segmenting the person area in the input image for each person. Each character can then be segmented into various parts.
  • the person to be highlighted in the image can be selected from the user. For example, one or several may be selected. If the clothes of people other than the person to be emphasized in the image are lowered in saturation or the clothes are colorful patterns, the patterns may be simply changed.
  • the user terminal 200 may be configured to replace a face in the image with a virtual face.
  • the user terminal 200 may generate a segmentation mask corresponding to the face region in the input image by using an artificial neural network model trained to derive the face region from the person image.
  • a new virtual face may be generated using a generative model such as deep learning GAN.
  • a face newly generated by using face landmark technology may be synthesized to an existing face part.
  • this may be notified or a warning message may be sent when a person performing a particular action in the image of a CCTV, black box, or the like is detected.
  • it may be sensed how to perform the pose of the person from the input image using the artificial neural network model that is trained to predict the pose in the input image including the person.
  • the act may include, but is not limited to, violent acts, theft acts, and riot acts.
  • the detected information may be transmitted to a required place to give a notification.
  • the image may be captured by setting a high resolution.
  • a warning message may be delivered using various methods such as voice or video based on the detected information. For example, depending on the action, a warning message in the form of different voices and / or images may be generated and transmitted.
  • a fire when a fire occurs from the input image, it may be detected that a fire has occurred from the image by detecting various environments as well as temperature and smoke in the image. To this end, by using an artificial neural network trained to predict fire in the input image, it may be detected whether there is an area where a fire occurs in the input image. Then, when a fire is detected, a warning voice may be generated. For example, voice, such as the location of the fire, the size of the fire can be automatically generated. Relevant information about the fire may be sent to the location and / or equipment needed.
  • a human's movement, denseness, location of stay, etc. may be detected from the input image to analyze the purchase pattern. For example, people's traffic, density, location of stay in an offline store may be analyzed.
  • a segmentation mask corresponding to the human area is generated from the input image using an artificial neural network model trained to derive the human area from the image, and based on the derived human area, the human line, the density of the human, and the human stay Location and the like can be identified.
  • FIG. 12 is an exemplary diagram illustrating a process of implementing a telephoto lens zoom effect by extracting a narrower area from a background in an image as the user terminal 200 according to an embodiment of the present disclosure increases in intensity of bokeh blur.
  • the user terminal 200 may be configured to separate the focusing area and the background area in the image. According to one embodiment, as shown, an area in which the separated background area is narrower than in the image may be extracted. The extracted background can be enlarged and used as a new background.
  • the user terminal 200 may fill in the empty space generated between the focusing area and the background area as the image is enlarged.
  • a method of filling an empty space may be used as an inpainting algorithm, a reflection padding, or a method of repolating by radially interpolating, but is not limited thereto.
  • a deep learning technique may be applied, but is not limited thereto.
  • the user terminal 200 may apply a super resolution technique to correct the deterioration of the enlarged image quality.
  • the super resolution technique may be a technique already known in the image processing distribution, for example, the deep learning technique may be applied, but is not limited thereto.
  • the zoom effect when the zoom effect is applied to the image by receiving an input for applying the zoom effect, the image quality may be deteriorated.
  • the super resolution technique may be applied in real time to correct the zoom applied image. .
  • the image may be received by the user terminal.
  • the user terminal may input the received image into the input layer of the first artificial neural network model to generate a depth map representing depth information of pixels in the image (S1320).
  • the user terminal may apply the bokeh effect for the pixels in the image based on the depth map representing the depth information for the pixels in the image (S1330).
  • the first artificial neural network model may be generated by receiving a plurality of reference images as an input layer and performing machine learning to infer depth information included in the plurality of reference images.
  • the neural network model may be learned by the machine learning module 250.
  • FIG. 14 is a block diagram of a bokeh effect application system 1400 according to an embodiment of the present disclosure.
  • the bokeh effect application system 1400 may include a data learner 1410 and a data recognizer 1420.
  • the data learning unit 1410 of the bokeh effect application system 1400 of FIG. 14 corresponds to the machine learning module of the bokeh effect application system 205 of FIG. 2, and the data recognition unit of the bokeh effect application system 1400 of FIG. 14.
  • 1420 may correspond to the depth map generation module 210, the bokeh effect application module 220, the segmentation mask generation module 230, and / or the detection area generation module 240 of the user terminal 200 of FIG. 2. have.
  • the data learner 1410 may input data to obtain a machine learning model.
  • the data recognizer 1420 may apply the data to the machine learning model to generate a depth map / information and a segmentation mask.
  • the bokeh effect application system 1400 as described above may include a processor and a memory.
  • the data learner 1410 may be a synthesis for image processing or effects of an image.
  • the data learner 1410 may learn a criterion about what image processing or effect to output according to the image.
  • the data learner 1410 may learn a criterion for generating a depth map / information corresponding to at least some regions of the image using a feature of an image or generating a segmentation mask in which region in the image.
  • the data learner 1410 acquires data to be used for learning, and applies the acquired data to a data learning model to be described later, so that the data learning unit 1410 may learn about image processing or an effect according to an image.
  • the data recognizer 1420 may generate a depth map / information or a segmentation mask for at least a portion of the image based on the image. Depth maps / information and / or segmentation masks for the image may be generated and output. The data recognizer 1420 may output the depth map / information and / or the segmentation mask from a predetermined image by using the learned data learning model. The data recognizer 1420 may acquire a predetermined image (data) according to a preset criterion by learning. In addition, the data recognizing unit 1420 may generate a depth map / information and / or a segmentation mask based on predetermined data by using the data learning model using the acquired data as an input value. Also, the result value output by the data learning model using the acquired data as an input value may be used to update the data learning model.
  • At least one of the data learner 1410 or the data recognizer 1420 may be manufactured in the form of at least one hardware chip and mounted on the electronic device.
  • at least one of the data learner 1410 or the data recognizer 1420 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general purpose processor (eg, a CPU).
  • AI artificial intelligence
  • the electronic device may be manufactured as a part of an application processor or a graphics dedicated processor (eg, a GPU) and mounted on various electronic devices.
  • the data learner 1410 and the data recognizer 1420 may be mounted on separate electronic devices, respectively.
  • one of the data learner 1410 and the data recognizer 1420 may be included in the electronic device, and the other may be included in the server.
  • the data learning unit 1410 and the data recognizing unit 1420 may provide the model information constructed by the data learning unit 1410 to the data recognizing unit 1420 via a wired or wireless connection.
  • the data input to 1420 may be provided to the data learner 1410 as additional learning data.
  • At least one of the data learner 1410 or the data recognizer 1420 may be implemented as a software module.
  • the software module may be a memory or computer readable non-readable. It may be stored in a non-transitory computer readable media.
  • at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • OS operating system
  • OS operating system
  • others may be provided by a given application.
  • the data learner 1410 may include a data acquirer 1411, a preprocessor 1412, a training data selector 1413, a model learner 1414, and a model evaluator 1415. It may include.
  • the data acquirer 1411 may acquire data necessary for machine learning. Since a large amount of data is required for learning, the data acquirer 1411 may receive a plurality of reference images, corresponding depth maps / information and segmentation masks.
  • the preprocessor 1412 may preprocess the acquired data so that the obtained data may be used for machine learning through an artificial neural network model.
  • the preprocessor 1412 may process the acquired data into a preset format so that the model learner 1414, which will be described later, may use.
  • the preprocessor 1412 may analyze and acquire image characteristics of each pixel or pixel group in the image.
  • the training data selector 1413 may select data necessary for learning from the preprocessed data.
  • the selected data may be provided to the model learner 1414.
  • the training data selector 1413 may select data required for learning from preprocessed data according to a preset criterion.
  • the training data selector 1413 may select data according to preset criteria by learning by the model learner 1414, which will be described later.
  • the model learner 1414 may learn a criterion about which depth map / information and segmentation mask to output based on the image based on the training data. In addition, the model learner 1414 may train the learning model outputting the depth map / information and the segmentation mask according to the image as the training data.
  • the data learning model may include a pre-built model.
  • the data learning model may include a model built in advance by receiving basic training data (eg, a sample image).
  • the data learning model may be constructed in consideration of the application field of the learning model, the purpose of learning, or the computer performance of the device.
  • the data learning model may include, for example, a model based on a neural network.
  • models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Networks (CNN) can be used as data learning models. But it is not limited thereto.
  • the model learning unit 1414 may determine a data learning model having a large correlation between the input learning data and the basic learning data as the data learning model to be trained.
  • the basic training data may be previously classified by the type of data, and the data learning model may be pre-built for each type of data.
  • the basic training data is classified based on various criteria such as the region where the training data is generated, the time at which the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, and the types of objects in the training data. It may be.
  • model learner 1414 may train the data learning model using, for example, a learning algorithm including an error back-propagation method or a gradient descent method.
  • model learner 1414 may learn the data learning model through, for example, supervised learning using the learning data as an input value.
  • the model learning unit 1414 learns data through unsupervised learning that finds a criterion for situation determination by, for example, self-learning a type of data necessary for situation determination without any guidance. You can train the model.
  • the model learner 1414 may learn the data learning model through, for example, reinforcement learning using feedback on whether the result of the situation determination according to the learning is correct.
  • the model learner 1414 may store the learned data learning model.
  • the model learner 1414 may store the learned data learning model in a memory of the electronic device including the data recognizer 1420.
  • the model learner 1414 may store the learned data learning model in a memory of a server connected to the electronic device through a wired or wireless network.
  • the memory in which the learned data learning model is stored may store, for example, commands or data related to at least one other element of the electronic device.
  • the memory may also store software and / or programs.
  • the program may include, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or 'application'), or the like.
  • the model evaluator 1415 inputs the evaluation data into the data learning model, and if the result output from the evaluation data does not satisfy a predetermined criterion, the model evaluator 1414 may retrain the model learner.
  • the evaluation data may include preset data for evaluating the data learning model.
  • the model evaluator 1415 does not satisfy a predetermined criterion when the number or ratio of the evaluation data that is not accurate among the results of the learned data learning model for the evaluation data exceeds a preset threshold. It can be evaluated as. For example, when a predetermined criterion is defined at a ratio of 2%, when the trained data learning model outputs an incorrect recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 1415 learns. The data learning model can be evaluated as not suitable.
  • the model evaluator 1415 evaluates whether each learned video learning model satisfies a predetermined criterion, and uses the model satisfying the predetermined criterion as the final data learning model. You can decide. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 1415 may determine any one or a predetermined number of models that are preset in the order of the highest evaluation score as the final data learning model.
  • At least one of the data acquirer 1411, the preprocessor 1412, the training data selector 1413, the model learner 1414, or the model evaluator 1415 in the data learner 1410 is at least one. May be manufactured in the form of a hardware chip and mounted on an electronic device.
  • at least one of the data acquirer 1411, the preprocessor 1412, the training data selector 1413, the model learner 1414, or the model evaluator 1415 is an artificial intelligence (AI). It may be manufactured in the form of a dedicated hardware chip, or may be manufactured as part of an existing general purpose processor (eg, a CPU or an application processor) or a graphics dedicated processor (eg, a GPU) and mounted on the above-mentioned various electronic devices.
  • AI artificial intelligence
  • the data acquirer 1411, the preprocessor 1412, the training data selector 1413, the model learner 1414, and the model evaluator 1415 may be mounted in one electronic device or may be separate. Each of the electronic devices may be mounted. For example, some of the data acquirer 1411, the preprocessor 1412, the training data selector 1413, the model learner 1414, and the model evaluator 1415 are included in the electronic device, and some of the data are included in the electronic device. Can be included on the server.
  • At least one of the data acquirer 1411, the preprocessor 1412, the training data selector 1413, the model learner 1414, or the model evaluator 1415 may be implemented as a software module.
  • Module may be stored on a computer readable non-transitory computer readable media.
  • at least one software module may be provided by an operating system (OS) or by a predetermined application.
  • some of the at least one software module may be provided by an operating system (OS) and others may be provided by a given application.
  • the data recognizer 1420 may include a data acquirer 1421, a preprocessor 1422, a recognition data selector 1423, a recognition result provider 1424, and a model updater 1425. It may include.
  • the data acquirer 1421 may acquire an image necessary to output the depth map / information and the segmentation mask. In contrast, the data acquirer 1421 may acquire a depth map / information and a segmentation mask required to output an image.
  • the preprocessor 1422 may preprocess the acquired data so that the obtained data may be used to output the depth map / information and the segmentation mask.
  • the preprocessor 1422 may process the acquired data in a preset format so that the recognition result providing unit 1424, which will be described later, may use the acquired data to output the depth map / information and the segmentation mask.
  • the recognition data selector 1423 may select data necessary for outputting the depth map / information and the segmentation mask from the preprocessed data.
  • the selected data may be provided to the recognition result providing unit 1424.
  • the recognition data selector 1423 may select some or all of the preprocessed data according to preset criteria for outputting the depth map / information and the segmentation mask.
  • the recognition data selector 1423 may select data according to a predetermined criterion by learning by the model learner 1414.
  • the recognition result providing unit 1424 may apply the selected data to the data learning model and output a depth map / information and a segmentation mask.
  • the recognition result provider 1424 may apply the selected data to the data learning model by using the data selected by the recognition data selector 1423 as an input value.
  • the recognition result may be determined by the data learning model.
  • the model updater 1425 may cause the data learning model to be updated based on the evaluation of the recognition result provided by the recognition result provider 1424. For example, the model updater 1425 may provide the recognition result provided by the recognition result provider 1424 to the model learner 1414 so that the model learner 1414 updates the data learning model. have.
  • At least one of the data acquisition unit 1421, the preprocessor 1422, the recognition data selection unit 1423, the recognition result providing unit 1424, or the model updater 1425 in the data recognizing unit 1420 is at least one of the data recognizing unit 1420. It may be manufactured in the form of one hardware chip and mounted on an electronic device. For example, at least one of the data acquirer 1421, the preprocessor 1422, the recognition data selector 1423, the recognition result providing unit 1424, or the model updater 1425 may be artificial intelligence (AI).
  • AI artificial intelligence
  • a dedicated hardware chip May be manufactured in the form of a dedicated hardware chip, or may be manufactured as a part of an existing general purpose processor (eg, a CPU or an application processor) or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various electronic devices.
  • a general purpose processor eg, a CPU or an application processor
  • a graphics dedicated processor eg, a GPU
  • the data obtaining unit 1421, the preprocessor 1422, the recognition data selecting unit 1423, the recognition result providing unit 1424, and the model updating unit 1425 may be mounted in one electronic device or may be separate. May be mounted on the electronic devices. For example, some of the data obtaining unit 1421, the preprocessor 1422, the recognition data selecting unit 1423, the recognition result providing unit 1424, and the model updating unit 1425 are included in the electronic device, and some of the remaining portions are included in the electronic device. May be included in the server.
  • At least one of the data acquirer 1421, the preprocessor 1422, the recognition data selector 1423, the recognition result providing unit 1424, or the model updater 1425 may be implemented as a software module.
  • At least one of the data obtaining unit 1421, the preprocessor 1422, the recognition data selecting unit 1423, the recognition result providing unit 1424, or the model updating unit 1425 includes a software module (or instruction). If implemented as a program module, the software module may be stored in a computer readable non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and others may be provided by a given application.
  • OS operating system
  • OS operating system
  • a user terminal providing a service for applying a bokeh effect to an image and the bokeh effect application system described herein includes a wireless telephone, a cellular telephone, a laptop computer, a wireless multimedia device, a wireless communication personal computer (PC) card, It may represent various types of devices such as a PDA, an external modem or an internal modem, a device communicating over a wireless channel, and the like.
  • the device may be an access terminal (AT), an access unit, a subscriber unit, a mobile station, a mobile device, a mobile unit, a mobile telephone, a mobile, a remote station, a remote terminal, a remote unit, a user device, user equipment, It may have various names, such as a handheld device. Any device described herein may have memory for storing instructions and data, as well as hardware, software, firmware, or combinations thereof.
  • the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), Field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein. May be implemented within a computer, or a combination thereof.
  • a general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
  • a processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
  • the techniques may include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM (On computer readable media such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, and the like. It may also be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor (s) to perform certain aspects of the functionality described herein.
  • Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
  • Storage media may be any available media that can be accessed by a computer.
  • such computer-readable media may be in the form of RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures.
  • any connection is properly termed a computer readable medium.
  • the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave
  • coaxial cable , Fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, wireless, and microwave
  • disks and disks include CDs, laser disks, optical disks, digital versatile discs, floppy disks, and Blu-ray disks, where the disks are usually Magnetically reproduce the data, while discs discs optically reproduce the data using a laser. Combinations of the above should also be included within the scope of computer-readable media.
  • the software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other type of storage medium known in the art.
  • An exemplary storage medium may be coupled to the processor such that the processor can read information from or write information to the storage medium.
  • the storage medium may be integral to the processor.
  • the processor and the storage medium may reside in an ASIC.
  • the ASIC may be present in the user terminal.
  • the processor and the storage medium may reside as discrete components in a user terminal.
  • example implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more standalone computer systems, the subject matter is not so limited, but rather in connection with any computing environment, such as a network or a distributed computing environment. It may be implemented. Moreover, aspects of the presently disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and handheld devices.
  • Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like.
  • the computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And, the functional program, code and code segments for implementing the embodiments can be easily inferred by programmers in the art to which the present invention belongs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un procédé réalisé par un terminal utilisateur permettant d'appliquer un effet Bokeh sur une image. Le procédé d'application d'un effet Bokeh peut comprendre les étapes consistant : à recevoir une image, et à entrer l'image reçue dans une couche d'entrée d'un premier modèle de réseau neuronal artificiel pour générer une carte de profondeur indiquant des informations de profondeur de pixels dans l'image ; et à appliquer un effet de Bokeh aux pixels dans l'image sur la base de la carte de profondeur indiquant les informations de profondeur des pixels dans l'image. Ici, le premier modèle de réseau neuronal artificiel peut être généré en réalisant un apprentissage machine en vue de recevoir une pluralité d'images de référence au niveau de la couche d'entrée et déduire des informations de profondeur incluses dans la pluralité d'images de référence.
PCT/KR2019/010449 2018-08-16 2019-08-16 Procédé d'application d'effet bokeh sur une image et support d'enregistrement WO2020036468A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/101,320 US20210073953A1 (en) 2018-08-16 2020-11-23 Method for applying bokeh effect to image and recording medium

Applications Claiming Priority (10)

Application Number Priority Date Filing Date Title
KR10-2018-0095255 2018-08-16
KR20180095255 2018-08-16
KR20180122100 2018-10-12
KR10-2018-0121628 2018-10-12
KR10-2018-0122100 2018-10-12
KR20180121628 2018-10-12
KR10-2018-0133885 2018-11-02
KR20180133885 2018-11-02
KR10-2019-0100550 2019-08-16
KR1020190100550A KR102192899B1 (ko) 2018-08-16 2019-08-16 이미지에 보케 효과를 적용하는 방법 및 기록매체

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/101,320 Continuation US20210073953A1 (en) 2018-08-16 2020-11-23 Method for applying bokeh effect to image and recording medium

Publications (1)

Publication Number Publication Date
WO2020036468A1 true WO2020036468A1 (fr) 2020-02-20

Family

ID=69525760

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2019/010449 WO2020036468A1 (fr) 2018-08-16 2019-08-16 Procédé d'application d'effet bokeh sur une image et support d'enregistrement

Country Status (1)

Country Link
WO (1) WO2020036468A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113643235A (zh) * 2021-07-07 2021-11-12 青岛高重信息科技有限公司 一种基于深度学习的芯片计数方法
CN116051361A (zh) * 2022-06-30 2023-05-02 荣耀终端有限公司 图像维测数据的处理方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140004592A (ko) * 2012-07-03 2014-01-13 우드맨 랩스, 인크. 3d 깊이 정보를 기반으로 한 이미지 블러
KR20150107605A (ko) * 2014-03-14 2015-09-23 후아웨이 테크놀러지 컴퍼니 리미티드 이미지 블러링 방법 및 장치, 그리고 전자 장치
US20160350930A1 (en) * 2015-05-28 2016-12-01 Adobe Systems Incorporated Joint Depth Estimation and Semantic Segmentation from a Single Image
KR20170018009A (ko) * 2014-06-17 2017-02-15 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 이미지 처리 방법 및 장치
KR101795952B1 (ko) * 2016-07-20 2017-11-09 연세대학교 산학협력단 2d 영상에 대한 깊이 영상 생성 방법 및 장치

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140004592A (ko) * 2012-07-03 2014-01-13 우드맨 랩스, 인크. 3d 깊이 정보를 기반으로 한 이미지 블러
KR20150107605A (ko) * 2014-03-14 2015-09-23 후아웨이 테크놀러지 컴퍼니 리미티드 이미지 블러링 방법 및 장치, 그리고 전자 장치
KR20170018009A (ko) * 2014-06-17 2017-02-15 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 이미지 처리 방법 및 장치
US20160350930A1 (en) * 2015-05-28 2016-12-01 Adobe Systems Incorporated Joint Depth Estimation and Semantic Segmentation from a Single Image
KR101795952B1 (ko) * 2016-07-20 2017-11-09 연세대학교 산학협력단 2d 영상에 대한 깊이 영상 생성 방법 및 장치

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113643235A (zh) * 2021-07-07 2021-11-12 青岛高重信息科技有限公司 一种基于深度学习的芯片计数方法
CN113643235B (zh) * 2021-07-07 2023-12-29 青岛高重信息科技有限公司 一种基于深度学习的芯片计数方法
CN116051361A (zh) * 2022-06-30 2023-05-02 荣耀终端有限公司 图像维测数据的处理方法及装置
CN116051361B (zh) * 2022-06-30 2023-10-24 荣耀终端有限公司 图像维测数据的处理方法及装置

Similar Documents

Publication Publication Date Title
KR102192899B1 (ko) 이미지에 보케 효과를 적용하는 방법 및 기록매체
WO2020085694A1 (fr) Dispositif de capture d'image et procédé de commande associé
WO2020159232A1 (fr) Procédé, appareil, dispositif électronique et support d'informations lisible par ordinateur permettant de rechercher une image
WO2019031714A1 (fr) Procédé et appareil de reconnaissance d'objet
WO2021251689A1 (fr) Dispositif électronique et procédé de commande de dispositif électronique
WO2018117704A1 (fr) Appareil électronique et son procédé de fonctionnement
WO2020130747A1 (fr) Appareil et procédé de traitement d'image pour transformation de style
US5912980A (en) Target acquisition and tracking
KR101679290B1 (ko) 영상 처리 방법 및 장치
WO2019059505A1 (fr) Procédé et appareil de reconnaissance d'objet
CN110959160A (zh) 一种手势识别方法、装置及设备
WO2021006482A1 (fr) Appareil et procédé de génération d'image
EP3539056A1 (fr) Appareil électronique et son procédé de fonctionnement
WO2019231130A1 (fr) Dispositif électronique et son procédé de commande
WO2020235852A1 (fr) Dispositif de capture automatique de photo ou de vidéo à propos d'un moment spécifique, et procédé de fonctionnement de celui-ci
WO2021045599A1 (fr) Procédé d'application d'effet bokeh sur une image vidéo et support d'enregistrement
WO2020036468A1 (fr) Procédé d'application d'effet bokeh sur une image et support d'enregistrement
WO2018117753A1 (fr) Dispositif électronique et procédé de commande associé
WO2020141907A1 (fr) Appareil de production d'image permettant de produire une image en fonction d'un mot clé et procédé de production d'image
WO2020091268A1 (fr) Appareil électronique et procédé de commande associé
WO2019190142A1 (fr) Procédé et dispositif de traitement d'image
EP3545685A1 (fr) Procédé et appareil de filtrage de vidéo
EP3707678A1 (fr) Procédé et dispositif de traitement d'image
WO2019190171A1 (fr) Dispositif électronique et procédé de commande associé
WO2022019390A1 (fr) Dispositif et procédé d'apprentissage de modèle d'analyse d'objet sur la base d'une augmentation de données

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19850421

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19850421

Country of ref document: EP

Kind code of ref document: A1