KR20230083743A - Method for providing image and electronic device supporting the same - Google Patents

Abstract

An electronic device according to various embodiments comprises: a memory for storing a plurality of groups corresponding to a plurality of representative facial features, respectively; and at least one processor, wherein the at least one processor may be configured to obtain a first image, detect a first region representing a face, the features of the face, and a second region requiring restoration in the first image, select a first group among the plurality of groups based on the features of the face and the plurality of representative facial features, obtain a facial region having the second region restored based on the representative facial features of the first group and the features of the face, and obtain a second image by synthesizing the facial region and the first image. Therefore, provided are a method for providing an image and an electronic device supporting the same, wherein the appearance of an actually photographed subject can be reflected more accurately on an image.

Description

Method for providing an image and electronic device supporting the same {METHOD FOR PROVIDING IMAGE AND ELECTRONIC DEVICE SUPPORTING THE SAME}

Various embodiments of this document relate to a method for providing an image and an electronic device supporting the same.

Image inpainting refers to a technique of restoring a damaged or obscured portion of an image.

When an image in which a part of a person's face is obscured by an accessory such as a mask or sunglasses is obtained, the electronic device may acquire an image in which the part of the face is partially restored by using image inpainting. there is.

When an image including a occluded part (or a damaged part) is acquired, the electronic device uses the artificial intelligence model to learn data ( For example, the masked part in the image can be restored using the features of a normal human face. Alternatively, when an image including an obscured portion is acquired, the electronic device matches the obtained image and a reference image using a warping technique, and matches the obtained image within the matched reference image. The obscured portion in the image may be restored by combining the portion corresponding to the obscured portion with the non-obscured portion of the acquired image.

In the case of restoring the obscured part in an image obtained from an electronic device using data learned using an artificial intelligence model (eg, general human facial features), the subject of the acquired image (eg, the user of the electronic device) face) and the subject of the image used for learning the artificial intelligence model (e.g., a general human face) are different, the restored part of the image in which the occluded part is restored may differ from the actual appearance of the subject of the acquired image. there is.

In addition, when a part obscured from an image obtained from an electronic device is restored using a warping technique, the shape of the object (eg, the face of the user of the electronic device) of the acquired image (eg, the face of the user of the electronic device is facing) Face shape that varies depending on direction), brightness of the acquired image, and facial expression and/or posture of the target of the acquired image are not considered, so the quality of the image in which the occluded part is restored may be degraded.

In various embodiments of the present document, by using a plurality of images stored in an electronic device to restore a part that needs to be restored in an image acquired by the electronic device, the appearance of an actual photographed target (eg, the user's face of the electronic device). It relates to a method for obtaining a more accurately reflected image, providing an image, and an electronic device supporting the same.

The technical tasks to be achieved by this document are not limited to the technical tasks mentioned above, and other technical tasks not mentioned can be clearly understood by those skilled in the art related to this document from the description below. There will be.

An electronic device according to various embodiments includes a memory for storing a plurality of groups corresponding to a plurality of representative facial features, and at least one processor electrically connected to the memory, the at least one processor comprising: A first image is acquired, a first region representing a face, a feature of the face, and a second region requiring restoration are detected in the first image, and the feature of the face and the plurality of regions are detected. Based on the representative facial features of, a first group is selected from among the plurality of groups, and based on the representative facial features of the first group and the facial features, a face region in which the second region is restored is obtained. and obtaining a second image by synthesizing the face region and the first image.

A method for providing an image in an electronic device according to various embodiments includes an operation of acquiring a first image, a first area representing a face in the first image, characteristics of the face, and a first area in the first area. An operation of detecting a second region requiring restoration, and selecting a first group from among the plurality of groups based on the facial features and a plurality of representative facial features corresponding to each of the plurality of groups stored in the memory of the electronic device. By selecting an operation, obtaining a face area in which the second area is restored based on the representative facial features of the first group and the facial features, and synthesizing the face area and the first image, 2 may include an operation of acquiring images.

A method for providing an image according to various embodiments and an electronic device supporting the same may restore a part that needs to be restored in an image acquired by the electronic device using a plurality of images stored in the electronic device, so that the actual photographing target (e.g. : An image in which a face of a user of an electronic device is more accurately (or more similarly) reflected may be obtained.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram of an electronic device, according to various embodiments.
3 is a block diagram of a processor, in accordance with various embodiments.
4 is a flowchart illustrating a method of providing an image, according to various embodiments.
5 is a flow chart describing a method of detecting facial elements, according to various embodiments.
6 is a flowchart illustrating a method of selecting a group from among a plurality of groups, according to various embodiments.
7 is a flowchart illustrating a method of selecting a group from among a plurality of groups or selecting an image including a predefined face, according to various embodiments.
8 is a flowchart illustrating a method of selecting a group from among a plurality of groups, according to various embodiments.
9 is a flowchart illustrating a method of obtaining a second image, according to various embodiments.
10 is an exemplary view illustrating a method of obtaining a second image, according to various embodiments.
11 is a flowchart illustrating a method of managing a plurality of groups, according to various embodiments.
12 is a flowchart describing a method of detecting a facial element, according to various embodiments.
13 is a flowchart illustrating a method of managing a plurality of groups, according to various embodiments.
14A and 14B are exemplary diagrams illustrating a method of providing an image, according to various embodiments.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments.

Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of other functionally related components (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, a : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

2 is a block diagram of an electronic device 101 according to various embodiments.

Referring to FIG. 2 , in one embodiment, an electronic device 101 may include a display 210, a camera module 220, a memory 230, and/or a processor 240.

In one embodiment, display 210 may be included in display module 160 of FIG. 1 .

In one embodiment, the display 210 may display various screens. For example, the display 210 may display a screen including an image while a camera application or an image application (eg, a gallery application) is executed on the electronic device 101 . As another example, the display 210 performs face inpainting (hereinafter, referred to as 'face inpainting') for restoring a region that needs to be restored within a face region included in an image. Various screens for face inpainting (eg, screens 1410, 1420, 1430, and 1440 of FIGS. 14A and 14B) may be displayed.

In one embodiment, the camera module 220 may be included in the camera module 180 of FIG. 1 .

In one embodiment, the camera module 220 may obtain an image to be subjected to face inpainting. In one embodiment, the image acquired through the camera module 220 and subject to face inpainting is a still image (eg, a still image obtained by a capture input while a preview image is displayed) and/or dynamic images (eg, videos, images).

In one embodiment, memory 230 may be included in memory 130 of FIG. 1 .

In one embodiment, the memory 230 may store information for performing at least part of an operation of providing an image. Information for performing at least a part of an operation of providing an image stored in the memory 230 will be described later.

In one embodiment, processor 240 may be included in processor 120 of FIG. 1 .

In one embodiment, the processor 240 may overall control an operation of providing an image. The processor 240 may include a plurality of components for performing an operation of providing an image. A plurality of components included in the processor 240 to provide an image will be described with reference to FIG. 3 . In one embodiment, the processor 240 may include one or more processors for performing an operation of providing an image.

In one embodiment, the electronic device 101 is illustrated in FIG. 2 as including a display 210, a camera module 220, a memory 230, and/or a processor 240, but is not limited thereto. . For example, the electronic device 101 may further include a communication module (eg, the communication module 190 of FIG. 1 ). When the electronic device 101 further includes a communication module, the electronic device 101 performs face recognition from an external electronic device (eg, the server 108, the electronic device 102, or the electronic device 104) through the communication module. Images for painting can be obtained. For another example, the electronic device 101 may not include the display 210 and/or the camera module 220 according to embodiments.

3 is a block diagram of a processor 240, in accordance with various embodiments.

Referring to FIG. 3 , in one embodiment, the processor 240 may include a group management unit 310 and a restoration unit 320 .

In one embodiment, the group management unit 310 may be a component for grouping a plurality of images stored in the memory 230 into a plurality of groups and managing the plurality of groups and group information for the plurality of groups. there is.

In one embodiment, the group management unit 310 may include a first face element detection unit 311, a face grouping unit 312, a face group management unit 313, and/or a new image group selection unit 314. there is.

In one embodiment, the facial element detection unit 311 may detect (eg, extract) facial elements within an image.

In an embodiment, the face element may include a face region representing a face in an image (eg, an entire face region), a region within the face region that needs to be restored, and facial features. Hereinafter, for convenience of description, a face region included in an image (eg, the entire face region) is referred to as a 'first region', and a region requiring restoration within the face region is referred to as a 'second region', An area (eg, an unoccluded area) excluding the area requiring restoration (the second area) within the face area (first area) will be referred to as a 'third area'.

In an embodiment, the second region (a region in the face region requiring restoration) may be a region in which the reliability of a feature detected within the region is less than or equal to a specified reliability level. For example, facial features may be extracted in the form of a vector having a constant size within the first region. Based on the degree of similarity between the vector corresponding to the extracted facial feature and the reference vectors stored in the memory 230 (eg, the average and/or minimum value of similarity), the reliability of the extracted facial feature is determined. can decide The second region may refer to a region in which the determined reliability level is equal to or less than the designated reliability level in the first region. In an embodiment, the second region may include a region in which a portion of a person's face is covered by an accessory such as a mask or sunglasses and/or a human body such as a person's hand or hair, within the first region. For another example, the second area may include an area in which part of data constituting the image in the first area is lost and/or damaged. However, examples of the second area are not limited to the above examples.

In an embodiment, the first facial element detector 311 may detect facial elements in an image using a generative adversarial network (GAN). For example, the first face element detector 311 may detect a face element in an image using style GAN-based Pixel2Style2Pixel and/or face identity disentanglement.

In one embodiment, the first facial element detector 311 may determine facial features, an identity feature of a person's face (hereinafter, referred to as a 'first facial feature'), and an environment when acquiring an image. And/or a feature reflecting the state of the face (hereinafter, referred to as 'second facial feature') (or also referred to as 'attribute feature') may be classified and detected.

In one embodiment, the first feature of the face is the shape of the ears, eyes, nose, and mouth, skin tone, and/or that is unique to a person's face (eg, that all human faces have in common). Or it may include the form of a beard. However, the first feature of the face is not limited to the above examples.

In one embodiment, the second feature of the face may be the environment at the time of image acquisition (eg, lighting (or brightness) around the subject) and/or the state of the person (subject) at the time of image acquisition (eg, the person's posture, the person ( It can include features that can change each time an image is acquired, such as facial expression). For example, the second feature of the face may include a relative direction of the face to the electronic device 101 (eg, the camera module 220) when acquiring an image. However, the second feature of the face is not limited to the above examples.

In the above examples, the first facial element detection unit 311 included in the group management unit 310 is illustrated as detecting the first facial feature and the second facial feature as facial features, but in one embodiment , the first facial element detector 311 included in the group management unit 310 may detect only the first feature of the face from each of all images stored in the memory 230 without detecting the second feature of the face. may be

In one embodiment, the first facial element detector 311 may perform an operation of detecting a facial element for each of all images stored in the memory 230 (eg, all images related to the gallery application). . For example, the first face element detection unit 311 detects the memory 230 when an application for face inpainting is installed (or when an application for face inpainting is executed after installing an application for face inpainting) or periodically. For each of all images stored in , an operation of detecting a face element may be performed.

In an embodiment, the first facial element detector 311 may perform an operation of detecting facial elements in the acquired image when obtaining an image. For example, when an image is acquired through the camera module 220, the first facial element detector 311 may perform an operation of detecting a facial element in the acquired image. However, the first facial element detecting unit 311 is an image in which an operation of detecting facial elements of an image has been performed by the second facial element detecting unit 321 included in the restoring unit 320 (eg, the result of face inpainting). image), an operation of separately detecting a face element may not be performed.

In one embodiment, the first face element detector 311 detects a face in an image when a face region does not exist in the image or when a ratio of a region requiring restoration to a face region in the image is greater than or equal to a specified ratio. An operation of acquiring (eg, extracting) a feature of may not be performed. For example, the first facial element detector 311 detects facial features in an image only when a face region exists in the image and a ratio of a region requiring restoration to the face region in the image is less than a specified ratio. Acquiring (eg, extracting) operations can be performed. When the first face element detection unit 311 does not perform an operation of extracting facial features from an image, the face grouping unit 312, the face group management unit 313, and the new image group selection unit 314 operate may not

In one embodiment, the face grouping unit 312, when a face element is detected for each of all images (eg, all images related to the gallery application) stored in the memory 230, all stored in the memory 230 Images may be grouped (eg, classified) into a plurality of groups (hereinafter referred to as 'a plurality of groups'). For example, the face grouping unit 312 may group images in which face elements are detected among all images stored in the memory 230 into a plurality of groups.

In an embodiment, the face grouping unit 312 may group the images into a plurality of groups based on first characteristics of each of the images (eg, images in which face elements are detected). In one embodiment, the face grouping unit 312 may group the images into a plurality of groups so that one or more images having a similar first feature are included in the same group by using a specified algorithm. For example, the face grouping unit 312 uses a designated algorithm (eg, a K-means clustering algorithm) so that the similarity between the first features of one or more images included in the same group is greater than or equal to the designated similarity. can be grouped into a plurality of groups.

In one embodiment, the face group management unit 313 may manage a plurality of groups.

In an embodiment, the face group management unit 313 may obtain (eg, calculate) group information for each of a plurality of groups. In one embodiment, the group information may include a group feature (hereinafter, mixed with 'representative facial feature of the group'), a representative image of the group, and/or a size of the group.

In an embodiment, the face group management unit 313 may obtain representative facial features of a group based on first features and weights of faces of one or more images belonging to the same group.

In one embodiment, the face group management unit 313 sets weights for one or more images belonging to the same group based on the angle of the face in the image (or the direction the face faces) and/or the resolution of the image. (e.g. granted). For example, with respect to one or more images belonging to the same group, the face group management unit 313 increases the face included in the image toward the front of the camera module 220 (eg, the direction the face faces and the camera module ( 220) is parallel) and/or the resolution of the image is higher, a higher weight may be set for the image.

In one embodiment, the face group management unit 313 may determine, for one or more images belonging to the same group, the angle of the face in the image, the resolution of the image, and/or the image used for face inpainting (eg: Based on whether the image is a result image of face inpainting), weights may be set (eg, assigned). For example, the face group management unit 313 determines that an image resulting from face inpainting performed by the restoration unit 320 (eg, a second image obtained in operation 409 of FIG. 4 to be described later) belongs to the same group. You can set a higher weight than other images (or add a weight value).

In one embodiment, the face group management unit 313 obtains a group feature (eg: can be calculated).

[Equation 1]

In [Equation 1], G Represents the number of images included in the group, F _G denotes a group feature (eg, a representative facial feature of the group), F _i denotes a first face feature of the image, and W _i denotes a weight of the image.

In one embodiment, the face group management unit 313 determines (eg, sets an image having the highest weight) an image having the highest weight (eg, an image with the highest weight set) as a representative image of the group among images included in the same group. )can do.

In one embodiment, the face group management unit 313 may determine the number of images included in each of the plurality of groups (eg, the number of images included in the corresponding group), or the total number of images stored in the memory 230 (or A ratio of the number of images included in the corresponding group to the total number of images included in the plurality of groups may be determined (eg, set) as the size of the corresponding group.

In an embodiment, the face group management unit 313 may transmit group information on a plurality of groups to the restoration unit 320 . For example, when receiving a request from the restoration unit 320, the face group management unit 313 may transfer group information on a plurality of groups to the restoration unit 320.

In one embodiment, when an image is input, the new image group selector 314 may select a group to include the input image from among a plurality of groups.

In an embodiment, an image input to the new image group selector 314 may be an image newly added to the gallery application (eg, an image managed by the gallery application). For example, an image input to the new image group selector 314 may be an image obtained through the camera module 220 or an external electronic device (eg, server 108, electronic device 102, electronic device 104 )) (for example, an image acquired through the camera module 220 or obtained from an external electronic device and immediately stored in the memory 230 without face inpainting). For another example, an image input to the new image group selector 314 may include an image on which face inpainting is performed (a result image of face inpainting).

In one embodiment, the new image group selection unit 314, when an image is input to the new image group selection unit 314, the input image (hereinafter referred to as 'input image') is face inpainting (eg : It is possible to check whether the face inpainting performed by the restoration unit 320 is an image (hereinafter, referred to as 'result image of face inpainting').

In an embodiment, the new image group selector 314 may check whether the input image corresponds to the resultant image of face inpainting based on a flag related to the input image. For example, the new image group selector 314 determines that the input image corresponds to the resulting image of face inpainting when a flag related to the input image indicates that the input image is an image in which face inpainting has been performed. You can check. For another example, the new image group selector 314 may indicate that a flag related to the input image indicates that the input image is an image on which face inpainting has not been performed, or a flag related to the input image does not exist. After the input image is acquired through the camera module 220 or acquired from an external electronic device, it can be confirmed that it corresponds to the image stored in the memory 230 without face inpainting.

In one embodiment, the new image group selection unit 314 selects a group selected while performing face inpainting related to the input image when it is confirmed that the input image is the result image of face inpainting. You can select as a group to include images. In an embodiment, the new image group selector 314 selects a selected group (eg, a first group selected in operation 405 to be described later) while face inpainting is performed on the input image, based on a flag related to the input image. group) may be determined as a group to include the input image. For example, the new image group selection unit 314 selects an input image to obtain an input image while face inpainting related to the input image is performed when the input image corresponds to a result image of face inpainting. Based on the flag indicating the selected group, a group selected to obtain an input image while face inpainting related to the input image is performed may be determined as a group to include the input image.

In one embodiment, the new image group selector 314 determines that the input image is not the result image of face inpainting (eg, obtained through the camera module 220 or obtained from an external electronic device). In the case of an image directly stored in the memory 230 without face inpainting), the first feature of the face of the input image and representative facial features of a plurality of groups (eg, the first feature of the face of the input image and the plurality of face features of the input image) The similarity between each of the representative facial features of the groups) may be compared. The new image group selector 314 selects a group having a representative facial feature most similar to a first feature of the face of the input image among the representative facial features of the plurality of groups (e.g., of the plurality of groups) among the plurality of groups. If the similarity between the representative facial feature most similar to the first facial feature of the input image among the representative facial features and the first facial feature of the input image is equal to or greater than the specified similarity, a group corresponding to the most similar representative facial feature) can be selected as a group to include the input image.

In one embodiment, the new image group selection unit 314 includes the input image in the selected group, and includes the selected group and group information on the selected group (eg, group characteristics of the selected group, representative image of the group, and / or group size). The new image group selector 314 is configured to select a representative facial feature of a plurality of groups when there is no group having a representative facial feature similar to the first feature of the face of the input image (e.g., the representative facial features of the plurality of groups). If the similarity between the representative facial feature most similar to the first facial feature of the input image and the first facial feature of the input image is less than the specified similarity), a new group is obtained in addition to the previously obtained plurality of groups ( e.g. create), and include the entered image in a new group. In this case, the new image group selector 314 acquires (eg creates) and manages group information (eg, group characteristics of the new group, representative image of the group, and/or size of the group) for the new group. can

In one embodiment, the group management unit 310 may operate in the background of the electronic device. For example, the group management unit 310 may perform operations including an operation of detecting a face element for each of all images stored in the memory 230 when an application for face inpainting is installed or periodically. there is. For another example, in one embodiment, the group management unit 310 (eg, the first facial element detection unit 311) performs operations including an operation of detecting a facial element in the acquired image whenever an image is acquired. can be done

In one embodiment, the reconstruction unit 320 may perform face inpainting on the image. In one embodiment, the restoration unit 320 includes a second facial element detection unit 321, an input image group selection unit 324, a basic face management unit 323, a facial feature combining unit 324, a face area generating unit, and/or a synthesis unit 326 .

In one embodiment, the second facial element detection unit 321 may detect (eg, extract) facial elements within an image. For example, the second facial element detector 321 is acquired through the camera module 220 (or obtained from an external electronic device (eg, server 108, electronic device 102, or electronic device 104)). A user input for performing face inpainting may be received with respect to a selected) image or an image selected based on a user input from among images stored in the memory 230 . When a user input for performing face inpainting is received, the second facial element detection unit 321 designates the obtained image or the selected image (hereinafter, the acquired image or the selected image as a 'first image'). ), a first region (a face region representing a face in an image), a second region (a region within the face region that needs to be restored), and/or facial features may be detected.

In an embodiment, the second facial element detector 321 may detect facial elements in the first image by using a generative adversarial network (GAN).

In one embodiment, the second facial element detector 321 determines the facial features of the first image, the first facial features (characteristics unique to a human face) and the second facial features (environmental and / or features reflecting the state of the face) can be classified and detected.

In one embodiment, the first feature of the face is the shape of the ears, eyes, nose, and mouth, skin tone, and/or that is unique to a person's face (eg, that all human faces have in common). Or it may include the form of a beard. However, the first feature of the face is not limited to the above examples.

In one embodiment, the second feature of the face may be the environment at the time of image acquisition (eg, lighting (or brightness) around the subject) and/or the state of the person (subject) at the time of image acquisition (eg, the person's posture, the person ( It can include features that can change each time an image is acquired, such as facial expression). For example, the second feature of the face may include a relative direction of the face to the electronic device 101 (eg, the camera module 220) when acquiring an image. However, the second feature of the face is not limited to the above examples.

In an exemplary embodiment, the second facial element detector 321 converts the first feature of the first image into the first feature of the second area and the third area (eg, the area within the face area excluding the second area that needs to be restored). .

In one embodiment, the second facial element detector 321 determines whether a face region does not exist in the first image or a ratio of a region requiring restoration to the face region in the first image is less than a specified ratio. An operation of obtaining (eg, extracting) facial features in the first image may not be performed. For example, the second face element detector 321 detects the first image only when a face region exists in the first image and the ratio of the region requiring reconstruction to the face region in the first image is greater than or equal to a specified ratio. It is possible to perform an operation of obtaining (eg, extracting) facial features in When the second facial element detection unit 321 does not perform an operation of extracting facial features from the first image, the input image group selection unit 322, the basic face management unit 323, and the facial feature combining unit 324 , the face area obtaining unit 325, and the synthesizing unit 326 may not operate.

In an embodiment, the input image group selector 322 selects a group for reconstructing a face region included in the first image, based on group information on a plurality of groups received from the group manager 310. can

In one embodiment, the input image group selector 322 receives group information (eg, group characteristics, representative image of the group, and/or size of the group) from the group management unit 310 for each of the plurality of groups. can receive

In one embodiment, the input image group selection unit 322 determines the similarity between the first feature of the face of the first image and group features (eg, representative facial features) corresponding to the plurality of groups, and/or a plurality of Based on the group sizes of the groups of , one or more candidate groups may be determined.

In one embodiment, the input image group selector 322 selects a third area of the face of the first image (eg, a second region requiring restoration within the face region) from among representative facial features corresponding to a plurality of groups. Features corresponding to the first feature (e.g., a feature of a face part corresponding to an unobscured region of the first image in the representative facial feature) may be identified. there is. For example, when the third region of the face of the first image includes the shape of the ears, eyes, and nose as representative facial features, the input image group selector 322 selects the mouth in the first image. in the first image, excluding the shape of the mouth (e.g., in each of the representative facial features) Except for the shape of the mouth corresponding to the mouth covered by the mask), the shape of the ears, eyes, and nose can be confirmed.

In an embodiment, the input image group selector 322 may select features corresponding to a first feature of a third region of the face of the first image from among representative facial features respectively corresponding to a plurality of groups; Similarities between the first features of the third region of the face of one image may be calculated.

In an embodiment, the input image group selector 322 may determine one or more candidate groups based on the calculated similarities and/or group sizes of the plurality of groups. For example, the input image group selector 322 may determine one or more candidate groups whose calculated similarity is greater than or equal to a specified similarity and whose group size is greater than or equal to a specified size.

In one embodiment, the face inpainting operation is performed based on an image including a face similar to the face of the first image by determining one or more candidate groups based on the calculated similarities by the input image group selector 322. can be performed. In an embodiment, the input image group selection unit 322 determines one or more candidate groups based on group sizes of a plurality of groups, thereby performing a face inpainting operation based on images of a group whose photographing frequency is higher than that of other groups. can be performed.

However, a method of determining one or more groups is not limited to the above-described example. For example, the input image group selector 322 may determine one or more candidate groups having a high similarity among the similarities without considering the size of the group.

In an embodiment, the input image group selector 322 may display representative images respectively corresponding to the determined one or more candidate groups through the display 210 . In one embodiment, while displaying representative images corresponding to the one or more candidate groups through the display 210, the input image group selector 322 selects a group corresponding to a representative image selected based on a user input. It can be selected as a group for face inpainting of the first image.

In the above examples, the input image group selector 322 determines one or more candidate groups and selects a group selected by the user from among the determined one or more candidate groups as a group for face inpainting of the first image. but is not limited to For example, the input image group selector 322 selects each of the features corresponding to the first feature of the third area of the face of the first image from among the representative facial features corresponding to the plurality of groups, and the first Similarities between the first features of the third region of the face of the image may be calculated. The input image group selection unit 322 selects a group having the highest similarity among the similarities (eg, a feature corresponding to a first feature of a third area of the face of the first image from among representative facial features of the group, and a first A group having the greatest similarity between the first features of the third region of the face of the image) may be selected as a group for face inpainting of the first image.

In one embodiment, the basic face management unit 323 may manage images including a predefined face. For example, among the representative facial features corresponding to the plurality of groups, each of the features corresponding to the first feature of the third region of the face of the first image and the first feature of the third region of the face of the first image All of the similarities between features may be less than the specified similarity. In this case, the basic face manager 323 may determine one or more images including a predefined face (eg, a mannequin model face, a celebrity face, and a normal human face) (eg, a predefined face). One or more images including the face may be determined as one or more candidate groups). The basic face management unit 323 may display one or more images including a predefined face through the display 210 . In one embodiment, while displaying one or more images including the predefined face through the display 210, the basic face management unit 323 selects a first, based on a user input, from among the one or more displayed images. You can select an image for face inpainting of an image.

In one embodiment, the basic face management unit 323 may acquire and manage facial features of each of one or more images including a predefined face.

In one embodiment, the facial feature combiner 324 combines the representative facial features of the group selected by the input image group selector 322 (hereinafter referred to as 'selected group') and the facial features of the first image, Facial features for obtaining a face region in which the second region is reconstructed may be obtained.

In an exemplary embodiment, the facial feature combiner 324 applies a first feature of a third region of the first image, a second feature of the first image, and a second region of the first image among representative facial features of the selected group. By combining the corresponding first features, a second region of the first image may constitute facial features to obtain a reconstructed first region. For example, the facial feature combiner 324 may use the first feature of the third region that is not covered in the first image (eg, when the nose and mouth are covered in the first image, the facial feature combiner 324 is not covered in the first image). shape of ears and eyes), second features of the first image (eg, direction of ears and eyes, facial expressions, and ambient lighting) and representative facial features of the selected group corresponding to the second region of the first image. By combining the first features (eg, the shape of the nose and the mouth), the second region of the first image may constitute a face feature for obtaining a reconstructed face region.

In an embodiment, the face region acquisition unit 325 may use an artificial neural network (eg, GAN) based on facial features for acquiring a face region in which the second region is restored, acquired by the facial feature combiner 324. ), a face region in which the second region is restored may be obtained. For example, the face region acquiring unit 325 obtains a first feature of a third area that is not covered in the first image, a second feature of the first image, and a representative facial feature of the selected group, using an artificial neural network. A first feature corresponding to the second area of 1 image may be input. The face area acquisition unit 325 determines that the third area of the output image of the artificial neural network (eg, the area corresponding to the third area of the first image within the output image of the artificial neural network) is the third area of the third area of the first image. The first feature and the second feature of the first image are applied, and the second region of the output image of the artificial neural network (eg, the region corresponding to the second region of the first image in the output image of the artificial neural network) is the selected group. An output image including a face region, which is a result of applying a first feature corresponding to a second region of the first image and a second feature of the first image, among representative facial features of , may be obtained.

In an embodiment, the synthesis unit 326 synthesizes the first image and the face area (eg, the output image of the artificial neural network) through the face area acquisition unit 325, so that the second area includes the restored face area. A resulting image (hereinafter, referred to as a 'second image') may be obtained. For example, the synthesis unit 326 extracts (eg, crops) a face region from the image acquired through the face region acquisition unit 325, and extracts the remaining region from the first image except for the face region. After extracting , a second image may be obtained by combining (eg, stitching) the extracted face region and the remaining region.

In one embodiment, the above examples illustrate an operation of obtaining a second image based on the representative facial features of the group selected by the new image group selection unit 314 and the facial features of the first image. Not limited. For example, the facial feature combiner 324, the face region obtainer 325, and/or the synthesizer 326 are managed by the basic face manager 323 using the same or similar method as the above examples. The second image may be obtained based on the facial features of the first image and features of predefined faces of each of the one or more images.

In one embodiment, the synthesis unit 326 may obtain (eg, generate) a flag related to the second image. For example, the synthesis unit 326 may use a flag indicating that the second image is a result image of face inpainting and/or a group selected while face inpainting related to the second image is being performed (eg, determined in operation 405 to be described later). A flag representing the first group) may be obtained.

In one embodiment, the restoration unit 320 (eg, the synthesis unit 326), when the second image is obtained, may transfer the obtained second image and a flag related to the second image to the group management unit 310. there is.

The electronic device 101 according to various embodiments includes a memory 230 storing a plurality of groups corresponding to each of a plurality of representative facial features, and at least one processor electrically connected to the memory 230 (eg, : a processor 240), wherein the at least one processor obtains a first image, and in the first image, a first area representing a face, a feature of the face, and a first area in the first area A second region requiring restoration is detected, a first group is selected from among the plurality of groups based on the facial features and the plurality of representative facial features, and the representative facial features of the first group and the face are selected. It may be configured to obtain a second image by acquiring a face region in which the second region is reconstructed based on the characteristics of and synthesizing the face region and the first image.

In various embodiments, the facial feature includes a first feature and a second feature, the first feature includes a feature unique to a human face, and the second feature includes an environment and a second feature when the first image is acquired. / or a feature reflecting the state of the face may be included.

In various embodiments, the memory 230 stores a plurality of group information corresponding to each of the plurality of groups, and the plurality of group information includes the plurality of representative facial features and each of the plurality of groups. corresponding representative images and group sizes, wherein the plurality of representative facial features are obtained based on first features and weights of faces of images included in the same group in each of the plurality of groups; , The representative images are images having the highest weight among the weights among the images included in the same group in each of the plurality of groups, and each of the group sizes is For , the ratio of the number of images included in the group or the number of images included in the group to the total number of images stored in the memory 230, and each of the weights, in each of the images, It can be set based on the angle of the face, the resolution of the image, and/or whether the image is a result image of face inpainting.

In various embodiments, the electronic device 101 further includes a display 210, and the at least one processor, based on the face feature, the plurality of representative facial features, and the group sizes, One or more candidate groups are determined from among the plurality of groups, one or more representative images corresponding to the one or more candidate groups are displayed among the representative images through the display 210, and the one or more representative images are displayed. A group corresponding to a representative image selected based on a user input among images may be selected as the first group.

In various embodiments, the at least one processor calculates similarities between the facial feature and each of the plurality of representative facial features, determines whether a similarity greater than or equal to a specified similarity exists among the similarities, and When a similarity equal to or greater than the similarity exists, the one or more candidate groups may be determined from among the plurality of groups based on the face feature, the plurality of representative facial features, and the group sizes.

In various embodiments, the at least one processor displays one or more images including a predefined face through the display 210 when there is no similarity greater than or equal to the specified similarity, and displays the one or more images It may be further configured to select an image selected based on a user input as the first group.

In various embodiments, the at least one processor calculates similarities between the facial feature and each of the plurality of representative facial features, and selects a group corresponding to the representative facial feature having the greatest similarity among the similarities. may be configured to select as a first group.

In various embodiments, the at least one processor may determine the first feature of a third region excluding the second region from the first region and the second feature of the first image, and a representative corresponding to the first group. It may be configured to obtain the facial region in which the second region is reconstructed using facial features.

In various embodiments, the at least one processor detects facial features included in a plurality of images stored in the memory 230, and converts the plurality of images into the plurality of images based on the detected facial features. group into groups, and obtain the plurality of group information corresponding to the plurality of groups.

In various embodiments, the at least one processor may include the second image in the first group after the second image is obtained, and may include the second image in the first group based on a feature of a face region included in the second image. It can be further configured to update 1 group.

4 is a flowchart 400 describing a method of presenting an image, according to various embodiments.

In one embodiment, FIG. 4 may be a flowchart illustrating a method of performing face inpainting on an image by the processor 240 (eg, the restoration unit 320).

Referring to FIG. 4 , in operation 401, in an embodiment, the processor 240 may acquire a first image. For example, the processor 240 may acquire an image acquired through the camera module 220 as a first image. For another example, the processor 240 may acquire an image selected from among images stored in the memory 230 (eg, an image selected by a user from among images related to a gallery application) as the first image. For another example, the processor 240 may acquire an image received from an external electronic device (eg, the server 108 , the electronic device 102 , or the electronic device 104 ) as the first image.

In one embodiment, the first image includes a still image (eg, a still image obtained by a capture input while a preview image is displayed) and/or a dynamic image (eg, video, image). can do.

In operation 403, in one embodiment, the processor 240 determines, within the first image, a first area representing a face (eg, an entire area of the face included in the first image), features of the face, and the first area. A second region (eg, a region hidden in the first image) that needs to be restored may be detected.

In one embodiment, the second region may be a region in which the reliability of a feature detected within the region is less than or equal to a specified reliability level. In an embodiment, the second region may include a region in which a portion of a person's face is covered by an accessory such as a mask or sunglasses and/or a human body such as a person's hand or hair, within the first region. For another example, the second area may include an area in which part of data constituting the first image is lost and/or damaged in the first area. However, examples of the second area are not limited to the above examples.

In an embodiment, the processor 240 may detect facial elements in an image using a generative adversarial network (GAN) model. For example, the processor 240 may detect a face element in the first image using style GAN-based Pixel2Style2Pixel and/or face identity disentanglement.

In one embodiment, the processor 240 classifies and detects facial features into a first facial feature unique to a person's face and a second facial feature reflecting the environment and/or facial state at the time of image acquisition. can do.

In one embodiment, the first feature of the face is the shape of the ears, eyes, nose, and mouth, skin tone, and/or facial hair that is unique to a person's face (eg, that all people have in common). may include the form of However, the first feature of the face is not limited to the above examples.

In one embodiment, the second feature of the face may be the environment at the time of image acquisition (eg, the lighting (or brightness) of the subject's surroundings) and/or the state of the person (subject) at the time of image acquisition (eg, the person's posture, the person's facial expression), it may include features that can change each time an image is acquired. For example, the second feature of the face may include a relative direction of the face to the electronic device 101 (eg, the camera module 220) when acquiring an image. However, the second feature of the face is not limited to the above examples.

In one embodiment, the processor 240 converts the first feature of the first image into the first feature of the second area and the third area (eg, a face area excluding the second area requiring reconstruction, or a face area). It can be detected by classifying it as the first feature of the unoccluded region within the region).

In an embodiment, the processor 240 may detect facial features based on whether a face exists in the first image and/or a ratio of the second area to the first area. Hereinafter, referring to FIG. 5 , an operation of the processor 240 detecting facial features based on whether or not a face exists in the first image and/or the ratio of the second area to the first area will be described. let it do

5 is a flowchart 500 describing a method of detecting facial elements, in accordance with various embodiments. For example, FIG. 5 may be a flowchart for explaining a method of detecting facial elements in the second facial element detecting unit 321 of the restoring unit 320 .

Referring to FIG. 5 , in operation 501, in an embodiment, the processor 240 may determine whether a first region exists in the first image. For example, when the first area (face area) is not detected in the first image, the processor 240 may determine that the first area does not exist in the first image. When the first area is detected in the first image, the processor 240 may determine that the first area exists in the first image. As another example, the processor 240 may, when the size (eg area) of the first region detected in the first image is smaller than the size (eg area) of the designated region, the first region in the first image It can be determined that this does not exist. The processor 240 may determine that the first region exists in the first image when the size of the first region detected in the first image is greater than or equal to the size of the designated region.

In one embodiment, when it is determined in operation 501 that the first region does not exist in the first image, the processor 240 performs an operation for face inpainting on the first image (eg, operations 405 to 409). may not perform.

In operation 503, in one embodiment, if it is determined in operation 501 that the first area exists in the first image, the processor 240 determines whether the ratio of the second area to the first area is greater than or equal to the specified ratio. You can check.

In an embodiment, when it is determined in operation 503 that the ratio of the second area to the first area is less than a specified ratio, the processor 240 performs an operation for face inpainting on the first image (eg, operations 405 to 405). Operation 409) may not be performed.

In operation 505, in one embodiment, the processor 240 may detect facial features in the first image when it is determined in operation 503 that the ratio of the second area to the first area is greater than or equal to the specified ratio. .

However, operations 501 to 505 are examples of operations for detecting facial elements within the first image, and operations for detecting facial elements within the first image are not limited thereto. For example, the processor 240 extracts a face from the first image regardless of the ratio of the second area to the first area (eg, even when the ratio of the second area to the first area is less than a specified ratio). features can be detected.

In operation 405, in one embodiment, the processor 240 selects a first group from among the plurality of groups based on the facial feature and the plurality of representative facial features.

In an embodiment, the processor 240 may select a first group for reconstructing a face region (eg, a second region) included in the first image, based on group information about a plurality of groups.

In an embodiment, the group information for each of the plurality of groups may include a group feature (representative facial feature) of each of the plurality of groups, a representative image of the group, and/or a size of the group.

In one embodiment, the group feature may be calculated using a weighted average of first face features of images included in the same group, as in [Equation 1] described above. In one embodiment, for one or more images belonging to the same group, weights are set (e.g., based on the angle of a face in the image, the resolution of the image, and/or whether the image is a result image of face inpainting). can be granted).

In one embodiment, among images included in the same group, an image having the highest weight (eg, an image with the highest weight set) is determined (eg, set) as the representative image of the group. can

In one embodiment, the size of the group may be the number of images included in each of a plurality of groups (eg, the number of images included in the corresponding group), or the total number of images stored in the memory 230 (or a plurality of groups). A ratio of the number of images included in the corresponding group to the total number of images included in the corresponding group may be determined (eg, set) as the size of the corresponding group.

Hereinafter, with reference to FIGS. 6 to 8 , methods for the processor 240 to select a first group (or a predefined face) from among a plurality of groups will be described.

6 is a flowchart 600 describing a method of selecting a group from among a plurality of groups, according to various embodiments.

Referring to FIG. 6 , in operation 601, in an embodiment, the processor 240 may obtain information about a plurality of groups (eg, group information about each of a plurality of groups). Since the operation of obtaining information on a plurality of groups by the processor 240 has been described above, a detailed description thereof will be omitted.

In operation 603, the processor 240 may determine one or more candidate groups from among the plurality of groups by comparing the facial features included in the first image and the plurality of representative facial features.

In one embodiment, the processor 240 determines the similarities between the first feature of the face of the first image and group features (eg, representative facial features) corresponding to the plurality of groups, and/or the plurality of groups. Based on the group sizes of s, one or more candidate groups may be determined.

In an embodiment, the processor 240 selects a third region of the face of the first image (eg, a region excluding the second region that needs to be restored within the face region), among the representative facial features corresponding to the plurality of groups. , or a non-occluded area within the facial area) corresponding to the first feature (eg, a feature of a facial part corresponding to the non-occluded area of the first image in the representative facial features) may be identified. For example, the processor 240 may determine if the third region of the face of the first image includes the shape of ears, eyes, and nose as representative facial features (eg, the mouth in the first image is masked by a mask). occluded), from representative facial features including the shape of the ears, eyes, nose, and/or mouth, respectively, excluding the shape of the mouth (e.g., from each of the representative facial features, occluded by a mask in the first image). Except for the shape of the mouth corresponding to the mouth), the shape of the ears, eyes, and nose can be confirmed.

In an embodiment, the processor 240 may select features corresponding to the first feature of the third region of the face of the first image among the representative facial features respectively corresponding to the plurality of groups, and the face of the first image. It is possible to calculate similarities between the first features of the third area of .

In an embodiment, the processor 240 may determine one or more candidate groups based on the calculated similarities and/or group sizes of the plurality of groups. For example, the processor 240 may determine one or more candidate groups whose calculated similarity is greater than or equal to a specified similarity and whose group size is greater than or equal to a specified size.

In an embodiment, the processor 240 may perform a face inpainting operation based on an image including a face similar to that of the first image by determining one or more candidate groups based on the calculated similarities. there is. In an embodiment, the processor 240 may determine one or more candidate groups based on the group sizes of the plurality of groups, thereby performing a face inpainting operation based on an image of a group whose photographing frequency was higher than that of other groups. there is.

However, a method of determining one or more groups is not limited to the above-described example. For example, the processor 240 may determine one or more candidate groups having a high similarity among the similarities without considering the size of the group.

In operation 605 , in one embodiment, the processor 240 may display representative images of one or more candidate groups through the display 210 . For example, the processor 240 may display representative images respectively corresponding to the determined one or more candidate groups through the display 210 .

In operation 607 , in one embodiment, a first group may be selected from one or more candidate groups based on a user input. For example, while displaying representative images corresponding to the one or more candidate groups through the display 210, the processor 240 selects a group corresponding to a representative image selected based on a user input as a face of the first image. It can be selected as the first group for inpainting.

7 is a flowchart 700 illustrating a method of selecting a group or selecting an image including a predefined face from among a plurality of groups, according to various embodiments.

Referring to FIG. 7 , in operation 701, in an embodiment, the processor 240 may obtain information about a plurality of groups (eg, group information about each of a plurality of groups). Since operation 701 is at least partially the same as or similar to operation 601, a detailed description thereof will be omitted.

In operation 703, in one embodiment, the processor 240 determines whether a representative facial feature corresponding to a facial feature (eg, a first facial feature) included in the first image exists among a plurality of representative facial features. can decide whether

In an embodiment, the processor 240 may select features corresponding to the first feature of the third region of the face of the first image among the representative facial features respectively corresponding to the plurality of groups, and the face of the first image. It is possible to calculate similarities between the first features of the third area of .

In one embodiment, the processor 240 may determine that a representative facial feature corresponding to a facial feature included in the first image does not exist when all of the similarities are less than a specified similarity. The processor 240 may determine that a representative facial feature corresponding to a facial feature included in the first image exists when a similarity level greater than or equal to a designated similarity level exists among the similarities.

When it is determined in operation 703 that there is a representative facial feature corresponding to a face included in the first image, in operation 705, in an embodiment, the processor 240 may determine one or more candidate groups among a plurality of groups. there is. For example, the processor 240 may determine similarities between a first feature of a face of a first image and group features (eg, representative facial features) corresponding to a plurality of groups, and/or values of a plurality of groups. Based on the group sizes, one or more candidate groups may be determined.

In operation 707 , in one embodiment, the processor 240 may display representative images of one or more candidate groups through the display 210 . For example, the processor 240 may display representative images respectively corresponding to the determined one or more candidate groups through the display 210 .

In operation 709 , in one embodiment, a first group may be selected from one or more candidate groups based on a user input. For example, while displaying representative images corresponding to the one or more candidate groups through the display 210, the processor 240 selects a group corresponding to a representative image selected based on a user input as a face of the first image. It can be selected as the first group for inpainting.

If it is determined in operation 703 that there is no representative facial feature corresponding to the face included in the first image, in operation 711, in one embodiment, the processor 240 selects a predefined face (eg, a mannequin). One or more images including a face of a model, a face of an entertainer, and a general face of a person may be displayed through the display 210 .

In one embodiment, the processor 240 determines that there is no representative facial feature corresponding to the face included in the first image when it is determined that there is no representative facial feature corresponding to the face included in the first image. Indicated information (eg, a message) may be displayed through the display 210 .

In operation 713, in one embodiment, the processor 240 selects an image (eg, one image to be used for face inpainting of the first image) from among one or more images including a predefined face, based on the user input. image) can be selected. For example, the processor 240 may select, as a first group, a selected image from among one or more images including a predefined face, based on a user input.

8 is a flowchart 800 describing a method of selecting a group from among a plurality of groups, according to various embodiments.

Referring to FIG. 8 , in operation 801, in an embodiment, the processor 240 may obtain information about a plurality of groups (eg, group information about each of a plurality of groups). Since operation 801 is at least partially the same as or similar to operation 601, a detailed description thereof will be omitted.

At operation 803, in one embodiment, the processor 240 may select a first group from among the plurality of groups by comparing the facial features and the plurality of representative facial features.

In an embodiment, the processor 240 may select features corresponding to the first feature of the third region of the face of the first image among the representative facial features respectively corresponding to the plurality of groups, and the face of the first image. It is possible to calculate similarities between the first features of the third area of . The processor 240 selects a group having the greatest similarity among the similarities (e.g., a feature corresponding to the first feature of the third area of the face of the first image among the representative facial features of the group and the face of the first image). A group having the greatest similarity between the first features of the third region) may be selected as the first group for face inpainting of the first image.

In operation 407, according to an embodiment, the processor 240 may obtain a facial region in which the second region of the first image is reconstructed, based on the representative facial features and facial features of the first group.

In operation 409, in one embodiment, the processor 240 generates a second image (a result in which the second area includes the reconstructed face area) by synthesizing the first image and the second area of the reconstructed face area in operation 407. image) can be obtained.

Hereinafter, operations 407 and 409 will be described in detail with reference to FIG. 9 .

9 is a flowchart 900 describing a method of obtaining a second image, according to various embodiments.

Referring to FIG. 9 , in operation 901, in one embodiment, the processor 240 determines features of a third region of the first image and representative facial features of a first group (the first group selected through operation 405). can For example, the processor 240 may determine the first and second features of a third region of the first image (eg, a third region that is not covered in the first image) and a representative face corresponding to the first group. A feature (eg, a feature corresponding to the second area of the first image among the representative facial features of the first group) may be identified.

In operation 903, in one embodiment, the processor 240 obtains a face region in which the second region of the first image is reconstructed using the features of the third region of the first image and the representative facial features of the first group. can do.

In an embodiment, the processor 240 combines the representative facial features of the first group and the facial features of the first image to obtain facial features to obtain a facial region from which a second region of the first image is reconstructed. can do.

In an embodiment, the processor 240 may correspond to a first feature of a third region of the first image, a second feature of the first image, and a second region of the first image among the selected representative facial features of the first group. By combining the first features of the second region, it is possible to configure facial features for obtaining a face region in which the second region is reconstructed. For example, the processor 240 determines the first feature of the third region that is not occluded in the first image (eg, if the nose and mouth are occluded in the first image, the ears and shape of the eyes), second features of the first image (eg, direction of ears and eyes, facial expression, and ambient lighting) and representative facial features of the first group corresponding to the second area of the first image. By combining features (eg, nose and mouth shapes), facial features may be configured to obtain a face region in which the second region is reconstructed.

In an embodiment, the processor 240 determines the facial region from which the second region is reconstructed, using an artificial neural network (eg, GAN), based on facial features for acquiring the face region from which the second region is reconstructed. can be obtained For example, the processor 240, by means of an artificial neural network, selects a first feature of a third area that is not covered in a first image, a second feature of the first image, and a first image among representative facial features of the first group. A first feature corresponding to the second area of can be input. The processor 240 determines that the third area of the output image of the artificial neural network (eg, the area corresponding to the third area of the first image in the output image of the artificial neural network) is the first feature of the third area of the first image and It is the result of applying the second feature of the first image, and the second area of the output image of the artificial neural network (eg, the area corresponding to the second area of the first image within the output image of the artificial neural network) is the representative face of the first group. Among the features, a feature corresponding to the second region of the first image and an output image including the face region as a result of applying the second feature of the first image may be obtained.

In operation 905, in one embodiment, the processor 240 obtains a second image including the reconstructed face region by synthesizing the face region (eg, an output image of the artificial neural network) and the first image. can do. For example, the processor 240 extracts (eg, crops) a face region from an output image of the artificial neural network, extracts the remaining region except for the face region from the first image, and then extracts the extracted face. The second image may be obtained by combining (eg, stitching) the area and the remaining area.

In one embodiment, in the above examples, the processor 240 illustrates an operation of acquiring a second image based on the representative facial features of the first group and the facial features of the first image, but is not limited thereto. . For example, the processor 240, using the same or similar method as the foregoing examples, selects through operation 713 a predefined face feature of an image including a predefined face and a face of the first image. Based on the feature, a second image may be obtained.

In one embodiment, the processor 240 may obtain (eg, generate) a flag related to the second image. For example, the processor 240 may set a flag indicating that the second image is a result image of face inpainting and/or a group selected while performing face inpainting related to the second image (eg, the first group determined in operation 405). ) can be obtained.

Although not shown in FIG. 4 , in one embodiment, when the second image is obtained, the processor 240 may store the second image and a flag related to the second image in the memory 230 . In one embodiment, the processor 240 may display the second image through the display 210 when the second image is acquired.

10 is an exemplary diagram 1000 illustrating a method of obtaining a second image, according to various embodiments.

Referring to FIG. 10 , in one embodiment, image 1 1010 may represent an image in which a first region 1011 including a second region requiring restoration is detected.

In one embodiment, processor 240 generates image 2 1020 (e.g., image 1 1010) including a first region 1021 corresponding to first region 1021, detected from image 1 1010. ), an image including the first region 1021 in which the first region 1021 is masked) may be obtained.

In one embodiment, image 3 1030 may represent an image including a face region 1031 in which the second region of image 1 1010 is reconstructed.

In an embodiment, the processor 240 may obtain image 4 1040 including the face region 1031 in which the second region is reconstructed, based on image 2 1020 and image 3 1030. . For example, the processor 240 extracts (eg, crops) the face region 1031 corresponding to the first region 1021 of image 2 1020 from image 3 1030 to obtain a second region. Image 4 1040 including the reconstructed face region 1031 may be obtained.

In one embodiment, the processor 240 may obtain image 5 (result image) 1050 by compositing image 1 1010 and image 4 1040 . For example, the processor 240 may perform other regions except for the first region 1011 in image 1 (1010) (eg, a region including the torso 1012) and a face region (in image 4 1040). 1031) may be combined (eg, stitched) to obtain image 5 (result image) 1050.

11 is a flowchart 1100 describing a method of managing a plurality of groups, according to various embodiments.

In one embodiment, FIG. 11 may be a flowchart illustrating a method of acquiring a plurality of groups and acquiring group information on the plurality of groups by the processor 240 (eg, the group management unit 310).

Referring to FIG. 11 , in operation 1101, in an embodiment, the processor 240 may detect a face element for each of a plurality of images stored in the memory 230.

In one embodiment, the processor 240 may detect a face element for each of all images stored in the memory 230 (eg, all images related to the gallery application). For example, the processor 240, when the application for face inpainting is installed (or after installing the application for face inpainting, when the application for face inpainting is executed) or periodically, all images stored in the memory 230 For each of them, an operation of detecting a face element may be performed.

In one embodiment, the processor 240, in each of the plurality of images (hereinafter, each of the plurality of images will be referred to as a 'third image'), a first area representing a face (eg, included in the first image) The entire face region), facial features, and a second region requiring restoration within the first region (eg, a region hidden in the first image) may be detected.

In an embodiment, the processor 240 may detect a face element in the third image using a generative adversarial network (GAN) model.

In one embodiment, the processor 240 classifies and detects facial features into a first facial feature unique to a person's face and a second facial feature reflecting the environment and/or facial state at the time of image acquisition. can do. In one embodiment, the processor 240 may detect only the first feature of the face in the third image without detecting the second feature of the face in order to manage the plurality of groups.

In an embodiment, the processor 240 may detect facial features based on whether a face exists in the third image and/or a ratio of the second area to the first area. Hereinafter, with reference to FIG. 12, an operation of the processor 240 to detect facial features based on whether or not a face exists in the first image and/or the ratio of the second area to the first area will be described. let it do

12 is a flowchart 1200 describing a method of detecting facial elements, in accordance with various embodiments. For example, FIG. 12 may be a flowchart for explaining a method of detecting facial elements in the first facial element detector 311 of the group management unit 310 .

Referring to FIG. 12 , in operation 1201, in one embodiment, processor 240 may determine whether a first region exists in a third image (in each of a plurality of images stored in memory 230). there is. For example, when the first area (face area) is not detected in the third image, the processor 240 may determine that the first area does not exist in the third image. When the first area is detected in the third image, the processor 240 may determine that the first area exists in the third image. As another example, the processor 240 may, when the size (eg area) of the first area detected in the third image is smaller than the size (eg area) of the designated area, the first area in the third image. It can be determined that this does not exist. The processor 240 may determine that the first region exists in the third image when the size of the first region detected in the third image is greater than or equal to the size of the designated region.

In an embodiment, when it is determined in operation 1201 that the first region does not exist in the third image, the processor 240 may not perform an operation of detecting facial features of the third image.

In operation 1203, in one embodiment, if it is determined in operation 1201 that the first area exists in the third image, the processor 240 determines whether the ratio of the second area to the first area is less than the specified ratio. You can check.

In an embodiment, when it is determined in operation 1203 that the ratio of the second area to the first area is greater than or equal to the specified ratio, the processor 240 may not perform an operation of detecting facial features of the third image. there is.

In operation 1205, in one embodiment, when it is determined in operation 1203 that the ratio of the second area to the first area is less than a specified ratio, the processor 240 may detect facial features in the third image. .

However, operations 1201 to 1205 are examples of operations for detecting facial elements in the third image, and operations for detecting facial elements in the third image are not limited thereto. For example, the processor 240 may select a face in the third image regardless of the ratio of the second area to the first area (eg, even when the ratio of the second area to the first area is less than a specified ratio). features can be detected.

In operation 1103, in one embodiment, the processor 240 selects a face element (eg, a first region (face) from among a plurality of images (eg, all images stored in the memory 230) stored in the memory 230. A plurality of groups may be obtained by grouping the images in which the region)) is detected.

In an embodiment, the processor 240 may group the images into a plurality of groups based on first characteristics of each of the images (eg, images in which facial elements are detected). In one embodiment, the processor 240 may group the images into a plurality of groups using a specified algorithm so that one or more images having a similar first characteristic are included in the same group. For example, the processor 240, by using a designated algorithm (eg, K-means clustering algorithm), selects a plurality of images such that the similarity between the first features of one or more images included in the same group is greater than or equal to the designated similarity. can be grouped into groups of

In operation 1105, in one embodiment, the processor 240 may obtain (eg, calculate) group information of each of a plurality of groups. In an embodiment, the group information may include a group feature (representative facial feature of the group), a representative image of the group, and/or a size of the group.

In an embodiment, the processor 240 may obtain a representative facial feature of a group based on first facial features and weights of one or more images belonging to the same group.

In one embodiment, the processor 240 sets weights (e.g., for one or more images belonging to the same group) based on the angle of a face in the image (or the direction the face is facing) and/or the resolution of the image. : granted) can be given. For example, with respect to one or more images belonging to the same group, the processor 240 may determine the face toward the front of the camera module 220 (eg, the direction in which the face faces and the direction in which the camera module 220 faces). parallel) and/or the higher the resolution of the image, the higher the weight can be set for the image.

In one embodiment, the processor 240, for one or more images belonging to the same group, based on the angle of the face in the image, the resolution of the image, and / or whether the image is an image used for face inpainting , weights can be set (eg, assigned). For example, the processor 240 sets a higher weight for the image used for face inpainting (eg, the representative image of the first group selected in operation 405 of FIG. 4) than other images belonging to the same group ( or add weight values).

In one embodiment, the processor 240 obtains a group feature (eg, calculates )can do.

In one embodiment, the processor 240 determines (eg, sets) an image having the highest weight (eg, an image with the highest weight set) among images included in the same group as a representative image of the group. can

In one embodiment, the processor 240 may determine the number of images included in each of a plurality of groups (eg, the number of images included in the corresponding group), or the total number of images stored in the memory 230 (or a plurality of images). A ratio of the number of images included in the corresponding group to the total number of images included in the groups may be determined (eg, set) as the size of the corresponding group.

In one embodiment, the processor 240 may perform operations 1101 to 1105 in the background. For example, the processor 240 performs operations including an operation of detecting a face element for each of all images stored in the memory 230 when an application for face inpainting is installed or periodically (eg, an operation Operations 1101 to 1105) may be performed.

13 is a flowchart 1300 describing a method of managing a plurality of groups, according to various embodiments.

In one embodiment, FIG. 13 illustrates an acquired image (hereinafter, an image acquired by the processor 240 (eg, the group manager 310) when the processor 240 (eg, the group manager 310) acquires the image. Referred to as a 'fourth image') in a previously obtained plurality of groups or a new group, it may be a flowchart explaining a method of managing a plurality of groups.

Referring to FIG. 13 , in operation 1301, in an embodiment, the processor 240 may obtain a fourth image. In one embodiment, the fourth image may be an image newly added to the gallery application (eg, an image managed by the gallery application). For example, the fourth image is an image acquired through the camera module 220 or an image acquired from an external electronic device (eg, obtained through the camera module 220 or acquired from an external electronic device without face inpainting). It may be an image stored in the memory 230 immediately). For another example, the fourth image may include an image on which face inpainting has been performed (a second image as a result image of face inpainting).

In operation 1303, in one embodiment, the processor 240 may determine whether the fourth image corresponds to the second image. For example, the processor 240 determines whether the fourth image is an image on which face inpainting has been performed, or is obtained through the camera module 220 or obtained from an external electronic device and then directly stored in the memory 230 without face inpainting. You can check whether the image is saved.

In an embodiment, the processor 240 may determine whether the fourth image corresponds to the second image based on a flag related to the fourth image. For example, if the flag associated with the fourth image indicates that the fourth image is an image on which face inpainting has been performed, the processor 240 may determine that the fourth image corresponds to the second image. For another example, if the flag related to the fourth image indicates that the fourth image is an image on which face inpainting has not been performed or the flag related to the fourth image does not exist, the fourth image is displayed. After being acquired through the camera module 220 or obtained from an external electronic device, it can be confirmed that the image corresponds to the image stored in the memory 230 without face inpainting.

If the fourth image corresponds to the second image in operation 1303, in operation 1305, the processor 240 may update the determined group to include the fourth image.

In one embodiment, the processor 240, when the fourth image corresponds to the second image, based on the flag associated with the fourth image, the selected group (eg: The first group determined in operation 405) may be determined as a group to include the fourth image. For example, if the fourth image corresponds to the second image, the processor 240 may perform face inpainting related to the fourth image based on a flag indicating a group selected to obtain the fourth image. , while face inpainting of the fourth image is being performed, a group selected to acquire the fourth image may be determined as a group to include the fourth image.

In one embodiment, the processor 240 may include the fourth image as the determined group and update the determined group. For example, the processor 240 may perform a representative facial feature of the determined group, a representative image of the group, and/or a representative facial feature of the determined group, based on the fourth image and one or more images previously included in the determined group. Size can be updated.

If the fourth image does not correspond to the second image in operation 1303, in operation 1307, the processor 240 may detect a face element in the fourth image. For example, the processor 240 may perform an operation at least partially the same as or similar to the operation of detecting the face element of operation 1101 with respect to the fourth image.

In operation 1309, in one embodiment, the processor 240 may determine a group to include the fourth image based on the detected facial elements.

In an embodiment, the processor 240 may select a first facial feature of the fourth image and representative facial features of a plurality of groups (eg, the first facial feature of the input image and representative facial features of a plurality of groups). similarities between each of them) can be compared. The processor 240 may select, among a plurality of groups, a group having a representative facial feature most similar to the first feature of the face of the fourth image among the representative facial features of the plurality of groups (eg, the representative facial features of the plurality of groups). If the similarity between the representative facial feature most similar to the first facial feature of the input image and the first facial feature of the input image is equal to or greater than the specified similarity, a group corresponding to the most similar representative facial feature) is selected as a fourth You can select as a group to include images.

In an embodiment, the processor 240 may, among the representative facial features of a plurality of groups, determine if there is no group having a representative facial feature similar to the first feature of the face of the fourth image (eg, a representative facial feature of the plurality of groups). If the similarity between the representative facial feature most similar to the first facial feature of the fourth image and the first facial feature of the fourth image is less than the specified similarity), a new group is obtained in addition to the previously obtained plurality of groups (eg, create), and include the fourth image in the new group.

In operation 1311, in one embodiment, the processor 240 may update the determined group to include the fourth image. For example, the processor 240 may perform a representative facial feature of the determined group, a representative image of the group, and/or a representative facial feature of the determined group based on the fourth image and one or more images previously included in the determined group. Size can be updated.

In one embodiment, the processor 240, when a new group is acquired, acquires and manages group information (eg, group characteristics of the new group, representative image of the group, and/or size of the group) for the new group. can

14A and 14B are exemplary diagrams illustrating a method of providing an image, according to various embodiments.

Referring to FIGS. 14A and 14B , in one embodiment, when an image is selected from among a plurality of images stored in the memory 230, the processor 240 may display the selected image through the display 210. The processor 240 may display a screen for editing the selected image through the display 210 based on a user input.

In one embodiment, when the selected image includes the face region, the processor 240 displays a screen for editing the selected image including an object (eg, an icon) for editing the face region 210 . can be displayed through For example, in FIG. 14A , if the selected image 1411 includes the face area 1412, the processor 240 includes the object 1414 for editing the face area 1412, the selected image ( A screen 1410 for editing 1411) may be displayed through the display 210 .

In one embodiment, the processor 240 displays a screen for editing the face region through the display 210 when a user input for an object (eg, object 1414) for editing the face region is received. can do.

In one embodiment, the processor 240, when the face region included in the selected image includes a second region that needs to be restored (eg, when the face region included in the selected image has a second region that needs to be restored is detected) ), and a screen including an object (eg, an icon) for face inpainting may be displayed through the display 210 . For example, in FIG. 14A , when the face region 1412 included in the selected image 1411 includes the second region 1413 that needs to be restored, the processor 240 executes a plurality of steps for editing the face region. Along with objects (e.g., object 1425-1 for changing eye size, object 1425-2 for removing blemishes in the face area, object 1425-3 for flowing the background behind the face area), A screen 1420 including an object 1425 for face inpainting may be displayed through the display 210 . However, the plurality of objects 1425-1, 1425-2, and 1425-3 for editing the face area are not limited to the above examples. In one embodiment, the processor 240 displays an object for face inpainting (eg, the object 1425) when the face region included in the selected image does not include the second region requiring restoration. It may not be displayed through (210).

In one embodiment, the processor, when the ratio of the second area (the second area to be restored) to the first area (face area) in the selected image is greater than or equal to a specified ratio, the object (eg, object for face inpainting) (1425)) may be displayed through the display 210. In one embodiment, the processor displays an object for face inpainting (eg, object 1425) through the display 210 when the ratio of the second area to the first area in the selected image is less than a specified ratio. may not

In one embodiment, when a plurality of face regions are detected in the selected image, the processor 240 creates a plurality of objects (eg, a first Circular ring-shaped objects having one color (eg, white) and displayed on each of the face regions may be displayed on the plurality of face regions through the display 210 . When one object is selected from among a plurality of objects representing a plurality of face regions, the processor 240 displays the selected object through the display 210 (eg, the selected object responds to the user input) In order to indicate selection by the selected object, the selected object may be displayed in a second color (eg, yellow) different from the first color (eg, white), and other objects other than the selected object may not be displayed through the display 210. . For example, the processor 240, such as the object 1424 included in the screen 1420, is selected by a user input (or when one face region is detected in the selected image, the object 1424 is automatically selected without a user input). ) objects may be displayed through the display 210 . In one embodiment, the processor 240 may determine a face region corresponding to the selected object (eg, the object 1424) as a face region to be subjected to face inpainting.

In one embodiment, the processor 240 performs face inpainting on a face area (eg, the face area 1412) based on a user input for an object (eg, the object 1425) for face inpainting. can be done

In an embodiment, the processor 240 selects representative images of one or more candidate groups (eg, one or more groups of operation 605) based on a user input for an object (eg, object 1425) for face inpainting. A screen including representative images) may be displayed through the display 210 . For example, the processor 240 displays a screen 1430 including representative images 1434, 1435, and 1436 of one or more candidate groups based on the input of the object 1425 for face inpainting. It can be displayed through (210).

In an embodiment, the processor 240 selects a face region (eg, representative images 1434, 1435, and 1436 of one or more candidate groups) of one or more candidate groups based on a user input. : A representative image to be used to reconstruct the face region 1412) may be selected. For example, the processor 240 may select a representative image 1434 from among representative images 1434, 1435, and 1436 of one or more candidate groups to reconstruct the face region 1412 based on a user input. there is.

In one embodiment, when a representative image (eg, representative image 1434) is selected, the processor 240 obtains a result image of face inpainting through the above-described operations (eg, operations 405 to 409). can do. For example, when the representative image 1434 is selected, the processor 240, through the above-described operations (eg, operations 405 to 409), determines that the second region 1413 that needs to be restored is the restored face region ( A result image 1441 of face inpainting including 1442 may be acquired.

A method of providing an image in an electronic device 101 according to various embodiments includes an operation of acquiring a first image, a first area representing a face in the first image, characteristics of the face, and the first image. Based on an operation of detecting a second region within the region that needs to be restored, the facial feature and a plurality of representative facial features corresponding to each of a plurality of groups stored in the memory 230 of the electronic device 101, an operation of selecting a first group from among the plurality of groups, an operation of obtaining a face region in which the second region is restored based on the representative facial features of the first group and the facial features, and the facial region and An operation of obtaining a second image by synthesizing the first image may be included.

In various embodiments, the facial feature includes a first feature and a second feature, the first feature includes a feature unique to a human face, and the second feature includes an environment and a second feature when the first image is acquired. / or a feature reflecting the state of the face may be included.

In various embodiments, the memory 230 stores a plurality of group information corresponding to each of the plurality of groups, and the plurality of group information includes the plurality of representative facial features and each of the plurality of groups. corresponding representative images and group sizes, wherein the plurality of representative facial features are obtained based on first features and weights of faces of images included in the same group in each of the plurality of groups; , The representative images are images having the highest weight among the weights among the images included in the same group in each of the plurality of groups, and each of the group sizes is For , the ratio of the number of images included in the group or the number of images included in the group to the total number of images stored in the memory 230, and each of the weights, in each of the images, It may be set based on the angle of the face, the resolution of the image, and/or whether the image is the result image of face inpainting.

In various embodiments, the operation of selecting a first group from among the plurality of groups may include one or more of the plurality of groups based on the facial features, the plurality of representative facial features, and the group sizes. Determining candidate groups, displaying one or more representative images corresponding to the one or more candidate groups among the representative images through the display 210 of the electronic device 101, and An operation of selecting a group corresponding to a representative image selected based on a user input from among images as the first group.

In various embodiments, the determining of the one or more candidate groups may include calculating similarities between the facial feature and each of the plurality of representative facial features, and determining whether a similarity greater than or equal to a designated similarity among the similarities exists. determining the one or more candidate groups from among the plurality of groups based on the facial features, the plurality of representative facial features, and the group sizes, when a similarity equal to or greater than the designated similarity exists; Actions may be included.

In various embodiments, the method may include an operation of displaying one or more images including a predefined face through the display 210 when there is no similarity greater than or equal to the specified similarity, and a user input among the one or more images. and selecting the selected image as the first group based on.

In various embodiments, the operation of selecting a first group from among the plurality of groups may include an operation of calculating similarities between the facial feature and each of the plurality of representative facial features; An operation of selecting a group corresponding to a representative facial feature as the first group.

In various embodiments, the obtaining of the face region in which the second region is reconstructed may include the first feature of a third region excluding the second region from the first region and the second feature of the first image; The method may further include obtaining the face region in which the second region is restored using the representative facial feature corresponding to the first group.

In various embodiments, the method may include an operation of detecting facial features included in a plurality of images stored in the memory 230, and combining the plurality of images into the plurality of groups based on the detected facial features. It may further include an operation of grouping into , and an operation of obtaining the plurality of group information corresponding to the plurality of groups.

In various embodiments, the method may include, after the second image is obtained, include the second image in the first group, and select the first group based on a feature of a face region included in the second image. An update operation may be further included.

In addition, the structure of data used in the above-described embodiments of this document can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical reading media (eg, CD-ROM, DVD, etc.).

101: electronic device 108: server

Claims (20)

In electronic devices,
a memory for storing a plurality of groups corresponding to each of a plurality of representative facial features; and
at least one processor electrically connected to the memory;
The at least one processor,
Acquiring a first image;
In the first image, detecting a first area representing a face, features of the face, and a second area that needs to be restored within the first area;
Selecting a first group from among the plurality of groups based on the facial feature and the plurality of representative facial features;
Obtaining a facial region in which the second region is reconstructed based on the representative facial feature of the first group and the facial feature; and
An electronic device configured to acquire a second image by synthesizing the face region and the first image.
According to claim 1,
The facial features include a first feature and a second feature,
The first feature includes a feature unique to a human face,
The second feature includes a feature reflecting an environment and/or a state of the face when the first image is acquired.
According to claim 2,
The memory stores a plurality of group information corresponding to each of the plurality of groups,
The plurality of group information includes the plurality of representative facial features, representative images corresponding to each of the plurality of groups, and group sizes;
The plurality of representative facial features are obtained based on first features and weights of faces of images included in the same group in each of the plurality of groups,
The representative images, in each of the plurality of groups, are images having the highest weight among the weights among the images included in the same group,
Each of the group sizes is a ratio of the number of images included in the group to the number of images included in the group or the total number of images stored in the memory, for each of the plurality of groups, and
Each of the weights is set based on a face angle, a resolution of the image, and/or whether the image is a resultant image of face inpainting in each of the images.
According to claim 3,
Including more displays,
The at least one processor,
determine one or more candidate groups from among the plurality of groups based on the facial feature, the plurality of representative facial features, and the group sizes;
Displaying one or more representative images corresponding to the one or more candidate groups among the representative images through the display; and
An electronic device configured to select a group corresponding to a representative image selected based on a user input from among the one or more representative images as the first group.
According to claim 4,
The at least one processor,
calculating similarities between the facial feature and each of the plurality of representative facial features;
determining whether there is a similarity greater than or equal to a designated similarity among the similarities, and
An electronic device configured to determine the one or more candidate groups from among the plurality of groups based on the face feature, the plurality of representative facial features, and the group sizes, when a similarity equal to or greater than the designated similarity exists.
According to claim 5,
The at least one processor,
If there is no similarity greater than or equal to the specified similarity, displaying one or more images including a predefined face through the display, and
The electronic device further configured to select an image selected based on a user input from among the one or more images as the first group.
According to claim 3,
The at least one processor,
calculating similarities between the facial feature and each of the plurality of representative facial features; and
An electronic device configured to select, as the first group, a group corresponding to a representative facial feature having the greatest similarity among the similarities.
According to claim 2,
The at least one processor,
The second region is restored using the first feature of the third region excluding the second region from the first region, the second feature of the first image, and the representative facial feature corresponding to the first group. An electronic device configured to acquire the face region.
According to claim 3,
The at least one processor,
detecting facial features included in a plurality of images stored in the memory;
grouping the plurality of images into the plurality of groups based on the detected facial features; and
The electronic device further configured to obtain the plurality of group information corresponding to the plurality of groups.
According to claim 9,
The at least one processor,
After the second image is obtained, the electronic device is further configured to include the second image in the first group and update the first group based on a feature of a face region included in the second image.
A method for providing an image in an electronic device,
obtaining a first image;
detecting, within the first image, a first area representing a face, features of the face, and a second area within the first area that needs to be restored;
selecting a first group from among the plurality of groups based on the facial feature and a plurality of representative facial features corresponding to each of the plurality of groups stored in a memory of the electronic device;
obtaining a face region in which the second region is restored, based on the representative facial features of the first group and the facial features; and
and acquiring a second image by synthesizing the face region and the first image.
According to claim 11,
The facial features include a first feature and a second feature,
The first feature includes a feature unique to a human face,
The second feature includes a feature reflecting an environment and/or a state of the face when the first image is acquired.
According to claim 12,
The memory stores a plurality of group information corresponding to each of the plurality of groups,
The plurality of group information includes the plurality of representative facial features, representative images corresponding to each of the plurality of groups, and group sizes;
The plurality of representative facial features are obtained based on first features and weights of faces of images included in the same group in each of the plurality of groups,
The representative images, in each of the plurality of groups, are images having the highest weight among the weights among the images included in the same group,
Each of the group sizes is a ratio of the number of images included in the group to the number of images included in the group or the total number of images stored in the memory, for each of the plurality of groups, and
Each of the weights is set based on, in each of the images, an angle of a face, a resolution of the image, and/or whether the image is a result image of face inpainting.
According to claim 13,
The operation of selecting a first group from among the plurality of groups,
determining one or more candidate groups from among the plurality of groups based on the face feature, the plurality of representative facial features, and the group sizes;
displaying one or more representative images corresponding to the one or more candidate groups among the representative images through a display of the electronic device; and
and selecting, as the first group, a group corresponding to a representative image selected based on a user input from among the one or more representative images.
15. The method of claim 14,
The operation of determining the one or more candidate groups,
calculating similarities between the facial feature and each of the plurality of representative facial features;
determining whether there is a similarity greater than or equal to a designated similarity among the similarities; and
Determining the one or more candidate groups from among the plurality of groups based on the facial features, the plurality of representative facial features, and the group sizes, when a similarity equal to or greater than the specified similarity exists. method.
According to claim 15,
displaying one or more images including a predefined face through the display when there is no similarity greater than or equal to the designated similarity; and
The method further comprising selecting an image selected based on a user input from among the one or more images as the first group.
According to claim 13,
The operation of selecting a first group from among the plurality of groups,
calculating similarities between the facial feature and each of the plurality of representative facial features; and
and selecting, as the first group, a group corresponding to a representative facial feature having the greatest similarity among the similarities.
According to claim 12,
The operation of acquiring the face region in which the second region is restored,
The second region is restored using the first feature of the third region excluding the second region from the first region, the second feature of the first image, and the representative facial feature corresponding to the first group. A method comprising obtaining the face region that has been selected.
According to claim 13,
detecting facial features included in a plurality of images stored in the memory;
grouping the plurality of images into the plurality of groups based on the detected facial features; and
The method further comprising obtaining the plurality of group information corresponding to the plurality of groups.
According to claim 19,
After the second image is acquired, including the second image in the first group and updating the first group based on a feature of a face region included in the second image.

Images