WO2021141216A1

WO2021141216A1 - Method and apparatus for processing image artifact by using electronic device

Info

Publication number: WO2021141216A1
Application number: PCT/KR2020/014433
Authority: WO
Inventors: 최우존; 최재성; 최지환
Original assignee: 삼성전자 주식회사
Priority date: 2020-01-07
Filing date: 2020-10-21
Publication date: 2021-07-15
Also published as: KR102677382B1; KR20210088960A

Abstract

An electronic device according to various embodiments comprises: a camera; a processor operatively connected to the camera; and a memory operatively connected to the processor, wherein the memory may store instructions that, when executed, cause the processor to: control the camera to, in a first state in which a subject is focused, obtain a first image by capturing an image of the subject; control the camera to, in a second state in which the subject is not focused, obtain a second image by capturing an image of the subject; determine whether or not a specified pattern is present in the first image and the second image; and on the basis of the determination that the pattern is present in the first image and that the pattern is not present in the second image, determine that the pattern is an artifact not present in the subject. Various other embodiments are also possible.

Description

Method and apparatus for processing image artifacts in electronic devices

Various embodiments relate to methods and apparatus for processing moire artifacts and/or pixelation artifacts in images.

When an EVD (electronic visual display) is photographed as a display of another electronic device with a camera, the spatial frequency of the optical image of the RGB pixel pattern of EVD formed on the image sensor before the EVD is sampled as digital data by the image sensor (spatial frequency) may be greater than a nyquist frequency determined by a spatial frequency of the Bayer pattern of the image sensor. Accordingly, an aliasing phenomenon may occur. For example, moire artifacts, moire artifacts, or grid patterns and pixelation artifacts that do not actually exist in the subject may be present in an image captured by a camera.

There is an approach that removes artifacts during processing after the image is acquired. For example, a method of processing (eg, removing) artifacts with a filter (eg, low pass filter (LPF) or anti-aliasing filter) may often have a problem in that pixel values having significant frequencies are removed. There is a method of processing (eg, removing) artifacts in software based on machine learning (eg, deep learning). This approach may require large amounts of data (eg training examples) for artifact processing (eg removal). Since the above-mentioned filter or machine learning method cannot automatically detect the presence of artifacts, unnecessary calculations are performed when there are no artifacts, and information included in the original photo can be partially transformed. There are ways to handle (eg remove) artifacts by adjusting the camera angle, position and/or focus. Since this method requires a technical background for the camera, it may be difficult for ordinary people to use it.

According to various embodiments, the electronic device may provide an artifact-processed image to a user by detecting an artifact before capturing and storing the subject, and processing (eg, preventing, suppressing, or removing) the occurrence of the artifact. In addition, various embodiments may provide, for example, a method and an electronic device for processing image artifacts.

An electronic device according to various embodiments may include a camera; a processor operatively coupled to the camera; and a memory operatively coupled to the processor, wherein the memory, when executed, causes the processor to: control the camera to acquire a first image by photographing the subject in a first state in which the subject is in focus; controlling the camera to acquire a second image by photographing the subject in a second state in which the subject is not in focus, and determining whether a specified pattern exists in the first image and the second image, and Based on it is determined that the pattern exists in the first image and it is determined that the pattern does not exist in the second image, instructions for determining the pattern as an artifact that does not exist in the subject may be stored. .

An electronic device according to various embodiments may include a camera; a processor operatively coupled to the camera; and a memory operatively coupled to the processor, wherein the memory, when executed, causes the processor to: control the camera to photograph a subject to obtain a first image having a first frequency band, and photograph the subject to control the camera to obtain a second image having a second frequency band lower than the first frequency band, determine whether a specified pattern exists in the first image and the second image, and Based on it is determined that the pattern exists in the image and it is determined that the pattern does not exist in the second image, instructions for determining the pattern as an artifact that does not exist in the subject may be stored.

According to various embodiments, the electronic device may detect an artifact before or after taking an image (eg, a photo or a video), and process (eg, suppress, prevent, remove) the artifact before or after storing the image. have. Using such an electronic device, a user may check a clear and distinct image in which artifacts are processed.

1 is a block diagram of an electronic device in a network environment, according to various embodiments of the present disclosure;

2 is a block diagram illustrating a camera, in accordance with various embodiments.

3 is a block diagram of an electronic device configured to process an artifact, according to various embodiments.

4A is a diagram illustrating a moire artifact, and FIG. 4B is a diagram illustrating a pixelated artifact.

5A is a diagram illustrating an image having an artifact, and FIG. 5B is a diagram illustrating an image without an artifact.

6 illustrates operations of a processor according to various embodiments.

7 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

8 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

9 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

10 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

11 illustrates operations related to the processing of artifacts, according to various embodiments.

12 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

13 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

14 illustrates operations related to processing of artifacts, according to various embodiments.

15 illustrates operations related to the processing of artifacts, in accordance with various embodiments.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , the electronic device 101 communicates with the electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input device 150 , a sound output device 155 , a display device 160 , an audio module 170 , and a sensor module ( 176 ), interface 177 , haptic module 179 , camera 180 , power management module 188 , battery 189 , communication module 190 , subscriber identification module 196 , or antenna module 197 . may include. In some embodiments, at least one of these components (eg, the display device 160 or the camera 180 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components may be implemented as one integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, the program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be loaded into the volatile memory 132 , process commands or data stored in the volatile memory 132 , and store the resulting data in the non-volatile memory 134 . According to an embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and a secondary processor 123 (eg, a graphic processing unit, an image signal processor) that can operate independently or together with the main processor 121 . , a sensor hub processor, or a communication processor). Additionally or alternatively, the auxiliary processor 123 may be configured to use less power than the main processor 121 or to be specialized for a designated function. The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 may be, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 123 (eg, image signal processor or communication processor) may be implemented as part of another functionally related component (eg, camera 180 or communication module 190 ). .

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input device 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output a sound signal to the outside of the electronic device 101 . The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display device 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to an embodiment, the display device 160 may include a touch circuitry configured to sense a touch or a sensor circuit (eg, a pressure sensor) configured to measure the intensity of a force generated by the touch. have.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input device 150 , or an external electronic device (eg, a sound output device 155 ) connected directly or wirelessly with the electronic device 101 . The sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera 180 may capture still images and moving images. According to one embodiment, camera 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct or IrDA (infrared data association)) or a second network 199 (eg, a cellular network, the Internet, or It may communicate with the external electronic device 104 through a computer network (eg, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified and authenticated.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be chosen. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as a part of the antenna module 197 .

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external

electronic devices

102 and 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more of the external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a function or service automatically or based on a request from a user or other device, the electronic device 101 may perform the function or service by itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. The one or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

2 is a block diagram 200 illustrating a camera 180, in accordance with various embodiments. Referring to FIG. 2 , the camera 180 includes a lens assembly 210 , a flash 220 , an image sensor 230 , an image stabilizer 240 , a memory 250 (eg, a buffer memory), or an image signal processor ( 260) may be included.

The lens assembly 210 may collect light emitted from a subject, which is an image to be captured. The lens assembly 210 may include one or more lenses. According to an embodiment, the camera 180 may include a plurality of lens assemblies 210 . In this case, the camera 180 may form, for example, a dual camera, a 360 degree camera, and/or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. may have lens properties different from those of . The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

The flash 220 may emit light used to enhance light emitted or reflected from the subject. According to an embodiment, the flash 220 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. can

The image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal. According to an embodiment, the image sensor 230 may be selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an infrared (IR) sensor, or an ultraviolet (UV) sensor. It may include one image sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

In response to the movement of the camera 180 or the electronic device 101 including the same, the image stabilizer 240 moves at least one lens or the image sensor 230 included in the lens assembly 210 in a specific direction or image. Operation characteristics of the sensor 230 may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken. According to one embodiment, the image stabilizer 240 is a sensor module (eg, the sensor module 176 of FIG. 1 ) disposed inside or outside the camera 180 (eg, a gyro sensor (not shown) or an acceleration sensor ( motion of the camera 180 or the electronic device 101 may be detected using the According to an embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer.

The image sensor 230 and the image stabilizer 240 may be formed as, for example, one module. For example, in order to quickly process an image acquired through the image sensor 230 , the image sensor 230 may be formed as one module with the image stabilizer 240 , and the time required for signal transmission between each module can reduce

The memory 250 may temporarily store at least a portion of the image acquired through the image sensor 230 for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, Bayer-patterned image or high-resolution image) is stored in the memory 250 and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display device 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 250 may be obtained and processed by, for example, the image signal processor 260 . According to an embodiment, the memory 250 may be configured as at least a part of the memory 130 of FIG. 1 or as a separate memory operated independently of the memory 130 .

The image signal processor 260 may perform image processing on an image acquired through the image sensor 230 or an image stored in the memory 250 . Image processing may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring), sharpening, or softening). Additionally or alternatively, the image signal processor 260 controls at least one of the components included in the camera 180 (eg, the image sensor 230 ) (eg, exposure time control, readout timing control, etc.) ) can be done. The image processed by the image signal processor 260 is stored back in the memory 250 for further processing, or an external component of the camera 180 (eg, the memory 130 , the display device 160 , the electronic device 102 ). ), the electronic device 104 , or the server 108). According to an embodiment, the image signal processor 260 may be configured as at least a part of the processor 120 of FIG. 1 or as a separate processor operated independently from the processor 120 of FIG. 1 . When the image signal processor 260 is configured as a processor separate from the processor 120 of FIG. 1 , at least one image processed by the image signal processor 260 may be processed by the processor 120 of FIG. 1 as it is or additionally. After image processing, it may be displayed through the display device 160 .

According to an embodiment, the electronic device 101 may include a plurality of cameras 180 each having different properties or functions. In this case, for example, the plurality of cameras 180 may include at least one of a wide-angle camera, a telephoto camera, and an IR camera (eg, a time of flight camera, a structured light camera). Similarly, at least one of the plurality of cameras 180 may be a front camera, and at least the other may be a rear camera.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as “first”, “second”, or “first” or “second” may simply be used to distinguish the component from other components in question, and may refer to components in other aspects (e.g., importance or order) is not limited. that one (eg first) component is “coupled” or “connected” to another (eg, second) component with or without the terms “functionally” or “communicatively” When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

As used herein, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). may be implemented as software (eg, the program 140) including For example, the processor (eg, the processor 120 ) of the device (eg, the electronic device 101 ) may call at least one of one or more instructions stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

3 is a block diagram of an electronic device 300 configured to process (eg, prevent and remove) artifacts, according to various embodiments. 4A is a diagram illustrating a moire artifact, and FIG. 4B is a diagram illustrating a pixelated artifact. 5A is a diagram illustrating an image having an artifact, and FIG. 5B is a diagram illustrating an image without an artifact.

Referring to FIG. 3 , an electronic device 300 (eg, the electronic device 101 of FIG. 1 ) includes a camera 310 , a processor 320 operatively connected to the camera 310 , and a processor 320 . ) and a memory 330 operatively connected.

The camera 310 may include a plurality of cameras (eg, the cameras 180 of FIG. 1 ). In an embodiment, the camera 310 may include a first camera 311 and a second camera 312 . The

cameras

311 and 312 may be disposed on the same plane. For example, although not shown, the housing structure of the electronic device 300 includes a first cover that faces in a first direction and forms a first surface of the electronic device 300, faces in a second direction opposite to the first direction, and The electronic device 300 may include a second cover that forms a second surface, and a side bezel structure that forms a side surface surrounding a space between the first surface and the second surface. The display may be visually exposed through at least a portion of the first surface, and accordingly, the first surface may be referred to as a front surface of the electronic device. The second surface may be referred to as a rear surface of the electronic device. The

cameras

311 and 312 are located in the space, and respective lenses may be visually exposed through the second surface (or rear surface). Although not illustrated, the camera 310 may further include one or more cameras in addition to the first camera 311 and the second camera 312 . For example, the camera 310 may further include one or more third cameras disposed on the second surface and one or more fourth cameras disposed on the first surface.

The first camera 311 is in a state in which a subject (eg, an electronic visual display (EVD) as a display of another electronic device) is focused (hereinafter, in focus state) or not in a state (hereinafter, out of focus (hereinafter, out of focus) Alternatively, the subject may be photographed in a defocus) state. For example, the focal length may mean a distance between a lens assembly (eg, the lens assembly 210 of FIG. 2 ) of a camera on which an image of a subject is formed and a focal plane. The lens assembly may include a plurality of lenses. It is possible to adjust the focal length by adjusting the spacing between these lenses. The focal plane may refer to a plane in which light reflected from the subject passes through the lens assembly to form a focal point. The in focus state may indicate that an image sensor of the camera is positioned on a focal plane. Accordingly, in the in focus state, the image sensor of the first camera 311 may acquire a clear image of the subject. The out of focus state may mean a state in which the image sensor is not positioned on the focal plane. Accordingly, in the out-of-focus state, the image sensor of the first camera 311 may acquire an image that is relatively less clear (or optically blurred) compared to the in-focus state.

A pass characteristic of a spatial frequency band may be optically different according to a focal length. The image acquired in the in focus state may have a first frequency band, and the image acquired in the out of focus state may have a second frequency band. The resolution of the image having the first frequency band may be higher than the resolution of the image having the second frequency band. For example, the image sensor of the first camera 311 acquires an image having a high frequency band (eg, 200 lines per millimeter (lpmm)) in an in focus state and a relatively low frequency band (eg, 200 lines per millimeter (lpmm)) in an out of focus state. : 80lpmm) can be acquired. Consequently, focal length (or sharpness) and spatial frequency can be correlated. That is, if one side (focal length or sharpness) changes, the other side (frequency band) can also change accordingly.

Hereinafter, in various embodiments, an image acquired in an in-focus state may be referred to as an in-focus image, and an image acquired in an out-of-focus state may be referred to as an out-of-focus image. According to the above correlation, the in focus state may be referred to as a high pass state and the out of focus state may be referred to as a low pass state.

In an embodiment, the second camera 312 may also acquire an in-focus image or an out-of-focus image by photographing a subject in an in-focus state or an out-of-focus state.

In an embodiment, the processor 320 (eg, the processor 120 of FIG. 1 ) may control the camera 310 to acquire an in-focus image and/or an out-of-focus image for image analysis. For example, the processor 320 may recognize that the electronic device 300 is in a stationary state based on an electrical signal or data value received from a sensor module (eg, a gyro sensor) after a camera application is executed. According to this recognition, the processor 320 may control the camera 310 to acquire an in-focus image and/or an out-of-focus image. As another example, the processor 320 may recognize that the electronic device 300 is in a moving state based on an electrical signal or data value received from a sensor module (eg, a gyro sensor), and accordingly, By controlling it, you can acquire an in-focus image and an out-of-focus image. As another example, the processor 320 may recognize that the subject moves from the image obtained from the camera 310, and accordingly control the camera 310 to obtain an in focus image and/or an out of focus image. can As another example, the processor 320 may compare the image obtained from the camera 310 with a reference image stored in the memory 330 . The processor 320 may control the camera 310 to obtain an in-focus image and/or an out-of-focus image when it is recognized that the EVD is present in the image obtained as a result of the comparison. As another example, when the EVD is present in the acquired image, the processor 320 may measure the distance between the subject EVD and the electronic device 300 using a sensor module (eg, an IR sensor). When the measured distance is within a specified reference value, the processor 320 may control the camera 310 to acquire an in-focus image and/or an out-of-focus image.

In an embodiment, the processor 320 may sequentially acquire an in-focus image and an out-of-focus image from the camera 310 . For example, the processor 320 may control the first camera 311 (or the second camera 312 ) to be in an in focus state and acquire an in focus image from the camera in such a state. The processor 320 may control the first camera 311 (or the second camera 312 ) to be in an out of focus state and acquire an out of focus image from the camera in such a state.

In one embodiment, the processor 320 controls one of the

cameras

311 , 312 to an In focus state and the other to an Out of focus state, and in such a state, an In focus image and an Out of focus image to the camera It can be obtained from the (311, 312).

Most analog objects (with the well-known exception being fabric) can be arranged in atomic or molecular patterns. This pattern cannot be resolved by the camera 310 . EVDs can generally be composed of RGB pixels. Since RGB pixels are arranged in a pattern of a relatively low spatial frequency compared to an analog object, they can be decomposed by the camera 310 . Although these pixels are invisible to the naked eye when viewed at normal viewing distances, users may often experience the need to take EVDs at close distances, a condition where optical resolution is sufficient to resolve individual RGB pixels. At this time, if the spatial frequency of the image formed on the image sensor (eg, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor) is greater than the nyquist frequency of the Bayer pattern of the image sensor, aliasing ( aliasing), a moire artifact as shown in FIG. 4A may be generated in the image. If it is less than the nyquist frequency, pixelation artifacts as shown in FIG. 4B may be generated in the image.

In an embodiment, the processor 320 may be configured to process (eg, suppress, prevent, etc.) artifacts that do not exist in the subject from occurring in the image acquired through the camera 310 . The processor 320 may, for example, be configured to process (eg, remove) artifacts present in the image without compromising the sharpness of the image as much as possible.

According to one embodiment, the processor 320 is configured to extract artifacts (eg, moire artifacts or pixelation artifacts) from the acquired images (in focus images and/or out of focus images). and may be configured to perform an operation of detecting. The processor 320 performs an operation to proactively prevent the occurrence of artifacts before storing the image (eg, an in focus image) and/or an image (eg, an in focus image, an out of focus image, or a combination thereof). image) may be configured to perform an operation of retroactively removing artifacts.

To perform the above operations, the processor 320 includes a domain transformation module 321 , an image analysis module 322 , a focus adjustment module 323 , a sharpness (or resolution) adjustment module 324 , and a notch filter design module 325 , de-artifacting module 326 , or image synthesis module 327 . At least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 may be configured in the electronic device 300 as separate hardware (eg, a coprocessor) different from the processor 320 . have. At least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 may be software (eg, instructions) stored in the memory 330 , and the processor 320 executes the software. can The memory 330 may be the memory 130 of FIG. 1 or a separate memory operated independently of the memory 130 .

In an embodiment, the domain transformation module 321 transforms a representation method of an image (eg, an in-focus image and/or an out-of-focus image) from a spatial domain to a frequency domain (eg, Fourier) can be converted). For example, the spatial domain may be a method in which an image is expressed as a pixel value, and the frequency domain may be a method in which an image is expressed as a spatial frequency (or wave number).

The frequency band of the in focus image may be higher than that of the out of focus image, for example. In other words, in an image obtained from the same subject, artifacts may exist in an image having a high frequency band and artifacts may not exist in an image having a relatively low frequency band. When a ripple or pattern having the same structure as an artifact exists in both images, the corresponding pattern may not be an artifact but may be a pattern actually existing in the subject.

In an embodiment, the image analysis (or image classification) module 322 may determine whether a specified fringe of a structure such as an artifact is present in the in focus image and/or the out of focus image. When the pattern does not exist in the in-focus image, the image analysis module 322 may determine that there is no artifact in the in-focus image (hereinafter, referred to as a first determination). According to the first determination, the processor 320 may obtain an image (or a video) from the camera in the in focus state and store it in the memory 330 . When the pattern is present in both the in-focus image and the out-of-focus image, the image analysis module 322 may determine that the pattern is an actual pattern existing in the subject (hereinafter, a second determination). According to the second determination, the processor 320 may obtain an image (or a video) from the camera in the in focus state and store it in the memory 330 . The image analysis module 322 applies the pattern to the subject when the pattern is present in the in-focus image (eg, the image as shown in FIG. 5A ) while the pattern does not exist in the out-of-focus image (eg, the image as shown in FIG. 5B ). Artifacts that do not actually exist (eg, moiré and/or pixelated artifacts) may be determined (hereinafter, referred to as the third determination). The image analysis module 322 may activate other modules based on the third determination. For example, the processor 320 may perform a post-processing operation (eg, focus adjustment, sharpness adjustment, notch filter design, artifact removal, or image synthesis) according to the third determination. The determination results (or classification results) of the image analysis module 322 may be organized as shown in Table 1 below.

	In focus 이미지in focus image	Out of focus 이미지out of focus images	판단 결과Judgment result
지정된 무늬 존재 여부Whether the specified pattern exists	NoNo	YesYes	(not physically possible)(not physically possible)
	YesYes	YesYes	제 1 판단 (No artifacts)first judgment (No artifacts)
	NoNo	NoNo	제 2 판단(No artifacts)Second judgment (No artifacts)
	YesYes	NoNo	제 3 판단(artifacts)Third judgments

In an embodiment, the image analysis module 322 may determine whether an artifact exists in the in-focus image using the domain transformation module 321 . For example, the image analysis module 322 may receive an in-focus image and an out-of-focus image converted into a frequency domain from the domain conversion module 321 . The image analysis module 322 may determine whether a spatial frequency corresponding to the specified pattern exists in the in-focus image and the out-of-focus image in the frequency domain. When the spatial frequency does not exist in the in-focus image in the frequency domain, the image analysis module 322 may determine that there are no artifacts in the in-focus image (eg, the first determination). When the spatial frequency exists in the in-focus image and the out-of-focus image in the frequency domain, the image analysis module 322 may determine (eg, the second determination) that the pattern is a pattern that is actually present in the subject. have. The image analysis module 322 determines the pattern as an artifact that does not actually exist in the subject when the spatial frequency is present in the in-focus image of the frequency domain while the spatial frequency does not exist in the out-of-focus image (eg, : The third judgment above) can be made.

In an embodiment, the focus adjustment module 323 is configured to tune the focal length of the first camera 311 or the second camera 312 according to the determination (or classification) of the pattern in the in focus image as an artifact. ) can be controlled. The processor 320 may obtain an image from the camera whose focal length is adjusted and store it in the memory 330 .

In one embodiment, the focus adjustment module 323 maintains the camera that has acquired the out of focus image in an out of focus state, based on the third determination, to fine tune the focal length of the camera. can For example, the focus adjustment module 323 may maintain the second camera 312 in an out of focus state, but may finely adjust a focal length of the second camera 312 . Accordingly, the occurrence of artifacts may be optically processed (eg, suppressed) and sharpness of an out of focus image may be improved. The degree of occurrence of artifacts and image sharpness may have a trade-off relationship. For example, as the image becomes sharper, the occurrence probability of the artifact may increase. As the amount of artifacts in the image decreases, the sharpness may decrease. In an embodiment, the focus adjustment module 323 may determine an appropriate focal plane movement amount Δa in consideration of such a trade-off using, for example, Equation 1.

In Equation 1, a may be a distance between the subject and the camera 310 (eg, the first camera 311 or the second camera 312 ). In an embodiment, the processor 320 may measure the distance a using, for example, a sensor module (eg, an IR sensor). Tmx,y may be a distance between adjacent pixels formed in an EVD, which is a subject. For example, the processor 320 may know the pixel density of the EVD (eg, obtain a pixel density value stored in the memory 330 ) and determine Tmx,y from this pixel density. For example, a pixel density of a monitor may be about 100 ppi (pixel per inch), and a pixel density of a display of a smart phone or a wearable device may be about 200 to 400 ppi. bfoc may be a distance between a lens (eg, the lens assembly 210 ) and an image sensor (eg, the image sensor 230 ) in an in focus state. b may be a distance when measuring the distance between the lens and the image sensor (eg, current). D may be the diameter of the lens.

In an embodiment, the focus adjustment module 323 converts a state of a camera (eg, the first camera 311 ) that has acquired an in-focus image from an in-focus state to an out-of-focus state based on the third determination. However, it is possible to fine tune the focal length of the camera step by step. For example, the focus adjustment module 323 acquires an in focus image before starting the fine adjustment of the camera (eg, fine adjustment) until the determination of the image analysis module 322 results in the first determination or the second determination. Fine adjustment of the focal length of the first camera 311) may be continued. Accordingly, the processor 320 may optically process (prevent, suppress, etc.) the occurrence of the artifact without impairing the sharpness of the out of focus image (eg, at the minimum sacrifice). In an embodiment, the movement amount Δa of the focal plane to be finely adjusted may be determined based on Equation 1 above.

In an embodiment, the processor 320 may check setting information related to the electronic device 300 (eg, the camera 310) and the subject, and based on the setting information, the moving amount Δa of the focal plane ) can be determined. The processor 320, for example, based on the movement amount Δa of the focal plane, the camera 310 (eg, the first camera 311, or the second camera 312) has a focal length It can be adjusted (eg fine-tuned)

In an embodiment, the sharpness adjustment module 324 is configured to configure an out of focus image obtained from a camera (eg, the first camera 311 or the second camera 312 ) whose focal length is adjusted by the focus adjustment module 323 . may be processed for sharpness enhancement. The processor 320 may store the image processed using the sharpness adjustment module 324 in the memory 330 .

In an embodiment, a difference between the in-focus image and the out-of-focus image in the frequency domain may include a spatial frequency (hereinafter, referred to as an artifact frequency) corresponding to an artifact. Accordingly, the notch filter design module 325 may be configured to design a notch filter for filtering out pixel values having an artifact frequency based on the third determination. For example, the notch filter design module 325 may receive an in-focus image and an out-of-focus image converted into a frequency domain from the domain transformation module 321 . The notch filter design module 325 may compare the received in-focus image and out-of-focus image, and obtain an artifact frequency from the comparison result. The notch filter design module 325 may generate a notch filter to filter out pixel values having artifact frequencies, for example. The processor 320 may process (eg, suppress) artifacts in the in focus image by using, for example, a notch filter generated using the notch filter design module 325 in the in focus image.

In an embodiment, the artifact removal module 326 may perform processing for removing artifacts while maintaining fine details on the in focus image. In an embodiment, the de-artifacting module 326 may use a machine-learned de-artifacting algorithm using various artifact images to remove artifacts from the in-focus image. In an embodiment, the artifact removal module 326 may remove the artifact from the in focus image by applying the spatial domain in focus image to the notch filter. The processor 320 may store the de-artifacted image in the memory 330 .

In an embodiment, the image synthesizing module 327 may improve the quality of an image to be stored in the memory 330 by synthesizing an in-focus image and an out-of-focus image. The processor 320 may store an image of improved quality in the memory 330 using the image synthesis module 327 .

For example, the image synthesizing module 327 may combine an in-focus image that has been de-artifacted by the de-artifacting module 326 with an out-of-focus image.

As another example, the image synthesis module 327 may acquire an out of focus image from a camera (eg, the first camera 311 or the second camera 312 ) whose focal length is adjusted by the focus adjustment module 323 . The image synthesizing module 327 may acquire the de-artifacting in-focus image from the de-artifacting module 326. The image synthesizing module 327 sends the de-artifacting module 326 to the acquired out-of-focus image. It is possible to synthesize in-focus images with artifacts removed by

As another example, the image synthesis module 327 is obtained from a camera whose focal length is adjusted by the focus adjustment module 323 (eg, the first camera 311 or the second camera 312) and the sharpness adjustment module 324 It is possible to obtain an out-of-focus image with improved sharpness by The image synthesizing module 327 may acquire the de-artifacting in-focus image from the de-artifacting module 326 . The image synthesizing module 327 may synthesize the in-focus image from which the artifacts have been removed by the de-artifacting module 326 with the out-of-focus image that has been processed for sharpness improvement.

In an embodiment, the processor 320 may detect the artifact without a user input from the outside of the electronic device 101 . For example, the processor 320 may perform an operation of detecting an artifact based on the activation of the camera 310 . Also, when acquiring an image of a subject from the camera 310 , the processor 320 may perform an operation of detecting an artifact in the acquired image. The processor 320 may perform an operation of automatically detecting the artifact without, for example, user input, feedback, or interaction.

In an embodiment, the processor 320 may detect the artifact before storing the image obtained from the camera 310 in the memory 330 (eg, the memory 130 of FIG. 1 ). The processor 320 may detect the artifact before, for example, a storage (eg, non-volatile storage) operation of the acquired image. For example, the processor 320 may store an image obtained from the camera 310 in the volatile memory 132 or the memory 250 (eg, a buffer memory), and detect an artifact from the stored image. Thereafter, the processor 320 processes (eg, suppresses) the artifact in the acquired image based on the detected artifact based on at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , 327 . , prevent, remove, etc.). The processor 320 may store, for example, the artifact-processed image in the memory 330 (eg, the non-volatile memory 134 ).

6 illustrates operations 600 of the processor 320 in accordance with various embodiments. In various embodiments, the processor 320 may perform the operations 600 using at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 610 , for example, the processor 320 may determine whether a specified pattern having a structure such as an artifact exists in the in-focus image obtained from the camera 310 .

When it is determined that the pattern specified in the in-focus image does not exist (operation 610, NO), in operation 620 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that the specified pattern exists in the in-focus image (operation 610, yes), in operation 630, the processor 320 determines whether the specified pattern exists in the out-of-focus image obtained from the camera 310, for example. can determine whether

The in-focus image used in operation 610 and the out-of-focus image used in operation 630 may be acquired simultaneously or sequentially. For example, the processor 320 controls the first camera 311 and the second camera 312 to be in focus state and out of focus state, respectively, prior to performing operation 610, and the first camera ( 311) and the second camera 312 may simultaneously acquire an in-focus image and an out-of-focus image. As another example, the processor 320 may control the first camera 311 or the second camera 312 to be in an in-focus state and acquire an in-focus image from the camera in such a state. The processor 320 controls the first camera 311 or the second camera 312 to an out of focus state when it is determined that the pattern specified in the in focus image exists (operation 610, Yes), and You can acquire an out of focus image from the camera.

When it is determined that the specified pattern exists in the out of focus image (operation 630 , Yes), the processor 320 may, for example, perform operation 620 .

If it is determined that the pattern specified in the out-of-focus image does not exist (No in operation 630), in operation 640, the processor 320 adjusts sharpness, for example, with respect to the in-focus image and/or the out-of-focus image. Image processing can be performed to suppress the occurrence of artifacts or to remove artifacts present in the image with minimal sacrifice. Various embodiments of operation 640 will be described below with reference to FIGS. 7 to 14 .

In operation 650 , the processor 320 may store, for example, an image obtained as a result of image processing in the memory 330 .

In some embodiments, when it is determined that the pattern specified in the out of focus image does not exist (operation 630, NO), the processor 320 may perform other operations instead of

operations

640 and 650 . For example, the processor 320 may store the in-focus image in the memory 330 together with information indicating the determination (operation 630, NO) (eg, a tag indicating that an artifact exists in the in-focus image). . Thereafter, when a user calls a specific image through an application (eg, a gallery), the processor 320 may determine whether an artifact exists in the called image through the presence or absence of the information. The processor 320 may determine that an artifact exists in the called image according to the existence of the information. According to this determination, the processor 320 displays, for example, a message for inquiring to the user whether to perform image processing for removing artifacts present in the called image (eg, the display device 160 of FIG. 1 ). ) can be printed. The processor 320 may receive a user input requesting image processing from an input device (eg, the input device 150 of FIG. 1 ) or a display, and in response to receiving the user input, image processing for artifact removal can be performed.

7 depicts operations 700 associated with the processing of artifacts, in accordance with various embodiments. For example, operations 700 may relate to suppressing the occurrence of artifacts and enhancing the sharpness of an image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 7 , the processor 320 may perform operations 700 using, for example, the image analysis module 322 and/or the focus adjustment module 323 . Also, the processor 320 may perform the operations 700 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 710 , the processor 320 may determine, for example, whether a specified pattern having a structure such as an artifact exists in the in focus image.

When it is determined that the pattern specified in the in-focus image does not exist (No in operation 710 ), in operation 720 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 710, Yes), in operation 730, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 730 , Yes), the processor 320 may, for example, perform operation 720 .

If it is determined that the pattern specified in the out-of-focus image does not exist (No in operation 730), in operation 740, the processor 320 sets the camera used to acquire the out-of-focus image to an out-of-focus state, for example. , but you can adjust the focal length of the camera.

In operation 750 , the processor 320 may store, for example, an image obtained from a camera whose focal length is adjusted in the memory 330 .

8 illustrates operations 800 associated with the processing of artifacts, in accordance with various embodiments. For example, the operations 800 may relate to suppressing the occurrence of artifacts and enhancing the sharpness of the image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 8 , the processor 320 may perform operations 800 using, for example, the image analysis module 322 , the focus adjustment module 323 , and/or the sharpness adjustment module 324 . have. Also, the processor 320 may perform the operations 800 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 810 , the processor 320 may determine, for example, whether a specified pattern having a structure such as an artifact exists in the in focus image.

When it is determined that the pattern specified in the in-focus image does not exist (No in operation 810 ), in operation 820 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 810, YES), in operation 830, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 830 , Yes), the processor 320 may, for example, perform operation 820 .

When it is determined that the pattern specified in the out-of-focus image does not exist (operation 830, NO), in operation 840, the processor 320 sets the camera used to acquire the out-of-focus image to the out-of-focus state, for example. , but you can adjust the focal length of the camera. For example, the processor 320 may determine the focal plane movement amount Δa using Equation 1, and adjust the focal length based on the movement amount Δa.

In operation 850, the processor 320 may, for example, perform a process for improving sharpness on an image obtained from a camera whose focal length is adjusted.

In operation 860 , the processor 320 may store, for example, a sharpness enhancement-processed image in the memory 330 .

9 illustrates operations 900 associated with the processing of artifacts, in accordance with various embodiments. For example, the operations 900 may relate to suppressing the occurrence of artifacts and enhancing the sharpness of the image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 9 , the processor 320 may perform operations 900 using, for example, the image analysis module 322 and/or the focus adjustment module 323 . Also, the processor 320 may perform the operations 900 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 910 , the processor 320 may determine, for example, whether a specified pattern having a structure such as an artifact exists in the in-focus image obtained from the first camera 311 in the in-focus state.

If it is determined that the pattern specified in the in-focus image does not exist (operation 910 , NO), in operation 920 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that the pattern specified in the in-focus image exists (operation 910, yes), in operation 930, the processor 320 determines that the pattern specified in the out-of-focus image acquired from the second camera 312 in the out-of-focus state is It can be determined whether it exists or not.

When it is determined that the specified pattern exists in the out of focus image (operation 930 , Yes), the processor 320 may, for example, perform operation 920 .

When it is determined that the pattern designated in the out of focus image does not exist (operation 930, NO), the processor 320, for example, when the pattern does not exist in the image acquired from the first camera 311 up to, the focal length of the first camera 311 may be adjusted in stages. For example, the processor 320 may repeatedly perform the following

operations

940 and 950 until a pattern specified in the image is not detected.

In operation 940 , the processor 320 may adjust, for example, a focal length of the first camera 311 . For example, the processor 320 may determine the focal plane movement amount Δa using Equation 1 and adjust the focal length based on the movement amount Δa.

In operation 950 , the processor 320 may determine, for example, whether a specified pattern exists in the image obtained from the first camera 311 whose focal length is adjusted.

When it is determined that a designated pattern exists in the image obtained from the first camera 311 whose focal length is adjusted (operation 950 , yes), the processor 320 may, for example, perform operation 940 again.

When it is determined that the designated pattern does not exist in the image obtained from the focal length-adjusted first camera 311 (operation 950, NO), in operation 960, the processor 320, for example, the first camera 311 ) may be stored in the memory 330 .

10 depicts operations 1000 associated with the processing of artifacts, in accordance with various embodiments. For example, the operations 1000 may relate to suppressing the occurrence of artifacts and enhancing the sharpness of an image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 10 , the processor 320 may perform operations 1000 using, for example, the domain transform module 321 , the image analysis module 322 , and/or the notch filter design module 325 . can Also, the processor 320 may perform the operations 1000 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 1010 , for example, the processor 320 may determine whether a specified pattern having a structure such as an artifact exists in the in focus image.

When it is determined that the pattern specified in the in-focus image does not exist (operation 1010 , NO), in operation 1020 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 1010, Yes), in operation 1030, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 1030 , yes), the processor 320 may, for example, perform operation 1020 .

When it is determined that the pattern specified in the out of focus image does not exist (operation 1030, NO), in operation 1040, the processor 320 generates a notch filter using, for example, the notch filter design module 325. can

In operation 1050 , the processor 320 may remove artifacts from the in focus image using, for example, a notch filter.

In operation 1060 , the processor 320 may store, for example, the de-artifacted image in the memory 330 .

11 illustrates operations 1100 associated with the processing of artifacts, in accordance with various embodiments. For example, operations 1100 may relate to removing an artifact. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 11 , the processor 320 may perform operations 1100 using, for example, the image analysis module 322 and/or the artifact removal module 326 . Also, the processor 320 may perform the operations 1100 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 1110 , for example, the processor 320 may determine whether a specified pattern having a structure such as an artifact exists in the in focus image.

If it is determined that the pattern specified in the in-focus image does not exist (No in operation 1110 ), in operation 1120 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 1110, YES), in operation 1130, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 1130 , Yes), the processor 320 may, for example, perform operation 1120 .

When it is determined that the specified pattern does not exist in the out-of-focus image (No in operation 1130), in operation 1140, the processor 320 removes artifacts from the in-focus image using, for example, a machine learning-based algorithm. can do.

In operation 1150 , the processor 320 may store, for example, the de-artifacted image in the memory 330 .

In some embodiments, when it is determined that the pattern specified in the out of focus image does not exist (operation 1130 , NO), the processor 320 may perform other operations instead of

operations

1140 and 1150 . For example, the processor 320 may store the in-focus image in the memory 330 together with information indicating the determination (operation 1130, NO) (eg, a tag indicating that an artifact exists in the in-focus image). . Thereafter, when a user calls a specific image through an application (eg, a gallery), the processor 320 may determine whether an artifact exists in the called image through the presence or absence of the information. The processor 320 may determine that an artifact exists in the called image according to the existence of the information. According to this determination, the processor 320 displays, for example, a message for inquiring to the user whether to perform image processing for removing artifacts present in the called image (eg, the display device 160 of FIG. 1 ). ) can be printed. The processor 320 may receive a user input requesting image processing from an input device (eg, the input device 150 of FIG. 1 ) or a display, and in response to receiving the user input, image processing for artifact removal can be performed.

12 illustrates operations 1200 associated with the processing of artifacts, in accordance with various embodiments. For example, operations 1200 may relate to removing artifacts and enhancing clarity of an image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 12 , the processor 320 uses, for example, the image analysis module 322 , the notch filter design module 325 , the de-artifact module 326 , and/or the image synthesis module 327 . Operations 1200 may be performed. Also, the processor 320 may perform the operations 1200 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 1210, for example, the processor 320 may determine whether a specified pattern having a structure such as an artifact exists in the in focus image.

If it is determined that the pattern specified in the in-focus image does not exist (operation 1210 , NO), in operation 1220 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 1210, YES), in operation 1230, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 1230 , Yes), the processor 320 may, for example, perform operation 1220 .

When it is determined that the specified pattern does not exist in the out-of-focus image (operation 1230, NO), in operation 1240, the processor 320 performs an in-focus image using, for example, a machine learning-based algorithm or a notch filter. Artifacts can be removed.

In operation 1250 , for example, the processor 320 may synthesize an in-focus image from which artifacts are removed with an out-of-focus image.

In operation 1260 , the processor 320 may store, for example, the composite image in the memory 330 .

13 illustrates operations 1300 related to the processing of artifacts, according to various embodiments. For example, operations 1300 may relate to removing artifacts and enhancing clarity of an image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 13 , the processor 320 may include, for example, an image analysis module 322 , a focus adjustment module 323 , a notch filter design module 325 , a de-artifact module 326 , and/or an image synthesis module. Module 327 may be used to perform operations 1300 . Also, the processor 320 may perform the operations 1300 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 1310 , for example, the processor 320 may determine whether a specified pattern having a structure such as an artifact exists in the in focus image.

When it is determined that the pattern specified in the in-focus image does not exist (No in operation 1310 ), in operation 1320 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 1310, Yes), in operation 1330, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 1330 , yes), the processor 320 may, for example, perform operation 1320 .

When it is determined that the specified pattern does not exist in the out-of-focus image (operation 1330, NO), in operation 1340, the processor 320 performs an in-focus image using, for example, a machine learning-based algorithm or a notch filter. You can remove artifacts and adjust the focal length of the camera that acquired the out of focus image.

In operation 1350 , for example, the processor 320 may synthesize an in-focus image from which artifacts are removed with an image obtained from a camera whose focal length is adjusted.

In operation 1360 , the processor 320 may store, for example, the composite image in the memory 330 .

14 illustrates operations 1400 related to the processing of artifacts, according to various embodiments. For example, operations 1400 may relate to removing artifacts and enhancing clarity of an image. As described above with reference to FIG. 6 , an in-focus image and an out-of-focus image may be acquired simultaneously or sequentially using the camera 310 . Referring to FIG. 14 , the processor 320 includes an image analysis module 322 , a focus adjustment module 323 , a sharpness adjustment module 324 , a notch filter design module 325 , an artifact removal module 326 , and/or Operations 1400 may be performed using the image synthesis module 327 . Also, the processor 320 may perform the operations 1400 using, for example, at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 1410, for example, the processor 320 may determine whether a specified pattern having a structure such as an artifact exists in the in focus image.

When it is determined that the pattern specified in the in-focus image does not exist (No in operation 1410 ), in operation 1420 , the processor 320 may store, for example, the in-focus image in the memory 330 .

When it is determined that a specified pattern exists in the in-focus image (operation 1410, YES), in operation 1430, the processor 320 may determine, for example, whether a specified pattern exists in the out-of-focus image.

When it is determined that the specified pattern exists in the out of focus image (operation 1430 , Yes), the processor 320 may, for example, perform operation 1420 .

When it is determined that the specified pattern does not exist in the out-of-focus image (No in operation 1430), in operation 1440, the processor 320 performs an in-focus image using, for example, a machine learning-based algorithm or a notch filter. You can remove artifacts and adjust the focal length of the camera that acquired the out of focus image. The processor 320 may perform a process for improving sharpness on the image obtained from the camera whose focal length is adjusted.

In operation 1450 , for example, the processor 320 may synthesize an in-focus image from which artifacts have been removed with an image obtained by processing an image obtained from a focal length-adjusted camera to improve clarity.

In operation 1460 , the processor 320 may store, for example, the composite image in the memory 330 .

15 illustrates operations 1500 of the processor 320 in accordance with various embodiments. In various embodiments, the processor 320 may perform the operations 1500 using at least one of the

modules

321 , 322 , 323 , 324 , 325 , 326 , and 327 .

In operation 1510, the processor 320 may control the camera 310 (eg, the first camera 311 or the second camera 312) to acquire a first image by photographing the subject while the subject is in focus. have.

In operation 1520 , the processor 320 may control the camera 310 to acquire a second image by photographing the subject in a state in which the subject is not focused. For example, the processor 320 may determine the camera that acquired the first image or another camera as the camera that will acquire the second image. That is, the processor 320 may control one camera (eg, the first camera 311 or the second camera 312 ) to acquire the first image and the second image. Alternatively, the processor 320 may control one of the first camera 311 and the second camera 312 to acquire a first image, and control the other to acquire a second image. Operation 1520 may precede operation 1510 . When a plurality of cameras are used to acquire the first image and the second image, operation 1520 may be performed together with operation 1510 .

In operation 1530, the processor 320 may determine whether a specified pattern exists in the first image and the second image.

Based on it is determined that the pattern exists in the first image and the pattern does not exist in the second image, in operation 1540, the processor 320 may determine the pattern as an artifact that does not exist in the subject.

The camera includes a first camera and a second camera disposed on one surface of the electronic device, wherein the instructions are configured to cause the processor to: control the first camera to photograph the subject in the first state, and In the second state, the second camera may be controlled to photograph the subject.

The instructions are configured to cause the processor to control the first camera or the second camera to adjust a focal length based on the pattern being determined as the artifact, and to store an image obtained from the focal length-adjusted camera. can do.

The instructions may cause the processor to: perform processing for sharpness enhancement on the image obtained from the focal length-adjusted camera.

The instructions cause the processor to: check setting information related to the electronic device and the subject, determine a focal plane movement amount based on the setting information, and adjust the focal length based on the movement amount can

The instructions include, by the processor, a distance between an electronic visual display (EVD), which is the subject, and the camera, a distance between adjacent pixels formed in the EVD, a distance between a lens and an image sensor when a subject is in focus, a distance between the lens and the At least one of a current distance between the image sensors and a diameter of the lens may be included in the setting information.

The instructions determine whether the pattern exists in an image acquired using the first camera after the processor controls the first camera to adjust a focal length based on the pattern being determined as the artifact. The determining operation may be repeatedly performed until it is determined that the pattern does not exist in the image acquired using the first camera.

The instructions are configured such that the processor converts the first image and the second image into a frequency domain based on whether the pattern is determined as the artifact, and compares the first image and the second image in the frequency domain. A frequency corresponding to the artifact may be recognized, a pixel value related to the recognized frequency may be removed from the first image, and an image from which the pixel value has been removed may be stored.

The instructions may cause the processor to remove the artifact from the first image based on determining that the pattern is the artifact, and to store the de-artifacted image.

The instructions include, by the processor, based on determining that the pattern is the artifact, remove the artifact from the first image, combine the second image with the de-artifact image, and store the synthesized image. can make it

The instructions are configured to cause the processor to control the camera to adjust a focal length based on the pattern being determined as the artifact, to remove the artifact from the first image, and to obtain an image obtained from the focal length-adjusted camera. and the de-artifacted image may be synthesized, and the synthesized image may be stored.

The instructions are configured to cause the processor to control the camera to adjust a focal length based on the pattern being determined as the artifact, perform processing for sharpness enhancement on an image obtained from the focal length-adjusted camera, and The artifact may be removed from the first image, the sharpness enhancement-processed image may be synthesized with the artifact-removed image, and the synthesized image may be stored.

The instructions may cause the processor to: acquire the first image and the second image by adjusting a focal length of the camera.

A method of operating an electronic device according to various embodiments may include a first obtained by photographing the subject in a first state in which a subject outside the electronic device is focused, using a camera operatively connected to the electronic device. determining whether a specified pattern exists in the image; Based on it is determined that the designated pattern exists in the first image, using the camera, in a second state in which the subject is not in focus, the designated pattern in the second image obtained by photographing the subject determining whether a pattern exists; determining the designated pattern as an artifact that does not exist in the subject based on it being determined that the designated pattern does not exist in the second image; removing the artifact from the first image; and storing the artifact-removed image.

The storing may include: synthesizing the second image and the de-artifacted image; and storing the synthesized image.

The storing may include controlling the camera to adjust a focal length; synthesizing the image obtained from the focal length-adjusted camera and the de-artifacting image; and storing the synthesized image.

The storing may include controlling the camera to adjust a focal length; performing processing for improving sharpness on the image obtained from the focal length-adjusted camera; synthesizing the sharpness-enhancing-processed image and the de-artifacting image; and storing the synthesized image.

The removing of the artifact may include: converting the first image and the second image into a frequency domain; recognizing a frequency corresponding to the artifact by comparing the first image and the second image in the frequency domain; and removing a pixel value related to the recognized frequency from the first image.

The embodiments of the present invention disclosed in the present specification and drawings are merely provided for specific examples in order to easily explain the technical contents according to the embodiments of the present invention and help the understanding of the embodiments of the present invention, and limit the scope of the embodiments of the present invention It's not what you want to do. Therefore, in the scope of various embodiments of the present invention, in addition to the embodiments disclosed herein, all changes or modifications derived from the technical ideas of various embodiments of the present invention should be interpreted as being included in the scope of various embodiments of the present invention. .

Claims

In an electronic device,

camera;

a processor operatively coupled to the camera; and

a memory operatively coupled to the processor, wherein the memory, when executed, causes the processor to:

controlling the camera to acquire a first image by photographing the subject in a first state in which the subject is in focus,

controlling the camera to acquire a second image by photographing the subject in a second state in which the subject is not in focus,

determining whether a specified pattern exists in the first image and the second image, and

an electronic device storing instructions for determining that the pattern is an artifact that does not exist in the subject, based on determining that the pattern exists in the first image and that the pattern does not exist in the second image Device.
According to claim 1, wherein the camera comprises a first camera and a second camera disposed on one surface of the electronic device,

The instructions allow the processor to:

controlling the first camera to photograph the subject in the first state,

an electronic device to control the second camera to photograph the subject in the second state.
The method of claim 2 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

controlling the first camera or the second camera to adjust a focal length, and

An electronic device configured to store an image obtained from the focal length-adjusted camera.
4. The method of claim 3, wherein the instructions enable the processor to:

An electronic device for performing a process for improving sharpness on an image obtained from the focal length-adjusted camera.
4. The method of claim 3, wherein the instructions enable the processor to:

check setting information related to the electronic device and the subject;

based on the setting information, determine a focal plane shift amount, and

an electronic device to adjust the focal length based on the movement amount.
6. The method of claim 5, wherein the instructions cause the processor to:

The distance between the electronic visual display (EVD) and the camera as the subject, the distance between adjacent pixels formed on the EVD, the distance between the lens and the image sensor when the subject is in focus, the current distance between the lens and the image sensor, and at least one of the diameters of the lenses included in the setting information.
The method of claim 2 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

After controlling the first camera to adjust the focal length, the operation of determining whether the pattern exists in the image acquired using the first camera is performed in the image acquired using the first camera An electronic device to repeatedly perform until it is determined that there is no .
The method of claim 1 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

converting the first image and the second image into a frequency domain;

Recognizing a frequency corresponding to the artifact by comparing the first image and the second image in the frequency domain;

removing a pixel value related to the recognized frequency from the first image, and

an electronic device to store the image from which the pixel value has been removed.
The method of claim 1 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

removing the artifact from the first image, and

An electronic device configured to store the de-artifacted image.
The method of claim 1 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

removing the artifact from the first image;

synthesizing the second image and the de-artifacting image; and

An electronic device for storing the synthesized image.
The method of claim 1 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

controlling the camera to adjust the focal length;

removing the artifact from the first image;

Combining the image obtained from the focal length-adjusted camera and the de-artifacting image, and

An electronic device for storing the synthesized image.
The method of claim 1 , wherein the instructions are configured by the processor based on determining that the pattern is the artifact;

controlling the camera to adjust the focal length;

Performing processing for sharpness improvement on the image obtained from the focal length-adjusted camera,

removing the artifact from the first image;

Combining the sharpness-enhancing-processed image and the de-artifacting image, and

An electronic device for storing the synthesized image.
In an electronic device,

camera;

a processor operatively coupled to the camera; and

a memory operatively coupled to the processor, wherein the memory, when executed, causes the processor to:

controlling the camera to obtain a first image having a first frequency band by photographing a subject,

controlling the camera to obtain a second image having a second frequency band lower than the first frequency band by photographing the subject,

determining whether a specified pattern exists in the first image and the second image, and

an electronic device storing instructions for determining that the pattern is an artifact that does not exist in the subject, based on determining that the pattern exists in the first image and that the pattern does not exist in the second image Device.
A method of operating an electronic device, comprising:

determining whether a specified pattern exists in a first image obtained by photographing the subject in a first state in which a subject outside the electronic device is focused, using a camera operatively connected to the electronic device;

Based on it is determined that the designated pattern exists in the first image, using the camera, in a second state in which the subject is not in focus, the designated pattern in the second image obtained by photographing the subject determining whether a pattern exists;

determining the designated pattern as an artifact that does not exist in the subject based on it being determined that the designated pattern does not exist in the second image;

removing the artifact from the first image; and

and storing the de-artifacted image.
The method of claim 14, wherein the storing comprises:

synthesizing the second image and the de-artifacted image; and

and storing the synthesized image.