KR20220014764A - Electronic device and method for generating training data of artificial intelligence learning model thereof - Google Patents

Abstract

In accordance with one embodiment of the present disclosure, an electronic device can include a communication circuit, a memory, and a processor operatively connected with the memory. In accordance with one embodiment, the memory can store instructions which, when executed, cause the processor to: acquire a first image; generate a virtual light source image, insert the virtual light source image into the first image to generate a first output image; insert the virtual light source image into the first image, and then, process the same through a first filter to generate a first input image; and generate the first input image and the first output image as a first training dataset of an artificial intelligence learning model for image compensation. Various embodiments identified through the present specifications can be possible. Therefore, the present invention is capable of maximizing a training effect and embodying a more precise and more accurate artificial intelligence learning model.

Description

Electronic device and method for generating training data of artificial intelligence learning model of an electronic device {Electronic device and method for generating training data of artificial intelligence learning model thereof}

Embodiments disclosed in this document relate to a technique for training an artificial intelligence learning model.

Recently, various electronic devices include a display and a camera. The display may display an image using a plurality of pixels located in the display area. The display may include drivers and signal wires for driving the pixels. When the electronic device captures a bright light source through a camera, pixels around the light source in the image are saturated and a desired image cannot be captured. In addition, when the camera and the display are arranged to affect each other, pixels or wires of the display may diffract light entering the camera, thereby degrading image resolution and causing light splitting or light bleeding around the light source.

Various embodiments of the present disclosure provide an electronic device and method for generating training data of an artificial intelligence learning model for compensating for an image captured using a camera disposed under a display.

Various embodiments of the present disclosure provide an electronic device for generating a training data set of an AI learning model for compensating an image including a light source, and a method for generating training data for an AI learning model of the electronic device.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. have.

An electronic device according to an embodiment disclosed in this document may include a communication circuit, a memory, and a processor operatively connected to the memory. According to an embodiment, the memory, when executed, causes the processor to acquire a first image, generate a virtual light source image, insert the virtual light source image into the first image to generate a first output image, , After inserting the virtual light source image into the first image, a first filter process is performed to generate a first input image, and an artificial intelligence learning model for compensating an image for the first input image and the first output image. Instructions for generating the first training data set may be stored.

In addition, the method for generating training data for an artificial intelligence learning model of an electronic device according to an embodiment disclosed in this document includes the operation of acquiring a first image, the operation of generating a virtual light source image, and the operation of generating the virtual light source image as the first generating a first output image by inserting it into an image, inserting the virtual light source image into the first image and performing a first filter process to generate a first input image, and the first input image and the first image and generating an output image as a first training data set of an artificial intelligence learning model for compensating for an image captured by the camera.

According to the embodiments disclosed in this document, it is possible to train an artificial intelligence learning model for compensating for an image captured by using a camera disposed under the display.

According to the embodiments disclosed in this document, it is possible to generate and provide a virtual training data set for training an artificial intelligence learning model for compensating an image including a light source.

According to the embodiments disclosed in this document, when generating the training data set of the artificial intelligence learning model, by including the virtual light source image, it is possible to solve the light splitting and light spreading phenomenon during image shooting, and improve the image quality An artificial intelligence learning model can be trained to

According to the embodiments disclosed in this document, by generating an infinite training data set based on a smaller number of shooting data and training an artificial intelligence learning model based on this, the training effect is maximized and more sophisticated and accurate artificial intelligence A learning model can be implemented.

In addition, various effects directly or indirectly identified through this document may be provided.

1 illustrates an electronic device in a network environment according to various embodiments of the present disclosure.
2A is a block diagram of an electronic device according to an exemplary embodiment.
2B is a block diagram of an electronic device according to an exemplary embodiment.
3A and 3B are examples of light source images according to various embodiments.
4 is a diagram for explaining an image compensation operation of an electronic device according to an exemplary embodiment.
5 is a diagram for explaining an operation of training an artificial intelligence learning model of an electronic device according to an exemplary embodiment.
6 is a diagram for explaining an operation of generating a virtual light source image of an electronic device according to an exemplary embodiment.
7 is a diagram for describing an operation of generating a training image set of an electronic device according to an exemplary embodiment.
8 is a flowchart of a method for training an artificial intelligence learning model of an electronic device according to an exemplary embodiment.
9 is a flowchart of an operation of generating a virtual light source image of an electronic device according to an exemplary embodiment.
10 is a flowchart of an image compensation operation of an electronic device according to an exemplary embodiment.
11 is a plan view of a display of an electronic device according to an exemplary embodiment.
12 is a cross-sectional view of a first display area of an electronic device according to an exemplary embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 may be included. In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 may use less power than the main processor 121 or may be set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the co-processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used in a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . A sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a LAN (local area network) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 includes various technologies for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be chosen. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2A is a block diagram of an electronic device according to an exemplary embodiment.

According to an embodiment, the electronic device 200a (eg, the electronic device 101 of FIG. 1 ) (eg, a PC or a server) includes a communication circuit 210a (eg, the communication module 190 of FIG. 1 ); It may include a memory 220a (eg, the memory 130 of FIG. 1 ), and a processor 230a (eg, the processor 120 of FIG. 1 ). According to various embodiments, the electronic device 200a is independently or externally configured (eg, (eg, the electronic devices 102 and 104 of FIG. 1 , the server 106 , or the electronic device 200b of FIG. 2B )). It is possible to train an AI learning model by being integrated with the AI learning system, for example, the AI learning system may be configured with the electronic device 200a, an external electronic device, or a combination of the electronic device 200a and an external electronic device.

According to an embodiment, the communication circuit 210a communicates data with an external electronic device (eg, (eg, the electronic devices 102 and 104 of FIG. 1 , the server 106 , or the electronic device 200b of FIG. 2B )). For example, the communication circuit 210a may transmit or receive an image and/or an image to an external electronic device. For example, the communication circuit 210a may be configured by the processor 230a. The training data set (eg, input image and output image) of the generated artificial intelligence learning model may be transmitted to an external electronic device For example, the communication circuit 210a may provide the artificial intelligence learning model to the external electronic device For example, the communication circuit 210a may provide a learning model for training image compensation to an external electronic device.

According to an embodiment, the memory 220a may include instructions executed by the processor 230a. According to an embodiment, the memory 220a may at least temporarily store an image and/or an image received from an external electronic device or a network. According to an embodiment, the memory 220a may store a training data set (eg, an input image and an output image pair) of the AI learning model.

According to an embodiment, the processor 230a may control the overall operation of the electronic device 200a. According to an embodiment, the processor 230a may train an artificial intelligence learning model. According to various embodiments, a system for training an artificial intelligence learning model is implemented internally in the electronic device 200a (eg, the processors 230a and 250), external to the electronic device 200a, or the electronic device It may be implemented as a combination of (200a) and the outside. For example, the AI learning model may be trained by at least one of the electronic device 200a and an external device. For example, an artificial intelligence learning system may be trained using federated learning. According to an embodiment, the AI learning model may include an image compensation model for compensating for an image. According to an embodiment, the AI learning model is at least one of a neural network, a support vector machine (SVN), a hidden Markov model (HMM), and a Baysian. It may be trained based on at least one of machine learning including According to various embodiments, the artificial intelligence learning model may be trained based on at least one of various pattern recognition algorithms that can find patterns from learning data and apply the patterns to real data in addition to the aforementioned machine learning.

According to an embodiment, the processor 230a may generate a training data set for training the AI learning model. For example, the training data set may include a pair of input images and output images.

According to an embodiment, the processor 230a may acquire a first image for generating a training data set. For example, the first image may include an image previously stored in the electronic device 200a (eg, the memory 220a), an image received from an external electronic device, and/or an image acquired through crawling. have. For example, the first image may include arbitrary images for generating a training data set (eg, an input image and an output image).

According to an embodiment, the processor 230a may generate a virtual light source image. For example, the processor 230a may generate a virtual light source image based on at least one parameter of the size, color, intensity, shape, brightness, position, light spread, and degree of light splitting of the virtual light source. have. According to an embodiment, the at least one parameter may be designated according to a user input or may be automatically designated by the processor 230a.

According to an embodiment, the processor 230a generates a raw image of the virtual light source, processes the raw image with a low-pass filter of less than or equal to a specified band, resizes the low-pass filter-processed image, and performs the resizing of the image. A virtual light source image may be generated by performing colorization and Bayerization, and demosaicing the colored and Bayer-ized image. According to an embodiment, the processor 230a may generate a raw image of the virtual light source in a Bayer domain. According to various embodiments, the processor 230a may generate a raw image of the virtual light source in a domain other than the Bayer domain (eg, an RGB domain) in a different manner. According to various embodiments, at least some of the operations of generating the virtual light source image may be omitted.

According to an embodiment, the processor 230a may generate a first output image by inserting the virtual light source image into the first image. For example, the first output image may be output data for training an artificial intelligence learning model. For example, the first output image may be a target image to be acquired through a trained artificial intelligence learning model when the first input image is input.

According to an embodiment, the processor 230a may generate a first output image by second filtering the virtual light source image and inserting the second filtered virtual light source image into the first image. For example, the second filter may be a filter for correcting the virtual light source image similarly to the case of photographing the light source through a real camera.

According to an embodiment, the processor 230a inserts the virtual light source image into the first image and then performs a first filter process to generate the first input image. According to an embodiment, the first filter may be a filter for correcting the first image in which the virtual light source image is inserted to correspond to the actual photographed image. For example, the first filter may be a filter for applying an effect (eg, a diffraction effect) appearing when an image is captured by a real camera to the first image into which the virtual light source image is inserted. For example, the first input image may be input data for training an artificial intelligence learning model.

According to an embodiment, the processor 230a may generate the first input image and the first output image as a training data set of an artificial intelligence learning model for compensating the image. For example, when the first input image is input, the artificial intelligence learning model may be trained to derive a first output image or a result image similar to the first output image.

According to an embodiment, the processor 230a may acquire a second image for generating a training data set. For example, the second image may be the same as or different from the first image. For example, the second image may include an image previously stored in the electronic device 200a (eg, the memory 220a), an image received from an external electronic device, and/or an image acquired through crawling. have. For example, the second image may include arbitrary images for generating a training data set (eg, an input image and an output image).

According to an embodiment, the processor 230a may generate a second image as a second output image, and may generate a second input image by performing a first filter process on the second image. According to an embodiment, the processor 230a may generate the second input image and the second output image as the second training data set of the AI learning model.

According to an embodiment, the first training data set may be a training data set including a virtual light source image, and the second training data set may be a training data set not including a virtual light source image.

According to various embodiments, the processor 230a trains an artificial intelligence learning model by itself using a training data set (eg, a first training data set and/or a second training data set), or the electronic device 200a An artificial intelligence learning model can be trained through an external artificial intelligence learning system. For example, the processor 230a may provide the generated training data set to an external artificial intelligence learning system through the communication circuit 210a. For example, when the electronic devices 200a and 200a and the external electronic device together form an AI learning system, the electronic devices 200a and 200a and the external electronic device may train the AI learning model together. .

According to various embodiments, some components of the electronic device 200a may be omitted or may further include at least some of the components of the electronic device 101 illustrated in FIG. 1 or the electronic device 200b illustrated in FIG. 2B . .

According to various embodiments of the present disclosure, by training an artificial intelligence learning model for image compensation through a training data set (eg, a first input image and a first output image) including a virtual light source, It can compensate for damage or deformation of an image, and can provide an artificial intelligence learning model that can compensate for light saturation, light blurring, or light splitting within an image. According to various embodiments of the present disclosure, in training an artificial intelligence learning model for image compensation, synthetic data (eg, a composite image) is generated instead of actual measurement data (eg, a photographed image) to train the training data By using it, the training effect of the artificial intelligence learning model can be increased because the parameter values required for learning can be set or changed as desired. According to various embodiments of the present disclosure, by generating a training data set (eg, a first input image and a first output image) required for an artificial intelligence learning model from the same source (eg, a first image), an actual photographed image is generated Disparity that may occur when used as training data for an artificial intelligence learning model, noise (noise change) depending on the surrounding environment, motion during actual shooting, and/or limitations of the camera (image sensor) (eg, light saturation value) ), errors and problems can be minimized.

2B is a block diagram of an electronic device according to an exemplary embodiment.

According to an embodiment, the electronic device 200b (eg, the electronic device 101 of FIG. 1 ) (eg, a mobile device) includes a display 210b (eg, the display module 160 of FIG. 1 ) and a camera 220b. ) (eg, camera module 180 of FIG. 1 ), communication circuit 230b (eg, communication module 190 of FIG. 1 ), memory 240b (eg, memory 130 of FIG. 1 ), and a processor 250b (eg, processor 120 of FIG. 1 ).

According to an embodiment, the display 210b may display an image and/or an image. According to an embodiment, the display 210b may include a plurality of pixels. According to an embodiment, the display 210b may include a plurality of regions having different pixel densities. According to an embodiment, the display 210b may include wires for driving respective pixels.

According to an embodiment, the camera 220b may be disposed below at least a partial area of the display 210b. For example, the camera 220b may include an under display camera (UDC). According to an embodiment, the camera 220b may generate an image based on the light incident through the display 210b. For example, when the camera 220b is disposed under the display 210b, the camera 220b (eg, the camera 220b hole) in the front of the electronic device 200b (the surface on which the display 210b is disposed). Without this, the area of the display 210b may be maximized. According to another embodiment, the image display area of the display 210b may include at least two sub-areas. For example, the at least two sub-regions may include a first region having a relatively high pixel density and a second region having a relatively low pixel density. According to an embodiment, at least a portion of the camera 220b may be disposed under the second area.

According to an embodiment, the communication circuit 230b may transmit/receive data to and from an external electronic device (eg, the electronic devices 102 and 104 of FIG. 1 , the server 106 , or the electronic device 200a of FIG. 2A ). have. For example, the communication circuit 230b may transmit or receive an image and/or an image to an external electronic device. For example, the communication circuit 230b may receive an artificial intelligence learning model from an external electronic device.

According to an embodiment, the memory 240b may include instructions executed by the processor 250b. According to an embodiment, the memory 240b may store an image and/or an image captured by the camera 220b or an image and/or an image received from an external electronic device.

According to an embodiment, the processor 250b may control the overall operation of the electronic device 200b.

According to an embodiment, the processor 250b may capture an image through the camera 220b. According to an embodiment, the processor 250b may compensate the captured image based on the AI learning model. For example, the processor 250b receives an artificial intelligence learning model from an external electronic device (eg, the electronic device 200a of FIG. 2A ) through the communication circuit 230b, and captures based on the received artificial intelligence learning model. image can be compensated.

For example, when the camera 220b is disposed under the display 210b, light incident to the camera 220b through the display 210b may cause diffraction due to pixels and wiring of the display 210b. For example, when photographing a light source, light may cause strong diffraction, which may cause light splitting and light spreading around the light source of the image. Alternatively, when the light source is photographed through the camera 220b, pixels in the image are saturated and information loss may occur. According to an embodiment, the processor 250b compensates for an image based on an artificial intelligence learning model trained based on a training data set including an input image and an output image generated using a virtual light source image, and compensates for an image within the captured image. It is possible to compensate for light splitting and light blurring, and to improve image quality.

According to various embodiments, some components of the electronic device 200b may be omitted or may further include at least some of the components of the electronic device 101 illustrated in FIG. 1 . According to various embodiments, the electronic device 200b may include at least a part of the configuration of the electronic device 200a of FIG. 2 or perform at least a part of the operation of the electronic device 200a of FIG. 2 .

3A and 3B are examples of light source images according to various embodiments.

Referring to FIG. 3A , the first image 310 represents an image when an actual light source is photographed using a camera (eg, an external electronic device (eg, the camera 220b of the electronic device 200b of FIG. 2B ). , the second image 320 represents a first input image including a virtual light source image generated by the electronic device (eg, the electronic device 200a of FIG. 2A ) according to an embodiment, and the third image 330 is A first output image generated by an electronic device (eg, the electronic device 200a of FIG. 2A ) according to an embodiment is shown.

Referring to FIG. 3B , the fourth image 340 is an image obtained when an actual light source is photographed using a camera (eg, the camera 220b of an external electronic device (eg, the electronic device 200b of FIG. 2B )). The fifth image 350 represents a first input image including a virtual light source image generated by an electronic device (eg, the electronic device 200a of FIG. 2A ) according to an embodiment, and a sixth image 360 . denotes a first output image generated by the electronic device (eg, the electronic device 200a of FIG. 2A ) according to an embodiment.

According to an embodiment, the electronic device (eg, the electronic device 200a of FIG. 2A ) may include diffraction of light that may occur in a photo taken with a real light source, such as the first image 310 and the fourth image 340 , In order to compensate for light scattering and/or light divergence, a training data set of an artificial intelligence learning model may be generated.

For example, the electronic device generates a virtual light source image, performs a first filter process to apply an effect similar to an image obtained by photographing a real light source to the virtual light source image, thereby generating the second image 320 and the fifth image 350 You can create the same input image. For example, the electronic device may generate a virtual light source image and generate first output images (ie, target images) such as the third image 330 and the sixth image 360 based on the generated virtual light source image.

According to an embodiment, the electronic device may generate and provide a pair of a first input image and a first output image as training data of the AI learning model. According to an embodiment, the external electronic device (eg, the electronic device 200b of FIG. 2B ) is an external electronic device based on an artificial intelligence learning model trained by the electronic device (eg, the electronic device 200a of FIG. 2A ) An image (for example, the first image 310 or the fourth image 340) captured through the camera of For example, the external electronic device compensates the captured image based on the artificial intelligence learning model to compensate for information loss due to light blurring, light splitting, and/or light saturation, such as the target image (eg, the third image). (330) or the sixth image 360) may be acquired.

According to various embodiments, the actual light source images 310 and 340, the first input images 320 and 350, and the first output images 330 and 360 shown in FIGS. 3A and 3B are an example, and the electronic device The type of light source (eg, an image including the light source) that can be compensated is not limited thereto, and various types and/or shapes of light sources may be compensated.

4 is a diagram for explaining an image compensation operation of an electronic device according to an exemplary embodiment.

According to an embodiment, in operation 410 , the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200b of FIG. 2B ) receives a camera (eg, the camera module 180 of FIG. 1 or the electronic device 200b of FIG. 2 ). An image may be acquired through the camera 220b). According to an embodiment, the camera of the electronic device may be disposed under an area of a display (eg, the display module 160 of FIG. 1 or the display 210b of FIG. 2B ). For example, when an image is taken using a camera disposed under the display, light saturation, light bleeding, and/or light splitting may occur in the image due to diffraction of light due to pixels and/or wiring of the display.

According to an embodiment, in operation 420 , the electronic device may pre-process the acquired image. For example, the electronic device may perform preprocessing (eg, resolution, brightness, or size adjustment) on at least a portion of the acquired image based at least on characteristics of the acquired image or characteristics of a display.

According to an embodiment, in operation 430 , the electronic device may compensate an image based on the artificial intelligence learning model. According to an embodiment, the AI learning model may be a learning model for compensating for an image acquired through a camera (eg, an under display camera (UDC)).

According to an embodiment, the AI learning model is a training data set (eg, an input image and an output image pair) generated by an external electronic device (not shown) (eg, the electronic device 200a of FIG. 2A ). can be trained using For example, the training data set may include an input image and an output image including a virtual light source image generated by the electronic device. For example, an electronic device receives a trained artificial intelligence learning model from an external electronic device, and compensates for an image based on the received artificial intelligence learning model, so that loss due to saturation of light contained in the image, light blur, and/or an image in which light splitting is compensated may be generated.

According to an embodiment, in operation 440 , the electronic device may post-process the image. For example, the electronic device may perform post-processing (eg, adjustment of resolution, brightness, or color) on at least a portion of the compensated image based on at least characteristics of the compensated image or display characteristics.

According to an embodiment, in operation 450 , the electronic device may acquire a processed image. For example, the finally acquired image may be an image in which light blurring, light splitting, and/or light saturation caused by a light source included in the image is compensated.

5 is a diagram for explaining an operation of training an artificial intelligence learning model of an electronic device according to an exemplary embodiment.

According to an embodiment, the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200a of FIG. 2A ) generates a training data set 517 for training an artificial intelligence learning model for image compensation. can do. For example, the training data set 517 may include a pair of input images and output images.

According to an embodiment, the electronic device may acquire the first image 501 . For example, the first image 501 may include an image pre-stored in the electronic device, an image received from an external electronic device, and/or an image acquired through crawling. For example, the first image 501 may include an arbitrary image for generating the training data set 517 (eg, an input image and an output image).

According to an embodiment, in operation 503 , the electronic device may generate a virtual light source image. According to an embodiment, the virtual light source image may include a composite image generated by the electronic device, or an image obtained by photographing actual lighting by the electronic device or an external electronic device. For example, the electronic device may generate a virtual light source image based on a virtual light source generation parameter. For example, the virtual light source generation parameter may include at least one of a size, color, intensity, shape, brightness, location, light spread, and degree of light splitting of the virtual light source. According to an embodiment, the virtual light source generation parameter may be designated according to a user input or may be automatically designated by the electronic device. According to an embodiment, the electronic device may generate a plurality of virtual light source images with respect to one image (eg, a first image) by applying a plurality of virtual light source generation parameters.

According to an embodiment, at least a part of the operation of generating the virtual light source image in operation 503 may be performed together as at least a part of the operation of inserting the virtual light source image in operation 507 or 511 . For example, at least a part of the virtual light source image generating operation (eg, disposing the virtual light source at a specific position in the image according to the position parameter) may be performed in operation 503 , operation 507 , or operation 511 .

According to an embodiment, in operation 505 , the electronic device may perform a second filter process on the virtual light source image. For example, the second filter may be a filter for correcting the virtual light source image similar to the case of photographing the light source through the real camera 220 . For example, the second filter may be set based on a light source design parameter. For example, the light source design parameter includes various values for adjusting the boundary of the virtual light source included in the virtual light image, light source shape (pattern) processing, light spreading, and light splitting into a form suitable for training of an artificial intelligence learning model. can do. According to an embodiment, operation 505 may be omitted as an optional operation.

According to an embodiment, in operation 507 , the electronic device may generate the first output image 509 by inserting the virtual light source image into the first image 501 .

According to an embodiment, in operation 511 , the electronic device may insert the virtual light source image into the first image 501 . According to an embodiment, operations 511 and 507 may be the same operation. For example, when operation 505 is omitted, operations 507 and 511 may be performed as a single operation.

According to an embodiment, in operation 513 , the electronic device may perform a first filter processing on the first image 501 in which the virtual light source image is inserted to generate the first input image 515 . According to an embodiment, the first filter may be a filter for correcting the first image 501 in which the virtual light source image is inserted to correspond to the actual photographed image. For example, the first filter may be a filter for applying an effect (eg, a diffraction effect) appearing when an image is captured through the real camera 220 to the first image 501 in which the virtual light source image is inserted. According to an embodiment, after inserting the virtual light source image into the first image 501 , the electronic device uses a point spread function (PSF) (eg, a convolution operation) to insert the first filter can be processed For example, the electronic device may acquire the PSF to process the first filter based on measurement data or simulation.

According to an embodiment, the electronic device generates various virtual light source images by applying a plurality of virtual light source generation parameters, thereby generating an input image (eg, a first image) or an output for one image (eg, a first image). An image (eg, a first output image) may be generated.

According to an embodiment, the electronic device may generate the first input image 515 and the first output image 509 as the first training data set 517 of the AI learning model.

According to an embodiment, the electronic device may acquire the second image 519 . For example, the second image 519 may include an image previously stored in the electronic device, an image received from an external electronic device, and/or an image acquired through crawling. For example, the second image 519 may be the same as or different from the first image 501 .

According to an embodiment, the electronic device may generate the second image 519 as the second output image 521 .

According to an embodiment, in operation 523 , the electronic device may perform a first filter process on the second image 519 to generate a second input image 525 . According to an embodiment, the first filter may be a filter for correcting the second image 519 to correspond to an actual photographed image. For example, the first filter may be a filter for applying an effect (eg, a diffraction effect) that appears when an image is actually captured by a camera to the second image 519 . According to various embodiments, the first filter applied in operation 513 and the first filter applied in operation 523 may be the same filter or different filters.

According to an embodiment, the electronic device may generate the second input image 525 and the second output image 521 as the second training data set 517 of the AI learning model.

According to an embodiment, in operation 527 , the electronic device may provide the generated training data set 517 to the AI learning model. According to various embodiments, the electronic device is configured with a training data set 517 (eg, a first training data set (a first input image and a first output image) and/or a second training data set (a second input image and a second input image) It is possible to train the AI learning model by itself using the output image)) or to train the AI learning model through an AI learning system external to the electronic device. For example, the electronic device may provide the generated training data set 517 to an external artificial intelligence learning system. According to an embodiment, the electronic device may receive a training data set generated by the external electronic device from the external electronic device, and train the AI learning model based on the received training data set. According to an embodiment, the electronic device may train an artificial intelligence learning model in conjunction with an external electronic device. For example, when the electronic device forms an artificial intelligence learning system with an external electronic device, the electronic device and the external electronic device may work together to train an artificial intelligence learning model.

According to an embodiment, the image compensation model 529 may be generated by training the AI learning model based on the training data set 517 generated by the electronic device. For example, the electronic device may compensate for an image captured by the camera based on the image compensation model 529 . For example, the electronic device may compensate for damage or deformation of an image due to diffraction of light in a photographed image, and may compensate for light saturation, light spreading, or light splitting in the photographed image.

6 is a diagram for explaining an operation of generating a virtual light source image of an electronic device according to an exemplary embodiment. According to an embodiment, the operations of FIG. 6 may be an example of generating the virtual light source image 503 of FIG. 5 .

According to an embodiment, in operation 610 , the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200a of FIG. 2A ) may generate a raw image of the virtual light source. For example, the electronic device may generate the raw image 660 including the virtual light source. In the following description, for convenience, one area 670 of the virtual light source in the image will be described.

According to an embodiment, the electronic device may generate a raw image of the virtual light source in a Bayer domain. For example, an electronic device may generate an initial image of a virtual light source in the Bayer domain similar to an image captured using a real camera (eg, complementary metal-oxide-semiconductor (CMOS) / charge coupled device (CCD) sensor). can For example, a boundary of a virtual light source may be formed in the image as 671 .

According to an embodiment, in operation 620 , the electronic device may perform a low pass filter (LPF) process on the raw image of the virtual light source below a specified band. For example, referring to 672 , the electronic device may reflect a blurring effect that may occur depending on a lens and/or an optical structure when an actual camera is photographed in a virtual light source of a raw image through the LPF.

According to an embodiment, in operation 630 , the electronic device may resize the low-pass filter-processed image. For example, the electronic device may resize the virtual light source to correspond to a desired size of the light source. For example, the electronic device may resize the virtual light source to correspond to a specified distance. For example, the electronic device may resize the virtual light source to correspond to an actual light source located at a specific distance. For example, the electronic device may downsample at least a portion of an image of the virtual light source to process an edge of the virtual light source similarly to an actual light source. For example, similar to 673, the electronic device may deform the edge of the virtual light source that is formed in a shape close to a straight line by processing the edge of the virtual light source to be more similar to the shape of the edge of the actually photographed light source.

According to an embodiment, in operation 640 , the electronic device may colorize or bayerize the resized image. For example, the electronic device may convert an image having a grayscale into three R, G, and B channels and set a color. For example, the electronic device may convert the re-imaged image into an image 674 having RGB colors.

According to an embodiment, in operation 650 , the electronic device may demosaicing the colored and Bayer-ized images. For example, the electronic device may perform demosaicing to generate a virtual light source image 675 having R, G, and B 3-channels.

According to an embodiment, the brightness of the virtual light source included in the virtual light source image generated by the electronic device may be set to be greater than a specified brightness reference. For example, when an image is actually captured by a camera, brightness saturation values of pixels may exist due to a performance limitation of the camera. For example, since the electronic device according to an embodiment generates an image including a virtual light source, the light source included in the virtual light source image may have a higher brightness value regardless of camera performance. According to an embodiment, by using the virtual light source image as training data of the artificial intelligence learning model, it may be possible to generate a more sophisticated learning model than when only real images are used as training data.

According to various embodiments, the electronic device may generate virtual lighting for learning in a different manner in other domains as well as in the Bayer domain.

7 is a diagram for explaining an operation of generating a training image set of an electronic device according to an exemplary embodiment.

According to an embodiment, the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200a of FIG. 2A ) uses a training data set (eg, an input image) of an artificial intelligence learning model based on a virtual light source generation algorithm. 715 and three output images 735 or three input images 725 and 745) may be generated.

According to an embodiment, the electronic device generates a virtual light source (711, 721), inserts the virtual light source into an arbitrary image (713, 723), and performs a first filter process on the image into which the virtual light source is inserted to obtain an input image ( 715, 725) can be created. According to an embodiment, the first filter may be a filter for applying a phenomenon due to diffraction of light that may occur when photographing an actual light source to the virtual light source image.

According to an embodiment, the electronic device generates a virtual light source (731, 741), performs a second filter process on the virtual light source (733, 743), and inserts the second filter-processed virtual light source into an arbitrary image to produce an output image (735, 745) can be created.

According to an embodiment, the electronic device generates a pair of an input image and an output image (eg, three input images 715 and 735, or three input images 725 and 745). can be provided as training data of an artificial intelligence learning model for image compensation. For example, the electronic device generates a first training data set (an input image 715 and an output image 735) for a virtual light source of a first type, or second training data for a virtual light source of a second type Three (input image 725 and output image 745) may be generated.

According to an embodiment, since the electronic device may generate a training data set (an input image and an output image) including a virtual light source image, it may generate a training data set for various light sources and various parameters. For example, an electronic device generates an infinite training data set based on a smaller number of shooting data and trains an AI learning model based on it, thereby maximizing the training effect and developing a more sophisticated and accurate AI learning model. can be implemented For example, the electronic device may generate a large amount of training data from a small number of source images by generating a plurality of virtual light source images and/or training data sets based on various parameters. According to an embodiment, by generating an input image and an output image set including a virtual light source based on the same source image, the electronic device generates a parallax ( disparity), noise (noise change) according to the surrounding environment, movement during actual shooting, and/or errors and problems caused by limitations (eg, light saturation value) of the camera 220 (image sensor) may be minimized.

An electronic device according to an embodiment includes a communication circuit; Memory; and a processor operatively coupled to the memory.

According to an embodiment, the memory, when executed, causes the processor to acquire a first image, generate a virtual light source image, insert the virtual light source image into the first image to generate a first output image, , After inserting the virtual light source image into the first image, a first filter process is performed to generate a first input image, and an artificial intelligence learning model for compensating for an image with the first input image and the first output image. Instructions for generating the first training data set may be stored.

According to an embodiment, the instructions include, by the processor, generating a raw image of a virtual light source, low-pass filtering the raw image below a specified band, resizing the low-pass filter-processed image, and the resizing One image may be colorized and Bayerized, and the colored and Bayerized image may be demosaiced to generate the virtual light source image.

According to an embodiment, the instructions may cause the processor to generate a raw image of the virtual light source in a Bayer domain.

According to an embodiment, the instructions include, by the processor, obtaining a second image, generating the second image as a second output image, and processing the second image with the first filter to obtain a second input image and generate the second input image and the second output image as a second training data set of the artificial intelligence learning model.

According to an embodiment, the instructions cause the processor to generate a first output image by second filtering the virtual light source image and inserting the second filtered virtual light source image into the first image. can

According to an embodiment, the instructions may be configured by the processor based on at least one parameter of a size, color, intensity, shape, brightness, location, light spread, and degree of light splitting of the virtual light source. You can create a light source image.

According to an embodiment, the instructions may cause the processor to process the first filter using a point spread function (PSF) after inserting the virtual light source image into the first image. .

According to an embodiment, the instructions may cause the processor to recognize a pattern from the training data set and train the AI learning model based on at least one of pattern recognition algorithms that can be applied to actual data. have.

According to an embodiment, the instructions include, by the processor, a neural network, a support vector machine (SVN), a hidden Markov model (HMM), and a Baysian. The artificial intelligence learning model may be trained based on at least one of machine learning including at least one.

An electronic device according to an embodiment includes a display; a camera disposed under at least a partial area of the display; communication circuit; Memory; and a processor operatively coupled to the display, the camera, the communication circuitry, and the memory. According to an embodiment, when the memory is executed, the processor may capture an image using the camera and compensate the captured image based on an artificial intelligence learning model. According to an embodiment, the AI learning model may be trained based on a training data set including an input image and an output image generated using a virtual light source image.

8 is a flowchart of a method for training an artificial intelligence learning model of an electronic device according to an exemplary embodiment.

According to an embodiment, in operation 810 , the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200a of FIG. 2A ) may acquire a first image. For example, the first image may include an image pre-stored in the electronic device, an image received from an external electronic device, and/or an image acquired through crawling. For example, the first image may include arbitrary images for generating a training data set (eg, an input image and an output image).

According to an embodiment, in operation 820, the electronic device may generate a virtual light source image. For example, the processor may generate the virtual light source image based on at least one parameter among a size, color, intensity, shape, brightness, location, light spread, and degree of light splitting of the virtual light source. According to an embodiment, the at least one parameter may be designated according to a user input or may be automatically designated by a processor.

According to an embodiment, in operation 830 , the electronic device may generate a first output image by inserting the virtual light source image into the first image. For example, the first output image may be output data for training an artificial intelligence learning model. For example, the first output image may be a target image to be acquired through a trained artificial intelligence learning model when the first input image is input. According to an embodiment, the electronic device may generate a first output image by performing a second filter process on the virtual light source image and inserting the second filtered virtual light source image into the first image. For example, the second filter may be a filter for correcting the virtual light source image similarly to the case of photographing the light source through a real camera.

According to an embodiment, in operation 840 , the electronic device inserts the virtual light source image into the first image and performs first filter processing to generate the first input image. According to an embodiment, the first filter may be a filter for correcting the first image in which the virtual light source image is inserted to correspond to the actual photographed image. For example, the first filter may be a filter for applying a diffraction effect appearing when an image is captured through a real camera to the first image in which the virtual light source image is inserted. For example, the first input image may be input data for training an artificial intelligence learning model.

According to an embodiment, in operation 850 , the electronic device may generate and/or provide the first input image and the first output image as a training data set of an artificial intelligence learning model for compensating for an image captured by the camera. . For example, the AI learning model may be trained to derive a first output image when a first input image is input. For example, when the first input image is input, the artificial intelligence learning model may be trained to derive a result image similar to the first output image. According to various embodiments, the electronic device trains an artificial intelligence learning model by itself using a training data set (eg, a first training data set and/or a second training data set), or learns an artificial intelligence external to the electronic device. It is possible to train an artificial intelligence learning model through the system. For example, the electronic device may provide the generated training data set to an external artificial intelligence learning system through a communication circuit.

According to an embodiment, the electronic device may generate a training data set that does not include a virtual light source image separately from the above operations. For example, the electronic device may acquire the second image, generate the second image as a second output image, and process the second image with a first filter to generate a second input image. For example, the electronic device may generate the second input image and the second output image as the second training data set of the AI learning model.

According to various embodiments of the present disclosure, in training an artificial intelligence learning model for image compensation, not only actual measurement data (eg, an image captured by the camera 220 ) but also synthetic data (eg, a composite image) By generating and using as training data, the training effect of the AI learning model can be increased because the parameter values required for learning can be set or changed as desired. According to various embodiments of the present disclosure, by generating a training data set (eg, a first input image and a first output image) required for an artificial intelligence learning model from the same source (eg, a first image), an actual photographed image is generated Disparity that may occur when used as training data for an artificial intelligence learning model, noise (noise change) depending on the surrounding environment, movement during actual shooting, and/or limitations of the camera 220 (image sensor) (eg: It is possible to minimize errors and problems due to the saturation value of light).

9 is a flowchart of an operation of generating a virtual light source image of an electronic device according to an exemplary embodiment.

According to an embodiment, in operation 910 , the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200a of FIG. 2A ) may generate a raw image of the virtual light source. According to an embodiment, the processor may generate a raw image of the virtual light source in a Bayer domain.

According to an embodiment, in operation 920 , the electronic device may low-pass filter the raw image. For example, the electronic device may reflect a blurring effect that may occur depending on a lens and/or an optical structure when an actual camera is photographed in a virtual light source of a raw image through the LPF.

According to an embodiment, in operation 930 , the electronic device may resize the low-pass filter-processed image. For example, the electronic device may resize the virtual light source to correspond to an actual light source located at a specific distance. For example, the electronic device may downsample at least a portion of an image of the virtual light source to process an edge of the virtual light source similarly to an actual light source.

According to an embodiment, in operation 940 , the electronic device may colorize and bayerize the resized image. . For example, the electronic device may convert an image having a grayscale into three R, G, and B channels and set a color.

According to an embodiment, in operation 950 , the electronic device may demosaicing the colored and Bayer-ized images. For example, the electronic device may perform demosaicing to finally generate a virtual light source image having R, G, and B 3-channels.

According to various embodiments, at least some of the operations of generating the virtual light source image may be omitted.

10 is a flowchart of an image compensation operation of an electronic device according to an exemplary embodiment.

According to an embodiment, in operation 1010 , the electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200b of FIG. 2B ) may capture an image using a camera. According to an embodiment, the camera of the electronic device may be disposed below at least a partial area of the display of the electronic device. For example, the camera may be an under display camera (UDC).

According to an embodiment, in operation 1020 , the electronic device may compensate an image based on the artificial intelligence learning model. For example, the AI learning model may be a learning model for rewarding an image. According to an embodiment, the AI learning model may be trained by an external electronic device (eg, the electronic device 200a of FIG. 2A ) in the manner described with reference to FIGS. 8 to 9 . For example, the electronic device may receive an artificial intelligence learning model trained based on a training data set including an input image and an output image generated using a virtual light source image from an external electronic device. For example, the electronic device may compensate an image based on an artificial intelligence learning model received from an external electronic device.

According to various embodiments of the present disclosure, by compensating an image using an artificial intelligence learning model for image compensation trained through a training data set (eg, a first input image and a first output image) including a virtual light source, , it is possible to compensate for damage or deformation of an image due to diffraction of light, and to improve image quality.

According to an embodiment, a method for generating training data for an artificial intelligence learning model of an electronic device includes: acquiring a first image; generating a virtual light source image; inserting the virtual light source image into the first image to generate a first output image; inserting the virtual light source image into the first image and performing a first filter process to generate a first input image; and generating the first input image and the first output image as a first training data set of an artificial intelligence learning model for compensating for an image captured by the camera.

According to an embodiment, the generating of the virtual light source image may include generating a raw image of the virtual light source; processing the raw image with a low-pass filter of less than or equal to a specified band; resizing the low-pass filter-processed image; Colorization and Bayerization of the resized image; and demosaicing the colored and Bayer-ized images to generate the virtual light source image.

According to an embodiment, the generating of the raw image of the virtual light source may include generating the raw image of the virtual light source in a Bayer domain.

According to an embodiment, the method includes: acquiring a second image; generating the second image as a second output image; generating a second input image by processing the second image with a first filter; and generating the second input image and the second output image as a second training data set of the artificial intelligence learning model.

According to an embodiment, the generating of the first output image may include: second filtering the virtual light source image; and inserting the second filter-processed virtual light source image into the first image to generate a first output image.

According to an embodiment, the generating of the virtual light source image is based on at least one parameter among the size, color, intensity, shape, brightness, location, light spread, and degree of light splitting of the virtual light source. and generating the virtual light source image.

According to an embodiment, the generating of the first input image includes inserting the virtual light source image into the first image and then processing the first filter using a point spread function (PSF). It can include actions.

According to an embodiment, the AI learning model may be trained based on at least one of pattern recognition algorithms that can recognize a pattern from the training data set and apply it to actual data.

According to an embodiment, the AI learning model is at least one of a neural network, a support vector machine (SVN), a hidden Markov model (HMM), and a Baysian. It may be trained based on at least one of machine learning including one.

According to an embodiment, an image compensation method of an electronic device includes: capturing an image using a camera of the electronic device; and compensating for the captured image based on an artificial intelligence learning model. According to an embodiment, the AI learning model may be trained based on a training data set including an input image and an output image generated using a virtual light source image.

11 is a plan view of a display 1200 of an electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200b of FIG. 2B ) according to an exemplary embodiment.

According to an embodiment, the display 1200 (eg, a display panel) may include a flexible substrate. The electronic device according to an embodiment may include a deformable display such as a foldable display, a rollable display, an extendable display, and a flexible display.

The display 1200 of the electronic device according to an embodiment may include a display area DA in which the plurality of pixels PX are located and a non-display area NA in which an image is not displayed. For example, the non-display area NA may be positioned to surround the display area DA. According to various exemplary embodiments, unlike illustrated in FIG. 11 , the non-display area NA may not contact at least a portion of an edge of the display area DA.

According to an embodiment, the display area DA may include a first display area DA1 and a second display area DA2. According to an embodiment, a camera module may be disposed under the first display area DA1. According to an embodiment, at least a portion of the first display area DA1 of the display 1200 may overlap the camera module in the first direction Dr1. For example, the first display area DA1 may be determined by an angle of view of a camera module (not shown) and/or a distance between the display 1200 and the camera module. The first display area DA1 may be a partial area of the display area DA through which light incident to the camera module passes.

For example, the camera module may be a color camera that acquires a color image. For example, the camera module may include a first image sensor. In an embodiment, the first image sensor of the camera module may include a first color filter that transmits only light in a wavelength band representing the first color, a second color filter that transmits only light in a wavelength band representing the second color, and/or a third color A third color filter that transmits only light in a wavelength band may be included. The first image sensor of the camera module may extract a chromaticity signal decomposed into three primary colors. For example, the first color may be red, the second color may be green, and the third color may be blue. According to various embodiments, the electronic device may further include a camera. For example, the electronic device according to an embodiment may further include a black-and-white camera and/or a camera including a time of flight (TOF) sensor.

According to an embodiment, the second display area DA2 may be the remaining area of the display area DA except for the first display area DA1. For example, the first display area DA1 may be surrounded by the second display area DA2 . In the electronic device according to an embodiment, when viewed in the first direction Dr1 , at least one pixel PX may be located in the first display area DA1 overlapping the camera module.

According to an embodiment, the pixel PX may be a minimum unit for displaying an image. For example, the pixel PX may display a color image including at least one of a red sub-pixel (not shown), a green sub-pixel (not shown), or a blue sub-pixel (not shown). As another example, the plurality of pixels PX may include an RGBG pentile structure in which red subpixels and blue subpixels are alternately positioned in the same column and green subpixels are positioned in adjacent columns. However, the present invention is not limited thereto, and the plurality of pixels PX may include an RGB stripe structure.

According to various embodiments, the structure of the electronic device is not limited to that illustrated in FIG. 11 , and the structure and arrangement of components of the electronic device may be changed.

12 is a cross-sectional view of a first display area DA1 of an electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 200b of FIG. 2B ) according to an exemplary embodiment.

The electronic device according to an embodiment may include a window 1250 , a display 1200 (eg, a display panel), and/or a camera module 1260 positioned below the display 1200 . The display 1200 may include a substrate 1210 , a first light blocking member 1290 , a pixel circuit layer 1220 , an organic emission layer 1230 , and/or an encapsulation layer 1240 .

The substrate 1210 may be positioned on the front surface of the display area DA. For example, the substrate 1210 may be positioned in the first display area DA1 and the second display area DA2 . The substrate 1210 may include a transparent material to allow light to pass therethrough. For example, the substrate 1210 may include polyethylene terephthalate (PET). As another example, the substrate 1210 may include polyimide (PI) or glass.

A pixel circuit layer 1220 may be positioned on the substrate 1210 . The pixel circuit layer 1220 may include wirings (not shown) that transmit a signal for driving a pixel, a plurality of transistors (not shown), and/or an interlayer insulating layer (not shown). The pixel circuit layer 1220 may not be located in the transmission area TA of the first display area DA1. The pixel circuit layer 1220 may not be disposed at a position that overlaps the transmissive area TA of the first display area DA1 when viewed in the first direction Dr1 . The pixel circuit layer 1220 may be located in the second display area DA2 and may be located in the first display area DA1 excluding the transparent area TA. The pixel circuit layer 1220 may be positioned in the first display area DA1 that does not overlap the transmissive area TA of the first display area DA1 when viewed in the first direction Dr1 . However, in some embodiments, a portion of the pixel circuit layer 1220 may be located in the transmission area TA of the display 1200 .

According to an embodiment, a first light blocking member 1290 positioned between the substrate 1210 and the pixel circuit layer 1220 may be positioned in the first display area DA1 . For example, the first light blocking member 1290 may float and may not be electrically connected to the pixel circuit layer 1220 . The first light blocking member 1290 may include a plurality of first openings OP1 passing through the first light blocking member 1290 in the first direction Dr1 .

An organic emission layer 1230 may be positioned on the pixel circuit layer 1220 . For example, the organic emission layer 1230 may include a low molecular weight organic material or a high molecular weight organic material such as poly 3,4-ethylenedioxythiophene (PEDOT). As another example, the organic light emitting layer 1230 may include a hole injection layer (HIL), a hole transporting layer (HTL), an electron transporting layer (ETL), or an electron injection layer (electron injection layer, EIL) may be a multilayer further comprising at least one of. The organic light emitting layer 1230 includes a first organic light emitting layer 1231 emitting a first color, a second organic light emitting layer 1232 emitting a second color, and/or a third organic light emitting layer 1233 emitting a third color. may include The organic emission layer 1230 may not be located in the transmission area TA of the first display area DA1 . The organic emission layer 1230 may not be disposed at a position overlapping the transmission area TA of the first display area DA1 when viewed in the first direction Dr1 . The organic emission layer 1230 may be positioned in the first display area DA1 that does not overlap the transmissive area TA of the first display area DA1 when viewed in the first direction Dr1 . The organic emission layer 1230 may be located in the second display area DA2 and may be located in the first display area DA1 except for the transmission area TA. The density of the organic emission layer 1030 of the first display area DA1 (the area occupied by the organic emission layer within a unit area) may be different from the density of the organic emission layer of the second display area DA2 . In one embodiment, the density of the organic emission layer 1030 of the first display area DA1 may be lower than the density of the organic emission layer of the second display area DA2 .

An encapsulation layer 1240 may be positioned on the organic emission layer 1230 . The encapsulation layer 1240 may cover and seal the pixel circuit layer 1220 and the organic emission layer 1230 . For example, since the organic light emitting diode (OLED) is very vulnerable to moisture and oxygen, the encapsulation layer 1240 may seal the pixel circuit layer 1220 and the organic light emitting layer 1230 to block the inflow of external moisture and oxygen. have. In an embodiment, the encapsulation layer 1240 may include a plurality of layers. For example, the encapsulation layer 1240 may include a composite film including both an inorganic film and an organic film. As another example, the encapsulation layer 1240 may include a triple layer in which an inorganic layer, an organic layer, and an inorganic layer are sequentially disposed.

A window 1250 may be positioned on the encapsulation layer 1240 . The window 1250 may be transparent and may transmit light. For example, the window 1250 may include tempered glass, reinforced plastic, or a flexible polymer material. The window 1250 may be attached to the display 1200 by a transparent adhesive layer (not shown). According to an embodiment, the window 1250 may include a plurality of layers. The electronic device according to an embodiment may further include a coating layer (or a protective layer) positioned on one surface of the window 1250 . According to an embodiment, the display 1200 may further include a color filter layer (not shown) positioned on the encapsulation layer 1240 .

The camera module 1260 may be located below the display 1200 . When viewed in the first direction Dr1 (or when viewed from above the display 1200 ), the camera module 1260 may overlap the first display area DA1 of the display 1200 . The camera module 1260 may acquire an image by using light passing through the first display area DA1 of the display 1200 . In an embodiment, the camera module 1260 may be a color camera that acquires a color image. For example, the camera module 1260 may include a first image sensor, and the first image sensor may include a color filter.

According to various embodiments, the structure of the electronic device is not limited to that illustrated in FIG. 12 , and the structure and arrangement of components of the electronic device (eg, display) may be changed.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, a processor (eg, processor 120 ) of a device (eg, electronic device 101 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term refers to the case where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store™) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly between smartphones (eg: smartphones) and online. In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

Claims (20)

In an electronic device,
communication circuit;
Memory; and
a processor operatively coupled to the memory;
The memory, when executed, the processor,
acquiring a first image,
create a virtual light image,
inserting the virtual light source image into the first image to generate a first output image,
After inserting the virtual light source image into the first image, a first filter process is performed to generate a first input image,
An electronic device for storing the instructions for generating the first input image and the first output image as a first training data set of an artificial intelligence learning model for compensating an image.
The method according to claim 1,
The instructions, the processor,
create a raw image of the virtual light source,
Low-pass filter processing for the raw image below a specified band,
Resizing the low-pass filter-processed image,
Colorization and Bayerization of the resized image,
An electronic device for generating the virtual light source image by demosaicing the colored and Bayer-ized images.
3. The method according to claim 2,
The instructions, the processor,
An electronic device for generating a raw image of the virtual light source in a Bayer domain.
The method according to claim 1,
The instructions, the processor,
acquiring a second image;
generating the second image as a second output image;
generating a second input image by processing the second image with the first filter;
An electronic device configured to generate the second input image and the second output image as a second training data set of the artificial intelligence learning model.
The method according to claim 1,
The instructions, the processor,
processing the virtual light source image with a second filter,
An electronic device for generating a first output image by inserting the virtual light source image processed by the second filter into the first image.
The method according to claim 1,
The instructions, the processor,
An electronic device configured to generate the virtual light source image based on at least one parameter of a size, color, intensity, shape, brightness, position, light spread, and degree of light splitting of the virtual light source.
The method according to claim 1,
The instructions, the processor,
After inserting the virtual light source image into the first image, the electronic device performs the first filter process using a point spread function (PSF).
The method according to claim 1,
The instructions, the processor,
An electronic device that recognizes a pattern from the training data set and trains the AI learning model based on at least one of pattern recognition algorithms that can be applied to actual data.
The method according to claim 1,
The instructions, the processor,
At least one of a neural network, a support vector machine (SVN), a hidden Markov model (HMM), and a machine learning comprising at least one of a Baysian. An electronic device for training the artificial intelligence learning model based on
A method for generating training data for an artificial intelligence learning model of an electronic device, the method comprising:
acquiring a first image;
generating a virtual light source image;
inserting the virtual light source image into the first image to generate a first output image;
inserting the virtual light source image into the first image and then performing a first filter process to generate a first input image; and
and generating the first input image and the first output image as a first training data set of an artificial intelligence learning model for compensating for an image captured by the camera.
11. The method of claim 10,
The operation of generating the virtual light source image includes:
generating a raw image of the virtual light source;
processing the raw image with a low-pass filter of less than or equal to a specified band;
resizing the low-pass filter-processed image;
Colorization and Bayerization of the resized image; and
and demosaicing the colored and Bayer-ized images to generate the virtual light source image.
12. The method of claim 11,
The generating of the raw image of the virtual light source includes generating a raw image of the virtual light source in a Bayer domain.
11. The method of claim 10,
acquiring a second image;
generating the second image as a second output image;
generating a second input image by processing the second image with a first filter; and
The method further comprising generating the second input image and the second output image as a second training data set of the artificial intelligence learning model.
11. The method of claim 10,
The operation of generating the first output image includes:
processing the virtual light source image with a second filter; and
and inserting the second filter-processed virtual light source image into the first image to generate a first output image.
11. The method of claim 10,
The operation of generating the virtual light source image includes:
and generating the virtual light source image based on at least one parameter selected from the group consisting of size, color, intensity, shape, brightness, location, light spreading, and light splitting of the virtual light source.
11. The method of claim 10,
The operation of generating the first input image includes:
and inserting the virtual light source image into the first image and then performing the first filter processing using a point spread function (PSF).
11. The method of claim 10,
The artificial intelligence learning model is trained based on at least one of pattern recognition algorithms that can be applied to actual data by recognizing a pattern from the training data set.
11. The method of claim 10,
The artificial intelligence learning model is machine learning including at least one of a neural network, a support vector machine (SVN), a hidden Markov model (HMM), and a Baysian. A method that is trained based on at least one of (machine learning).
In an electronic device,
display;
a camera disposed under at least a partial area of the display;
communication circuit;
Memory; and
a processor operatively coupled with the display, the camera, the communication circuitry, and the memory;
The memory, when executed, the processor,
Capture an image using the camera,
to compensate the captured image based on an artificial intelligence learning model,
The artificial intelligence learning model is an electronic device trained based on a training data set including an input image and an output image generated using a virtual light source image.
A method for compensating an image of an electronic device, the method comprising:
capturing an image using a camera of the electronic device; and
Compensating the captured image based on an artificial intelligence learning model,
The artificial intelligence learning model is trained based on a training data set including an input image and an output image generated using a virtual light source image.

Images