WO2024076101A1

WO2024076101A1 - Artificial-intelligence-based image processing method and electronic device for supporting same

Info

Publication number: WO2024076101A1
Application number: PCT/KR2023/015018
Authority: WO
Inventors: 송진우
Original assignee: 삼성전자 주식회사
Priority date: 2022-10-04
Filing date: 2023-09-27
Publication date: 2024-04-11

Abstract

Provided in an embodiment of the present disclosure are an image processing method and an electronic device for supporting same, the method enabling image processing of a video to be supported using artificial intelligence when capturing (or recording) the video The electronic device may comprise a display, a camera, a processor and an image signal processor included in the processor. The image signal processor can receive, through the camera, input data (for example, raw data) corresponding to the sensor output size of a sensor while a user captures video. The image signal processor can determine parameters corresponding to settings by which the video is captured or a change in the settings. The image signal processor can perform, on the basis of the parameters, image processing for converting the input data into a full color image and changing the output size of the full color image to match a video output size. The image signal processor can output image data according to image processing.

Description

Artificial intelligence-based image processing method and electronic devices that support it

An embodiment of the present disclosure provides an image processing method that can support image processing of a video using an artificial intelligence (AI) method when shooting (or recording) a video, and an electronic device that supports the same.

With the development of digital technology, various types of electronic devices such as smart phones, digital cameras, and/or wearable devices are widely used. In order to support and increase the functionality of electronic devices, the hardware and/or software portions of electronic devices are continuously being developed.

Recently, with the increase in the use of electronic devices and image capture functions using electronic devices, various studies on image processing technology in electronic devices are being conducted. For example, recent electronic devices support artificial intelligence (AI)-based image processing when capturing images. For example, AI-based image processing technology is being proposed that can detect full color images even with minimal lighting (e.g., dark environments such as at night). For example, AI-based image processing technology may refer to processing technology that allows high-quality images to be obtained even in a minimal lighting environment.

Meanwhile, AI-based image processing technology is currently used for image processing for images (e.g. still images). For example, in the process of generating (e.g., recording) a video, an electronic device provides video by processing video signals using technology based on a general video processing algorithm.

For example, conventionally, an image signal processor (ISP) is implemented as a technology based on an image processing algorithm, its pipeline is defined, and its structure is simplified. Therefore, conventional image processing methods may have limitations, and there may be no progress in improving the performance of demosaic and/or noise reduction, which have a significant impact on the resolution of the final image. For example, existing image processing algorithms may have lower performance compared to recent AI-based image processing algorithms. However, AI-based image processing algorithms consume a lot of current to be used in scenarios where video is captured (e.g., recorded), so to date, general image processing algorithms are being applied to videos.

In one embodiment of the present disclosure, an image processing method capable of supporting image processing of a video using an artificial intelligence (AI) method when shooting a video and an electronic device supporting the same are provided.

In one embodiment of the present disclosure, through the structure of an image signal processor (ISP) that can improve resolution and noise reduction ability during artificial intelligence-based video shooting (e.g., recording), real-time demosaic (demosaic) ) Provides an image processing method that can support processing and an electronic device that supports the same.

The technical problem to be achieved in this document is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An electronic device according to an embodiment of the present disclosure may include a display, a camera, a processor operatively connected to the display and the camera, and an image signal processor (ISP) included in the processor. The image signal processor according to one embodiment may operate to receive input data corresponding to the sensor output size of the sensor through the camera while a user is capturing a video. The image signal processor according to one embodiment may operate to determine a parameter corresponding to a setting or setting change in which the video is captured. The image signal processor according to an embodiment is configured to perform image processing to convert the input data into a full color image based on the parameters and vary the output size of the full color image to match the video output size. It can work. The image signal processor according to one embodiment may operate to output image data according to image processing.

An image processing device of an electronic device according to an embodiment of the present disclosure may include a camera, an image signal processor (ISP) operatively connected to the camera, and a memory operatively connected to the image signal processor. You can. According to one embodiment, the memory, when executed, causes the image signal processor to receive input data corresponding to a sensor output size of a sensor through the camera while a video capture is performed by a user, and Determining parameters corresponding to the settings or setting changes to be captured, converting the input data to a full color image based on the parameters, and changing the output size of the full color image to match the video output size. Instructions for performing processing and outputting image data according to image processing may be stored.

A method of operating an electronic device according to an embodiment of the present disclosure includes, while a user is capturing a video, an image signal processor (ISP) transmits input data corresponding to the sensor output size of the sensor through a camera. It may include performing the operation of receiving. The operating method may include performing an operation to determine a parameter corresponding to a setting or setting change in which the video is captured. The operating method may include performing an operation of converting the input data into a full color image based on the parameters and performing image processing to vary the output size of the full color image to suit the video output size. there is. The operation method may include performing an operation of outputting image data according to image processing.

In order to solve the above problems, various embodiments of the present disclosure may include a computer-readable recording medium on which a program for executing the method on a processor is recorded.

According to one embodiment, a non-transitory computer-readable storage medium (or computer program product) storing one or more programs is described. According to one embodiment, one or more programs, when executed by a processor of an electronic device, may output a sensor output size of a sensor through a camera while a user performs video capture. An operation of receiving input data corresponding to the video, an operation of determining a parameter corresponding to a setting or setting change in which the video is captured, converting the input data into a full color image based on the parameter, and the full color image. It may include a command for performing image processing to change the output size of the image to match the video output size, and an operation for outputting image data according to the image processing.

Additional scope of applicability of the present disclosure will become apparent from the detailed description that follows. However, since various changes and modifications within the spirit and scope of the present disclosure may be clearly understood by those skilled in the art, the detailed description and specific embodiments, such as preferred embodiments of the present disclosure, should be understood as being given by way of example only.

According to an electronic device, its operating method, and a recording medium according to an embodiment of the present disclosure, power consumption can be reduced and resolution can be increased when processing artificial intelligence-based images in video. According to one embodiment, the image signal processor generates a full color image based on parameter replacement of the AI network and simultaneously performs resize, so that when shooting (e.g. recording) an artificial intelligence-based video, Current consumption can be reduced. According to one embodiment, if image quality is improved by processing calculations for image processing based on artificial intelligence, power consumption due to a reduction in the amount of calculation can be minimized. According to one embodiment, by reducing power consumption due to artificial intelligence-based image processing, it is possible to use the AI network of the image signal processor even during high-complexity video shooting (e.g., recording).

In addition, various effects that can be directly or indirectly identified through this document may be provided. The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

In relation to the drawing description, the same or similar reference numerals may be used for the same or similar components.

1 is a block diagram of an electronic device in a network environment according to various embodiments.

2 is a block diagram illustrating a camera module according to various embodiments.

FIG. 3 is a diagram schematically showing the configuration of an electronic device according to an embodiment of the present disclosure.

FIG. 4 is a diagram schematically showing the configuration of an image signal processor according to an embodiment of the present disclosure.

FIG. 5 is a diagram illustrating an example of operations of a measurement unit and an alignment unit in an image signal processor according to an embodiment of the present disclosure.

FIG. 6 is a diagram illustrating an example of the operation of a preprocessor in an image signal processor according to an embodiment of the present disclosure.

FIG. 7 is a diagram illustrating an example of a full color image in an image signal processor according to an embodiment of the present disclosure.

FIG. 8 is a diagram illustrating an example of the output of an image signal processor according to an embodiment of the present disclosure.

FIG. 9 is a diagram illustrating an example of a network cache of an image signal processor according to an embodiment of the present disclosure.

FIG. 10 is a diagram illustrating an example of an operation of an image signal processor according to an embodiment of the present disclosure.

FIGS. 11A, 11B, and 11C are diagrams illustrating an example of a scaling operation in an image signal processor according to an embodiment of the present disclosure.

FIG. 12 is a flowchart illustrating a method of operating an electronic device according to an embodiment of the present disclosure.

FIG. 1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments.

Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores instructions or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 is a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (e.g., a central processing unit (CPU) or an application processor (AP)) that can be operated independently or together. 123) (e.g., graphic processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor hub processor, or communication processor (CP, communication processor)) may be included. For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, replace the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or when the main processor 121 While in an active (e.g., application execution) state, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or At least some of the functions or states related to the communication module 190 can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system (OS) 142, middleware 144, or applications 146. there is.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a telecommunication network such as a LAN or wide area network (WAN)). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 to communicate within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (eMBB, enhanced mobile broadband), minimization of terminal power and access to multiple terminals (mMTC, massive machine type communications), or high-reliability and low-latency (URLLC, ultra-reliable and low-latency). communications) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO, full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 198 or the second network 199, is connected to the plurality of antennas by, for example, the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external

electronic devices

102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external

electronic devices

102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

2 is a block diagram 200 illustrating a camera module 180 according to various embodiments.

Referring to FIG. 2, the camera module 180 includes a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, It may include a memory 250 (e.g., buffer memory) or an image signal processor 260.

The lens assembly 210 may collect light emitted from a subject that is the target of image capture. Lens assembly 210 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 210. In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 have the same lens properties (e.g., angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly is different from another lens assembly. It may have one or more lens properties that are different from the lens properties of . The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

The flash 220 may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash 220 may include one or more light emitting diodes (eg, red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.

The image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal. According to one embodiment, the image sensor 230 is one image sensor selected from among image sensors with different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, and the same It may include a plurality of image sensors having different properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 240 moves at least one lens or image sensor 230 included in the lens assembly 210 in a specific direction in response to the movement of the camera module 180 or the electronic device 101 including the same. The operating characteristics of the image sensor 230 can be controlled (e.g., adjusting read-out timing, etc.). This allows to compensate for at least some of the negative effects of said movement on the image being captured. According to one embodiment, the image stabilizer 240 uses a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180 to stabilize the camera module 180 or the electronic device 101. ) can detect such movements. According to one embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer.

The memory 250 may at least temporarily store at least a portion of the image acquired through the image sensor 230 for the next image processing task. For example, when image acquisition is delayed due to the shutter or when multiple images are acquired at high speed, the acquired original image (e.g., Bayer-patterned image or high-resolution image) is stored in the memory 250. , the corresponding copy image (eg, low-resolution image) may be previewed through the display module 160. Thereafter, when a specified condition is satisfied (eg, user input or system command), at least a portion of the original image stored in the memory 250 may be obtained and processed, for example, by the image signal processor 260. According to one embodiment, the memory 250 may be configured as at least part of the memory 130 or as a separate memory that operates independently.

The image signal processor 260 may perform one or more image processes on an image acquired through the image sensor 230 or an image stored in the memory 250. The one or more image processes may include, for example, depth map creation, three-dimensional modeling, panorama creation, feature point extraction, image compositing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring). may include blurring, sharpening, or softening. Additionally or alternatively, the image signal processor 260 provides control (e.g., exposure time control, or read-out timing control) for at least one of the components included in the camera module 180 (e.g., image sensor 230). etc.) can be performed. Images processed by the image signal processor 260 are stored back in memory 250 for further processing or are stored in external components of the camera module 180 (e.g., memory 130, display module 160, electronics ( 102), an electronic device 104, or a server 108). According to one embodiment, the image signal processor 260 may be configured as at least a part of the processor 120, or may be configured as a separate processor that operates independently from the processor 120. When the image signal processor 260 is configured as a separate processor from the processor 120, at least one image processed by the image signal processor 260 is displayed as is or after additional image processing by the processor 120. It may be displayed through module 160.

According to one embodiment, the electronic device 101 may include a plurality of camera modules 180, each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least another one may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least another one may be a rear camera.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one element from another, and may be used to distinguish such elements in other respects, such as importance or order) is not limited. One (e.g. first) component is said to be "coupled" or "connected" to another (e.g. second) component, with or without the terms "functionally" or "communicatively". Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order. may be removed, omitted, or one or more other operations may be added.

Referring to FIG. 3 , the electronic device 101 according to an embodiment of the present disclosure may include a display module 160, a camera module 180, a memory 130, and/or a processor 120. According to one embodiment, the electronic device 101 may include all or at least some of the components of the electronic device 101 as described in the description with reference to FIG. 1 .

According to one embodiment, the display module 160 may include the same or similar configuration as the display module 160 of FIG. 1. According to one embodiment, the display module 160 may include a display and visually provide various information to the outside of the electronic device 101 (eg, a user). According to one embodiment, the display module 160 may visually provide various information (eg, content, images, videos) related to the executing application and its use under the control of the processor 120.

According to one embodiment, the display module 160 includes a touch sensor, a pressure sensor capable of measuring the intensity of touch, and/or a touch panel that detects a magnetic field-type stylus pen. (e.g. digitizer) may be included. According to one embodiment, the display module 160 receives a signal (e.g., voltage, amount of light, resistance, electromagnetic signal, and/or Touch input and/or hovering input (or proximity input) can be detected by measuring changes in electric charge. According to one embodiment, the display module 160 may include a liquid crystal display (LCD), an organic light emitted diode (OLED), or an active matrix organic light emitted diode (AMOLED). According to one embodiment, the display module 160 may include a flexible display.

According to one embodiment, the camera module 180 may correspond to the camera module 180 of FIG. 1 or FIG. 2. According to one embodiment, when activated, the camera module 180 may capture a subject and transmit related results (eg, a captured image) to the processor 120 and/or the display module 160.

According to one embodiment, the memory 130 may correspond to the memory 130 of FIG. 1 . According to one embodiment, the memory 130 may store various data used by the electronic device 101. In one embodiment, data may include, for example, input data or output data for an application (e.g., program 140 of FIG. 1) and instructions associated with the application. In one embodiment, the data may include camera image data acquired through a camera module. In one embodiment, the data may include various learning data acquired based on the user's learning through interaction with the user. In one embodiment, data may include various schemas (or algorithms, models, networks, or functions) to support artificial intelligence-based image processing.

According to one embodiment, the memory 130 may store instructions that cause the processor 120 to operate when executed. For example, an application may be stored as software (eg, program 140 in FIG. 1) on the memory 130 and may be executable by the processor 120. According to one embodiment, the application may be a variety of applications that can provide various functions or services (eg, an image capture function based on artificial intelligence) on the electronic device 101.

According to one embodiment, the processor 120 may perform an application layer processing function required by the user of the electronic device 101. According to one embodiment, the processor 120 may provide commands and control of functions for various blocks of the electronic device 101. According to one embodiment, the processor 120 may perform operations or data processing related to control and/or communication of each component of the electronic device 101. For example, the processor 120 may include at least some of the components and/or functions of the processor 120 of FIG. 1 . The processor 120 may be operatively connected to components of the electronic device 101, for example. The processor 120 may load commands or data received from other components of the electronic device 101 into the memory 130, process the commands or data stored in the memory 130, and store the resulting data. there is.

According to one embodiment, the processor 120 may be an application processor (AP). According to one embodiment, the processor 120 may be a system semiconductor responsible for calculation and multimedia driving functions of the electronic device 101. According to one embodiment, the processor 120 is configured in the form of a system-on-chip (SoC), a technology-intensive semiconductor chip that integrates several semiconductor technologies and implements system blocks into one chip. It can be included. According to one embodiment, the system blocks of the processor 120 include a graphics processing unit (GPU) 310 and an image signal processor (ISP) 320, as illustrated in FIG. 3. , central processing unit (CPU) 330, neural processing unit (NPU) 340, digital signal processor 350, modem 360, connectivity It may include (connectivity) 370 and/or security (security) 380 blocks.

According to one embodiment, GPU 310 may be responsible for graphics processing. According to one embodiment, the GPU 310 receives instructions from the CPU 330 and performs graphics processing to express the shape, position, color, shading, movement, and/or texture of objects (or objects) on the display. can do.

According to one embodiment, the ISP 320 may be responsible for processing and correcting images and videos. According to one embodiment, the ISP 320 receives raw data (e.g., input data, raw data, original data, Alternatively, it can play a role in correcting raw data to create an image in the user's preferred form. According to one embodiment, the ISP 320 may correct physical limitations that may occur in the camera module 180, interpolate red, green, blue (R/G/B) values, and remove noise. According to one embodiment, the ISP 320 may perform post-processing such as adjusting partial brightness of the image and emphasizing detailed parts. For example, the ISP 320 can generate a result preferred by the user by independently tuning and correcting the image quality of the image acquired through the camera module 180.

According to one embodiment, the ISP 320 may support artificial intelligence-based image processing technology to improve image quality, speed image processing, and reduce current consumption (e.g., low power). For example, the ISP 320 can maintain low power while improving video quality, and for this purpose, it can support artificial intelligence-based video shooting. According to one embodiment, the ISP 320 may support artificial intelligence-based image processing related to improving video quality in a dark, low-light environment. According to one embodiment, the ISP 320 may support scene segmentation (e.g., image segmentation) technology that recognizes and/or classifies parts of the scene being shot in conjunction with the NPU 340. there is. For example, the ISP 320 may include processing functions by applying different parameters to objects such as the sky, bushes, and/or skin. According to one embodiment, the ISP 320 detects and displays a human face when shooting an image through an artificial intelligence function, or uses the coordinates and information of the face to adjust the brightness, focus, and/or color of the image. .

According to one embodiment, the configuration and detailed operation of the ISP 320 of the processor 120 will be described with reference to the drawings described later.

According to one embodiment, the CPU 330 may play a role corresponding to the processor 120. According to one embodiment, the CPU 330 may decode user commands and perform arithmetic and logical operations and/or data processing. For example, the CPU 330 may be responsible for the functions of memory, interpretation, calculation, and control. According to one embodiment, the CPU 330 may control the overall functions of the electronic device 101. For example, the CPU 330 can execute all software (eg, applications) of the electronic device 101 on an operating system (OS) and control hardware devices.

According to one embodiment, the NPU 340 may be responsible for processing optimized for an artificial intelligence deep learning algorithm. According to one embodiment, the NPU 340 is a processor optimized for deep learning algorithm calculations (e.g., artificial intelligence calculations), and can process big data quickly and efficiently like a human neural network. For example, the NPU 340 can be mainly used for artificial intelligence calculations. According to one embodiment, the NPU 340 recognizes objects, environments, and/or people in the background when taking an image through the camera module 180 and automatically adjusts the focus, or uses the camera module 180 when taking a picture of food. It can automatically switch the shooting mode to food mode and/or erase only unnecessary subjects from the captured results.

According to one embodiment, the electronic device 101 supports integrated machine learning processing by interacting with all processors such as GPU 310, ISP 320, CPU 330, and NPU 340. You can.

According to one embodiment, the DSP 350 may represent an integrated circuit that helps quickly process digital signals. According to one embodiment, the DSP 350 may perform a high-speed processing function by converting an analog signal into a digital signal.

According to one embodiment, the modem 360 may perform a role that allows the electronic device 101 to use various communication functions. For example, the modem 360 can support communications such as phone calls and data transmission and reception by exchanging signals with a base station. According to one embodiment, the modem 360 is an integrated modem that supports communication technologies such as LTE and 2G to 5G (e.g., cellular modem, LTE modem, 5G modem, and 5G-Advanced modem, and 6G modem). may include. According to one embodiment, the modem 360 may include an artificial intelligence modem to which an artificial intelligence algorithm is applied.

According to one embodiment, connectivity 370 may support wireless data transmission based on IEEE 802.11. According to one embodiment, connectivity 370 may support communication services based on IEEE 802.11 (eg, Wi-Fi) and/or 802.15 (eg, Bluetooth, ZigBee, UWB). For example, the connectivity 370 can support communication services for an unspecified number of people in a localized area, such as indoors, using an unlicensed band.

According to one embodiment, the security 380 may provide an independent security execution environment between data or services stored in the electronic device 101. According to one embodiment, the security 380 prevents hacking from occurring through software and hardware security during user authentication when providing services such as biometrics, mobile ID, and/or payment of the electronic device 101. can play a role in preventing. For example, the security 380 is based on device security to strengthen the security of the electronic device 101 and user information such as mobile ID, payment, and car key in the electronic device 101. An independent security execution environment can be provided by the security service.

According to one embodiment of the present disclosure, the processor 120 (eg, ISP 320) may include processing circuitry and/or executable program elements. According to one embodiment, the processor 120 (e.g., ISP 320) supports image processing of a video in an AI manner when capturing (e.g., recording) a video based on processing circuits and/or executable program elements. You can control (or process) actions related to doing something.

According to one embodiment, the processor 120 (e.g., ISP 320) collects raw data (e.g., input data, raw data) through the camera module 180 while a video is captured by a user. or original data) may be performed. According to one embodiment, the processor 120 (e.g., ISP 320) may perform an operation to determine parameters corresponding to settings or settings changes in which the video is captured. According to one embodiment, the processor 120 (e.g., ISP 320) may perform image processing operations including demosaicing and scaling of the raw data based on the parameters. According to one embodiment, the processor 120 (eg, ISP 320) may perform an operation of outputting image data according to image processing.

According to one embodiment, detailed operations of the processor 120 and/or the ISP 320 of the electronic device 101 will be described with reference to the drawings described later.

According to one embodiment, operations performed by the processor 120 (eg, ISP 320) may be implemented as a recording medium (or computer program product). For example, the recording medium may include a non-transitory computer-readable recording medium on which a program for executing various operations performed by the processor 120 is recorded.

Embodiments described in this disclosure may be implemented in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof. According to hardware implementation, the operations described in one embodiment include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). ), processors, controllers, micro-controllers, microprocessors, and/or other electrical units to perform functions. .

In one embodiment, the recording medium (or computer program product) includes an image signal processor (ISP) receiving raw data through a camera module while a user performs video capture. An operation of determining a parameter corresponding to a setting or setting change in which the video is captured, an operation of performing image processing including demosaicing and scaling of the raw data based on the parameter, and It may include a computer-readable recording medium on which a program for executing an operation of outputting image data according to image processing is recorded.

According to one embodiment, in Figure 4, when shooting a video in the electronic device 101, the image acquired through the camera module 180 is received from the ISP 320, and artificial intelligence-based image processing is performed. An example of the operation can be shown.

Referring to FIG. 4, the electronic device 101 includes a camera module 180 (e.g., the camera module 180 in FIG. 1 or 2) and an image signal processor 320 (e.g., the ISP 320 in FIG. 3). , and a memory 480 (eg, the memory 130 of FIG. 1 or FIG. 3 or the memory 250 of FIG. 2). In one embodiment, the camera module 180 may include an image sensor 410, a receiver unit 420, and a measurement unit 430. According to one embodiment, the camera module 180 may include a lens (eg, the lens assembly 210 of FIG. 2), although not shown. In one embodiment, ISP 320 may include an alignment unit 440, a preprocessor 450, an AI network 460, and a post-processor 470. In one embodiment, the memory 480 is a memory included in the electronic device 101 (e.g., memory 130 of FIG. 1 or 3), and/or a memory included in the camera module 180 (e.g., It may be the memory 250 of FIG. 2).

According to one embodiment, the image sensor 410 (e.g., the image sensor 230 of FIG. 2) receives raw data from the lens of the camera module 180 (e.g., the lens assembly 210 of FIG. 2). can receive. According to one embodiment, the image sensor 410 may receive CFA data (or CFA image) according to a color filter array (CFA) pattern (e.g., color filter shape (Bayer pattern)) of the image sensor 410. .

According to one embodiment, the receiver unit 420 may receive raw data from the image sensor 410 in real time. According to one embodiment, the receiver unit 420 may transmit raw data received in real time to the memory 480. According to one embodiment, the receiver unit 420 may perform a role of directly transmitting raw data to the memory 480 (eg, direct memory access, DMA) according to the output characteristics of the image sensor 410. For example, the receiver unit 420 may perform the role of directly transmitting (e.g., sharing) data between the image sensor 410 and the memory 480 without intervention of the CPU (e.g., CPU 330 in FIG. 3). You can.

According to one embodiment, the measurement unit 430 may receive raw data from the image sensor 410 and extract measurement data based on the received raw data. According to one embodiment, the measurement data may include motion data and 3A (eg, Auto exposure, Auto white balance, Auto focus) data. According to one embodiment, the measurement data may include tone correction histogram data to be used in the post-processing unit 470.

According to one embodiment, the measurement unit 430 may operate flexibly according to the CFA of the image sensor 410. According to one embodiment, the measurement unit 430 may perform measurement according to the CFA pattern of the image sensor 410. For example, when generating a luminance image for calculating a histogram, the measurement unit 430 may change a coefficient for combining each input color channel into luminance. According to one embodiment, the measurement unit 430 may perform measurement for each channel and perform measurement according to the changed CFA when the CFA of the image sensor 410 is changed. According to one embodiment, the measurement unit 430 may determine the movement of the image through at least one frame (eg, 1 frame or N frames) from the image sensor 410.

According to one embodiment, the ISP 320 (e.g., an AI front end) performs demosaic, shadow correction, color correction, noise removal, and sharpness adjustment based on raw data and measurement data. The same image processing can be performed. According to one embodiment, the ISP 320 uses raw data captured by the camera module 180 using an artificial intelligence method (e.g., artificial neural network (e.g., UNET, artificial neural network (ANN), deep neural network (DNN), CNN) (AI network structures such as convolutional neural network, RNN (recurrent neural network), AE (autoencoder), and/or GAN (generative adversarial network)) to full color conversion (AI front end) ) can play a role. According to one embodiment, the ISP 320 may include an alignment unit 440, a pre-processing unit 450, an AI network 460, and a post-processing unit 470.

According to one embodiment, the alignment unit 440 acquires measurement data such as image movement and/or brightness from the measurement unit 430, and uses the obtained measurement data to divide N-1 frames of the image into N Motion compensation can be performed to fit the frame. In one embodiment, the N-1 frame and N frame may represent raw data (eg, CFA data (or frame)) sequentially output from the image sensor 410. For example, N frame may represent the currently output frame, and N-1 frame may represent the previous frame of one frame (e.g., N frame). The operation of the alignment unit 440 according to one embodiment is illustrated in FIG. 5 .

Referring to FIG. 5, the alignment unit 440 according to one embodiment receives measurement data (e.g., motion data) from the measurement unit 430 and aligns the N-1 frame 510 with the N frame 520. Motion compensation can be performed. In one embodiment, N-1 frames 510 may represent an example where there is motion, and N frames 520 may represent an example where there is no motion. For example, the motion for each frame is distorted due to parallax (or parallax in reading speed) due to image reading in the camera module 180 and characteristics of the image sensor 410 (e.g., rolling shutter type image sensor). (e.g. image distortion) may occur. According to one embodiment, the alignment unit 440 performs an operation (e.g., warping) on the N-1 frame 510 to match the N-1 frame 510 with the N frame 520 for motion compensation for distortion. operation) can be performed.

According to one embodiment, the alignment unit 440 may measure and compensate for motion between images and match them so that at least one image (eg, one or multiple frames) can be input at a time. For example, the alignment unit 440 can match multiple images and provide them to the AI network 460, thereby helping the AI network 460 increase image resolution and reduce noise during demosaicing. According to one embodiment, the measurement unit 430 may measure the movement of the image for each frame and provide the measurement to the alignment unit 440 so that the alignment unit 440 can match multiple images.

According to one embodiment, the preprocessor 450 may calculate the raw data using different arithmetic operations for each pixel and then transmit it to the AI network 460. According to one embodiment, the preprocessing unit 450 is located at the front of the AI network 460, and can optimize the preprocessing operation performed in the existing ISP 320 into a form suitable for the AI network 460. . The operation of the preprocessor 450 according to one embodiment is illustrated in FIG. 6.

Referring to FIG. 6, the preprocessor 450 may perform an operation consisting of arithmetic operations for each pixel on the N frame 520 and the N-1 frame 510. According to one embodiment, the preprocessor 450 performs four arithmetic operations based on, for example, Pedestal (610), white balance (WB) 620, lens shading correction (LSC) 630, and Gamma (640). can be performed. According to one embodiment, the preprocessor 450 may sequentially input at least two frames (e.g., N-1 frame 510 and N frame 520) to the AI network 460. According to one embodiment, the AI network 460 may perform tone correction by reordering the order of each pixel when at least two or more frames are sequentially input. For example, the AI network 460 performs global tone correction (GTM, global tone mapping) 650 and local tone correction (LTM, local tone mapping) 660 for at least two sequentially input frames. It is possible to obtain image quality gains.

According to one embodiment, the preprocessor 450 may adjust coefficients for each color channel for the luminance image. According to one embodiment, the coefficient (Y) for generating a luminance image can be expressed as an example in <Equation 1> below.

Y = (0.257 * R) + (0.504 * G) + (0.098 * B)

In one embodiment, <Equation 1> represents an example of calculating a coefficient for generating a luminance image, but is not limited thereto.

According to one embodiment, the AI network 460 may include an AI accelerator such as a microprocessor (eg, MPU, microprocessor unit). According to one embodiment, the AI network 460 may perform demosaic to convert the CFA pattern of the image sensor 410 into full color form using a designated artificial intelligence algorithm. According to one embodiment, the AI network 460 may perform an operation of interpolating and filling empty parts of pixels using peripheral pixels (e.g., color-filled pixels) in raw data, based on a designated artificial intelligence algorithm. . According to one embodiment, the AI network 460 performs demosaicing (e.g., a function of converting to full color) and, at approximately the same time, outputs the current video input from the preprocessor 450 (e.g., a function of converting to full color). By changing the output size of the image (e.g., scaling) to match the resolution of the raw data (e.g., sensor output size), the image output speed can be improved and/or power consumption can be minimized.

According to one embodiment, the AI network 460 performs demosaicing in full color based on the raw data (e.g., CFA image) of the image sensor 410 processed in the pre-processor 450 and then performs demosaicing in the post-processor 470. ) can be transmitted. The AI network 460 according to one embodiment uses parameters learned through AI (e.g., AI parameters) and uses one image or N images matched in the measurement unit 430 and the alignment unit 440 to generate one image. Full color images can be output. In one embodiment, the parameters may include information learned according to the CFA of the image sensor 410 (eg, by sensor output size). For example, when a change in sensor output size (e.g., mode switching between FHD and UHD) occurs, the AI network 460 changes the network layer information, weight parameters, and/or bias parameters to match the changed mode, thereby changing the desired mode. May operate to obtain an output (e.g., video output size). For example, the output of AI network 460 may represent a full-color image in FHD or UHD. The output of AI network 460 according to one embodiment is illustrated in FIG. 7 .

Referring to FIG. 7, the full-

color image

701, 703, or 705 of the AI network 460 is adjusted to the video resolution (e.g., HD, FHD, UHD, or 8K) currently set by the user without a separate scaling process. can be printed. For example, element 701 represents an example of a full-color image from AI network 460 for the output of a 4x4 sensor, and element 703 represents an example of a full-color image from AI network 460 for the output of a 2x2 sensor, Element 705 may represent an example of a full-color image from the AI network 460 for the output of a Bayer sensor. For example, if the user sets FHD resolution recording in the camera module 180 using a 12M sensor, the output (e.g. video output size) of the AI network 460 may be a 3M full color image. . According to one embodiment, the output of the ISP 320 (e.g., the AI network 460) and its operation are described with reference to the drawings described below.

According to one embodiment, the post-processing unit 470 may perform tone correction on the output of the AI network 460 (e.g., demosaic data or full color data). For example, the post-processing unit 470 can adjust the color of the image. According to one embodiment, the post-processing unit 470 performs real-time tone correction, such as GTM and/or LTM, on the full color image output from the AI network 460 using measurement data, as illustrated in FIG. 6. It can be done. According to one embodiment, the measurement unit 430 may provide tone-related data such as an image histogram necessary for tone correction of the post-processing unit 470 to the post-processing unit 470.

According to one embodiment, the memory 480 may store the output of the receiver unit 420 and/or the post-processing unit 470.

According to an embodiment of the present disclosure, based on the structure of the ISP 320 illustrated in FIG. 4, we will look at the operation of processing artificial intelligence-based images when capturing (e.g., recording) a video in the electronic device 101. see.

Referring to FIG. 8, the full color image output from the AI network 460 of the ISP 320 is displayed at the video resolution (e.g. HD, FHD, UHD, or 8K) set by the user and/or at the camera module 180. It can be output according to AI parameters that automatically change based on situational awareness. According to one embodiment, the output size of the AI network 460 may change depending on the user's video resolution setting and/or AI parameter change. For example, the AI network 460 may scale the output image according to the image size of various scenarios when shooting (e.g., recording) a video. For example, the output of the AI network 460 of the ISP 320 may be changed in response to the CFA type of the image sensor 410 or the sensor output size that varies depending on the lens.

According to one embodiment, the AI network 460 measures measurement data (e.g., exposure value (EV), bright value (BV), and/or zoom ratio) of the measurement unit 430. Using this, you can change the parameters to suit the situation. For example, the AI network 460 can use exposure value (EV) or brightness value (BV) to distinguish the illuminance at which an image is captured, and change parameters to suit the situation of the classified illuminance. According to one embodiment, when the AI network 460 determines that the illumination is low according to a low exposure value (EV), it can use an AI parameter that reduces noise significantly. According to one embodiment, the AI network 460 can automatically switch to AI parameters that enhance details when it is determined to be outdoors based on a high exposure value (EV). According to one embodiment, the AI network 460 processes demosaic based on AI parameters appropriate for the situation and outputs an image with a corresponding resolution (e.g., FHD, UHD, or 8K) (or video output size). You can.

According to one embodiment, the AI network 460 of the ISP 320 processes demosaicing and scaling at once to a size that matches the output size of the selected video shooting scenario (e.g., setting) regardless of the input image size. can do. For example, the AI network 460 performs artificial intelligence-based learning to output 3M full-color images in the case of FHD settings and 9M full-color images in UHD settings, and operates using the learned training data. can do.

Referring to FIG. 9, the AI network 460 of the ISP 320 can switch parameters according to sensors of various CFA patterns (e.g., Bayer, 2x2, 3x3, 4x4, or RGBW). For example, in recent years, sensors with various patterns in addition to sensors with Bayer-type patterns have been developed. According to one embodiment, it may be difficult for a general ISP to support all of the various patterns. The ISP 320 according to an embodiment of the present disclosure is based on artificial intelligence-based operation using the AI network 460 and can flexibly respond to various sensor patterns through various learning. According to one embodiment, the AI network 460 may include its own memory (e.g., network cache 900) that stores various types of parameters so that they can be quickly and easily switched. In one embodiment, network cache 900 may be included within AI network 460 or ISP 320.

According to one embodiment, the network cache 900 may previously cache parameters frequently used when shooting (eg, recording) a video. For example, the AI network 460 loads both first parameters (e.g., parameters for low light) and second parameters (e.g., parameters for outdoor use) into the network cache 900 and then loads them at a high speed according to the measurement data. It can be supplied to the AI network 460. According to one embodiment, the AI network 460 determines (A) video resolution (e.g., FHD, UHD, and 8K) as in the example, (B) image condition (or 3A stats) (e.g., as in the example) : outdoor, indoor, and lowlight), and/or (C) As an example, various parameters corresponding to CFA patterns (e.g., Bayer, 2x2, 3x3, and 4x4) can be cached in the network cache 900, and video When shooting, you can quickly switch to parameters corresponding to measurement data.

According to one embodiment, FIG. 10 may show an example of the computational area of the AI network 460 during digital zoom when shooting (eg, recording) a video.

According to one embodiment, in general, when shooting (eg, recording) a video, functions such as VDIS (video digital image stabilizer) or SAT (scene aligned transform) may be included. In one embodiment, VDIS may represent a method of reducing shaking by changing the crop area of the image according to the shaking of the camera module 180. In one embodiment, SAT may represent a method of switching without a sense of heterogeneity by changing the crop area of the image when switching between camera modules 180 of different angles of view. According to one embodiment, both the VDIS method and the SAT method have something in common in cropping the image, and for this purpose, the output of the AI network 460 is an image margin (image) in the basic video output resolution (e.g., Output Image Size (1010)). The area including margin) (1020) (e.g., Image Margin Area (1030)) can be calculated. According to one embodiment, the size of the output image of the AI network 460 may be determined and output in consideration of the margins of the VDIS and SAT of the camera module 480. For example, it can be output in a size that takes the margin into consideration, such as FHD 2M + 50% margin = 3M. According to one embodiment, an example of output calculated based on the area 1030 including the image margin 1020 is shown in FIGS. 11A to 11C.

According to one embodiment, FIGS. 11A, 11B, and 11C may represent an example of a scaling operation in the decoding (eg, pool) and encoding (eg, up convolution) process of the AI network 460. According to one embodiment, the AI network 460 changes the

output images

1103, 1105, and 1107 to a resolution (e.g., 8K, UHD, FHD) suitable for the current shooting environment (or situation) based on parameter adjustment. can do. According to one embodiment, the AI network 460 can adjust the size of the image based on AI parameter changes and determine the output size according to the user scenario.

According to one embodiment, the AI network 460 may adjust the size of the output image based on an encoder-decoder based model (e.g., UNET) such as an autoencoder (AE). According to one embodiment, the AI network 460 may have a network structure divided into a decoding part (eg, encoder 1110) and an encoding part (eg, decoder 1120). For example, the AI network 460 has an encoder 1110 (or contracting path) and decoder 1120 (or expanding path) structures, and may be symmetrical to each other.

According to one embodiment, the AI network 460 may perform an operation of concatenating features obtained from each layer of the encoding step to each layer of the decoding step. In one embodiment, the direct connection between each layer of the encoder 1110 and each layer of the decoder 1120 may be referred to as a skip connection. According to one embodiment, the AI network 460 arranges the layers so that the neural network structure has skip connections in parallel and the left and right sides are symmetrical with respect to the center, as illustrated in FIGS. 11A, 11B, and 11C, e.g. For example, it may include a U-shaped neural network structure.

According to one embodiment, the part connecting the encoder 1110 and the decoder 1120 is called a bridge 1130. For example, in FIGS. 11A, 11B, and 11C, the bottom portion of the U-shaped structure may represent a bridge 1130. According to one embodiment, a rectangular box in FIGS. 11A, 11B, and 11C may represent a layer block 1140.

According to one embodiment, in the structure of the encoder 1110 and decoder 1120 of FIGS. 11A, 11B, and 11C, the vertical direction of the layer block 1140 represents the dimension of the map, and the layer block 1140 The horizontal direction of can indicate the number of channels. For example, if the vertical direction is 256x256 and the horizontal direction is 128, this may indicate that the image of the corresponding layer has a size of 256x256x128. For example, if the input image is 512x512x3, an image with 3 RGB channels and a size of 512x512x3 can be displayed.

According to one embodiment, the encoder 1110 may down-scale (or down-sample) each layer and proceed to the next step of the encoder 1110 (e.g., in the direction of the arrow below).

According to one embodiment, the decoder 1120 may perform up convolution for each layer and proceed to the next step of the decoder 1120 (eg, in the direction of the arrow above).

According to one embodiment, in the structure of the decoder 1120, the right portion shown for each layer in FIGS. 11A, 11B, and 11C is an example of the

output images

1103, 1105, and 1107 of the AI network 460. can represent. For example, the AI network 460 may provide

output images

1103, 1105, and 1107 from each layer.

According to one embodiment, the image in the middle of encoding (e.g., layer block 1140) and the image in the middle of decoding (e.g., layer block 1140) of the encoder 1110 and decoder 1120 may not be meaningful as images. there is. However, the intermediate images may contain important information regarding the detail and/or resolution of the output image. For example, in the AI network 460, the closer it is to the input end (e.g., the top layer block of the encoder 1110 in FIG. 11A) and the output end (e.g., the top layer block of the decoder 1120 in FIG. 11A), the better the image (e.g., Although the size of the label block is large and can contain a lot of information, current consumption due to image processing is high and the amount of computation can increase. Therefore, as in the example of Figure 11b or Figure 11c, even if only one layer of the output stage of the decoder 1120 is removed, it can have a significant effect in terms of current consumption and computation time, and the effect of resizing the image with a minimum reduction in resolution can be achieved. You can get it.

The AI network 460 according to an embodiment of the present disclosure may have an output terminal capable of outputting an image for each layer (e.g., supporting image output for each layer), the user's video resolution setting, and/or a camera module. The output location (e.g. layer) can be adaptively changed according to the situation measured in (180). For example, the AI network 460 may use the first layer in FIG. 11A, the second layer in FIG. 11B, or the second layer in FIG. 11C based on parameters related to the input image (e.g., the CFA 1101 of the image sensor 410). The corresponding video (e.g., 8K video (1103), FHD video (1105), or UHD video (1107)) can be output in three layers.

According to one embodiment, the AI network 460 can learn with different input/output sizes even during learning, and by generating and learning a resized reference file using a complex resizing algorithm, AI The resolution of the network 460 can be improved, and computational efficiency and image downscaling can be performed all at once.

According to an embodiment of the present disclosure, a full-color image can be generated by changing the parameters of one fixed network (e.g., AI network 460) and resizing can be performed at the same time, thereby reducing current consumption and computation amount. According to an embodiment of the present disclosure, the AI network 460 can be used even in high-complexity video shooting (e.g., recording) by maximizing image quality due to AI computation and minimizing power consumption by reducing the amount of computation. .

According to one embodiment, the AI network 460 performs demosaicing and scaling at the same time based on learned training data, thereby reducing complexity while maintaining image quality performance and reducing current consumption. there is. According to one embodiment, as the computation on the video is performed based on the AI network 460, there is no need for additional scaling in the computation of the ISP 320 that calculates the video, so the computation steps can be simplified. .

According to one embodiment, the ISP 320 changes the hardware of various CFA patterns (e.g., Bayer, 2x2, 4x4, RGBW, and/or CMYK) of various image sensors 410 through learning of the AI network 460. You can apply without it.

The electronic device 101 according to an embodiment of the present disclosure includes a display (e.g., the display module 160 of FIGS. 1 or 3), a camera (e.g., the camera module 180 of FIGS. 1 to 3), and the A processor operatively connected to a display and the camera (e.g., the processor 120 of FIG. 1 or 3), and an image signal processor (ISP) included in the processor (e.g., the processor 120 of FIG. 3 or 4) It may include an ISP (320).

According to one embodiment, the image signal processor 320 receives input data (e.g., raw data) corresponding to the sensor output size of the sensor through the camera while the user performs video shooting. It can operate to receive. According to one embodiment, the image signal processor 320 may operate to determine parameters corresponding to settings or settings changes in which the video is captured. According to one embodiment, the image signal processor 320 converts the input data into a full color image based on the parameters and changes the output size of the full color image to suit the video output size. Operates to perform processing. According to one embodiment, the image signal processor 320 may operate to output image data according to image processing.

According to one embodiment, the camera 180 has an image sensor (e.g., image sensor 410 in FIG. 4) set to receive the input data through the lens of the camera, and receives the input data from the image sensor. It may include a measurement unit (eg, the measurement unit 430 in FIG. 4) configured to receive and extract measurement data based on the received input data.

According to one embodiment, the measurement unit 430 may operate to perform measurement according to a CFA (color filter array) pattern of the image sensor and determine the movement of the image through at least one frame from the image sensor. there is.

According to one embodiment, the image signal processor 320 includes an alignment unit (e.g., alignment unit 440 in FIG. 4) set to match input data from the image sensor and measurement data from the measurement unit, and an alignment unit 440 of FIG. 4 of the input data. A preprocessor (e.g., the preprocessor 450 in FIG. 4) that processes arithmetic operations for each pixel, converts the input data into the full color image based on artificial intelligence (AI) while shooting the video, and An AI network (e.g., AI network 460 in FIG. 4) that processes scaling (e.g., variable output size) corresponding to the video output size, and a full-color image output from the AI network using the measurement data. It may include a post-processing unit (e.g., the post-processing unit 470 of FIG. 4) that performs real-time tone correction (tone mapping).

According to one embodiment, while performing the video capture, the image signal processor 320 detects a change in the video mode currently set for video capture or a video mode set during video capture, and uses AI in response to the detected result. The network may operate to determine learned parameters.

According to one embodiment, the image signal processor 320 detects a video mode automatically changed by the camera while performing the video shooting, and determines parameters learned from the AI network in response to the detected result. It can operate to do so.

According to one embodiment, the image signal processor 320 determines the parameter based on at least one element of the CFA pattern of the image sensor, lens characteristics, measurement data of the measurement unit, sensor output size, and/or video output size. It can operate to do so.

According to one embodiment, the parameters can be changed in real time.

According to one embodiment, the image signal processor 320 may operate to change parameters for the AI network to parameters with different input/output sizes in real time in response to the settings or changes to the settings.

According to one embodiment, the image signal processor 320 processes demosaic to convert the input data into a full-color image based on parameters determined through an AI network and output size of the full-color image. may be operated to process variable scaling to match the current video output size.

According to one embodiment, the parameters may include at least one of network layer information, weight parameters, and/or bias parameters.

An image processing device for an electronic device 101 according to an embodiment of the present disclosure includes a camera (e.g., the camera module 180 of FIGS. 1 to 3) and an image signal processor (ISP) operatively connected to the camera. image signal processor) (e.g., ISP 320 of FIG. 3 or 4), and a memory operatively connected to the image signal processor (e.g., memory 130 of FIG. 1 or FIG. 3 or memory 480 of FIG. 4) ))) may be included.

According to one embodiment, the

memory

130, 480, when executed, the image signal processor 320, while performing video capture by the user, corresponds to the sensor output size of the sensor through the camera. Instructions for receiving input data (e.g. raw data) can be stored. According to one embodiment, the

memories

130 and 480 may store instructions that, when executed, allow the image signal processor 320 to determine parameters corresponding to settings or settings changes in which the video is captured. . According to one embodiment, when the

memories

130 and 480 are executed, the image signal processor 320 converts the input data into a full color image based on the parameters and displays the full color image. Instructions that perform image processing to change the output size of the image to match the video output size can be stored. According to one embodiment, the

memories

130 and 480 may store instructions that, when executed, cause the image signal processor 320 to output image data according to image processing.

Hereinafter, methods of operating the electronic device 101 in various embodiments will be described in detail. Operations performed in the electronic device 101 according to various embodiments include a processor 120 including various processing circuitry and/or executable program elements of the electronic device 101. It can be executed by (e.g. ISP 320). According to one embodiment, operations performed by the electronic device 101 are stored in the memory 130 and, when executed, include instructions that cause the processor 120 (e.g., ISP 320) to operate. It can be executed by .

According to one embodiment, FIG. 12 may show an example of supporting image processing of a video using an AI method when capturing (e.g., recording) a video in the electronic device 101 according to an embodiment.

In the electronic device 101 according to an embodiment of the present disclosure, an image processing method for a video may be performed, for example, according to the flowchart shown in FIG. 12. The flowchart shown in FIG. 12 is merely a flowchart according to an embodiment of image processing of the electronic device 101, and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some of the operations. Other operations may be performed complementary to at least some operations. According to an embodiment of the present disclosure, operations 1201 to 1211 may be performed by at least one processor 120 (eg, ISP 320) of the electronic device 101.

According to one embodiment, the operation described in FIG. 12 is, for example, performed heuristically in combination with the operations described in FIGS. 3 to 11, or is performed as a detailed operation of some of the described operations. It can be performed heuristically.

As shown in FIG. 12, an operation method performed by the electronic device 101 according to an embodiment (e.g., an operation method for supporting image processing of a video) includes an operation 1201 of performing video capture, video An operation 1203 of receiving raw data through the camera module 180 while shooting, an operation 1205 of identifying a setting or a setting change in which a video is captured, and an operation of determining parameters corresponding to the setting or setting change. It may include (1207), an operation (1209) of performing image processing on raw data based on parameters, and an operation (1211) of outputting image data according to image processing.

Referring to FIG. 12 , in operation 1201, the processor 120 (eg, ISP 320) of the electronic device 101 may perform an operation of capturing a video. According to one embodiment, the processor 120 may execute an application related to video shooting based on user input. According to one embodiment, the processor 120 may execute (or activate) the camera module 180 based on application execution. According to one embodiment, the processor 120 may control the display module 160 to display an image (eg, a preview image) acquired through the camera module 180. According to one embodiment, the processor 120 may start video capture based on detecting a user input that performs video capture (eg, recording).

In operation 1203, the processor 120 may receive raw data through the camera module 180 while capturing a video. According to one embodiment, the ISP 320 of the processor 120 may receive unprocessed raw data transmitted from the camera module 180 (e.g., the image sensor 410) while performing video shooting. . According to one embodiment, the image sensor 410 acquires CFA data (or CFA image) according to the CFA pattern of the image sensor 410, and transmits the acquired CFA data to the ISP 320 (e.g. : Can be transmitted to the alignment unit 440 of the ISP 320. According to one embodiment, the measurement unit 430 may extract measurement data based on raw data and transmit the measurement data to the ISP 320. In one embodiment, the measurement data may include motion data and 3A (eg, Auto exposure, Auto white balance, Auto focus) data. According to one embodiment, the measurement unit 430 may determine the movement of the image based on one or several frames (eg, N frames) input from the image sensor 430. According to one embodiment, the ISP 320 may acquire raw data of the image sensor 410 and measurement data of the measurement unit 430, respectively.

In operation 1205, the processor 120 may perform an operation to identify settings in which a video is captured or settings change. According to one embodiment, the ISP 320 of the processor 120 may change the video mode (e.g., video resolution) currently set for video capture (e.g., HD, FHD, UHD, or 8K) or the video mode set during video capture. can be detected. According to one embodiment, the ISP 320 may detect a video mode automatically changed by the camera module 180.

In operation 1207, the processor 120 may perform an operation of determining a parameter corresponding to a setting or setting change. According to one embodiment, the ISP 320 of the processor 120 includes the CFA pattern of the image sensor 410, lens characteristics (e.g., micro lens), measurement data (e.g., motion data) of the measurement unit 430, and sensor output. The parameter may be determined based on at least one factor of size and/or video output size. For example, the ISP 320 may switch parameters to be used in the AI network 460 of the ISP 320 in real time in response to setting or changing settings. For example, the ISP 320 can change the parameters for the AI network 460 to parameters with different input/output sizes in real time based on the measurement data of the measurement unit 430. For example, the ISP 320 may change the input/output size to a different parameter in real time in response to the user's change in video mode while shooting a video.

In operation 1209, the processor 120 may process an image for raw data based on parameters. According to one embodiment, the ISP 320 of the processor 120 may simultaneously process demosaicing (e.g., full color image) and resizing of raw data through the AI network 460 based on parameters. According to one embodiment, the ISP 320 performs demosaicing of raw data (e.g., CFA image) in full color through the AI network 460, and uses learned parameters corresponding to settings or settings changes (e.g., AI parameters), one full-color image can be output using at least one image (eg, one image or multiple frames) matched by the measurement unit 430 and the alignment unit 440. According to one embodiment, the ISP 320 changes network layer information, weight parameters, and/or bias parameters through the AI network 460 to match settings or change settings, and outputs correspondingly (e.g., video output size). It can be resized and printed.

In operation 1211, the processor 120 may perform an operation of outputting image data (eg, output data) according to image processing. According to one embodiment, the ISP 320 of the processor 120 may output image data to the memory 130 (eg, the memory 480 of FIG. 4) and/or the display module 160.

An operation method performed by the electronic device 101 according to an embodiment of the present disclosure includes using an image signal processor (ISP) (e.g., the ISP of FIG. 3 or 4) while a user is capturing a video. (320)) includes an operation of receiving input data (e.g., raw data) corresponding to the sensor output size of the sensor through a camera (e.g., the camera module 180 in FIGS. 1 to 3). can do. According to one embodiment, the operating method may include determining a parameter corresponding to a setting or setting change in which the video is captured. According to one embodiment, the operating method performs image processing to convert the input data into a full color image based on the parameters and vary the output size of the full color image to match the video output size. Can include actions. According to one embodiment, the operating method may include outputting image data according to image processing.

According to one embodiment, the operation of receiving the input data includes receiving input data from an image sensor of the camera and receiving measurement data extracted based on the input data by a measurement unit of the camera. can do.

According to one embodiment, the operation of performing the image processing may include an operation of matching input data of the image sensor and measurement data of the measurement unit by the image signal processor (eg, alignment unit). According to one embodiment, the operation of performing the image processing may include processing an arithmetic operation for each pixel of the input data by the image signal processor (eg, a preprocessor). According to one embodiment, the operation of performing the image processing is to process the input data based on artificial intelligence (AI) while performing the video capture by the image signal processor (e.g., AI network). It may include converting a full color image and scaling it to correspond to the video output size. According to one embodiment, the operation of performing the image processing is to perform real-time tone correction (tone correction) on the full color image output from the AI network by the image signal processor (e.g., post-processing unit) using the measurement data. It may include an operation to perform mapping).

According to one embodiment, the operation of determining the parameter includes, while performing the video capture, an operation of detecting a change in the video mode currently set for video capture or a video mode set during video capture, and AI in response to the detected result. It may include an operation to determine parameters learned in the network.

According to one embodiment, the operation of determining the parameter includes detecting a video mode automatically changed by the camera while shooting the video, and determining a parameter learned from an AI network in response to the detected result. It may include actions such as:

According to one embodiment, the operation of determining the parameter includes determining the parameter based on at least one element of the CFA pattern of the image sensor, lens characteristics, measurement data of the measurement unit, sensor output size, and/or video output size. Can include actions.

According to one embodiment, the operation of determining the parameter may include an operation of changing the parameter for the AI network to a parameter with a different input/output size in real time in response to the setting or change in the setting.

According to one embodiment, the operation of determining the parameter includes processing demosaic to convert the input data into a full color image based on parameters determined through an AI network, and processing the demosaic of the full color image. It may include an operation to process scaling to change the output size to match the current video output size.

According to one embodiment, the parameters can be changed in real time.

The various embodiments of the present disclosure disclosed in the specification and drawings are merely provided as specific examples to easily explain the technical content of the present disclosure and aid understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be construed as including all changes or modified forms derived based on the technical idea of the present disclosure in addition to the embodiments disclosed herein.

Claims

In electronic devices,

display;

camera;

a processor operatively connected to the display and the camera; and

An image signal processor (ISP, image signal processor) included in the processor, wherein the image signal processor includes,

While performing video capture by the user, receive input data corresponding to the sensor output size of the sensor through the camera,

Determine parameters corresponding to settings or settings changes in which the video is captured,

Based on the parameters, perform image processing to convert the input data into a full color image and vary the output size of the full color image to match the video output size, and

An electronic device set to output image data according to image processing.
The method of claim 1, wherein the camera:

an image sensor configured to receive the input data through a lens of the camera; and

A measurement unit configured to receive the input data from the image sensor and extract measurement data based on the received input data,

The measurement unit is configured to perform measurement according to a color filter array (CFA) pattern of the image sensor and determine movement of an image through at least one frame from the image sensor.
The method of claim 2, wherein the image signal processor,

an alignment unit configured to match input data from the image sensor and measurement data from the measurement unit;

a preprocessor that processes arithmetic operations for each pixel of the input data;

An AI network that converts the input data into the full color image and performs scaling corresponding to the video output size based on artificial intelligence (AI) while capturing the video; and

An electronic device including a post-processing unit that performs real-time tone mapping on a full-color image output from the AI network using the measurement data.
The method of claim 1, wherein the image signal processor:

While performing the video capture, an electronic device configured to detect a change in the video mode currently set for video capture or a video mode set during video capture, and determine parameters learned from the AI network in response to the detected results.
The method of claim 1, wherein the image signal processor:

An electronic device configured to detect a video mode automatically changed by the camera while capturing the video, and determine parameters learned from an AI network in response to the detected result.
The method of claim 1, wherein the image signal processor:

An electronic device configured to determine a parameter based on at least one element of an image sensor's CFA pattern, lens characteristics, measurement data of a measurement unit, sensor output size, and/or video output size.
According to paragraph 1,

An electronic device in which the parameters can be changed in real time.
The method of claim 1, wherein the image signal processor:

An electronic device configured to change parameters for an AI network in real time to parameters with different input/output sizes in response to the settings or changes to the settings.
The method of claim 1, wherein the image signal processor:

Based on parameters determined through an AI network, demosaic processing is performed to convert the input data into a full-color image, and scaling is performed to vary the output size of the full-color image to match the current video output size. An electronic device configured to do so.
According to paragraph 1,

The electronic device wherein the parameters include at least one of network layer information, weight parameters, and/or bias parameters.
In an image processing device of an electronic device,

camera;

an image signal processor (ISP) operatively connected to the camera; and

comprising a memory operatively coupled to the image signal processor;

The memory may, when executed, cause the image signal processor to:

While performing video capture by the user, receive input data corresponding to the sensor output size of the sensor through the camera,

Determine parameters corresponding to settings or settings changes in which the video is captured,

Based on the parameters, perform image processing to convert the input data into a full color image and vary the output size of the full color image to match the video output size, and

An image processing device that stores instructions that output image data according to image processing.
In a method of operating an electronic device,

An operation of an image signal processor (ISP) receiving input data corresponding to the sensor output size of a sensor through a camera while a user performs video shooting;

An operation of determining a parameter corresponding to a setting or setting change in which the video is captured;

converting the input data into a full color image based on the parameters and performing image processing to vary the output size of the full color image to match the video output size; and

A method including the operation of outputting image data according to image processing.
The method of claim 12, wherein the operation of receiving the input data includes:

An operation of receiving input data from an image sensor of the camera,

A method comprising receiving measurement data extracted based on the input data by a measurement unit of the camera.
The method of claim 13, wherein the operation of performing the image processing includes:

An operation of matching the input data and the measurement data by the image signal processor,

An operation of processing an arithmetic operation for each pixel of the input data,

An operation of converting the input data into the full color image and processing scaling corresponding to the video output size based on artificial intelligence (AI) while shooting the video, and

A method comprising performing real-time tone mapping on the full color image using the measurement data.
According to clause 12,

The method is capable of changing in real time and includes at least one of network layer information, weight parameters, and/or bias parameters.