KR20230095782A - Electronic device and method for generating high-definition image - Google Patents

Abstract

An electronic device and its operating method are disclosed. An electronic device according to the technical idea of the present disclosure may generate first image data by receiving light from the outside, generate second image data based on the first image data, and generate the first image data and the second image data. An image sensor outputting image data, and a resolution of at least one region of an image corresponding to the first image data is increased using a machine learning algorithm, and at least a portion of the first image data having an increased resolution and the second image It may include an application processor that outputs third image data including at least a portion of the data.

Description

Electronic device and high-definition image generation method {ELECTRONIC DEVICE AND METHOD FOR GENERATING HIGH-DEFINITION IMAGE}

The technical spirit of the present disclosure relates to an electronic device, and more particularly, to a method for generating a high-definition image using an electronic device.

An image sensor is a device that captures a two-dimensional or three-dimensional image of an object. An image sensor generates an image of an object using a photoelectric conversion element that reacts according to the intensity of light reflected from the object. With the recent development of CMOS (Complementary Metal-Oxide Semiconductor) technology, CMOS image sensors using CMOS are widely used. The image sensor may include a pixel array, and as the image sensor is implemented with high pixels, the pixel array may include a large number of color pixels. Remosaic processing based on interpolation and/or extrapolation may be performed to convert the raw image output from the image sensor into a predetermined pattern such as an RGB image.

The image sensor may use a merged pixel and an ISP calculation function to generate a high-definition image. An electronic device may generate a high-definition image using merged pixels such as tetra cells and Qcells. ISP calculation functions that can be performed by the image sensor may include functions such as interpolation, auto white balance adjustment, color correction, and gamma correction.

Meanwhile, in order for an image sensor to generate a complex image (eg, an image with many edges) as a high-definition image, the image sensor needs to process a lot of information. As one of the methods for processing such a lot of information, the electronic device can generate a complex image with high quality using machine learning. The electronic device may generate a high-definition image using, for example, an artificial neural network. Artificial neural networks are a subset of machine learning and are the core of machine learning algorithms that include one input layer, one or more hidden layers and one output layer. Machine learning for generating a high-definition image may be performed in an image sensor or an application processor.

When machine learning is performed on an image sensor to generate a high-definition image, there is a problem in that image generation takes too long. In addition, there is a limit to generating a high-definition image due to a lack of resources possessed by the image sensor.

The technical idea of the present disclosure may provide a method and electronic device for generating a high-definition image by performing machine learning in an application processor in order to reduce the time required to generate the image.

In addition, the technical idea of the present disclosure may provide an electronic device that performs machine learning in an application processor to secure resources of an image sensor and a method for generating a high-definition image.

An electronic device for generating a high-definition image according to the technical idea of the present disclosure may generate first image data by receiving light from the outside, generate second image data based on the first image data, and generate the first image data. An image sensor outputting data and the second image data, and a resolution of at least one region of the image corresponding to the first image data is increased using a machine learning algorithm, and at least a portion of the first image data having the increased resolution and an application processor outputting third image data including at least a portion of the second image data.

In addition, a method for generating a high-definition image of an electronic device including an image sensor and an application processor according to the technical idea of the present disclosure includes generating first image data by receiving light from the outside, and generating a first image data based on the first image data. generating 2 image data, outputting the first image data and the second image data, and increasing a resolution of at least one region of an image corresponding to the first image data using a machine learning algorithm; and outputting third image data including at least a portion of the first image data having an increased resolution and at least a portion of the second image data.

In addition, the recording medium of the high-definition image generation method according to the technical idea of the present disclosure includes an operation of generating first image data by receiving light from the outside, an operation of generating second image data based on the first image data, The operation of outputting the first image data and the second image data, and increasing the resolution of at least one region of the image corresponding to the first image data using a machine learning algorithm, and the first image data having an increased resolution and outputting third image data including at least a portion of and at least a portion of the second image data.

According to the technical concept of the present disclosure, an application processor performs a part of the process of creating immortality, thereby securing resources of an image sensor and improving system performance.

1 is a diagram for explaining a high-definition image generating system according to an embodiment of the present disclosure.
2 is a diagram for explaining an operation of generating a high-definition image between sensor hardware and an application processor according to an embodiment of the present disclosure.
3A and 3B are diagrams for explaining a method of performing a high-definition image generation operation by sensor hardware according to an embodiment of the present disclosure.
FIG. 4 is a diagram for explaining in detail another embodiment of the high-definition image generating operation shown in FIG. 3A.
5 is a flowchart of a method for generating a high-definition image by an electronic device according to an embodiment of the present disclosure.
6 is a diagram for explaining an operation of increasing the resolution of an image by using machine learning by an electronic device according to an embodiment of the present disclosure.
7 is a diagram for explaining an operation of generating first image data and second image data by an electronic device according to an embodiment of the present disclosure.
8 is a block diagram of an electronic device including a multi-camera module according to an embodiment of the present disclosure. 9 is a detailed block diagram of the camera module of FIG. 8 .

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a high-definition image generating system according to an embodiment of the present disclosure.

Referring to FIG. 1 , an electronic device 100 may include an image sensor 110 and an application processor 120 . The image sensor 110 may include a pixel array including a plurality of two-dimensionally arranged pixels and a readout circuit, and the pixel array may convert received optical signals into electrical signals. The pixel array may be implemented with photoelectric conversion elements such as charge coupled devices (CCDs) or complementary metal oxide semiconductors (CMOS), and may be implemented with various types of photoelectric conversion elements. Hereinafter, the pixel array included in the image sensor 110 will be described as meaning CMOS, but the pixel array used to implement the present disclosure is not limited thereto. The image sensor 110 may be implemented as a semiconductor chip or package including a pixel array and a readout circuit.

According to one example, the image sensor 110 may include a pixel array, and a color filter array (CFA) may be disposed in the pixel array so that a color of a predetermined component is sensed for each pixel. Hereinafter, in describing the embodiments of the present disclosure, the terms of color filter, color pixel, filter array, and pixel array may be defined in various ways, and, for example, the color filter array is disposed on a pixel array including a light sensing element. may be defined as a separate component, or the color filter array may be defined as a component included in the pixel array. Each color pixel may be defined as including a corresponding color filter. In addition, in describing the following embodiments, each of the color filter array cell and the CFA block may be defined as including the above-described color pixel.

The color filter array may include a plurality of CFA blocks repeatedly disposed along horizontal and vertical directions, and each CFA block may include color pixels having a predetermined size. According to one example, CFA blocks may be arranged according to a pattern. According to one example, the CFA blocks may be arranged according to a Bayer pattern. For example, in a 2x2 pattern, CFA blocks may be arranged such that the upper left side corresponds to blue, the lower left and upper right portions correspond to green, and the lower right side corresponds to red. According to another example, the CFA block has various patterns such as a Tetra pattern in which one color unit is arranged in 2x2, a Nona pattern in which one color unit is arranged in 3x3, and a Hexadeca pattern in which one color unit is arranged in 4x4 can be arranged according to Hereinafter, for convenience of description, it is assumed that the CFA blocks included in the color filter array correspond to the Bayer pattern, but it will be fully understood that the spirit of the present disclosure is not limited thereto.

According to one embodiment, the image sensor 110 may be part of sensor hardware. The sensor hardware may perform a re-mosaic processing operation on image data received from the image sensor 110 or an image processing operation including a demosaicing operation.

The application processor 120 may perform pre-processing such as crosstalk correction and despeckle operations on image data, and further perform post-processing such as sharpening operations on full image data. can also be done Also, for example, the application processor 120 may perform auto dark level compensation (ADLC), bad pixel correction, lens shading correction, gamma correction, Image signal processing may be performed to reduce noise and improve image quality, such as color filter array interpolation, color correction, and color enhancement. In addition, the application processor 120 may generate an image file by compressing image data generated by image signal processing for image quality improvement, or may restore image data from the image file. The application processor 120 may perform machine learning (eg, deep learning) for image quality improvement. An operation of the application processor 120 improving image quality using machine learning will be described later with reference to FIGS. 3A and 3B. Configurations of the application processor 120 and modules described below may be software blocks executed by a predetermined processor or may be implemented as a combination of dedicated hardware blocks and processing units.

The application processor 120 may convert the format of image data by performing an image processing operation. The application processor 120 may convert image data corresponding to a color filter pattern such as a Bayer pattern, a tetra pattern, or a hexadeca pattern into full color image data in an RGB format.

Input image data including color information may be input to the application processor 120 . The input image data may be image data received from the image sensor 110 or may be corrected data after the image data is received by the application processor 120 .

The application processor 120 may store an image output to the preview without a separate command for real-time operation (zero shutter lag) of the camera. Shutter lag refers to a time delay from when a shutter is triggered when a user presses a photographing button of a camera until a picture is actually recorded. The application processor 120 may store an image output to the camera preview without a separate command in order to minimize such a time delay. When the shutter is triggered, the application processor 120 may check a frame of time when the shutter is actually triggered, and record pictures and images so that no time delay occurs.

For example, when a user executes a camera function and a preview screen is output to the display from the first time, the application processor 120 may store an image of the preview screen from the first time. When the user presses the camera shutter at the second time, shooting may actually start from the third time due to shutter lag.

2 is a diagram for explaining an operation of generating a high-definition image between sensor hardware and an application processor according to an embodiment of the present disclosure.

The sensor hardware 210 may include an image sensor 212 and a first ISP block 214 . The image sensor 212 may perform the operation of the image sensor 110 described above with reference to FIG. 1 . For example, the image sensor 212 may generate image data by receiving light from the outside. Hereinafter, image data is described as meaning image data for convenience, but data to which the present disclosure can be applied is not limited to image data and may include various types of data such as image data.

The sensor hardware 210 may generate first image data d_1 and second image data d_2. For example, the first image data d_1 may be raw data generated by receiving light from the outside, and the second image data d_2 may be Bayer data converted to generate an image with raw data. (Bayer data). According to an embodiment, the sensor hardware 210 generates first image data d_1 using light received from the outside and converts a format of the generated first image data d_1 to form second image data ( d_2) can be created. The first image data d_1 and the second image data d_2 may be data used by the sensor hardware 210 to generate images or images. The raw data may include, but is not limited to, data composed of cells such as tetra cells arranged in 2x2, nona cells arranged in 3x3, hexadeca cells arranged in 4x4, and Q cells arranged in one color unit. Bayer data is data in which a pattern of color pixels appears as a Bayer pattern, and is data calculated from raw data to generate an image.

The image sensor 212 may transmit the first image data d_1 to the first ISP block 214 in the sensor hardware 210 . The first ISP block 214 is a block for preprocessing in the sensor, and may perform an ISP operation for generating the second image data d_2. For example, when the image sensor 212 transmits the first image data d_1 to the first ISP block 214, the first ISP block 214 receives the first image data d_1 and performs an ISP operation ( Example: Remosaic) may be performed to generate the second image data d_2. The first ISP block 214 may transmit the generated second image data d_2 to the second ISP block 222 .

The second image data d_2 may be used to perform auto focus (AF), auto exposure (AE), and auto white balance (AWB) functions. The second image data d_2 transmitted from the second ISP block 222 to the first ISP block 214 may be used to perform AE, AF, and AWB functions. The first ISP block 214 may change the received first image data d_1 into second image data d_2 so that AE, AF, and AWB functions may be performed.

The AP 220 may include a second ISP block 222 , a target region selection block 224 , and an image quality improvement block 226 . The target region selection block 224 may receive first image data d_1 from the image sensor 212 and detect a target region to be increased in resolution in the first image corresponding to the first image data d_1. . The target region selection block 224 may detect a target region according to various criteria. According to an embodiment, the target region selection block 224 selects the corresponding region as the target region when an edge included in at least one region of the first image corresponding to the received first image data d_1 satisfies a specific condition. can be selected as An edge may refer to a boundary line between areas where brightness rapidly changes in a video or image. For example, the target region selection block 224 may check the brightness value of each region of the image corresponding to the received data. In the checked brightness values for each area, if the difference between the brightness values between the two adjacent areas is equal to or greater than a predetermined value, the boundary between the two adjacent areas may be determined as an edge. The target region selection block 224 may determine the corresponding region as the target region when the number of edges distributed per unit area in the first image corresponding to the received first image data d_1 is greater than a predetermined criterion. According to an embodiment, the target region selection block 224 generates an ambiguity map based on the first image data d_1, and uses the ambiguity map to generate a second image corresponding to the first image data d_1. A target region may be determined in one image. According to an embodiment, the target region selection block 224 may determine a plurality of regions as target regions when there are a plurality of regions in the image corresponding to the received data that satisfy a predetermined criterion. Also, the target area may be the same area in the first image corresponding to the first image data d_1 and the second image corresponding to the second image data d_2. The target region selection block 224 may generate target region information on at least one target region generated by the method described above. The target region selection block 224 may transfer the first image data d_1 and the image quality improvement information d_11 including target region information to the image quality improvement block 226 .

According to an embodiment, the image sensor 210 may generate first image data d_1 based on target area information. The image sensor 210 may transmit the second image data d_2 to the application processor 220 . The application processor 220 may generate target area information based on the received second image data d_2. The application processor 220 may transmit the generated target area information to the image sensor 210 . The image sensor 210 may generate first image data d_1 including only a partial area (eg, a cropped area) rather than the entire area of the camera preview. The image sensor 210 may generate first image data d_1 based on target area information. For example, the image sensor may generate the first image data d_1 such that the first image corresponding to the first image data d_1 includes only the target area. The image sensor 210 may transmit the first image data d_1 generated so that the first image includes only the target region to the application processor 220 . When the first image data d_1 is data corresponding to the target area, it is much smaller in size than data including all areas of the preview, enabling fast operation. Even in an embodiment in which the size of the first image data d_1 is small, the second image data d_2 may be data including the entire area of the preview. The application processor 220 may generate third image data d_3 with improved image quality by using the first image data d_1 and the second image data d_2 including data on the target region.

The picture quality improvement block 226 may perform an operation to improve picture quality of the selected target region. The picture quality improvement block 226 may receive first image data d_1 and picture quality improvement information d_11 including target area information from the target area selection block 224 . The picture quality improvement block 226 identifies a target area in the first image corresponding to the first image data d_1 based on target area information, and a first correction image corresponding to the first corrected image in which the picture quality of the target area is improved. Correction image data D_1 may be generated. The first corrected image may be an image obtained by changing only the resolution of the target region in the first image. According to an embodiment, the image quality improvement block 226 may improve the image quality of the target region using machine learning (or deep learning). The picture quality improvement block 226 uses, for example, machine learning algorithms and filtering techniques such as convolutional neural network (CNN), de-blocking filter (DF), sample adaptive offset (SAO), and variational autoencoder (VAE). A machine learning algorithm capable of improving the image quality of the first image target region and used by the image quality improvement block 226 to generate a high-quality image is not limited to the above-mentioned embodiment. The picture quality improvement block 226 may transmit the first corrected image data D_1 corresponding to the first corrected image to the second ISP block 222 .

The second ISP block 222 may receive the first corrected image data D_1 from the picture quality improvement block 226 and receive the second image data d_2 from the first ISP block 214 . The second ISP block 222 is a block that performs an ISP operation within the application processor 220 and may perform a different function from that of the first ISP block 214 within the sensor hardware 210 . The second ISP block 222 may perform an ISP operation to convert the format of the second image data d_2 received from the first ISP block 214 . For example, the second ISP block 222 may convert the second image data d_2 into second corrected image data in RGB format using various interpolation methods (eg, bilinear interpolation, pixel doubling interpolation). The second image data d_2 includes only one color filter (one of red, blue, and green) per pixel, but the second correction image data in RGB format has three colors of red, blue, and green per pixel. All can be included.

The second ISP block 222 may generate a high-definition image using the second corrected image data and the first corrected image data D_1. For example, the second ISP block 222 may generate a third image in which only the target region is replaced with the first corrected image in the second corrected image corresponding to the second corrected image data. The third image data d_3 corresponding to the third image may be data obtained by replacing (or inserting) only a portion corresponding to the target region in the second corrected image data with the first corrected image data D_1.

The second ISP block 222 may determine whether real-time processing is required in the image sensor 212 and retransmit the second image data d_2 to the first ISP block 214 if real-time processing is required. For example, the second ISP block 222 may transmit the second image data d_2 to the first ISP block 214 by determining that real-time processing is required when AE, AF, and AWB are to be executed.

3A and 3B are diagrams for explaining a method of performing a high-definition image generation operation by sensor hardware according to an embodiment of the present disclosure.

The sensor hardware 310 may generate first image data d_1 and second image data d_2. 3A illustrates a process in which the sensor hardware 310 generates first image data d_1 and second image data d_2 and transmits them to an application processor to generate an image with improved quality. The first image data d_1 is raw data generated by the sensor hardware 310 based on light received from the outside, and the second image data d_2 is raw data generated by the sensor hardware 310 based on the first image data d_1. It may be Bayer data transformed using an ISP operation. The sensor hardware 310 may transmit the generated first image data d_1 and second image data d_2 to the application processor. Blocks 320, 330, and 340 of FIG. 3A are all included in the application processor. The application processor may include an ambiguity map block 320, a machine learning block 330, and a refinement block 340.

The ambiguity map block 320 may generate an ambiguity map based on the received first image data d_1. The ambiguity map may be a data file generated by the application processor to detect a region (week point, target region) requiring image quality improvement in image data. The first image data d_1 that the application processor can use to generate the ambiguity map may include tetra cell, nona cell, hexadeca cell, and Q cell data. The ambiguity map block 320 may generate an ambiguity map based on the received data and determine at least one target region for improving image quality by using the generated ambiguity map. The ambiguity map block 320 may generate target region information for at least one determined target region. The ambiguity map block 320 may transmit image quality improvement information d_11 including target region information and the first image data d_1 to the machine learning block 330 .

The machine learning block 330 may receive the first image data d_1 and the image quality improvement information d_11 including target region information from the ambiguity map block 320 . The machine learning block 330 may improve the quality of the first image corresponding to the received first image data d_1. The machine learning block 330 may improve the quality of at least one target region of the first image by using machine learning. For example, the machine learning block 330 may use a machine learning algorithm such as a convolutional neural network (CNN), a de-blocking filter (DF), a sample adaptive offset (SAO), or a variational autoencoder (VAE) to improve image quality. there is. The machine learning block 330 may generate first corrected image data D_1 by applying a machine learning algorithm to the first image data d_1. The first corrected image corresponding to the first corrected image data D_1 may be an image obtained by increasing the resolution of the target region in the first image. The machine learning block 330 may transmit the generated first corrected image data D_1 to the quality improvement block 340 .

The picture quality improvement block 340 may receive the first corrected image data D_1 from the machine learning block 330 and receive the second image data d_2 from the sensor hardware 310 . The picture quality improvement block 340 may generate second corrected image data by transforming the format of the second image data d_2 from Bayer data to RGB. Unlike Bayer data, an RGB file can contain all three colors of red, green, and blue in one pixel. The picture quality improvement block 340 may generate third image data d_3 based on the second corrected image data and the first corrected image data D_1. According to an embodiment, the picture quality improvement block 340 may generate third image data d_3 by replacing at least a portion of the second corrected image data with a portion of the first corrected image data D_1. For example, the picture quality improvement block 340 replaces data corresponding to the target region in the second corrected image data with data corresponding to the target region in the first corrected image data D_1 to obtain third image data d_3. can create Since the first corrected image data D_1 is data corresponding to an image of which the image quality of the target region is improved, the image quality of the target region may be improved in the third image corresponding to the third image data d_3.

FIG. 3B shows a still image 350 and a target region 352 of an image to be image quality improved. The ambiguity map block 320 can determine a target region 352 . For example, the ambiguity map block 320 may determine, as the target region 352, a region having the most edges in which the brightness rapidly changes in the image. Thereafter, the machine learning block 330 and the image quality improvement block 340 may increase the resolution of the target region 352 using machine learning and combine with Bayer data to generate a high-definition image.

FIG. 4 is a diagram for explaining in detail another embodiment of the high-definition image generating operation shown in FIG. 3A.

Referring to FIG. 4 , the sensor hardware 310 may generate a high-quality zoom image. For example, when a user using an electronic device executes a camera app and executes a zoom function while a photographed screen is displayed on a display, the sensor hardware may generate a high-definition zoom image using the algorithm of the present disclosure. Hereinafter, contents overlapping those described with reference to FIGS. 1 to 3B will be omitted, and an embodiment of generating a high-definition zoom image will be described.

The sensor hardware 310 may capture real-time images using merged cells (eg, tetra cells, nona cells, hexadeca cells, and Q cells). Referring to FIG. 4 , the sensor hardware 310 may capture real-time images using hexadeca cells in which color pixels are arranged in 4x4 bundles. For example, the first object 402 located in the first frame 400 may be photographed in real time.

The sensor hardware 310 may perform a zoom function of enlarging an area within the first frame 400 based on a user input. A zoomed region of interest (ROI) in the first frame 400 may be positioned inside the second frame 410, and the first object 412 is magnified and displayed inside the second frame 410. It can be. The sensor hardware 310 may configure the third frame 420 by rearranging color pixels inside the second frame 410 by performing an operation such as remosaicing. The sensor hardware 310 uses merged cells in the first frame 400 and the second frame 410 , but may perform remosaicing to increase resolution in the third frame 420 .

The application processor may perform machine learning to increase the resolution of the zoomed object 412 based on a user input. The application processor may generate an ambiguity map based on data corresponding to the image included in the second frame 410, and based on data corresponding to the image included in the remosaic third frame 420, the ambiguity map may be generated. can do running. Data corresponding to the image included in the second frame 410 of FIG. 4 and data corresponding to the image included in the third frame 420 are the first correction image data and the second correction image data in the embodiment of FIG. 3 , respectively. It may correspond to image data.

5 is a flowchart of a method for generating a high-definition image by an electronic device according to an embodiment of the present disclosure.

In operation 502, the image sensor may generate first image data by receiving light from the outside. The pixel array included in the image sensor may generate first image data by converting the received optical signal into an electrical signal. For example, the image sensor may capture a first image and generate first image data corresponding to the first image. The first image data may be raw data. The raw data may include data composed of cells such as tetra cells arranged in 2x2, nona cells arranged in 3x3, hexadeca cells arranged in 4x4, and Q cells arranged in one color unit.

In operation 504, the sensor hardware may generate second image data based on the first image data. The second image data may be Bayer data obtained by converting raw data to generate an image. The Bayer data may be data having a pattern in which color pixel arrays of the image sensor are arranged in a 2x2 shape. The sensor hardware may perform an ISP operation for generating second image data based on the first image data.

In step 506, the image sensor may output first image data and second image data and transmit them to the application processor. In this case, the second image data may be directly transmitted to a block performing an ISP operation within the application processor, but the first image data may be transmitted to a block performing an image quality improvement operation.

In operation 510, the application processor may determine whether real-time processing of the second image data is required. For example, the application processor may determine whether to perform auto focus (AF), auto exposure (AE), and auto white balance (AWB) functions in the current image sensor. When functions such as AF, AE, and AWB need to be performed, the application processor may directly retransmit the second image data to the image sensor without performing an operation to improve the quality of the received second image data. When functions such as AF, AE, and AWB are not performed, the application processor may perform a picture quality improvement operation using the received first image data and second image data.

In step 512, the application processor may increase the resolution of the first image data using machine learning. The application processor may determine the target area using the received first image data. For example, the target area may be determined as an area including more edges than a predetermined criterion, but the criterion for determining the target area is not limited thereto. The application processor may improve image quality by performing machine learning on data corresponding to the determined target region.

The application processor may generate first corrected image data obtained by improving the image quality of a target region in the first image data and second corrected image data obtained by converting a file format of the second image data into RGB.

In step 514, the application processor may generate third image data using the first corrected image data and the second corrected image data. The application processor may generate third image data obtained by changing data corresponding to the target region of the second corrected image into data corresponding to the target region of the first corrected image.

6 is a diagram for explaining an operation of increasing the resolution of an image by using machine learning by an electronic device according to an embodiment of the present disclosure.

The embodiment shown in FIG. 6 may be an embodiment in which step 512 shown in FIG. 5 is embodied.

In step 602, the application processor may receive first image data. The first image data is raw data generated by the image sensor receiving external light.

In step 604, the application processor may determine a target region to increase the resolution of the first image data. The application processor may determine the target area according to various criteria. For example, when the number of edges distributed per unit area of the first image corresponding to the received first image data is greater than a predetermined criterion, the application processor may determine the corresponding region as the target region. According to an embodiment, the application processor may generate an ambiguity map based on the first image data and determine a target region in the first image corresponding to the first image data by using the ambiguity map. When there are a plurality of regions in the image corresponding to the received data that satisfy a predetermined criterion, the application processor may determine the plurality of regions as the target region.

In operation 606, the application processor may increase the resolution of the target region in the first image data by using machine learning. The application processor may generate first correction image data obtained by increasing the resolution of the target region in the first image data. The application processor may generate second corrected image data obtained by converting a file format of the second image data into RGB.

In operation 608, the application processor may generate third image data by inserting data corresponding to the target region in the first corrected image data into data corresponding to the target region in the second corrected image data.

7 is a diagram for explaining an operation of generating first image data and second image data by an electronic device according to an embodiment of the present disclosure.

In operation 702, the image sensor may generate first image data by receiving external light. For example, when a user executes a camera app on an electronic device, an image sensor recognizing an object currently being photographed by a camera converts received light into an electrical signal to generate first image data.

In operation 704, the image sensor may determine a region of interest based on a user input in the first image data. When the user zooms in, the image sensor may determine the zoomed area as the area of interest. For example, when a user touches the display and stretches a finger, the electronic device may recognize it as a zoom operation and determine a region of interest.

In operation 706, the application processor may generate rearranged second image data by remosaicing the pixel array based on the region of interest.

8 is a block diagram of an electronic device including a multi-camera module according to an embodiment of the present disclosure. 9 is a detailed block diagram of the camera module of FIG. 8 .

Referring to FIG. 8 , an electronic device 800 may include a camera module group 900, an application processor 1000, a PMIC 1100, and an external memory 1200.

The camera module group 900 may include a plurality of camera modules 900a, 900b, and 900c. Although the drawing shows an embodiment in which three camera modules 900a, 900b, and 900c are disposed, the embodiments are not limited thereto. In some embodiments, the camera module group 900 may be modified to include only two camera modules. Also, in some embodiments, the camera module group 900 may be modified to include n camera modules (where n is a natural number equal to or greater than 4).

Hereinafter, a detailed configuration of the camera module 900b will be described in more detail with reference to FIG. 9 , but the following description may be equally applied to other camera modules 900a and 900b according to embodiments.

Referring to FIG. 9 , the camera module 900b includes a prism 905, an optical path folding element (hereinafter referred to as “OPFE”) 910, an actuator 930, an image sensing device 940, and storage. A portion 950 may be included.

The prism 905 may include a reflective surface 907 of a light reflective material to transform a path of light L incident from the outside.

In some embodiments, the prism 905 may change the path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism 905 rotates the reflective surface 907 of the light reflecting material in the direction A around the central axis 906 or rotates the central axis 906 in the direction B to move in the first direction X. A path of the incident light L may be changed in a second direction Y, which is perpendicular to the second direction Y. At this time, the OPFE 910 may also move in a third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

In some embodiments, as shown, the maximum angle of rotation of the prism 905 in the A direction may be less than 15 degrees in the plus A direction and greater than 15 degrees in the minus A direction. However, the embodiments are not limited thereto.

In some embodiments, prism 905 can move around 20 degrees, or between 10 and 20 degrees, or between 15 and 20 degrees in the positive or negative B direction, where the angle of movement is positive. It can move at the same angle in the (+) or minus (-) B direction, or it can move to an almost similar angle within the range of 1 degree.

In some embodiments, the prism 905 can move the reflective surface 906 of the light reflective material in a third direction parallel to the direction of extension of the central axis 906 (eg, the Z direction).

The OPFE 910 may include, for example, an optical lens consisting of m (where m is a natural number) number of groups. The m lenses may move in the second direction (Y) to change the optical zoom ratio of the camera module 900b. For example, when the basic optical zoom magnification of the camera module 900b is Z, when m optical lenses included in the OPFE 910 are moved, the optical zoom magnification of the camera module 900b is 3Z or 5Z or It can be changed to an optical zoom magnification of 5Z or higher.

The actuator 930 may move the OPFE 910 or an optical lens (hereinafter referred to as an optical lens) to a specific position. For example, the actuator 930 may adjust the position of the optical lens so that the image sensor 942 is located at the focal length of the optical lens for accurate sensing.

The image sensing device 940 may include an image sensor 942 , a control logic 944 and a memory 946 . The image sensor 942 may sense an image of a sensing target using light L provided through an optical lens. The control logic 944 may control the overall operation of the camera module 900b. For example, the control logic 944 may control the operation of the camera module 900b according to a control signal provided through the control signal line CSLb.

The memory 946 may store information required for operation of the camera module 900b, such as calibration data 947. The calibration data 947 may include information necessary for the camera module 900b to generate image data using light L provided from the outside. The calibration data 947 may include, for example, information about a degree of rotation, information about a focal length, information about an optical axis, and the like, as described above. When the camera module 900b is implemented in the form of a multi-state camera in which the focal length changes according to the position of the optical lens, the calibration data 947 is the focal length value for each position (or state) of the optical lens and It may include information related to auto focusing.

The storage unit 950 may store image data sensed through the image sensor 942 . The storage unit 950 may be disposed outside the image sensing device 940 and may be implemented in a stacked form with a sensor chip constituting the image sensing device 940 . In some embodiments, the storage unit 950 may be implemented as an Electrically Erasable Programmable Read-Only Memory (EEPROM), but the embodiments are not limited thereto.

Referring to FIGS. 8 and 9 together, in some embodiments, each of the plurality of camera modules 900a, 900b, and 900c may include an actuator 930. Accordingly, each of the plurality of camera modules 900a, 900b, and 900c may include the same or different calibration data 947 according to the operation of the actuator 930 included therein.

In some embodiments, one camera module (eg, 900b) of the plurality of camera modules 900a, 900b, 900c is a folded lens including the prism 905 and the OPFE 910 described above. camera module, and the remaining camera modules (for example, 900a and 900b) may be vertical camera modules that do not include the prism 905 and the OPFE 910, but embodiments are limited thereto. it is not going to be

In some embodiments, one camera module (eg, 900c) among the plurality of camera modules 900a, 900b, and 900c extracts depth information using infrared rays (IR), for example. It may be a depth camera of the form. In this case, the application processor 1000 merges image data provided from the depth camera and image data provided from other camera modules (eg, 900a or 900b) to obtain a 3D depth image. can create

In some embodiments, at least two camera modules (eg, 900a, 900b) among the plurality of camera modules 900a, 900b, and 900c may have different fields of view (viewing angles). In this case, for example, optical lenses of at least two of the plurality of camera modules 900a, 900b, and 900c (eg, 900a and 900b) may have different optical lenses, but is not limited thereto.

Also, in some embodiments, each of the plurality of camera modules 900a, 900b, and 900c may have different viewing angles. In this case, optical lenses included in each of the plurality of camera modules 900a, 900b, and 900c may also be different from each other, but are not limited thereto.

In some embodiments, each of the plurality of camera modules 900a, 900b, and 900c may be disposed physically separated from each other. That is, the sensing area of one image sensor 942 is not divided and used by a plurality of camera modules 900a, 900b, and 900c, but an independent image inside each of the plurality of camera modules 900a, 900b, and 900c. A sensor 942 may be placed.

Referring back to FIG. 8 , the application processor 1000 may include an image processing device 1010 , a memory controller 1020 , and an internal memory 1030 . The application processor 1000 may be implemented separately from the plurality of camera modules 900a, 900b, and 900c. For example, the application processor 1000 and the plurality of camera modules 900a, 900b, and 900c may be separately implemented as separate semiconductor chips.

The image processing device 1010 may include a plurality of sub image processors 1012a, 1012b, and 1012c, an image generator 1014, and a camera module controller 1016.

The image processing device 1010 may include a plurality of sub image processors 1012a, 1012b, and 1012c corresponding to the number of the plurality of camera modules 900a, 900b, and 900c.

Image data generated from each of the camera modules 900a, 900b, and 900c may be provided to the corresponding sub image processors 1012a, 1012b, and 1012c through separate image signal lines ISLa, ISLb, and ISLc. For example, image data generated from the camera module 900a is provided to the sub image processor 1012a through the image signal line ISLa, and image data generated from the camera module 900b is provided to the image signal line ISLb. Image data generated from the camera module 900c may be provided to the sub image processor 1012c through the image signal line ISLc. Such image data transmission may be performed using, for example, a Camera Serial Interface (CSI) based on MIPI (Mobile Industry Processor Interface), but embodiments are not limited thereto.

Meanwhile, in some embodiments, one sub image processor may be arranged to correspond to a plurality of camera modules. For example, the sub image processor 1012a and the sub image processor 1012c are not implemented separately from each other as shown, but integrated into one sub image processor, and the camera module 900a and the camera module 900c Image data provided from may be selected through a selection element (eg, multiplexer) and the like, and then provided to the integrated sub image processor.

Image data provided to each of the sub image processors 1012a, 1012b, and 1012c may be provided to the image generator 1014. The image generator 1014 may generate an output image using image data provided from each of the sub image processors 1012a, 1012b, and 1012c according to image generating information or a mode signal.

Specifically, the image generator 1014 merges at least some of the image data generated from the camera modules 900a, 900b, and 900c having different viewing angles according to image generation information or a mode signal to obtain an output image. can create Also, the image generator 1014 may generate an output image by selecting any one of image data generated from the camera modules 900a, 900b, and 900c having different viewing angles according to image generation information or a mode signal. .

In some embodiments, the image creation information may include a zoom signal or zoom factor. Also, in some embodiments, the mode signal may be a signal based on a mode selected by a user, for example.

When the image generation information is a zoom signal (zoom factor) and each of the camera modules 900a, 900b, and 900c has different fields of view (viewing angles), the image generator 1014 operates differently according to the type of zoom signal. can be performed. For example, when the zoom signal is the first signal, after merging the image data output from the camera module 900a and the image data output from the camera module 900c, the merged image signal and the camera module not used for merging An output image may be generated using the image data output from step 900b. If the zoom signal is a second signal different from the first signal, the image generator 1014 does not merge the image data and uses any one of the image data output from each of the camera modules 900a, 900b, and 900c. You can choose to generate an output image. However, the embodiments are not limited thereto, and a method of processing image data may be modified and implemented as needed.

In some embodiments, the image generator 1014 receives a plurality of image data having different exposure times from at least one of the plurality of sub image processors 1012a, 1012b, and 1012c, and performs high dynamic range (HDR) processing on the plurality of image data. ) processing, it is possible to generate merged image data with increased dynamic range.

The camera module controller 1016 may provide a control signal to each of the camera modules 900a, 900b, and 900c. Control signals generated from the camera module controller 1016 may be provided to corresponding camera modules 900a, 900b, and 900c through separate control signal lines CSLa, CSLb, and CSLc.

One of the plurality of camera modules 900a, 900b, and 900c is designated as a master camera (eg, 900b) according to image generation information including a zoom signal or a mode signal, and the remaining camera modules (eg, 900b) For example, 900a and 900c) may be designated as slave cameras. Such information may be included in a control signal and provided to corresponding camera modules 900a, 900b, and 900c through separate control signal lines CSLa, CSLb, and CSLc.

Camera modules operating as a master and a slave may be changed according to a zoom factor or an operation mode signal. For example, when the viewing angle of the camera module 900a is wider than that of the camera module 900b and the zoom factor indicates a low zoom factor, the camera module 900b operates as a master and the camera module 900a operates as a slave. can act as Conversely, when the zoom factor indicates a high zoom factor, the camera module 900a may operate as a master and the camera module 900b may operate as a slave.

In some embodiments, the control signal provided from the camera module controller 1016 to each of the camera modules 900a, 900b, and 900c may include a sync enable signal. For example, when the camera module 900b is a master camera and the camera modules 900a and 900c are slave cameras, the camera module controller 1016 may transmit a sync enable signal to the camera module 900b. The camera module 900b receiving such a sync enable signal generates a sync signal based on the provided sync enable signal, and transmits the generated sync signal to the camera modules (900b) through the sync signal line SSL. 900a, 900c). The camera module 900b and the camera modules 900a and 900c may transmit image data to the application processor 1000 in synchronization with the sync signal.

In some embodiments, a control signal provided from the camera module controller 1016 to the plurality of camera modules 900a, 900b, and 900c may include mode information according to the mode signal. Based on this mode information, the plurality of camera modules 900a, 900b, and 900c may operate in a first operation mode and a second operation mode in relation to sensing speed.

The plurality of camera modules 900a, 900b, and 900c generate an image signal at a first rate (eg, generate an image signal having a first frame rate) in a first operation mode, and generate an image signal at a second frame rate higher than the first rate. encoding (eg, encoding an image signal having a second frame rate higher than the first frame rate) and transmitting the encoded image signal to the application processor 1000 . In this case, the second speed may be 30 times or less than the first speed.

The application processor 1000 stores the received image signal, that is, the encoded image signal, in the internal memory 1030 or the external storage 1200 of the application processor 1000, and then the memory 1030 or storage 1030. The encoded image signal may be read and decoded from 1200, and image data generated based on the decoded image signal may be displayed. For example, a corresponding sub-processor among the plurality of sub-processors 1012a, 1012b, and 1012c of the image processing device 1010 may perform decoding and may also perform image processing on the decoded image signal.

The plurality of camera modules 900a, 900b, and 900c generate image signals at a third rate lower than the first rate in the second operation mode (eg, image signals having a third frame rate lower than the first frame rate). generation), and transmit the image signal to the application processor 1000. An image signal provided to the application processor 1000 may be an unencoded signal. The application processor 1000 may perform image processing on a received image signal or store the image signal in the memory 1030 or the storage 1200 .

The PMIC 1100 may supply power, eg, a power supply voltage, to each of the plurality of camera modules 900a, 900b, and 900c. For example, the PMIC 1100 supplies first power to the camera module 900a through the power signal line PSLa under the control of the application processor 1000, and supplies the first power to the camera module 900a through the power signal line PSLb ( Second power may be supplied to 900b) and third power may be supplied to the camera module 900c through the power signal line PSLc.

The PMIC 1100 may generate power corresponding to each of the plurality of camera modules 900a, 900b, and 900c in response to a power control signal PCON from the application processor 1000, and may also adjust the level of the power. . The power control signal PCON may include a power control signal for each operation mode of the plurality of camera modules 900a, 900b, and 900c. For example, the operation mode may include a low power mode, and in this case, the power control signal PCON may include information about a camera module operating in the low power mode and a set power level. Levels of the powers provided to each of the plurality of camera modules 900a, 900b, and 900c may be the same or different from each other. Also, the level of power can be dynamically changed.

As above, exemplary embodiments have been disclosed in the drawings and specifications. Although the embodiments have been described using specific terms in this specification, they are only used for the purpose of explaining the technical idea of the present disclosure, and are not used to limit the scope of the present disclosure described in the claims. . Therefore, those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical scope of protection of the present disclosure should be determined by the technical spirit of the appended claims.

Claims (22)

an image sensor that generates first image data by receiving light from the outside, generates second image data based on the first image data, and outputs the first image data and the second image data; and
A third method comprising increasing the resolution of at least one region of the image corresponding to the first image data using a machine learning algorithm and including at least a portion of the first image data and at least a portion of the second image data with increased resolution. An electronic device including an application processor that outputs image data. According to claim 1,
The image sensor,
An electronic device generating the second image data by performing an image signal processor (ISP) operation based on the first image data. According to claim 2,
The ISP operation includes an operation of generating super resolution second image data using a re-mosaic algorithm. According to claim 3,
The second image data is Bayer pattern data. According to claim 4,
The application processor,
An electronic device that converts the second image data from Bayer pattern data to red green blue (rgb) data. According to claim 5,
The application processor,
determining whether real-time processing of the second image data is performed based on whether at least one of auto exposure (AE), auto focus (AF), and auto white balance (AWB) is executed;
In response to determining that real-time processing of the second image data is performed, the electronic device transmits the second image data converted to rgb data to the image sensor without applying deep learning to the first image data. According to claim 1,
The application processor,
An electronic device that receives the first image data and determines at least one target region to increase resolution in an image corresponding to the first image data. According to claim 7,
The application processor,
generating an ambiguity map for the first image data;
An electronic device that determines the at least one target region in an image corresponding to the first image data by using the ambiguity map. According to claim 8,
The application processor,
An electronic device that increases a resolution of the at least one target region by using a machine learning algorithm based on the first image data. According to claim 9,
The application processor,
An electronic device generating the third image data by inserting data corresponding to a target region in the first image data into data corresponding to the target region in the second image data. According to claim 1,
The application processor,
Receiving the second image data, and determining at least one target region to increase resolution in an image corresponding to the second image data;
An electronic device that transmits information about the at least one target area to the image sensor. According to claim 11,
The image sensor,
Receiving information on the at least one target region;
An electronic device that generates first image data including data corresponding to an image of the at least one target region based on received information about the target region. According to claim 1,
The first image data is a Qcell image or a tetra cell image. A method for generating a high-definition image of an electronic device including an image sensor and an application processor,
generating first image data by receiving light from the outside;
Generating second image data based on the first image data;
outputting the first image data and the second image data; and
A third method comprising increasing the resolution of at least one region of the image corresponding to the first image data using a machine learning algorithm, and including at least a portion of the first image data and at least a portion of the second image data with increased resolution. A method comprising outputting image data. According to claim 14,
The operation of generating the second image data,
A method configured to generate the second image data by performing an image signal processor (ISP) operation based on the first image data. According to claim 15,
The ISP operation includes an operation of generating super resolution second image data using a re-mosaic algorithm. According to claim 14,
The operation of generating the third image data,
and receiving the first image data and determining at least one target region to increase resolution in an image corresponding to the first image data. According to claim 17,
The operation of determining the target region,
generating an ambiguity map for the first image data; and
and determining the at least one target region in the image corresponding to the first image data by using the ambiguity map. According to claim 18,
The operation of generating the third image data,
and increasing a resolution of the at least one target region using a machine learning algorithm based on the first image data. According to claim 19,
The operation of generating the third image data,
and generating the third image data by inserting data corresponding to a target region in the first image data into data corresponding to the target region in the second image data. According to claim 14,
The first image data is a Qcell image or a tetra cell image. generating first image data by receiving light from the outside by controlling an image sensor;
generating second image data based on the first image data by controlling the image sensor; and
Instructions for implementing a method including controlling an application processor to generate third image data based on at least one region of the first image data and at least one region of the second image data using a deep learning algorithm A computer-readable recording medium for storage.

Images