WO2022025574A1

WO2022025574A1 - Electronic device comprising image sensor and image signal processor, and method therefor

Info

Publication number: WO2022025574A1
Application number: PCT/KR2021/009661
Authority: WO
Inventors: 전혜경; 강호석; 김동수; 이성구; 홍웅호; 원종훈
Original assignee: 삼성전자 주식회사
Priority date: 2020-07-28
Filing date: 2021-07-26
Publication date: 2022-02-03
Also published as: KR20220014150A

Abstract

An electronic device may comprise an image sensor and at least one processor electrically connected to the image sensor. The at least one processor may: acquire first image data generated by adding dummy data having a second number of bits for each pixel to image data in which each pixel has a first number of effective bits, in a first mode of the image sensor, each pixel of the first image data having a third number of bits; and acquire, from the image sensor, second image data in which each pixel has a third number of effective bits, in a second mode of the image sensor. Various other embodiments identified through the specification are possible.

Description

Electronic device including image sensor and image signal processor and method thereof

Embodiments disclosed in this document relate to driving of an image sensor and an image signal processor.

In general, image processing is performed from the original image to suit the purpose of using the image. In particular, as image processing technology develops, the range of color expression in an image is becoming more important. In a dark or bright shooting environment, if the range of color expression is narrow, it may be difficult to obtain a clear image. Therefore, in order to obtain a clear image in which subjects are clearly distinguished from each other or in which various colors are expressed, a human can recognize the dynamic range of an image having a limited color expression range compared to the human perceptible dynamic range (hereafter, dynamic range). High dynamic range (hereafter, HDR, high dynamic range) processing that expands to an existing level is performed.

In a relatively low dynamic range environment and a relatively high dynamic range environment, the size of effective data output by the image sensor may be different from each other. Therefore, in order to acquire image data having different sizes of effective data in the image signal processor, the image signal processor is changed to match the size of the effective data to obtain data and process the image.

When the range of data output from the image sensor is changed by changing the output mode of the image sensor according to the change of the surrounding environment, the image signal processor stops the operation in progress and resets the image signal processor according to the change in the data output from the image sensor After that, it worked again.

Since it takes time to reset the setting mode of the image signal processor, there is a disconnection between the images before and after changing the range of data output from the image sensor. In addition, the interruption time between images was 300 ms or more, which has been a factor hindering usability.

According to various embodiments of the present disclosure, when the effective data size output from the image sensor is changed, dummy data is added to keep the size of data acquired by the image signal processor constant to perform image processing without stopping the operation. It is possible to provide an electronic device and a method for controlling the electronic device that can process the data without interruption.

An electronic device according to an embodiment disclosed in this document may include an image sensor and at least one processor electrically connected to the image sensor. In the first mode of the image sensor, the at least one processor is configured to generate a first image generated by adding dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of significant bits. obtain data, wherein each pixel of the first image data has a third number of bits, in a second mode of the image sensor, a second from the image sensor wherein each pixel has the third number of significant bits Image data can be acquired.

In addition, in the method of operating an electronic device according to an embodiment disclosed in this document, in the first mode of the image sensor, in image data in which each pixel has a first number of valid bits, a second number of bits in each pixel obtaining first image data generated by adding dummy data having It may include an action to

Also, the electronic device according to an embodiment disclosed in this document may include an image sensor, at least one processor electrically connected to the image sensor, and an interface connecting the image sensor and the at least one processor. The image sensor adds dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of valid bits, and adds the dummy data so that each pixel receives a third number of bits The branch may generate first image data, and provide the generated first image data to the at least one processor through the interface. The at least one processor may identify the valid bit and the dummy data included in the first image data provided through the interface, and perform image processing based on the identified valid bit.

According to various embodiments disclosed in this document, the size of data output from the image sensor may be constantly maintained even when the surrounding environment changes, such as a bright outdoor environment or a backlight environment.

Also, according to various embodiments, data output from the image sensor may be acquired without resetting the image signal processor.

In addition, according to various embodiments, it is possible to provide a seamless image capturing function even when the shooting environment changes.

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a diagram illustrating a structure of an electronic device and a camera module according to an embodiment.

2 illustrates a hardware configuration of an electronic device according to an embodiment.

3 illustrates image processing when an image having a first dynamic range (eg, a normal mode) is output by an electronic device according to an exemplary embodiment.

4 illustrates image processing when an image having a second dynamic range (eg, HDR) higher than a first dynamic range is output by the electronic device, according to an exemplary embodiment.

5 illustrates an operation of an image signal processor for each mode in an electronic device according to an embodiment.

6 illustrates a process of processing an image according to a mode by an image sensor, an image signal processor, and a display according to an exemplary embodiment.

7 is a flowchart illustrating an operation process in an image signal processor of an electronic device according to an exemplary embodiment.

8 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;

9 is a block diagram illustrating a camera module according to various embodiments.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. However, this document is not intended to limit the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present invention.

1 is a diagram schematically illustrating an exterior and a camera module 180 of an electronic device 100 on which a camera module 180 is mounted, according to an embodiment. Although the embodiment of FIG. 1 has been illustrated and described on the premise of a mobile device, in particular, a smart phone, it is to those skilled in the art that it can be applied to various electronic devices or electronic devices equipped with a camera among mobile devices. will be clearly understood.

Referring to FIG. 1 , the display 110 may be disposed on the front surface of the electronic device 100 according to an embodiment. In an embodiment, the display 110 may occupy most of the front surface of the electronic device 100 . A display 110 and a bezel 190 region surrounding at least some edges of the display 110 may be disposed on the front surface of the electronic device 100 . The display 110 may include a flat area and a curved area extending from the flat area toward the side of the electronic device 100 . The electronic device 100 illustrated in FIG. 1 is an example, and various embodiments are possible. For example, the display 110 of the electronic device 100 may include only a flat area without a curved area, or may include a curved area only at one edge instead of both sides. Also, in an embodiment, the curved area may extend toward the rear surface of the electronic device, so that the electronic device 100 may include an additional planar area.

According to an embodiment, the electronic device 100 may additionally include a speaker, a receiver, a front camera, a proximity sensor, a home key, and the like. The electronic device 100 according to an embodiment may be provided in which the rear cover 150 is integrated with the main body of the electronic device. In another embodiment, the rear cover 150 may be separated from the main body of the electronic device 100 to have a form in which the battery can be replaced. The back cover 150 may be referred to as a battery cover or a back cover.

In an embodiment, a fingerprint sensor 171 for recognizing a user's fingerprint may be included in the first area 170 of the display 110 . Since the fingerprint sensor 171 is disposed on a lower layer of the display 110 , the fingerprint sensor 171 may not be recognized by the user or may be difficult to recognize. Also, in addition to the fingerprint sensor 171 , a sensor for additional user/biometric authentication may be disposed in a portion of the display 110 . In another embodiment, a sensor for user/biometric authentication may be disposed on one area of the bezel 190 . For example, the IR sensor for iris authentication may be exposed through one area of the display 110 or may be exposed through one area of the bezel 190 .

In an embodiment, the front camera 161 may be disposed in the second area 160 on the front side of the electronic device 100 . In the embodiment of FIG. 1 , the front camera 161 is shown to be exposed through one area of the display 110 , but in another embodiment, the front camera 161 may be exposed through the bezel 190 . The electronic device 100 may include one or more front cameras 161 . For example, the electronic device 100 may include two front cameras, such as a first front camera and a second front camera. In an embodiment, the first front camera and the second front camera may be cameras of the same type having the same specifications (eg, pixels), but the first front camera and the second front camera may be implemented as cameras of different specifications. . The electronic device 100 may support a function (eg, 3D imaging, auto focus, etc.) related to a dual camera through two front cameras. The above-mentioned description of the front camera may be equally or similarly applied to the rear camera of the electronic device 100 .

In an embodiment, in the electronic device 100 , various hardware or sensors 163 to assist photographing, such as a flash, may be additionally disposed. For example, a distance sensor (eg, TOF sensor) for detecting the distance between the subject and the electronic device 100 may be further included. The distance sensor may be applied to both a front camera and/or a rear camera. The distance sensor may be separately disposed or included and disposed on the front camera and/or the rear camera.

In an embodiment, at least one physical key may be disposed on a side portion of the electronic device 100 . For example, the first function key 151 for turning on/off the display 110 or turning on/off the power of the electronic device 100 may be disposed on the right edge with respect to the front surface of the electronic device 100 . have. In an embodiment, the second function key 152 for controlling the volume or screen brightness of the electronic device 100 may be disposed on the left edge with respect to the front surface of the electronic device 100 . In addition to this, additional buttons or keys may be disposed on the front or rear of the electronic device 100 . For example, a physical button or a touch button mapped to a specific function may be disposed in a lower region of the front bezel 190 .

The electronic device 100 illustrated in FIG. 1 corresponds to one example, and the shape of the device to which the technical idea disclosed in the present disclosure is applied is not limited. For example, by adopting a flexible display and a hinge structure, a foldable electronic device that can be folded horizontally or vertically, a rollable electronic device that can be rolled, a tablet or a notebook computer, the technical idea of the present disclosure This can be applied. In addition, the present technical idea can be applied even when it is possible to arrange the first camera and the second camera facing in the same direction to face different directions through rotation, folding, deformation, etc. of the device.

Referring to FIG. 1 , an electronic device 100 according to an embodiment may include a camera module 180 . The camera module 180 includes a lens assembly 111 , a housing 113 , an infrared cut filter 115 , an image sensor 120 , and an image signal processor (ISP) 130 . may include

In an embodiment, the lens assembly 111 may have a different number, arrangement, type, etc. of lenses depending on the front camera and the rear camera. Depending on the type of lens assembly, the front camera and the rear camera may have different characteristics (eg, focal length, maximum magnification, etc.). The lens may be moved forward and backward along the optical axis, and may operate so that a target object, which is a subject, can be clearly captured by changing a focal length.

In one embodiment, the camera module 180 includes a housing 113 for mounting at least one coil surrounding the periphery of the barrel around the optical axis and a barrel for mounting at least one or more lenses aligned on the optical axis. can

In an embodiment, the infrared cut filter 115 may be disposed on the upper surface of the image sensor 120 . The image of the subject passing through the lens may be partially filtered by the infrared cut filter 115 and then detected by the image sensor 120 .

In an embodiment, the image sensor 120 may be disposed on the upper surface of the printed circuit board 140 . The image sensor 120 may be electrically connected to the image signal processor 130 connected to the printed circuit board 140 by a connector. A flexible printed circuit board (FPCB) or a cable may be used as the connector.

In an embodiment, the image sensor 120 may be a complementary metal oxide semiconductor (CMOS) sensor or a charged coupled device (CCD) sensor. A plurality of individual pixels are integrated in the image sensor 120 , and each individual pixel may include a micro lens, a color filter, and a photodiode. Each individual pixel is a kind of photodetector that can convert incoming light into an electrical signal. Photodetectors generally cannot detect the wavelength of the captured light by themselves and cannot determine color information. The photodetector may include a photodiode.

In an embodiment, light information of a subject incident through the lens assembly 111 may be converted into an electrical signal by the image sensor 120 and input to the image signal processor 130 .

In an embodiment, the camera module 180 may be disposed on the front side as well as the rear side of the electronic device 100 . Also, the electronic device 100 may include a plurality of camera modules 180 as well as one camera module 180 to improve camera performance. For example, the electronic device 100 may further include a front camera 161 for video call or self-camera photography. The front camera 161 may support a relatively low number of pixels compared to the rear camera module. The front camera may be relatively smaller than the rear camera module.

2 illustrates a hardware configuration of an electronic device according to an embodiment. In the description of FIG. 2 , the configuration illustrated in FIG. 1 may be briefly described or a description thereof may be omitted.

Referring to FIG. 2 , in an embodiment, the electronic device 100 may include a camera module 180 , a processor 220 , a display 110 , and a memory 230 . The camera module 180 may include an image sensor 120 and an image signal processor 130 . In the description of FIG. 2 , descriptions of the same reference numerals as those of FIG. 1 may be omitted.

The components illustrated in FIG. 2 are exemplary, and the electronic device 100 may further include additional components. For example, the electronic device 100 may further include at least one microphone for recording audio data. Also, for example, the electronic device 100 may include at least one sensor for determining a direction in which the front or rear of the electronic device 100 faces and/or posture information of the electronic device 100 . In an embodiment, the at least one sensor may include an acceleration sensor, a gyro sensor, and the like.

In an embodiment, the image sensor 120 may include a complementary metal oxide semiconductor (CMOS) sensor or a charged coupled device (CCD) sensor. The light information of the subject incident through the lens assembly 111 may be converted into an electrical signal by the image sensor 120 and input to the image signal processor 130 . An infrared cut filter (hereinafter, IR cut filter) may be disposed on the upper surface of the image sensor 120 , and the image of the subject passing through the lens is partially filtered by the IR cut filter and then the image sensor 120 ) can be detected by

In an embodiment, when the image signal processor 130 and the image sensor 120 are physically separated, a sensor interface conforming to an appropriate standard may electrically connect the image sensor 120 and the image signal processor 130 .

In an embodiment, the image signal processor 130 may perform image processing on the electrically converted image data. A process in the image signal processor 130 may be divided into a pre-ISP (hereinafter, pre-processing) and an ISP chain (hereinafter, post-processing). Image processing before the demosaicing process may mean pre-processing, and image processing after the demosaicing process may mean post-processing. The preprocessing process may include 3A processing, lens shading correction, edge enhancement, dead pixel correction, knee correction, and the like. 3A may include at least one of auto white balance (AWB), auto exposure (AE), and auto focusing (AF). The post-processing process may include at least one of changing a sensor index value, changing a tuning parameter, and adjusting an aspect ratio. The post-processing process may include processing the image data output from the image sensor 120 or image data output from the scaler. The image signal processor 130 may adjust contrast, sharpness, saturation, dithering, etc. of the image through a post-processing process. Here, the contrast, sharpness, and saturation adjustment procedures are performed in the YUV color space, and the dithering procedure may be performed in the RGB (Red Green Blue) color space. . A part of the pre-processing process may be performed in the post-processing process, or a part of the post-processing process may be performed in the pre-processing process. In addition, a part of the pre-processing process may be overlapped with a part of the post-processing process.

In an embodiment, the display 110 may display contents such as an execution screen of an application executed by the processor 220 or images and/or videos stored in the memory 230 on the display 110 . In addition, the processor 220 may display the image data acquired through the camera module 180 on the display 110 in real time.

In an embodiment, the display 110 may be implemented integrally with the touch panel. The display 110 may support a touch function, detect a user input such as a touch using a finger, and transmit it to the processor 220 . The display 110 may be connected to a display driver integrated circuit (DDIC) for driving the display 110 , and the touch panel may be connected to a touch IC that detects touch coordinates and processes a touch-related algorithm. In one embodiment, the display driving circuit and the touch IC may be integrally formed, and in another embodiment, the display driving circuit and the touch IC may be formed separately. The display driving circuit and/or the touch IC may be electrically connected to the processor 220 .

In an embodiment, the processor 220 may execute/control various functions supported by the electronic device 100 . For example, the processor 220 may execute an application by executing a code written in a programming language stored in the memory 230 , and may control various hardware. For example, the processor 220 may execute an application supporting a photographing function stored in the memory 230 . In addition, the processor 220 may execute the camera module 180 and set and support an appropriate shooting mode so that the camera module 180 may perform an operation intended by the user.

In an embodiment, the memory 230 may store instructions executable by the processor 220 . The memory 230 may be understood as a concept including a component in which data is temporarily stored, such as a random access memory (RAM), and/or a component in which data is permanently stored, such as a solid state drive (SSD). . For example, the processor 220 may implement a software module in the RAM space by calling instructions stored in the SSD. In various embodiments, the memory 230 may include various types, and an appropriate type may be adopted according to the purpose of the device.

In an embodiment, an application related to the camera module 180 may be stored in the memory 230 . For example, a camera application may be stored in the memory 230 . The camera application may support various shooting functions, such as photo shooting, video shooting, panoramic shooting, and slow motion shooting.

In an embodiment, an application associated with the camera module 180 may correspond to various types of applications. For example, a chatting application, a web browser application, an email application, a shopping application, etc. may also use the camera module 180 to support a video call, photo/video attachment, streaming service, product image, or product-related virtual reality (VR) shooting function. can

3 illustrates image processing when an image having a first dynamic range (eg, a normal mode) is output by an electronic device according to an exemplary embodiment. When the electronic device 100 outputs an image having the first dynamic range (eg, a normal mode), dummy data is added to the image data having the first dynamic range through the image sensor 120 or a separate hardware module. can be added.

The valid bit referred to in this specification may relate to image data output through the image sensor 120 . The effective bit may mean data output by each pixel of the image sensor 120 by light incident through the camera lens. The effective bit may mean a color value of each pixel. The color value may include color information and brightness information. For example, when the color filter array is composed of red (R, red), green (G, green), and blue (B, blue) colors, effective bits of a pixel include color information of at least one of R, G, and B may include The green, red, and blue are only examples of color values, and the color values are not limited. The color value may be at least one of red, green, blue, yellow, emerald, white, cyan, and magenta. . According to various embodiments, the color filter array may include a red, green, blue, emerald (RGBE) pattern, a cyan, yellow, magenta (CYYM) pattern, a cyan, yellow, green, magenta (CYGM) pattern, or a red, green (RGBW) pattern. , blue, white) may include a color filter array of a pattern.

In an embodiment, the image sensor 120 may output image data 310 having a color value for each pixel. In a configuration separate from the image sensor 120 , new first image data 320 may be generated by adding dummy data to the output image data 310 . For example, the image sensor 120 outputs image data in which the number of valid bits of each pixel is a first number (eg, 10), and a second number (eg, 2) for each pixel in a separate hardware module Dummy data having a number of bits) can be added. The separate hardware module may be positioned between the image sensor 120 and the image signal processor 130 and may be electrically connected to the image sensor 120 and the image signal processor 130 . The image data output by the image sensor 120 may be provided to the image signal processor 130 through the hardware module.

In an embodiment, the image sensor 120 may add dummy data to the image data 310 acquired through the pixel array. The image sensor 120 may generate the first image data 320 by adding the dummy data. For example, the image sensor 120 adds dummy data having a second number (eg, 2) bits per pixel to the image data 310 having a first number (eg, 10) of effective bits. The first image data 320 having the third number (eg, 12) bits of each pixel may be generated. The image sensor 120 may output the first image data 320 generated by adding the dummy data. The first image data 320 may include identification information for identifying unique data and dummy data included in the image data. The identification information may be referred to as a delimiter or identifier that distinguishes the unique data from the dummy data. The dummy data may be referred to as a dummy bit. The first number, the second number, and the third number mentioned in this document are not limited to the number given as an example, and it can be assumed that the third number is the same as the sum of the first number and the second number. .

In an embodiment, the image signal processor 130 may acquire the generated first image data 320 in a first setting state. The image signal processor 130 may acquire the first image data 320 and, at the same time, acquire identification information included in the first image data 320 .

In an embodiment, the image signal processor 130 may perform image analysis on the first image data 320 . Image analysis of the first image data 320 may be performed through the processor 220 . The image analysis may be to determine whether the acquired image data needs to be processed in a second dynamic range (eg, HDR) higher than the first dynamic range. The processor 220 may determine that HDR processing is necessary according to a surrounding environment or a main subject. For example, when a surrounding environment such as a bright outdoor or backlight environment is detected, it may be determined that HDR processing is necessary.

In an embodiment, the processor 220 may control the mode of the image sensor 120 based on a result of image analysis. As a result of the image signal processor 130 or the processor 220 analyzing the image, when the photographing environment is not changed, the processor 220 may maintain the output mode of the image sensor 120 . As a result of the image signal processor 130 or the processor 220 analyzing the image, in an environment requiring HDR processing, the processor 220 sets the output mode of the image sensor 120 to the image sensor 120 illustrated in FIG. 4 below. ) can be changed to the output mode.

In an embodiment, the image signal processor 130 may perform image processing on the first image data 320 . The image signal processor 130 may perform image processing based on the image data 310 included in the first image data 320 . In other words, the image signal processor 130 may perform image processing on valid data among the acquired image data.

In an embodiment, the image signal processor 130 may provide image data on which image processing is performed to the processor 220 . The processor 220 may control the provided image data to be stored in the memory 230 or to be displayed on the display 110 .

In an embodiment, the image sensor 120 may output second image data 410 having a color value for each pixel. The second image data 410 may have a larger number of valid bits in each pixel than the image data 310 of FIG. 3 . Accordingly, the second image data 410 may include a larger number of color values or a wider color range than the image data 310 of FIG. 3 . For example, the image sensor 120 may output image data in which the number of valid bits of each pixel is a third number (eg, 12).

In an embodiment, the image signal processor 130 may acquire the output second image data 410 in a first setting state, similar to the embodiment of FIG. 3 .

In an embodiment, the image signal processor 130 may perform image analysis on the second image data 410 . Image analysis of the second image data 410 may be performed through the processor 220 .

In an embodiment, the processor 220 may control the mode of the image sensor 120 based on a result of image analysis. As a result of the image signal processor 130 or the processor 220 analyzing the image, the processor 220 may maintain the output mode of the image sensor 120 when the photographing environment is not changed. As a result of the image signal processor 130 or the processor 220 analyzing the image, if the HDR processing is not required, the processor 220 sets the output mode of the image sensor 120 to the image sensor 120 illustrated in FIG. 3 . ) can be changed to the output mode.

In an embodiment, the image signal processor 130 may perform image processing on the second image data 410 . When the second image data 410 conforms to HDR processing, the image signal processor 130 may perform HDR processing. The HDR processing may be a process to extend the dynamic range of an image having a limited dynamic region to a level that human vision can perceive in order to distinguish objects. . Here, the HDR processing is not limited to a specific method, and may be performed by a method widely known in the art.

5 illustrates an operation of an image signal processor for each mode in an electronic device according to an embodiment. The operating subject of the flowchart illustrated in FIG. 5 may be understood as a processor (eg, the processor 220 of FIG. 2 ) or an image signal processor (eg, the image signal processor 130 of FIG. 1 ).

In operation 510 according to an embodiment, in the first mode of the image sensor 120 , the image signal processor 130 includes a second number of each pixel in the image data in which each pixel has a first number of significant bits. First image data generated by adding dummy data having bits may be obtained. The dummy data is for making the number of bits of each pixel of the first image data obtained by the image signal processor 130 equal to the number of bits of each pixel of the second image data obtained in the HDR environment. It can be data.

In an embodiment, the image signal processor 130 may acquire the first image data in a first setting state. The first setting state may be a setting mode operated when the number of bits of the image data acquired by the image signal processor 130 is the third number.

In operation 520 according to an embodiment, a second mode change event may occur. The second mode change event may be an event related to changing the output mode of the image sensor 120 from the first mode to the second mode.

In an embodiment, in response to the mode change event, the processor 220 may control the mode of the image sensor 120 to be changed from the first mode to the second mode. The first mode may be a mode for outputting image data having the first number of effective bits of each pixel of the image data, and the second mode may be a mode for outputting image data having the third number of effective bits of each pixel of the image data. have. The third number may be greater than the first number. For example, the first number may be 10, and the third number may be 12. Also, the definitions of the first mode and the second mode may be equally applied to the first mode and the second mode mentioned in this document.

In operation 530 according to an embodiment, in the second mode of the image sensor 120 , the image signal processor 130 may acquire second image data having a third number of valid bits. The image signal processor 130 may acquire the second image data in a first setting state. Since the second image data has the same third number of bits in each pixel as in the first image data, the image signal processor 130 may acquire the second image data in the first setting state.

6 illustrates a process of processing an image according to a mode by an image sensor, an image signal processor, and a display according to an exemplary embodiment. An operation performed by the image signal processor illustrated in FIG. 6 (eg, the image signal processor 130 of FIG. 1 ) may be performed by a processor (eg, the processor 220 of FIG. 2 ).

In operation 610 according to an embodiment, in the first mode of the image sensor 120 , the image sensor 610 may output image data having a first number of valid bits of each pixel.

In an embodiment, the image sensor 120 may add dummy data having a second number of bits to image data having a first number of effective bits of each pixel. The image sensor 120 may output image data to which the dummy data is added.

In operation 615 according to an embodiment, in the first mode of the image sensor 120 , the image signal processor 130 transmits the image data in which the number of effective bits of each pixel is the first number to the second image data for each pixel. Image data to which dummy data having the number of bits is added may be obtained. In the image data to which the dummy data is added, the number of bits of each pixel may be the third number. The image signal processor 130 may acquire image data to which the dummy data is added in a first set state.

In an embodiment, the image signal processor 130 may perform image processing while maintaining the same setting state regardless of the mode change of the image sensor 120 . For example, when the image sensor 120 is in the first mode, the image signal processor 130 adds a second number of dummy data to the first number of valid data for each pixel so that each pixel has a third number of bits may operate in a first setting state for processing the first image data having

In operation 620 according to an embodiment, the display 110 may output a preview. The display 110 may output a preview based on image data in which the number of effective bits of each pixel is the first number. The display 110 may receive a preview format including the image data from the processor 220 and may output a preview based on the preview format.

In operation 625 according to an embodiment, the image signal processor 130 may analyze the first image data to which the obtained dummy data is added. When analyzing the first image data to which the dummy data is added, the image signal processor 130 analyzes the first image data including the dummy data or calculates a first number of valid bits per pixel except for the dummy data. It can analyze image data.

In operation 630 according to an embodiment, a second mode change event may occur. This may correspond to operation 520 of FIG. 5 .

In operation 635 according to an embodiment, the image sensor 120 may receive a second mode change signal. When a second mode change event occurs, the processor 220 may provide a second mode change signal to the image sensor 120 through the designated interface. The image sensor 120 may receive the second mode change signal provided by the processor 220 through the designated interface.

In operation 640 according to an embodiment, the image sensor 120 may change the output mode from the first mode to the second mode in response to receiving the mode change signal.

In operation 645 according to an embodiment, in the second mode of the image sensor 120 , the image sensor 610 may output image data in which each pixel has a third number of valid bits.

In operation 650 according to an embodiment, in the second mode of the image sensor 120 , the image signal processor 130 may obtain image data in which each pixel has a third number of significant bits. In comparison with operation 615 , the image signal processor 130 may acquire second image data in which the number of bits included in each pixel is the same as that of the first image data.

In an embodiment, the image signal processor 130 may perform image processing while maintaining operation regardless of a mode change of the image sensor 120 . For example, when the mode of the image sensor 120 is changed from the first mode to the second mode, the image signal processor 130 may maintain the first set state. Since the number of bits of each pixel is the same, in the second mode, the image signal processor 130 sets the first set state as in the first mode with respect to the second image data in which the number of effective bits of each pixel is the third number. can operate as

In operation 655 according to an embodiment, the display 110 may output a preview. The display 110 may output a preview based on image data in which the number of effective bits of each pixel is the third number. For example, the display 110 may display image data output from the image sensor 120 on which image processing is performed.

7 is a flowchart illustrating an operation process in an image signal processor of an electronic device according to an exemplary embodiment. The operating subject of the flowchart illustrated in FIG. 7 may be understood as a processor (eg, the processor 220 of FIG. 2 ) or an image signal processor (eg, the image signal processor 130 of FIG. 1 ).

In operation 710 according to an embodiment, the image signal processor 130 may obtain identification information on a valid bit. Alternatively, the image signal processor 130 may obtain identification information on the dummy data. The image signal processor 130 may acquire identification information on the dummy data by acquiring image data including identification information indicating that the dummy data is included. The identification information may be referred to as a delimiter or identifier (ID) that distinguishes the unique data from the dummy data. The identification information may be included in a header area called embedded data and transmitted to the image signal processor 130 together with image data acquired through the image sensor 120 .

In an embodiment, the image signal processor 130 may recognize the number of valid bits based on the identification information on the dummy data. The image signal processor 130 may classify valid bits and dummy data among bits of each pixel based on identification information included in the dummy data.

In an embodiment, the image signal processor 130 may recognize the output mode of the image sensor 120 based on the identification information on the dummy data. When dummy data is included in the acquired image data, the image signal processor 130 may determine that the output mode of the image sensor 120 is the first mode. For example, when each pixel of the image data acquired by the image signal processor 130 includes a first number (eg, 10) of valid bits and a second number (eg, two) of dummy data, the image The signal processor 130 may determine that the output mode of the image sensor 120 is the first mode. When the dummy data is not included in the acquired image data, the image signal processor 130 may determine that the output mode of the image sensor 120 is the second mode. For example, when each pixel of the image data acquired by the image signal processor 130 includes a third number (eg, 12) of valid bits without dummy data, the image signal processor 130 or the processor 220 may determine that the output mode of the image sensor 120 is the second mode.

In operation 720 according to an embodiment, the image signal processor 130 may perform image processing based on the number of valid bits of each pixel of the image data and dummy data.

In an embodiment, the image signal processor 130 may determine a range of data to be image processed based on identification information included in the image data. For example, when the acquired image data includes identification information on dummy data, the image signal processor 130 may generate image data (eg, a figure in which the number of valid bits of each pixel is a first number (eg, 10)) Image processing may be performed based on the image data 310 of FIG. 3 . As another example, if the image signal processor 130 does not include identification information in the acquired image data, the image data (eg, figure Image processing may be performed based on the second image data 410 of FIG. 4 .

In operation 730 according to an embodiment, the image signal processor 130 may determine whether it is necessary to change the mode through image analysis. The image signal processor 130 (or the processor 220 ) may determine the HDR environment through analysis of the acquired image data. In an embodiment, the processor 220 may determine that it is necessary to change the mode when determining that the HDR environment is present.

In an embodiment, the image signal processor 130 may determine whether it is an HDR environment based on brightness information of the image. The image signal processor 130 (or the processor 220 ) uses an auto exposure (AE) function so that the first area of the image has a first brightness greater than or equal to the first threshold, and the second area differentiated from the first area is When the second brightness is lower than the second threshold, which is darker than the first brightness, the HDR environment may be determined. The image signal processor 130 and/or the processor 220 may determine that the output mode of the image sensor 120 needs to be changed when detecting the HDR environment.

In an embodiment, the processor 220 may analyze a scene of the captured image to determine whether it is an HDR environment. The processor 220 may analyze the scene of the captured image to determine the HDR environment when the shooting background is a backlight environment or a bright outdoor environment. The processor 220 may determine whether the captured image is a backlight environment or a bright outdoor environment through a function (eg, a scene optimizer) on the image obtained through the camera. The one function (eg, a scene optimizer) may be a function capable of discriminating an object, a background, etc. based on data based on machine learning. For example, in the case of taking a photo or video under sunlight, when the light is located behind the subject and the subject is expressed below a specific brightness, the processor 220 may determine that it is a backlight environment. The image signal processor 130 (or the processor 220 ) may determine that it is necessary to change the output mode of the image sensor 120 when detecting that the HDR environment is present.

In operation 740 according to an embodiment, when a mode change of the image sensor 120 is required, the image signal processor 130 (or the processor 220 ) may provide a change signal to the image sensor 120 . . The processor 220 may provide a signal for changing to the second mode to the image sensor 120 and simultaneously control the mode of the image sensor 120 to be changed to the second mode.

8 is a block diagram of an electronic device (eg, the electronic device 100 of FIG. 1 ) 801 in the network environment 800 according to various embodiments of the present disclosure.

Referring to FIG. 8 , in a network environment 800 , the electronic device 801 communicates with the electronic device 802 through a first network 898 (eg, a short-range wireless communication network) or a second network 899 . It may communicate with the electronic device 804 or the server 808 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 801 may communicate with the electronic device 804 through the server 808 . According to an embodiment, the electronic device 801 includes a processor 820 , a memory 830 , an input module 850 , a sound output module 855 , a display module 860 , an audio module 870 , and a sensor module ( 876), interface 877, connection terminal 878, haptic module 879, camera module 880, power management module 888, battery 889, communication module 890, subscriber identification module 896 , or an antenna module 897 . In some embodiments, at least one of these components (eg, the connection terminal 878 ) may be omitted or one or more other components may be added to the electronic device 801 . In some embodiments, some of these components (eg, sensor module 876 , camera module 880 , or antenna module 897 ) are integrated into one component (eg, display module 860 ). can be

The processor (eg, the processor 220 of FIG. 2 ) 820 , for example, executes software (eg, the program 840 ) to be at least one other component of the electronic device 801 connected to the processor 820 . It can control elements (eg, hardware or software components) and can perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 820 stores commands or data received from other components (eg, the sensor module 876 or the communication module 890 ) into the volatile memory 832 . may be stored in , process commands or data stored in the volatile memory 832 , and store the result data in the non-volatile memory 834 . According to an embodiment, the processor 820 is a main processor 821 (eg, a central processing unit or an application processor) or a secondary processor 823 (eg, a graphics processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 801 includes a main processor 821 and a sub-processor 823 , the sub-processor 823 uses less power than the main processor 821 or is set to be specialized for a specified function. can The coprocessor 823 may be implemented separately from or as part of the main processor 821 .

The coprocessor 823 may, for example, act on behalf of the main processor 821 while the main processor 821 is in an inactive (eg, sleep) state, or when the main processor 821 is active (eg, executing an application). ), together with the main processor 821, at least one of the components of the electronic device 801 (eg, the display module 860, the sensor module 876, or the communication module 890) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 823 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 880 or the communication module 890 ). have. According to an embodiment, the auxiliary processor 823 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 801 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 808). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 830 may store various data used by at least one component (eg, the processor 820 or the sensor module 876 ) of the electronic device 801 . The data may include, for example, input data or output data for software (eg, the program 840 ) and commands related thereto. The memory 830 may include a volatile memory 832 or a non-volatile memory 834 .

The program 840 may be stored as software in the memory 830 , and may include, for example, an operating system 842 , middleware 844 , or an application 846 .

The input module 850 may receive a command or data to be used in a component (eg, the processor 820 ) of the electronic device 801 from the outside (eg, a user) of the electronic device 801 . The input module 850 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 855 may output a sound signal to the outside of the electronic device 801 . The sound output module 855 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.

The display module 860 may visually provide information to the outside (eg, a user) of the electronic device 801 . The display module 860 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 860 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 870 may convert a sound into an electrical signal or, conversely, convert an electrical signal into a sound. According to an embodiment, the audio module 870 acquires a sound through the input module 850 or an external electronic device (eg, a sound output module 855 ) directly or wirelessly connected to the electronic device 801 . A sound may be output through the electronic device 802 (eg, a speaker or headphones).

The sensor module 876 detects an operating state (eg, power or temperature) of the electronic device 801 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 876 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 877 may support one or more designated protocols that may be used for the electronic device 801 to directly or wirelessly connect with an external electronic device (eg, the electronic device 802 ). According to an embodiment, the interface 877 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 878 may include a connector through which the electronic device 801 can be physically connected to an external electronic device (eg, the electronic device 802 ). According to an embodiment, the connection terminal 878 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 879 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 879 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 880 may capture still images and moving images. According to an embodiment, the camera module 880 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 888 may manage power supplied to the electronic device 801 . According to an embodiment, the power management module 888 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 889 may supply power to at least one component of the electronic device 801 . According to an embodiment, the battery 889 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 890 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 801 and an external electronic device (eg, the electronic device 802, the electronic device 804, or the server 808). It can support establishment and communication performance through the established communication channel. The communication module 890 may include one or more communication processors that operate independently of the processor 820 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 890 is a wireless communication module 892 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 894 (eg, : It may include a LAN (local area network) communication module, or a power line communication module). Among these communication modules, a corresponding communication module is a first network 898 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 899 (eg, legacy). It may communicate with the external electronic device 804 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunication network such as a computer network (eg, LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 892 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 896 within a communication network, such as the first network 898 or the second network 899 . The electronic device 801 may be identified or authenticated.

The wireless communication module 892 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 892 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 892 uses various technologies for securing performance in a high frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 892 may support various requirements specified in the electronic device 801 , an external electronic device (eg, the electronic device 804 ), or a network system (eg, the second network 899 ). According to an embodiment, the wireless communication module 892 includes a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

The antenna module 897 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 897 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 897 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 898 or the second network 899 is connected from the plurality of antennas by, for example, the communication module 890 . can be selected. A signal or power may be transmitted or received between the communication module 890 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 897 .

According to various embodiments, the antenna module 897 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a specified high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, a command or data may be transmitted or received between the electronic device 801 and the external electronic device 804 through the server 808 connected to the second network 899 . Each of the external electronic devices 802 or 804 may be the same as or different from the electronic device 801 . According to an embodiment, all or part of operations performed by the electronic device 801 may be executed by one or more external electronic devices 802 , 804 , or 808 . For example, when the electronic device 801 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 801 may instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 801 . The electronic device 801 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 801 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. According to another embodiment, the external electronic device 804 may include an Internet of things (IoT) device. The server 808 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 804 or the server 808 may be included in the second network 899 . The electronic device 801 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The electronic device according to various embodiments disclosed in this document may be a device of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as “first”, “second”, or “first” or “second” may simply be used to distinguish the component from other components in question, and may refer to components in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, for example, and interchangeably with terms such as logic, logic block, component, or circuit. can be used A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present disclosure, one or more instructions stored in a storage medium (eg, internal memory 836 or external memory 838) readable by a machine (eg, electronic device 801) may be implemented as software (eg, the program 840) including For example, the processor (eg, the processor 820 ) of the device (eg, the electronic device 801 ) may call at least one of one or more instructions stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to an embodiment, the methods according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed online (eg download or upload), directly between smartphones (eg smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. . According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repetitively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

9 is a block diagram 900 illustrating a camera module (eg, the camera module 180 of FIG. 1 ) 880 according to various embodiments of the present disclosure.

Referring to FIG. 9 , the camera module 880 includes a lens assembly 910 , a flash 920 , an image sensor 930 , an image stabilizer 940 , a memory 950 (eg, a buffer memory), or an image signal processor. (960). The lens assembly 910 may collect light emitted from a subject, which is an image to be captured. The lens assembly 910 may include one or more lenses. According to an embodiment, the camera module 880 may include a plurality of lens assemblies 910 . In this case, the camera module 880 may form, for example, a dual camera, a 360 degree camera, or a spherical camera. Some of the plurality of lens assemblies 910 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. It may have one or more lens properties different from the lens properties of . The lens assembly 910 may include, for example, a wide-angle lens or a telephoto lens.

The flash 920 may emit light used to enhance light emitted or reflected from the subject. According to an embodiment, the flash 920 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 930 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 910 into an electrical signal. According to an embodiment, the image sensor 930 is, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 930 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 940 moves at least one lens or image sensor 930 included in the lens assembly 910 in a specific direction in response to the movement of the camera module 880 or the electronic device 801 including the same. Operation characteristics of the image sensor 930 may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken. According to an embodiment, the image stabilizer 940 is, according to an embodiment, the image stabilizer 940 is a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 880 . Such a movement of the camera module 880 or the electronic device 801 may be detected using . According to an embodiment, the image stabilizer 940 may be implemented as, for example, an optical image stabilizer. The memory 950 may temporarily store at least a portion of the image acquired through the image sensor 930 for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 950 and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display module 860 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a portion of the original image stored in the memory 950 may be obtained and processed by, for example, the image signal processor 960 . According to an embodiment, the memory 950 may be configured as at least a part of the memory 830 or as a separate memory operated independently of the memory 830 .

The image signal processor 960 may perform one or more image processing on an image acquired through the image sensor 930 or an image stored in the memory 950 . The one or more image processes may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening. Additionally or alternatively, the image signal processor 960 may include at least one of the components included in the camera module 880 (eg, an image sensor). 930), for example, exposure time control, readout timing control, etc. The image processed by the image signal processor 960 is stored back in the memory 950 for further processing. or may be provided as an external component of the camera module 880 (eg, the memory 830, the display module 860, the electronic device 802, the electronic device 804, or the server 808). According to an example, the image signal processor 960 may be configured as at least a part of the processor 820 or as a separate processor operated independently of the processor 820. The image signal processor 960 may be configured as the processor 820 and a separate processor, at least one image processed by the image signal processor 960 may be displayed through the display module 860 either as it is by the processor 820 or after additional image processing.

According to an embodiment, the electronic device 801 may include a plurality of camera modules 880 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 880 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 880 may be a front camera, and at least the other may be a rear camera.

According to various embodiments, the electronic device 100 includes an image sensor 120 and at least one processor electrically connected to the image sensor 120 (eg, the image signal processor 130 of FIG. 2 and/or the image signal processor 130 of FIG. 2 ). processor 220). In the first mode of the image sensor 120 , the at least one processor is configured to generate a second image generated by adding dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of valid bits. 1 Image data can be acquired. Each pixel of the first image data may have a third number of bits. In the second mode of the image sensor 120 , the at least one processor may obtain second image data in which each pixel has the third number of significant bits from the image sensor 120 .

According to an embodiment, the first image data may include identification information indicating that the dummy data is included in the first image data. The at least one processor may identify the image data having the first number of valid bits from the first image data based on the identification information.

According to an embodiment, the at least one processor may determine whether the dummy data is included in the first image data.

According to an embodiment, the first mode is a mode for outputting image data in which the effective bits of each pixel is the first number, and the second mode is image data in which the effective bits of each pixel is the third number. It may be a mode that outputs . The third number may be greater than the first number.

According to an embodiment, when the analysis result of the first image data obtained in the first mode satisfies a predefined condition, the at least one processor sets the mode of the image sensor 120 to the first mode. can be changed to the second mode.

According to an embodiment, when the analysis result of the second image data obtained in the second mode does not satisfy a predefined condition, the at least one processor sets the mode of the image sensor 120 from the second mode. It can be changed to the first mode.

According to an embodiment, the at least one processor may determine the mode change of the image sensor 120 based on brightness information of at least a partial region of the first image data or the second image data.

According to an embodiment, the at least one processor may detect the mode change event of the image sensor 120 by analyzing the M-th frame. In response to the mode change event, the at least one processor may acquire image data corresponding to the changed mode of the image sensor 120 from an N-th frame that is a frame after the M-th frame.

According to an embodiment, the at least one processor may include an image signal processor (ISP). According to an embodiment, when the image sensor 120 is in the first mode, the image signal processor (eg, the image signal processor 130 of FIG. 2 ) acquires the first image data in a first setting state to perform image processing. can be done When the image sensor 120 is changed from the first mode to the second mode, the image signal processor 130 may obtain the second image data in the first set state and perform image processing.

According to an embodiment, the electronic device 100 may further include a display (eg, the display 110 of FIG. 1 ). The at least one processor may display on the display 110 based on the image data or the second image data included in the acquired first image data.

According to various embodiments of the present disclosure, in the method of operating the electronic device 100 , in the first mode of the image sensor 120 , each pixel includes a second number of bits in the image data having the first number of valid bits. acquiring first image data generated by adding the dummy data to the image sensor; and acquiring second image data having the second number of valid bits from the image sensor in a second mode of the image sensor. can

According to an embodiment, the first image data may include identification information indicating that the dummy data is included in the first image data. The method of operating the electronic device 100 may include identifying the image data having the first number of valid bits in the first image data based on the identification information.

According to an embodiment, in the method of operating the electronic device 100, when the analysis result of the obtained first image data in the first mode satisfies a predefined condition, the mode of the image sensor is changed from the first mode. Changing the mode of the image sensor from the second mode to the first mode when the operation of changing to the second mode and the analysis result of the second image data obtained in the first mode do not satisfy a predefined condition It may include an action to

According to an embodiment, the method of operating the electronic device 100 includes acquiring the first image data in a first setting state and performing image processing when the image sensor is in the first mode, wherein the image sensor performs the image processing. When changing from the first mode to the second mode, the method may include acquiring the second image data in the first setting state and performing image processing.

According to an embodiment, the method of operating the electronic device 100 may include displaying on the display based on the image data or the second image data included in the acquired first image data.

According to various embodiments, the electronic device 100 includes an image sensor (eg, the image sensor 120 of FIG. 2 ) and at least one processor electrically connected to the image sensor 120 (eg, the image signal processor of FIG. 2 ) 130) and/or the processor 220 of FIG. 2), and an interface connecting the image sensor 120 and the at least one processor. The image sensor 120 may add dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of valid bits. The image sensor 120 may generate first image data in which each pixel has a third number of bits by adding the dummy data. The image sensor 120 may provide the generated first image data to the at least one processor through the interface. The at least one processor may identify the valid bit and the dummy data included in the first image data provided through the interface. The at least one processor may perform image processing based on the identified valid bit.

Claims

In an electronic device,

image sensor; and

At least one processor electrically connected to the image sensor,

The at least one processor comprises:

In the first mode of the image sensor, to obtain first image data generated by adding dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of effective bits, each pixel of the first image data has a third number of bits;

and, in a second mode of the image sensor, obtain from the image sensor second image data in which each pixel has the third number of significant bits.
The method according to claim 1,

The first image data includes identification information indicating that the dummy data is included in the first image data,

and the at least one processor identifies the image data having the first number of valid bits in the first image data based on the identification information.
The method according to claim 1,

The at least one processor determines whether the dummy data is included in the first image data.
The method according to claim 1,

The first mode is a mode for outputting image data in which the effective bit of each pixel is the first number, and the second mode is a mode for outputting image data in which the effective bit of each pixel is the third number, electronic device.
The method according to claim 1,

The at least one processor,

If the analysis result of the obtained first image data in the first mode satisfies a predefined condition, changing the mode of the image sensor from the first mode to the second mode,

and changing the mode of the image sensor from the second mode to the first mode when a result of analyzing the obtained second image data in the second mode does not satisfy a predefined condition.
The method according to claim 1,

The at least one processor determines the mode change of the image sensor based on brightness information of at least a partial region of the first image data or the second image data.
The method according to claim 1,

The at least one processor detects a mode change event of the image sensor by analyzing the M-th frame,

In response to the mode change event, the electronic device acquires image data corresponding to the changed mode of the image sensor from an N-th frame that is a frame after the M-th frame.
The method according to claim 1,

and the third number is greater than the first number.
The method according to claim 1,

The at least one processor includes an image signal processor (ISP),

The image signal processor performs image processing by acquiring the first image data in a first setting state when the image sensor is in the first mode,

The image signal processor obtains the second image data in the first set state and performs image processing when the image sensor is changed from the first mode to the second mode.
The method according to claim 1,

The electronic device further includes a display,

The at least one processor displays on the display based on the image data or the second image data included in the acquired first image data.
A method of operating an electronic device, comprising:

obtaining first image data generated by adding dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of significant bits in a first mode of the image sensor; each pixel of the first image data has a third number of bits; and

and obtaining, in a second mode of the image sensor, second image data having the second number of significant bits from the image sensor.
12. The method of claim 11,

The first image data includes identification information indicating that the dummy data is included in the first image data,

and identifying the image data having the first number of significant bits in the first image data based on the identification information.
12. The method of claim 11,

changing the mode of the image sensor from the first mode to the second mode when the analysis result of the acquired first image data in the first mode satisfies a predefined condition; and

and changing the mode of the image sensor from the second mode to the first mode when the analysis result of the acquired second image data in the first mode does not satisfy a predefined condition.
12. The method of claim 11,

The image signal processor may perform image processing by acquiring the first image data in a first setting state when the image sensor is in the first mode; and

and when the image sensor is changed from the first mode to the second mode, the image signal processor acquires the second image data in the first set state and performs image processing.
12. The method of claim 11,

and displaying on a display based on the image data or the second image data included in the acquired first image data.