WO2022025574A1 - Dispositif électronique comprenant un capteur d'image et un processeur de signal d'image, et son procédé - Google Patents

Dispositif électronique comprenant un capteur d'image et un processeur de signal d'image, et son procédé Download PDF

Info

Publication number
WO2022025574A1
WO2022025574A1 PCT/KR2021/009661 KR2021009661W WO2022025574A1 WO 2022025574 A1 WO2022025574 A1 WO 2022025574A1 KR 2021009661 W KR2021009661 W KR 2021009661W WO 2022025574 A1 WO2022025574 A1 WO 2022025574A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
mode
image data
processor
image sensor
Prior art date
Application number
PCT/KR2021/009661
Other languages
English (en)
Korean (ko)
Inventor
전혜경
강호석
김동수
이성구
홍웅호
원종훈
Original Assignee
삼성전자 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성전자 주식회사 filed Critical 삼성전자 주식회사
Publication of WO2022025574A1 publication Critical patent/WO2022025574A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H04N25/581—Control of the dynamic range involving two or more exposures acquired simultaneously
    • H04N25/583—Control of the dynamic range involving two or more exposures acquired simultaneously with different integration times
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H01L27/14601—Structural or functional details thereof
    • H01L27/14609—Pixel-elements with integrated switching, control, storage or amplification elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/71—Circuitry for evaluating the brightness variation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H04N25/581—Control of the dynamic range involving two or more exposures acquired simultaneously
    • H04N25/585—Control of the dynamic range involving two or more exposures acquired simultaneously with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith

Definitions

  • Embodiments disclosed in this document relate to driving of an image sensor and an image signal processor.
  • image processing is performed from the original image to suit the purpose of using the image.
  • the range of color expression in an image is becoming more important.
  • a human can recognize the dynamic range of an image having a limited color expression range compared to the human perceptible dynamic range (hereafter, dynamic range).
  • High dynamic range (hereafter, HDR, high dynamic range) processing that expands to an existing level is performed.
  • the size of effective data output by the image sensor may be different from each other. Therefore, in order to acquire image data having different sizes of effective data in the image signal processor, the image signal processor is changed to match the size of the effective data to obtain data and process the image.
  • the image signal processor stops the operation in progress and resets the image signal processor according to the change in the data output from the image sensor After that, it worked again.
  • the interruption time between images was 300 ms or more, which has been a factor hindering usability.
  • dummy data is added to keep the size of data acquired by the image signal processor constant to perform image processing without stopping the operation. It is possible to provide an electronic device and a method for controlling the electronic device that can process the data without interruption.
  • An electronic device may include an image sensor and at least one processor electrically connected to the image sensor.
  • the at least one processor is configured to generate a first image generated by adding dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of significant bits. obtain data, wherein each pixel of the first image data has a third number of bits, in a second mode of the image sensor, a second from the image sensor wherein each pixel has the third number of significant bits Image data can be acquired.
  • a second number of bits in each pixel obtaining first image data generated by adding dummy data having It may include an action to
  • the electronic device may include an image sensor, at least one processor electrically connected to the image sensor, and an interface connecting the image sensor and the at least one processor.
  • the image sensor adds dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of valid bits, and adds the dummy data so that each pixel receives a third number of bits
  • the branch may generate first image data, and provide the generated first image data to the at least one processor through the interface.
  • the at least one processor may identify the valid bit and the dummy data included in the first image data provided through the interface, and perform image processing based on the identified valid bit.
  • the size of data output from the image sensor may be constantly maintained even when the surrounding environment changes, such as a bright outdoor environment or a backlight environment.
  • data output from the image sensor may be acquired without resetting the image signal processor.
  • FIG. 1 is a diagram illustrating a structure of an electronic device and a camera module according to an embodiment.
  • FIG. 2 illustrates a hardware configuration of an electronic device according to an embodiment.
  • FIG 3 illustrates image processing when an image having a first dynamic range (eg, a normal mode) is output by an electronic device according to an exemplary embodiment.
  • a first dynamic range eg, a normal mode
  • FIG. 4 illustrates image processing when an image having a second dynamic range (eg, HDR) higher than a first dynamic range is output by the electronic device, according to an exemplary embodiment.
  • a second dynamic range eg, HDR
  • FIG 5 illustrates an operation of an image signal processor for each mode in an electronic device according to an embodiment.
  • FIG. 6 illustrates a process of processing an image according to a mode by an image sensor, an image signal processor, and a display according to an exemplary embodiment.
  • FIG. 7 is a flowchart illustrating an operation process in an image signal processor of an electronic device according to an exemplary embodiment.
  • FIG. 8 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.
  • FIG. 9 is a block diagram illustrating a camera module according to various embodiments.
  • FIG. 1 is a diagram illustrating a structure of an electronic device and a camera module according to an embodiment.
  • FIG. 1 is a diagram schematically illustrating an exterior and a camera module 180 of an electronic device 100 on which a camera module 180 is mounted, according to an embodiment.
  • FIG. 1 has been illustrated and described on the premise of a mobile device, in particular, a smart phone, it is to those skilled in the art that it can be applied to various electronic devices or electronic devices equipped with a camera among mobile devices. will be clearly understood.
  • the display 110 may be disposed on the front surface of the electronic device 100 according to an embodiment.
  • the display 110 may occupy most of the front surface of the electronic device 100 .
  • a display 110 and a bezel 190 region surrounding at least some edges of the display 110 may be disposed on the front surface of the electronic device 100 .
  • the display 110 may include a flat area and a curved area extending from the flat area toward the side of the electronic device 100 .
  • the electronic device 100 illustrated in FIG. 1 is an example, and various embodiments are possible.
  • the display 110 of the electronic device 100 may include only a flat area without a curved area, or may include a curved area only at one edge instead of both sides.
  • the curved area may extend toward the rear surface of the electronic device, so that the electronic device 100 may include an additional planar area.
  • the electronic device 100 may additionally include a speaker, a receiver, a front camera, a proximity sensor, a home key, and the like.
  • the electronic device 100 may be provided in which the rear cover 150 is integrated with the main body of the electronic device.
  • the rear cover 150 may be separated from the main body of the electronic device 100 to have a form in which the battery can be replaced.
  • the back cover 150 may be referred to as a battery cover or a back cover.
  • a fingerprint sensor 171 for recognizing a user's fingerprint may be included in the first area 170 of the display 110 . Since the fingerprint sensor 171 is disposed on a lower layer of the display 110 , the fingerprint sensor 171 may not be recognized by the user or may be difficult to recognize. Also, in addition to the fingerprint sensor 171 , a sensor for additional user/biometric authentication may be disposed in a portion of the display 110 . In another embodiment, a sensor for user/biometric authentication may be disposed on one area of the bezel 190 . For example, the IR sensor for iris authentication may be exposed through one area of the display 110 or may be exposed through one area of the bezel 190 .
  • the front camera 161 may be disposed in the second area 160 on the front side of the electronic device 100 .
  • the front camera 161 is shown to be exposed through one area of the display 110 , but in another embodiment, the front camera 161 may be exposed through the bezel 190 .
  • the electronic device 100 may include one or more front cameras 161 .
  • the electronic device 100 may include two front cameras, such as a first front camera and a second front camera.
  • the first front camera and the second front camera may be cameras of the same type having the same specifications (eg, pixels), but the first front camera and the second front camera may be implemented as cameras of different specifications.
  • the electronic device 100 may support a function (eg, 3D imaging, auto focus, etc.) related to a dual camera through two front cameras. The above-mentioned description of the front camera may be equally or similarly applied to the rear camera of the electronic device 100 .
  • various hardware or sensors 163 to assist photographing such as a flash, may be additionally disposed.
  • a distance sensor eg, TOF sensor
  • the distance sensor may be applied to both a front camera and/or a rear camera.
  • the distance sensor may be separately disposed or included and disposed on the front camera and/or the rear camera.
  • At least one physical key may be disposed on a side portion of the electronic device 100 .
  • the first function key 151 for turning on/off the display 110 or turning on/off the power of the electronic device 100 may be disposed on the right edge with respect to the front surface of the electronic device 100 .
  • the second function key 152 for controlling the volume or screen brightness of the electronic device 100 may be disposed on the left edge with respect to the front surface of the electronic device 100 .
  • additional buttons or keys may be disposed on the front or rear of the electronic device 100 .
  • a physical button or a touch button mapped to a specific function may be disposed in a lower region of the front bezel 190 .
  • the electronic device 100 illustrated in FIG. 1 corresponds to one example, and the shape of the device to which the technical idea disclosed in the present disclosure is applied is not limited.
  • a foldable electronic device that can be folded horizontally or vertically, a rollable electronic device that can be rolled, a tablet or a notebook computer
  • the technical idea of the present disclosure This can be applied.
  • the present technical idea can be applied even when it is possible to arrange the first camera and the second camera facing in the same direction to face different directions through rotation, folding, deformation, etc. of the device.
  • an electronic device 100 may include a camera module 180 .
  • the camera module 180 includes a lens assembly 111 , a housing 113 , an infrared cut filter 115 , an image sensor 120 , and an image signal processor (ISP) 130 .
  • ISP image signal processor
  • the lens assembly 111 may have a different number, arrangement, type, etc. of lenses depending on the front camera and the rear camera.
  • the front camera and the rear camera may have different characteristics (eg, focal length, maximum magnification, etc.).
  • the lens may be moved forward and backward along the optical axis, and may operate so that a target object, which is a subject, can be clearly captured by changing a focal length.
  • the camera module 180 includes a housing 113 for mounting at least one coil surrounding the periphery of the barrel around the optical axis and a barrel for mounting at least one or more lenses aligned on the optical axis.
  • the infrared cut filter 115 may be disposed on the upper surface of the image sensor 120 .
  • the image of the subject passing through the lens may be partially filtered by the infrared cut filter 115 and then detected by the image sensor 120 .
  • the image sensor 120 may be disposed on the upper surface of the printed circuit board 140 .
  • the image sensor 120 may be electrically connected to the image signal processor 130 connected to the printed circuit board 140 by a connector.
  • a flexible printed circuit board (FPCB) or a cable may be used as the connector.
  • the image sensor 120 may be a complementary metal oxide semiconductor (CMOS) sensor or a charged coupled device (CCD) sensor.
  • CMOS complementary metal oxide semiconductor
  • CCD charged coupled device
  • a plurality of individual pixels are integrated in the image sensor 120 , and each individual pixel may include a micro lens, a color filter, and a photodiode.
  • Each individual pixel is a kind of photodetector that can convert incoming light into an electrical signal. Photodetectors generally cannot detect the wavelength of the captured light by themselves and cannot determine color information.
  • the photodetector may include a photodiode.
  • light information of a subject incident through the lens assembly 111 may be converted into an electrical signal by the image sensor 120 and input to the image signal processor 130 .
  • the camera module 180 may be disposed on the front side as well as the rear side of the electronic device 100 .
  • the electronic device 100 may include a plurality of camera modules 180 as well as one camera module 180 to improve camera performance.
  • the electronic device 100 may further include a front camera 161 for video call or self-camera photography.
  • the front camera 161 may support a relatively low number of pixels compared to the rear camera module.
  • the front camera may be relatively smaller than the rear camera module.
  • FIG. 2 illustrates a hardware configuration of an electronic device according to an embodiment.
  • the configuration illustrated in FIG. 1 may be briefly described or a description thereof may be omitted.
  • the electronic device 100 may include a camera module 180 , a processor 220 , a display 110 , and a memory 230 .
  • the camera module 180 may include an image sensor 120 and an image signal processor 130 .
  • descriptions of the same reference numerals as those of FIG. 1 may be omitted.
  • the electronic device 100 may further include additional components.
  • the electronic device 100 may further include at least one microphone for recording audio data.
  • the electronic device 100 may include at least one sensor for determining a direction in which the front or rear of the electronic device 100 faces and/or posture information of the electronic device 100 .
  • the at least one sensor may include an acceleration sensor, a gyro sensor, and the like.
  • the image sensor 120 may include a complementary metal oxide semiconductor (CMOS) sensor or a charged coupled device (CCD) sensor.
  • CMOS complementary metal oxide semiconductor
  • CCD charged coupled device
  • the light information of the subject incident through the lens assembly 111 may be converted into an electrical signal by the image sensor 120 and input to the image signal processor 130 .
  • An infrared cut filter (hereinafter, IR cut filter) may be disposed on the upper surface of the image sensor 120 , and the image of the subject passing through the lens is partially filtered by the IR cut filter and then the image sensor 120 ) can be detected by IR cut filter.
  • a sensor interface conforming to an appropriate standard may electrically connect the image sensor 120 and the image signal processor 130 .
  • the image signal processor 130 may perform image processing on the electrically converted image data.
  • a process in the image signal processor 130 may be divided into a pre-ISP (hereinafter, pre-processing) and an ISP chain (hereinafter, post-processing).
  • Image processing before the demosaicing process may mean pre-processing, and image processing after the demosaicing process may mean post-processing.
  • the preprocessing process may include 3A processing, lens shading correction, edge enhancement, dead pixel correction, knee correction, and the like.
  • 3A may include at least one of auto white balance (AWB), auto exposure (AE), and auto focusing (AF).
  • the post-processing process may include at least one of changing a sensor index value, changing a tuning parameter, and adjusting an aspect ratio.
  • the post-processing process may include processing the image data output from the image sensor 120 or image data output from the scaler.
  • the image signal processor 130 may adjust contrast, sharpness, saturation, dithering, etc. of the image through a post-processing process.
  • the contrast, sharpness, and saturation adjustment procedures are performed in the YUV color space, and the dithering procedure may be performed in the RGB (Red Green Blue) color space.
  • a part of the pre-processing process may be performed in the post-processing process, or a part of the post-processing process may be performed in the pre-processing process.
  • a part of the pre-processing process may be overlapped with a part of the post-processing process.
  • the display 110 may display contents such as an execution screen of an application executed by the processor 220 or images and/or videos stored in the memory 230 on the display 110 .
  • the processor 220 may display the image data acquired through the camera module 180 on the display 110 in real time.
  • the display 110 may be implemented integrally with the touch panel.
  • the display 110 may support a touch function, detect a user input such as a touch using a finger, and transmit it to the processor 220 .
  • the display 110 may be connected to a display driver integrated circuit (DDIC) for driving the display 110 , and the touch panel may be connected to a touch IC that detects touch coordinates and processes a touch-related algorithm.
  • DDIC display driver integrated circuit
  • the display driving circuit and the touch IC may be integrally formed, and in another embodiment, the display driving circuit and the touch IC may be formed separately.
  • the display driving circuit and/or the touch IC may be electrically connected to the processor 220 .
  • the processor 220 may execute/control various functions supported by the electronic device 100 .
  • the processor 220 may execute an application by executing a code written in a programming language stored in the memory 230 , and may control various hardware.
  • the processor 220 may execute an application supporting a photographing function stored in the memory 230 .
  • the processor 220 may execute the camera module 180 and set and support an appropriate shooting mode so that the camera module 180 may perform an operation intended by the user.
  • the memory 230 may store instructions executable by the processor 220 .
  • the memory 230 may be understood as a concept including a component in which data is temporarily stored, such as a random access memory (RAM), and/or a component in which data is permanently stored, such as a solid state drive (SSD).
  • the processor 220 may implement a software module in the RAM space by calling instructions stored in the SSD.
  • the memory 230 may include various types, and an appropriate type may be adopted according to the purpose of the device.
  • an application related to the camera module 180 may be stored in the memory 230 .
  • a camera application may be stored in the memory 230 .
  • the camera application may support various shooting functions, such as photo shooting, video shooting, panoramic shooting, and slow motion shooting.
  • an application associated with the camera module 180 may correspond to various types of applications.
  • a chatting application may also use the camera module 180 to support a video call, photo/video attachment, streaming service, product image, or product-related virtual reality (VR) shooting function.
  • VR virtual reality
  • FIG 3 illustrates image processing when an image having a first dynamic range (eg, a normal mode) is output by an electronic device according to an exemplary embodiment.
  • a first dynamic range eg, a normal mode
  • dummy data is added to the image data having the first dynamic range through the image sensor 120 or a separate hardware module. can be added.
  • the valid bit referred to in this specification may relate to image data output through the image sensor 120 .
  • the effective bit may mean data output by each pixel of the image sensor 120 by light incident through the camera lens.
  • the effective bit may mean a color value of each pixel.
  • the color value may include color information and brightness information.
  • effective bits of a pixel include color information of at least one of R, G, and B may include
  • the green, red, and blue are only examples of color values, and the color values are not limited.
  • the color value may be at least one of red, green, blue, yellow, emerald, white, cyan, and magenta. .
  • the color filter array may include a red, green, blue, emerald (RGBE) pattern, a cyan, yellow, magenta (CYYM) pattern, a cyan, yellow, green, magenta (CYGM) pattern, or a red, green (RGBW) pattern.
  • RGBE emerald
  • CYYM yellow, magenta
  • CYGM cyan, yellow, green, magenta
  • RGBW red, green
  • blue, white may include a color filter array of a pattern.
  • the image sensor 120 may output image data 310 having a color value for each pixel.
  • new first image data 320 may be generated by adding dummy data to the output image data 310 .
  • the image sensor 120 outputs image data in which the number of valid bits of each pixel is a first number (eg, 10), and a second number (eg, 2) for each pixel in a separate hardware module Dummy data having a number of bits) can be added.
  • the separate hardware module may be positioned between the image sensor 120 and the image signal processor 130 and may be electrically connected to the image sensor 120 and the image signal processor 130 .
  • the image data output by the image sensor 120 may be provided to the image signal processor 130 through the hardware module.
  • the image sensor 120 may add dummy data to the image data 310 acquired through the pixel array.
  • the image sensor 120 may generate the first image data 320 by adding the dummy data.
  • the image sensor 120 adds dummy data having a second number (eg, 2) bits per pixel to the image data 310 having a first number (eg, 10) of effective bits.
  • the first image data 320 having the third number (eg, 12) bits of each pixel may be generated.
  • the image sensor 120 may output the first image data 320 generated by adding the dummy data.
  • the first image data 320 may include identification information for identifying unique data and dummy data included in the image data.
  • the identification information may be referred to as a delimiter or identifier that distinguishes the unique data from the dummy data.
  • the dummy data may be referred to as a dummy bit.
  • the first number, the second number, and the third number mentioned in this document are not limited to the number given as an example, and it can be assumed that the third number is the same as the sum of the first number and the second number. .
  • the image signal processor 130 may acquire the generated first image data 320 in a first setting state.
  • the image signal processor 130 may acquire the first image data 320 and, at the same time, acquire identification information included in the first image data 320 .
  • the image signal processor 130 may perform image analysis on the first image data 320 .
  • Image analysis of the first image data 320 may be performed through the processor 220 .
  • the image analysis may be to determine whether the acquired image data needs to be processed in a second dynamic range (eg, HDR) higher than the first dynamic range.
  • the processor 220 may determine that HDR processing is necessary according to a surrounding environment or a main subject. For example, when a surrounding environment such as a bright outdoor or backlight environment is detected, it may be determined that HDR processing is necessary.
  • the processor 220 may control the mode of the image sensor 120 based on a result of image analysis. As a result of the image signal processor 130 or the processor 220 analyzing the image, when the photographing environment is not changed, the processor 220 may maintain the output mode of the image sensor 120 . As a result of the image signal processor 130 or the processor 220 analyzing the image, in an environment requiring HDR processing, the processor 220 sets the output mode of the image sensor 120 to the image sensor 120 illustrated in FIG. 4 below. ) can be changed to the output mode.
  • the image signal processor 130 may perform image processing on the first image data 320 .
  • the image signal processor 130 may perform image processing based on the image data 310 included in the first image data 320 . In other words, the image signal processor 130 may perform image processing on valid data among the acquired image data.
  • the image signal processor 130 may provide image data on which image processing is performed to the processor 220 .
  • the processor 220 may control the provided image data to be stored in the memory 230 or to be displayed on the display 110 .
  • FIG. 4 illustrates image processing when an image having a second dynamic range (eg, HDR) higher than a first dynamic range is output by the electronic device, according to an exemplary embodiment.
  • a second dynamic range eg, HDR
  • the image sensor 120 may output second image data 410 having a color value for each pixel.
  • the second image data 410 may have a larger number of valid bits in each pixel than the image data 310 of FIG. 3 .
  • the second image data 410 may include a larger number of color values or a wider color range than the image data 310 of FIG. 3 .
  • the image sensor 120 may output image data in which the number of valid bits of each pixel is a third number (eg, 12).
  • the image signal processor 130 may acquire the output second image data 410 in a first setting state, similar to the embodiment of FIG. 3 .
  • the image signal processor 130 may perform image analysis on the second image data 410 .
  • Image analysis of the second image data 410 may be performed through the processor 220 .
  • the processor 220 may control the mode of the image sensor 120 based on a result of image analysis. As a result of the image signal processor 130 or the processor 220 analyzing the image, the processor 220 may maintain the output mode of the image sensor 120 when the photographing environment is not changed. As a result of the image signal processor 130 or the processor 220 analyzing the image, if the HDR processing is not required, the processor 220 sets the output mode of the image sensor 120 to the image sensor 120 illustrated in FIG. 3 . ) can be changed to the output mode.
  • the image signal processor 130 may perform image processing on the second image data 410 .
  • the image signal processor 130 may perform HDR processing.
  • the HDR processing may be a process to extend the dynamic range of an image having a limited dynamic region to a level that human vision can perceive in order to distinguish objects.
  • the HDR processing is not limited to a specific method, and may be performed by a method widely known in the art.
  • the image signal processor 130 may provide image data on which image processing is performed to the processor 220 .
  • the processor 220 may control the provided image data to be stored in the memory 230 or to be displayed on the display 110 .
  • FIG. 5 illustrates an operation of an image signal processor for each mode in an electronic device according to an embodiment.
  • the operating subject of the flowchart illustrated in FIG. 5 may be understood as a processor (eg, the processor 220 of FIG. 2 ) or an image signal processor (eg, the image signal processor 130 of FIG. 1 ).
  • the image signal processor 130 in the first mode of the image sensor 120 , includes a second number of each pixel in the image data in which each pixel has a first number of significant bits. First image data generated by adding dummy data having bits may be obtained. The dummy data is for making the number of bits of each pixel of the first image data obtained by the image signal processor 130 equal to the number of bits of each pixel of the second image data obtained in the HDR environment. It can be data.
  • the image signal processor 130 may acquire the first image data in a first setting state.
  • the first setting state may be a setting mode operated when the number of bits of the image data acquired by the image signal processor 130 is the third number.
  • a second mode change event may occur.
  • the second mode change event may be an event related to changing the output mode of the image sensor 120 from the first mode to the second mode.
  • the processor 220 may control the mode of the image sensor 120 to be changed from the first mode to the second mode.
  • the first mode may be a mode for outputting image data having the first number of effective bits of each pixel of the image data
  • the second mode may be a mode for outputting image data having the third number of effective bits of each pixel of the image data.
  • the third number may be greater than the first number.
  • the first number may be 10, and the third number may be 12.
  • the definitions of the first mode and the second mode may be equally applied to the first mode and the second mode mentioned in this document.
  • the image signal processor 130 may acquire second image data having a third number of valid bits.
  • the image signal processor 130 may acquire the second image data in a first setting state. Since the second image data has the same third number of bits in each pixel as in the first image data, the image signal processor 130 may acquire the second image data in the first setting state.
  • FIG. 6 illustrates a process of processing an image according to a mode by an image sensor, an image signal processor, and a display according to an exemplary embodiment.
  • An operation performed by the image signal processor illustrated in FIG. 6 may be performed by a processor (eg, the processor 220 of FIG. 2 ).
  • the image sensor 610 may output image data having a first number of valid bits of each pixel.
  • the image sensor 120 may add dummy data having a second number of bits to image data having a first number of effective bits of each pixel.
  • the image sensor 120 may output image data to which the dummy data is added.
  • the image signal processor 130 transmits the image data in which the number of effective bits of each pixel is the first number to the second image data for each pixel.
  • Image data to which dummy data having the number of bits is added may be obtained.
  • the number of bits of each pixel may be the third number.
  • the image signal processor 130 may acquire image data to which the dummy data is added in a first set state.
  • the image signal processor 130 may perform image processing while maintaining the same setting state regardless of the mode change of the image sensor 120 . For example, when the image sensor 120 is in the first mode, the image signal processor 130 adds a second number of dummy data to the first number of valid data for each pixel so that each pixel has a third number of bits may operate in a first setting state for processing the first image data having
  • the display 110 may output a preview.
  • the display 110 may output a preview based on image data in which the number of effective bits of each pixel is the first number.
  • the display 110 may receive a preview format including the image data from the processor 220 and may output a preview based on the preview format.
  • the image signal processor 130 may analyze the first image data to which the obtained dummy data is added.
  • the image signal processor 130 analyzes the first image data including the dummy data or calculates a first number of valid bits per pixel except for the dummy data. It can analyze image data.
  • a second mode change event may occur. This may correspond to operation 520 of FIG. 5 .
  • the image sensor 120 may receive a second mode change signal.
  • the processor 220 may provide a second mode change signal to the image sensor 120 through the designated interface.
  • the image sensor 120 may receive the second mode change signal provided by the processor 220 through the designated interface.
  • the image sensor 120 may change the output mode from the first mode to the second mode in response to receiving the mode change signal.
  • the image sensor 610 may output image data in which each pixel has a third number of valid bits.
  • the image signal processor 130 may obtain image data in which each pixel has a third number of significant bits. In comparison with operation 615 , the image signal processor 130 may acquire second image data in which the number of bits included in each pixel is the same as that of the first image data.
  • the image signal processor 130 may perform image processing while maintaining operation regardless of a mode change of the image sensor 120 .
  • the image signal processor 130 may maintain the first set state. Since the number of bits of each pixel is the same, in the second mode, the image signal processor 130 sets the first set state as in the first mode with respect to the second image data in which the number of effective bits of each pixel is the third number. can operate as
  • the display 110 may output a preview.
  • the display 110 may output a preview based on image data in which the number of effective bits of each pixel is the third number.
  • the display 110 may display image data output from the image sensor 120 on which image processing is performed.
  • FIG. 7 is a flowchart illustrating an operation process in an image signal processor of an electronic device according to an exemplary embodiment.
  • the operating subject of the flowchart illustrated in FIG. 7 may be understood as a processor (eg, the processor 220 of FIG. 2 ) or an image signal processor (eg, the image signal processor 130 of FIG. 1 ).
  • the image signal processor 130 may obtain identification information on a valid bit. Alternatively, the image signal processor 130 may obtain identification information on the dummy data. The image signal processor 130 may acquire identification information on the dummy data by acquiring image data including identification information indicating that the dummy data is included. The identification information may be referred to as a delimiter or identifier (ID) that distinguishes the unique data from the dummy data. The identification information may be included in a header area called embedded data and transmitted to the image signal processor 130 together with image data acquired through the image sensor 120 .
  • ID delimiter or identifier
  • the image signal processor 130 may recognize the number of valid bits based on the identification information on the dummy data.
  • the image signal processor 130 may classify valid bits and dummy data among bits of each pixel based on identification information included in the dummy data.
  • the image signal processor 130 may recognize the output mode of the image sensor 120 based on the identification information on the dummy data. When dummy data is included in the acquired image data, the image signal processor 130 may determine that the output mode of the image sensor 120 is the first mode. For example, when each pixel of the image data acquired by the image signal processor 130 includes a first number (eg, 10) of valid bits and a second number (eg, two) of dummy data, the image The signal processor 130 may determine that the output mode of the image sensor 120 is the first mode. When the dummy data is not included in the acquired image data, the image signal processor 130 may determine that the output mode of the image sensor 120 is the second mode.
  • a first number eg, 10
  • a second number eg, two
  • the image signal processor 130 or the processor 220 may determine that the output mode of the image sensor 120 is the second mode.
  • the image signal processor 130 may perform image processing based on the number of valid bits of each pixel of the image data and dummy data.
  • the image signal processor 130 may determine a range of data to be image processed based on identification information included in the image data. For example, when the acquired image data includes identification information on dummy data, the image signal processor 130 may generate image data (eg, a figure in which the number of valid bits of each pixel is a first number (eg, 10)) Image processing may be performed based on the image data 310 of FIG. 3 . As another example, if the image signal processor 130 does not include identification information in the acquired image data, the image data (eg, figure Image processing may be performed based on the second image data 410 of FIG. 4 .
  • image data eg, figure Image processing may be performed based on the second image data 410 of FIG. 4 .
  • the image signal processor 130 may determine whether it is necessary to change the mode through image analysis.
  • the image signal processor 130 (or the processor 220 ) may determine the HDR environment through analysis of the acquired image data. In an embodiment, the processor 220 may determine that it is necessary to change the mode when determining that the HDR environment is present.
  • the image signal processor 130 may determine whether it is an HDR environment based on brightness information of the image.
  • the image signal processor 130 (or the processor 220 ) uses an auto exposure (AE) function so that the first area of the image has a first brightness greater than or equal to the first threshold, and the second area differentiated from the first area is When the second brightness is lower than the second threshold, which is darker than the first brightness, the HDR environment may be determined.
  • the image signal processor 130 and/or the processor 220 may determine that the output mode of the image sensor 120 needs to be changed when detecting the HDR environment.
  • the processor 220 may analyze a scene of the captured image to determine whether it is an HDR environment.
  • the processor 220 may analyze the scene of the captured image to determine the HDR environment when the shooting background is a backlight environment or a bright outdoor environment.
  • the processor 220 may determine whether the captured image is a backlight environment or a bright outdoor environment through a function (eg, a scene optimizer) on the image obtained through the camera.
  • the one function eg, a scene optimizer
  • the one function may be a function capable of discriminating an object, a background, etc. based on data based on machine learning.
  • the processor 220 may determine that it is a backlight environment.
  • the image signal processor 130 (or the processor 220 ) may determine that it is necessary to change the output mode of the image sensor 120 when detecting that the HDR environment is present.
  • the image signal processor 130 may provide a change signal to the image sensor 120 .
  • the processor 220 may provide a signal for changing to the second mode to the image sensor 120 and simultaneously control the mode of the image sensor 120 to be changed to the second mode.
  • FIG. 8 is a block diagram of an electronic device (eg, the electronic device 100 of FIG. 1 ) 801 in the network environment 800 according to various embodiments of the present disclosure.
  • the electronic device 801 communicates with the electronic device 802 through a first network 898 (eg, a short-range wireless communication network) or a second network 899 . It may communicate with the electronic device 804 or the server 808 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 801 may communicate with the electronic device 804 through the server 808 .
  • the electronic device 801 includes a processor 820 , a memory 830 , an input module 850 , a sound output module 855 , a display module 860 , an audio module 870 , and a sensor module ( 876), interface 877, connection terminal 878, haptic module 879, camera module 880, power management module 888, battery 889, communication module 890, subscriber identification module 896 , or an antenna module 897 .
  • at least one of these components eg, the connection terminal 878
  • some of these components are integrated into one component (eg, display module 860 ). can be
  • the processor 820 executes software (eg, the program 840 ) to be at least one other component of the electronic device 801 connected to the processor 820 . It can control elements (eg, hardware or software components) and can perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 820 stores commands or data received from other components (eg, the sensor module 876 or the communication module 890 ) into the volatile memory 832 . may be stored in , process commands or data stored in the volatile memory 832 , and store the result data in the non-volatile memory 834 .
  • software eg, the program 840
  • the processor 820 stores commands or data received from other components (eg, the sensor module 876 or the communication module 890 ) into the volatile memory 832 .
  • the processor 820 is a main processor 821 (eg, a central processing unit or an application processor) or a secondary processor 823 (eg, a graphics processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor).
  • a main processor 821 eg, a central processing unit or an application processor
  • a secondary processor 823 eg, a graphics processing unit, a neural network processing unit
  • NPU neural processing unit
  • an image signal processor e.g., a sensor hub processor, or a communication processor.
  • the electronic device 801 includes a main processor 821 and a sub-processor 823
  • the sub-processor 823 uses less power than the main processor 821 or is set to be specialized for a specified function.
  • the coprocessor 823 may be implemented separately from or as part of the main processor 821 .
  • the coprocessor 823 may, for example, act on behalf of the main processor 821 while the main processor 821 is in an inactive (eg, sleep) state, or when the main processor 821 is active (eg, executing an application). ), together with the main processor 821, at least one of the components of the electronic device 801 (eg, the display module 860, the sensor module 876, or the communication module 890) It is possible to control at least some of the related functions or states.
  • the coprocessor 823 eg, an image signal processor or a communication processor
  • may be implemented as part of another functionally related component eg, the camera module 880 or the communication module 890 ). have.
  • the auxiliary processor 823 may include a hardware structure specialized for processing an artificial intelligence model.
  • Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 801 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 808).
  • the learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited
  • the artificial intelligence model may include a plurality of artificial neural network layers.
  • Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example.
  • the artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.
  • the memory 830 may store various data used by at least one component (eg, the processor 820 or the sensor module 876 ) of the electronic device 801 .
  • the data may include, for example, input data or output data for software (eg, the program 840 ) and commands related thereto.
  • the memory 830 may include a volatile memory 832 or a non-volatile memory 834 .
  • the program 840 may be stored as software in the memory 830 , and may include, for example, an operating system 842 , middleware 844 , or an application 846 .
  • the input module 850 may receive a command or data to be used in a component (eg, the processor 820 ) of the electronic device 801 from the outside (eg, a user) of the electronic device 801 .
  • the input module 850 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).
  • the sound output module 855 may output a sound signal to the outside of the electronic device 801 .
  • the sound output module 855 may include, for example, a speaker or a receiver.
  • the speaker can be used for general purposes such as multimedia playback or recording playback.
  • the receiver may be used to receive an incoming call. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.
  • the display module 860 may visually provide information to the outside (eg, a user) of the electronic device 801 .
  • the display module 860 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device.
  • the display module 860 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.
  • the audio module 870 may convert a sound into an electrical signal or, conversely, convert an electrical signal into a sound. According to an embodiment, the audio module 870 acquires a sound through the input module 850 or an external electronic device (eg, a sound output module 855 ) directly or wirelessly connected to the electronic device 801 . A sound may be output through the electronic device 802 (eg, a speaker or headphones).
  • an external electronic device eg, a sound output module 855
  • a sound may be output through the electronic device 802 (eg, a speaker or headphones).
  • the sensor module 876 detects an operating state (eg, power or temperature) of the electronic device 801 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do.
  • the sensor module 876 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.
  • the interface 877 may support one or more designated protocols that may be used for the electronic device 801 to directly or wirelessly connect with an external electronic device (eg, the electronic device 802 ).
  • the interface 877 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
  • connection terminal 878 may include a connector through which the electronic device 801 can be physically connected to an external electronic device (eg, the electronic device 802 ).
  • the connection terminal 878 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).
  • the haptic module 879 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense.
  • the haptic module 879 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
  • the camera module 880 may capture still images and moving images. According to an embodiment, the camera module 880 may include one or more lenses, image sensors, image signal processors, or flashes.
  • the power management module 888 may manage power supplied to the electronic device 801 .
  • the power management module 888 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).
  • PMIC power management integrated circuit
  • the battery 889 may supply power to at least one component of the electronic device 801 .
  • the battery 889 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.
  • the communication module 890 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 801 and an external electronic device (eg, the electronic device 802, the electronic device 804, or the server 808). It can support establishment and communication performance through the established communication channel.
  • the communication module 890 may include one or more communication processors that operate independently of the processor 820 (eg, an application processor) and support direct (eg, wired) communication or wireless communication.
  • the communication module 890 is a wireless communication module 892 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 894 (eg, : It may include a LAN (local area network) communication module, or a power line communication module).
  • a corresponding communication module is a first network 898 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 899 (eg, legacy).
  • the wireless communication module 892 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 896 within a communication network, such as the first network 898 or the second network 899 .
  • the electronic device 801 may be identified or authenticated.
  • the wireless communication module 892 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR).
  • NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)).
  • eMBB enhanced mobile broadband
  • mMTC massive machine type communications
  • URLLC ultra-reliable and low-latency
  • the wireless communication module 892 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example.
  • a high frequency band eg, mmWave band
  • the wireless communication module 892 uses various technologies for securing performance in a high frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna.
  • the wireless communication module 892 may support various requirements specified in the electronic device 801 , an external electronic device (eg, the electronic device 804 ), or a network system (eg, the second network 899 ).
  • the wireless communication module 892 includes a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).
  • a peak data rate eg, 20 Gbps or more
  • loss coverage eg, 164 dB or less
  • U-plane latency for realizing URLLC
  • the antenna module 897 may transmit or receive a signal or power to the outside (eg, an external electronic device).
  • the antenna module 897 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern.
  • the antenna module 897 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 898 or the second network 899 is connected from the plurality of antennas by, for example, the communication module 890 . can be selected. A signal or power may be transmitted or received between the communication module 890 and an external electronic device through the selected at least one antenna.
  • other components eg, a radio frequency integrated circuit (RFIC)
  • RFIC radio frequency integrated circuit
  • the antenna module 897 may form a mmWave antenna module.
  • the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a specified high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.
  • peripheral devices eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)
  • GPIO general purpose input and output
  • SPI serial peripheral interface
  • MIPI mobile industry processor interface
  • a command or data may be transmitted or received between the electronic device 801 and the external electronic device 804 through the server 808 connected to the second network 899 .
  • Each of the external electronic devices 802 or 804 may be the same as or different from the electronic device 801 .
  • all or part of operations performed by the electronic device 801 may be executed by one or more external electronic devices 802 , 804 , or 808 .
  • the electronic device 801 may instead of executing the function or service itself.
  • one or more external electronic devices may be requested to perform at least a part of the function or the service.
  • One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 801 .
  • the electronic device 801 may process the result as it is or additionally and provide it as at least a part of a response to the request.
  • cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used.
  • the electronic device 801 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing.
  • the external electronic device 804 may include an Internet of things (IoT) device.
  • the server 808 may be an intelligent server using machine learning and/or neural networks.
  • the external electronic device 804 or the server 808 may be included in the second network 899 .
  • the electronic device 801 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.
  • the electronic device may be a device of various types.
  • the electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device.
  • a portable communication device eg, a smart phone
  • a computer device e.g., a laptop, a desktop, a tablet, or a portable multimedia device
  • portable medical device e.g., a portable medical device
  • camera e.g., a camera
  • a wearable device e.g., a smart watch
  • a home appliance device e.g., a smart bracelet
  • first”, “second”, or “first” or “second” may simply be used to distinguish the component from other components in question, and may refer to components in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.
  • module used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, for example, and interchangeably with terms such as logic, logic block, component, or circuit.
  • a module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions.
  • the module may be implemented in the form of an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • one or more instructions stored in a storage medium may be implemented as software (eg, the program 840) including
  • the processor eg, the processor 820 of the device (eg, the electronic device 801 ) may call at least one of one or more instructions stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command.
  • the one or more instructions may include code generated by a compiler or code executable by an interpreter.
  • the device-readable storage medium may be provided in the form of a non-transitory storage medium.
  • 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.
  • a signal eg, electromagnetic wave
  • the methods according to various embodiments disclosed in this document may be provided by being included in a computer program product.
  • Computer program products may be traded between sellers and buyers as commodities.
  • the computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store TM ) or on two user devices ( It can be distributed online (eg download or upload), directly between smartphones (eg smartphones).
  • a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.
  • each component eg, a module or a program of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. .
  • one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added.
  • a plurality of components eg, a module or a program
  • the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. .
  • operations performed by a module, program, or other component are executed sequentially, in parallel, repetitively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.
  • FIG. 9 is a block diagram 900 illustrating a camera module (eg, the camera module 180 of FIG. 1 ) 880 according to various embodiments of the present disclosure.
  • a camera module eg, the camera module 180 of FIG. 1
  • the camera module 880 includes a lens assembly 910 , a flash 920 , an image sensor 930 , an image stabilizer 940 , a memory 950 (eg, a buffer memory), or an image signal processor. (960).
  • the lens assembly 910 may collect light emitted from a subject, which is an image to be captured.
  • the lens assembly 910 may include one or more lenses.
  • the camera module 880 may include a plurality of lens assemblies 910 . In this case, the camera module 880 may form, for example, a dual camera, a 360 degree camera, or a spherical camera.
  • Some of the plurality of lens assemblies 910 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. It may have one or more lens properties different from the lens properties of .
  • the lens assembly 910 may include, for example, a wide-angle lens or a telephoto lens.
  • the flash 920 may emit light used to enhance light emitted or reflected from the subject.
  • the flash 920 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.
  • the image sensor 930 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 910 into an electrical signal.
  • the image sensor 930 is, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having properties, or a plurality of image sensors having different properties.
  • Each image sensor included in the image sensor 930 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.
  • CCD charged coupled device
  • CMOS complementary metal oxide semiconductor
  • the image stabilizer 940 moves at least one lens or image sensor 930 included in the lens assembly 910 in a specific direction in response to the movement of the camera module 880 or the electronic device 801 including the same. Operation characteristics of the image sensor 930 may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken.
  • the image stabilizer 940 is, according to an embodiment, the image stabilizer 940 is a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 880 . Such a movement of the camera module 880 or the electronic device 801 may be detected using .
  • the image stabilizer 940 may be implemented as, for example, an optical image stabilizer.
  • the memory 950 may temporarily store at least a portion of the image acquired through the image sensor 930 for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 950 and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display module 860 .
  • the acquired original image eg, a Bayer-patterned image or a high-resolution image
  • a copy image corresponding thereto eg, a low-resolution image
  • the memory 950 may be configured as at least a part of the memory 830 or as a separate memory operated independently of the memory 830 .
  • the image signal processor 960 may perform one or more image processing on an image acquired through the image sensor 930 or an image stored in the memory 950 .
  • the one or more image processes may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening.
  • the image signal processor 960 may include at least one of the components included in the camera module 880 (eg, an image sensor). 930), for example, exposure time control, readout timing control, etc.
  • the image processed by the image signal processor 960 is stored back in the memory 950 for further processing.
  • the image signal processor 960 may be configured as at least a part of the processor 820 or as a separate processor operated independently of the processor 820.
  • the image signal processor 960 may be configured as the processor 820 and a separate processor, at least one image processed by the image signal processor 960 may be displayed through the display module 860 either as it is by the processor 820 or after additional image processing.
  • the electronic device 801 may include a plurality of camera modules 880 each having different properties or functions.
  • at least one of the plurality of camera modules 880 may be a wide-angle camera, and at least the other may be a telephoto camera.
  • at least one of the plurality of camera modules 880 may be a front camera, and at least the other may be a rear camera.
  • the electronic device 100 includes an image sensor 120 and at least one processor electrically connected to the image sensor 120 (eg, the image signal processor 130 of FIG. 2 and/or the image signal processor 130 of FIG. 2 ). processor 220).
  • the at least one processor is configured to generate a second image generated by adding dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of valid bits. 1 Image data can be acquired. Each pixel of the first image data may have a third number of bits.
  • the at least one processor may obtain second image data in which each pixel has the third number of significant bits from the image sensor 120 .
  • the first image data may include identification information indicating that the dummy data is included in the first image data.
  • the at least one processor may identify the image data having the first number of valid bits from the first image data based on the identification information.
  • the at least one processor may determine whether the dummy data is included in the first image data.
  • the first mode is a mode for outputting image data in which the effective bits of each pixel is the first number
  • the second mode is image data in which the effective bits of each pixel is the third number. It may be a mode that outputs .
  • the third number may be greater than the first number.
  • the at least one processor sets the mode of the image sensor 120 to the first mode. can be changed to the second mode.
  • the at least one processor sets the mode of the image sensor 120 from the second mode. It can be changed to the first mode.
  • the at least one processor may determine the mode change of the image sensor 120 based on brightness information of at least a partial region of the first image data or the second image data.
  • the at least one processor may detect the mode change event of the image sensor 120 by analyzing the M-th frame. In response to the mode change event, the at least one processor may acquire image data corresponding to the changed mode of the image sensor 120 from an N-th frame that is a frame after the M-th frame.
  • the at least one processor may include an image signal processor (ISP).
  • ISP image signal processor
  • the image signal processor when the image sensor 120 is in the first mode, acquires the first image data in a first setting state to perform image processing. can be done
  • the image signal processor 130 may obtain the second image data in the first set state and perform image processing.
  • the electronic device 100 may further include a display (eg, the display 110 of FIG. 1 ).
  • the at least one processor may display on the display 110 based on the image data or the second image data included in the acquired first image data.
  • each pixel in the method of operating the electronic device 100 , in the first mode of the image sensor 120 , each pixel includes a second number of bits in the image data having the first number of valid bits. acquiring first image data generated by adding the dummy data to the image sensor; and acquiring second image data having the second number of valid bits from the image sensor in a second mode of the image sensor.
  • the first image data may include identification information indicating that the dummy data is included in the first image data.
  • the method of operating the electronic device 100 may include identifying the image data having the first number of valid bits in the first image data based on the identification information.
  • the mode of the image sensor is changed from the first mode. Changing the mode of the image sensor from the second mode to the first mode when the operation of changing to the second mode and the analysis result of the second image data obtained in the first mode do not satisfy a predefined condition It may include an action to
  • the method of operating the electronic device 100 includes acquiring the first image data in a first setting state and performing image processing when the image sensor is in the first mode, wherein the image sensor performs the image processing.
  • the method may include acquiring the second image data in the first setting state and performing image processing.
  • the method of operating the electronic device 100 may include displaying on the display based on the image data or the second image data included in the acquired first image data.
  • the electronic device 100 includes an image sensor (eg, the image sensor 120 of FIG. 2 ) and at least one processor electrically connected to the image sensor 120 (eg, the image signal processor of FIG. 2 ) 130) and/or the processor 220 of FIG. 2), and an interface connecting the image sensor 120 and the at least one processor.
  • the image sensor 120 may add dummy data having a second number of bits to each pixel to image data in which each pixel has a first number of valid bits.
  • the image sensor 120 may generate first image data in which each pixel has a third number of bits by adding the dummy data.
  • the image sensor 120 may provide the generated first image data to the at least one processor through the interface.
  • the at least one processor may identify the valid bit and the dummy data included in the first image data provided through the interface.
  • the at least one processor may perform image processing based on the identified valid bit.
  • the first image data may include identification information indicating that the dummy data is included in the first image data.
  • the at least one processor may identify the image data having the first number of valid bits from the first image data based on the identification information.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Power Engineering (AREA)
  • Electromagnetism (AREA)
  • Condensed Matter Physics & Semiconductors (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Studio Devices (AREA)

Abstract

Un dispositif électronique peut comprendre un capteur d'image et au moins un processeur connecté électriquement au capteur d'image. L'au moins un processeur peut : acquérir des premières données d'image générées par ajout de données factices ayant un deuxième nombre de bits pour chaque pixel à des données d'image dans lesquelles chaque pixel a un premier nombre de bits efficaces, dans un premier mode du capteur d'image, chaque pixel des premières données d'image ayant un troisième nombre de bits ; et acquérir, à partir du capteur d'image, des secondes données d'image dans lesquelles chaque pixel a un troisième nombre de bits efficaces, dans un second mode du capteur d'image. Divers autres modes de réalisation identifiés à partir de la description sont possibles.
PCT/KR2021/009661 2020-07-28 2021-07-26 Dispositif électronique comprenant un capteur d'image et un processeur de signal d'image, et son procédé WO2022025574A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2020-0093865 2020-07-28
KR1020200093865A KR20220014150A (ko) 2020-07-28 2020-07-28 이미지 센서와 이미지 시그널 프로세서를 포함하는 전자 장치 및 그의 방법

Publications (1)

Publication Number Publication Date
WO2022025574A1 true WO2022025574A1 (fr) 2022-02-03

Family

ID=80036553

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/009661 WO2022025574A1 (fr) 2020-07-28 2021-07-26 Dispositif électronique comprenant un capteur d'image et un processeur de signal d'image, et son procédé

Country Status (2)

Country Link
KR (1) KR20220014150A (fr)
WO (1) WO2022025574A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150099263A (ko) * 2014-02-21 2015-08-31 삼성전자주식회사 이미지를 획득하기 위한 전자 장치 및 방법
KR20170123661A (ko) * 2015-11-25 2017-11-08 후아웨이 테크놀러지 컴퍼니 리미티드 촬영 방법, 촬영 장치, 및 단말
KR101823256B1 (ko) * 2016-08-11 2018-01-29 삼성전기주식회사 이미지 처리 장치 및 이미지 처리 방법
KR20180079519A (ko) * 2016-12-30 2018-07-11 삼성전자주식회사 이미지 센서
US20190208111A1 (en) * 2017-12-28 2019-07-04 Waymo Llc Multiple Operating Modes to Expand Dynamic Range

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150099263A (ko) * 2014-02-21 2015-08-31 삼성전자주식회사 이미지를 획득하기 위한 전자 장치 및 방법
KR20170123661A (ko) * 2015-11-25 2017-11-08 후아웨이 테크놀러지 컴퍼니 리미티드 촬영 방법, 촬영 장치, 및 단말
KR101823256B1 (ko) * 2016-08-11 2018-01-29 삼성전기주식회사 이미지 처리 장치 및 이미지 처리 방법
KR20180079519A (ko) * 2016-12-30 2018-07-11 삼성전자주식회사 이미지 센서
US20190208111A1 (en) * 2017-12-28 2019-07-04 Waymo Llc Multiple Operating Modes to Expand Dynamic Range

Also Published As

Publication number Publication date
KR20220014150A (ko) 2022-02-04

Similar Documents

Publication Publication Date Title
WO2022030855A1 (fr) Dispositif électronique et procédé permettant de générer une image par application d'un effet sur un sujet et un arrière-plan
WO2022039424A1 (fr) Procédé de stabilisation d'images et dispositif électronique associé
WO2022149654A1 (fr) Dispositif électronique pour réaliser une stabilisation d'image, et son procédé de fonctionnement
WO2022119372A1 (fr) Appareil électronique effectuant un traitement d'image et procédé pour le faire fonctionner
WO2022092706A1 (fr) Procédé de prise de photographie à l'aide d'une pluralité de caméras, et dispositif associé
WO2022196993A1 (fr) Dispositif électronique et procédé de capture d'image au moyen d'un angle de vue d'un module d'appareil de prise de vues
WO2022235075A1 (fr) Dispositif électronique et son procédé de fonctionnement
WO2022244970A1 (fr) Procédé de capture d'image de dispositif électronique, et dispositif électronique associé
WO2021251631A1 (fr) Dispositif électronique comprenant une fonction de réglage de mise au point, et procédé associé
WO2021246736A1 (fr) Appareil de traitement d'images et procédé de traitement d'images
WO2022025574A1 (fr) Dispositif électronique comprenant un capteur d'image et un processeur de signal d'image, et son procédé
WO2022240186A1 (fr) Procédé de correction de distorsion d'image et dispositif électronique associé
WO2022092607A1 (fr) Dispositif électronique comportant un capteur d'image et procédé de fonctionnement de celui-ci
WO2022240183A1 (fr) Procédé de génération de fichier comprenant des données d'image et des données de mouvement, et dispositif électronique associé
WO2021230567A1 (fr) Procédé de capture d'image faisant intervenir une pluralité d'appareils de prise de vues et dispositif associé
WO2022245148A1 (fr) Procédé de traitement d'image et dispositif électronique pour celui-ci
WO2024106746A1 (fr) Dispositif électronique et procédé d'augmentation de la résolution d'une image de bokeh numérique
WO2022080737A1 (fr) Procédé de correction de distorsion d'image, et dispositif électronique associé
WO2022250344A1 (fr) Dispositif électronique comprenant un capteur d'image et un capteur de vision dynamique, et son procédé de fonctionnement
WO2022173236A1 (fr) Dispositif électronique comprenant un capteur d'image et son procédé de fonctionnement
WO2022270870A1 (fr) Procédé pour effectuer une photographie de manière multidirectionnelle par l'intermédiaire d'une caméra et dispositif électronique associé
WO2024085487A1 (fr) Dispositif électronique, procédé et support de stockage lisible par ordinateur non transitoire destinés au changement de réglage de caméra
WO2022225248A1 (fr) Procédé et dispositif électronique de stabilisation d'image pendant la capture d'image
WO2023080495A1 (fr) Dispositif électronique pour capturer des images à plage dynamique élevée, et son procédé de fonctionnement
WO2022203355A1 (fr) Dispositif électronique comprenant une pluralité de caméras

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21849771

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21849771

Country of ref document: EP

Kind code of ref document: A1