WO2023018201A1 - 이미지 안정화를 수행하는 전자 장치 및 그의 동작 방법 - Google Patents

이미지 안정화를 수행하는 전자 장치 및 그의 동작 방법 Download PDF

Info

Publication number
WO2023018201A1
WO2023018201A1 PCT/KR2022/011891 KR2022011891W WO2023018201A1 WO 2023018201 A1 WO2023018201 A1 WO 2023018201A1 KR 2022011891 W KR2022011891 W KR 2022011891W WO 2023018201 A1 WO2023018201 A1 WO 2023018201A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
electronic device
processor
blur
image
Prior art date
Application number
PCT/KR2022/011891
Other languages
English (en)
French (fr)
Korean (ko)
Inventor
송원석
양경동
윤재무
조지연
최송하
Original Assignee
삼성전자 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성전자 주식회사 filed Critical 삼성전자 주식회사
Priority to EP22856198.1A priority Critical patent/EP4340342A1/de
Priority to US18/075,750 priority patent/US12015848B2/en
Publication of WO2023018201A1 publication Critical patent/WO2023018201A1/ko

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/685—Vibration or motion blur correction performed by mechanical compensation
    • H04N23/687—Vibration or motion blur correction performed by mechanical compensation by shifting the lens or sensor position
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/684—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/71—Circuitry for evaluating the brightness variation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B2205/00—Adjustment of optical system relative to image or object surface other than for focusing
    • G03B2205/0007—Movement of one or more optical elements for control of motion blur

Definitions

  • the present disclosure relates to an electronic device that performs image stabilization or video digital image stabilization (VDIS) and an operating method thereof.
  • VDIS video digital image stabilization
  • image stabilization technology for minimizing the effect of motion/shake on an image acquired by a user is also being developed.
  • image stabilization an image stabilization technique that considers motion information acquired by an electronic device or an illumination condition (or exposure value) has been proposed.
  • an electronic device including a plurality of cameras, it may be difficult to adaptively control an effect of motion/shake on an image acquired by a user in a scenario in which switching between cameras occurs.
  • An aspect of the present disclosure is to address at least the problems and/or disadvantages noted above and provide at least the advantages described below. Accordingly, one aspect of the present disclosure is to provide an electronic device and method for performing image stabilization or video digital image stabilization (VDIS).
  • VDIS video digital image stabilization
  • the electronic device includes a camera including a lens assembly, at least one sensor configured to detect a movement of the camera or the lens assembly, the camera and the at least one sensor and a processor electrically connected to the processor, wherein the processor determines an exposure value to be applied when photographing using the camera, obtains image frames based on the exposure value, and while the image frames are obtained, the at least one Acquire motion information on the camera or lens assembly using a sensor of , determine image stabilization strength based on the exposure value and the motion information, and determine VDIS for the image frames based on the image stabilization strength. (video digital image stabilization) can be performed.
  • the electronic device includes a first camera including a first lens assembly, a second camera including a second lens assembly, the first camera, the second camera, the a processor electrically connected to at least one sensor configured to detect a motion of a first lens assembly or a second lens assembly, the first camera, a second camera, and the at least one sensor, wherein the processor comprises: , Performing a first VDIS based on the first camera, detecting an event of switching the main camera from the first camera to the second camera while the first VDIS is being performed, and detecting the first camera and the second camera.
  • Identify a parameter difference of a camera obtain image frames based on the parameter difference, obtain motion information for the second camera or the second lens assembly while acquiring the image frames, and obtain the parameter difference and the Determining a degree of blur that may occur based on motion information, determining image stabilization strength based on the blur degree, and performing a second VDIS on the image frames based on the image stabilization strength.
  • the effect of motion/shake can be minimized by considering both motion information and illumination conditions (eg, exposure values).
  • a degree of blur that may occur is predicted based on motion information and an illumination condition (eg, an exposure value), and an image stabilization strength is determined based on the predicted degree of blur.
  • the image stabilization technology may be adaptively applied by considering parameters between cameras to apply the image stabilization technology.
  • FIG. 1 is a block diagram of an electronic device in a network environment, according to an embodiment.
  • FIG. 2 is a block diagram illustrating a camera module, according to one embodiment.
  • FIG. 3 is a diagram schematically illustrating a configuration of an electronic device according to an exemplary embodiment.
  • VDIS video digital image stabilization
  • FIG. 5 is a flowchart illustrating a flow of performing VDIS by an electronic device according to an embodiment.
  • FIG. 6 is a diagram illustrating how VDIS is performed according to image stabilization strength, according to an exemplary embodiment.
  • FIG. 7 is a diagram illustrating a specific state in which VDIS is performed according to image stabilization strength, according to an exemplary embodiment.
  • FIG. 8 is a flowchart illustrating a flow of performing VDIS when an electronic device including a plurality of cameras performs a camera change, according to an embodiment.
  • FIG. 1 is a block diagram of an electronic device in a network environment, according to an embodiment.
  • an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 .
  • the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included.
  • at least one of these components eg, the connection terminal 178) may be omitted or one or more other components may be added.
  • some of these components eg, sensor module 176, camera module 180, or antenna module 197) are integrated into one component (eg, display module 160). It can be.
  • the processor 120 for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers commands or data received from other components (eg, sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 .
  • software eg, the program 140
  • the processor 120 transfers commands or data received from other components (eg, sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 .
  • the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor).
  • a main processor 121 eg, a central processing unit or an application processor
  • a secondary processor 123 eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor.
  • NPU neural network processing unit
  • the secondary processor 123 may be implemented separately from or as part of the main processor 121 .
  • the secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states.
  • the auxiliary processor 123 eg, an image signal processor or a communication processor
  • the auxiliary processor 123 may include a hardware structure specialized for processing an artificial intelligence model.
  • AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where artificial intelligence is performed, or may be performed through a separate server (eg, the server 108).
  • the learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited.
  • the artificial intelligence model may include a plurality of artificial neural network layers.
  • Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples.
  • the artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.
  • the memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 .
  • the data may include, for example, input data or output data for software (eg, program 140) and commands related thereto.
  • the memory 130 may include volatile memory 132 or non-volatile memory 134 .
  • the program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .
  • the input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user).
  • the input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).
  • the sound output module 155 may output sound signals to the outside of the electronic device 101 .
  • the sound output module 155 may include, for example, a speaker or a receiver.
  • the speaker can be used for general purposes such as multimedia playback or recording playback.
  • a receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.
  • the display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user).
  • the display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device.
  • the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.
  • the audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).
  • the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).
  • the sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do.
  • the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.
  • the interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102).
  • the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
  • HDMI high definition multimedia interface
  • USB universal serial bus
  • SD card interface Secure Digital Card interface
  • audio interface audio interface
  • connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102).
  • the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).
  • the haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses.
  • the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
  • the camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.
  • the power management module 188 may manage power supplied to the electronic device 101 .
  • the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.
  • PMIC power management integrated circuit
  • the battery 189 may supply power to at least one component of the electronic device 101 .
  • the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
  • the communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported.
  • the communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication.
  • the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module).
  • a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module).
  • the wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199.
  • IMSI International Mobile Subscriber Identifier
  • the wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology).
  • NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)).
  • eMBB enhanced mobile broadband
  • mMTC massive machine type communications
  • URLLC ultra-reliable and low latency
  • -latency communications can be supported.
  • the wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example.
  • the wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported.
  • the wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199).
  • the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.
  • eMBB peak data rate for eMBB realization
  • a loss coverage for mMTC realization eg, 164 dB or less
  • U-plane latency for URLLC realization eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less
  • the antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device).
  • the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB).
  • the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna.
  • other components eg, a radio frequency integrated circuit (RFIC) may be additionally formed as a part of the antenna module 197 in addition to the radiator.
  • RFIC radio frequency integrated circuit
  • the antenna module 197 may form a mmWave antenna module.
  • the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.
  • peripheral devices eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)
  • signal e.g. commands or data
  • commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 .
  • Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 .
  • all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 .
  • the electronic device 101 when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself.
  • one or more external electronic devices may be requested to perform the function or at least part of the service.
  • One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 .
  • the electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed.
  • cloud computing distributed computing, mobile edge computing (MEC), or client-server computing technology may be used.
  • the electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing.
  • the external electronic device 104 may include an internet of things (IoT) device.
  • Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or server 108 may be included in the second network 199 .
  • the electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.
  • FIG. 2 is a block diagram 200 illustrating a camera module 180, according to one embodiment.
  • the camera module 180 includes a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (eg, a buffer memory), or an image signal processor. (260).
  • the lens assembly 210 may collect light emitted from a subject that is an image capturing target.
  • the lens assembly 210 may include one or more lenses.
  • the camera module 180 may include a plurality of lens assemblies 210 . In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera.
  • Some of the plurality of lens assemblies 210 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may have the same lens properties as other lens assemblies. may have one or more lens properties different from the lens properties of .
  • the lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.
  • the flash 220 may emit light used to enhance light emitted or reflected from a subject.
  • the flash 220 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.
  • the image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal.
  • the image sensor 230 is, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having a property, or a plurality of image sensors having other properties.
  • Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.
  • CCD charged coupled device
  • CMOS complementary metal oxide semiconductor
  • the image stabilizer 240 moves at least one lens or image sensor 230 included in the lens assembly 210 in a specific direction in response to movement of the camera module 180 or the electronic device 101 including the same. Operation characteristics of the image sensor 230 may be controlled (eg, read-out timing is adjusted, etc.). This makes it possible to compensate at least part of the negative effect of the movement on the image being taken.
  • the image stabilizer 240 may include a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180. Such a movement of the camera module 180 or the electronic device 101 may be detected using .
  • the image stabilizer 240 may be implemented as, for example, an optical image stabilizer.
  • the memory 250 may at least temporarily store at least a portion of an image acquired through the image sensor 230 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 250 and , a copy image (eg, a low resolution image) corresponding thereto may be previewed through the display device 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 250 may be obtained and processed by the image signal processor 260 , for example. According to one embodiment, the memory 250 may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .
  • the image signal processor 260 may perform one or more image processes on an image obtained through the image sensor 230 or an image stored in the memory 250 .
  • the one or more image processes for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening.
  • the image signal processor 260 may include at least one of the components included in the camera module 180 (eg, an image sensor). 230) may be controlled (eg, exposure time control, read-out timing control, etc.)
  • the image processed by the image signal processor 260 is stored again in the memory 250 for further processing.
  • the image signal processor 260 may be configured as at least a part of the processor 120 or may be configured as a separate processor that operates independently of the processor 120.
  • the image signal processor 260 may be configured as a processor 120 When configured as a separate processor, at least one image processed by the image signal processor 260 may be displayed through the display device 160 as it is or after additional image processing by the processor 120 .
  • the electronic device 101 may include a plurality of camera modules 180 each having different properties or functions.
  • at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least the other may be a telephoto camera.
  • at least one of the plurality of camera modules 180 may be a front camera, and at least another one may be a rear camera.
  • FIG. 3 briefly illustrates a configuration of an electronic device according to an embodiment.
  • the electronic device 101 may include at least a camera 310, a sensor 320, and a processor 350.
  • the electronic device 101 may further include a VDIS module 330 and a display 340 .
  • the camera 310, the sensor 320, the display 340, and the processor 350 may include the camera module 180, sensor module 176, display module 160, and processor of FIG. 1, respectively. (120) can be corresponded to.
  • camera 310 may include a plurality of cameras.
  • the camera 310 may include a first camera 311 and a second camera 312, but is not limited thereto and may include more cameras.
  • the first camera 311 and the second camera 312 included in the camera 310 may have different fields of view (FOV), characteristics of image sensors, optical hand-shake prevention ( There may be differences in the existence and characteristics of optical image stabilization (OIS), the focal length, the presence or absence of auto focus (AF), and the characteristics of zoom operation.
  • FOV fields of view
  • OIS optical image stabilization
  • AF auto focus
  • each of the first camera 311 and the second camera 312 may include components included in the camera module 180 of FIG. 1 .
  • the camera 310 may obtain image frames of an external object, background, and light source of the electronic device 101 .
  • the senor 320 obtains at least motion information of the electronic device 101, motion information of the camera 310, or motion information of a lens assembly (eg, the lens assembly 210) according to an OIS operation. can do.
  • the senor 320 may include a gyro sensor.
  • the electronic device 101 may identify photographing parameters for image frames through the camera 310 and/or the sensor 320 .
  • the electronic device 101 may identify an illumination condition (eg, an exposure value) using the camera 310 and/or the sensor 320 .
  • an illumination condition eg, an exposure value
  • the VDIS module 330 may be implemented in hardware or software.
  • the VDIS module 330 may perform image stabilization or operations for performing VDIS on image frames acquired by the camera 310 . A more detailed description of the VDIS module 330 will be described later with reference to FIG. 4 .
  • the display 340 may visually provide image frames acquired by the camera 310 to a user.
  • the display 340 may visually provide a user with image frames obtained by performing image stabilization/VDIS on image frames acquired by the camera 310 .
  • the processor 350 may control elements of the electronic device 101 to perform designated operations.
  • the processor 350 controls the camera 310, the sensor 320, the VDIS module 330, and the display 340, thereby controlling the camera 310, the sensor 320, the VDIS module 330, and the display. (340)
  • Each can be made to perform designated operations.
  • the processor 350 may include the first camera 311 and the second camera 312 can be controlled respectively.
  • VDIS video digital image stabilization
  • the VDIS module 330 includes a motion detection module 410, a path calculation module 420, a blur prediction module 430, an image stabilization strength setting module 440, and a VDIS execution module 450. can do.
  • the motion detection module 410 may obtain the motion degree (or motion amount) of the electronic device 101 or the camera 310 .
  • the motion detection module 410 may include image frames obtained by the camera 310 and motion information of the electronic device 101 detected by the sensor 320, motion information of the camera 310, or OIS operation. Based on the motion information of the lens assembly (eg, the lens assembly 210) according to , the degree of motion of the electronic device 101 and/or the camera 310 may be calculated/estimated.
  • the path calculation module 420 may perform cumulative calculation on motion information (eg, motion degree) of the electronic device 101 and/or the camera 310 acquired by the motion detection module 410.
  • motion information eg, motion degree
  • the path calculation module 420 uses the cumulatively calculated movement information (eg, degree of movement) of the electronic device 101 and/or the camera 310 to detect the electronic device 101 and/or the camera ( 310) may calculate/estimate the motion path (or motion trajectory).
  • the cumulatively calculated movement information eg, degree of movement
  • the camera 310 may calculate/estimate the motion path (or motion trajectory).
  • the path calculation module 420 based on the calculated/estimated motion path (or motion trajectory), provides a stabilization path (or trajectory) such that the motion path (or motion trajectory) becomes a smooth path (or trajectory). stabilization trajectory).
  • the path calculation module 420 may estimate a motion path (or a motion trajectory) corresponding to an average value of motion degrees of the electronic device 101 and/or the camera 310, A motion path corresponding to the value may be created as a stabilization path.
  • the average value is an example, and may not be limited thereto.
  • the blur prediction module 430 may predict the degree of blur that may occur for each image frame based on information about image frames and motion information that may be acquired by the electronic device 101 .
  • the blur prediction module 430 may include the obtained illumination condition (eg, exposure value), motion information of the electronic device 101, motion information of the camera 310, or a lens assembly according to an OIS operation (eg, A degree of blur (eg, amount of blur, size of blur, probability of occurrence of blur) that may occur in each image frame may be calculated/obtained using motion information of the lens assembly 210 .
  • the blur may be caused by factors such as shaking of the electronic device 101/camera 310, focal length, shutter speed, whether AF is performed, or whether OIS is performed.
  • the image stabilization strength setting module 440 may set the image stabilization strength based on the expected degree of blur.
  • the image stabilization strength may be applied to image frames according to the movement of the electronic device 101, the movement of the camera 310, or the movement of a lens assembly (eg, the lens assembly 210) according to the OIS operation.
  • shaking it may mean the intensity of performing image stabilization (or shake correction) on image frames in which motion/shake occurs.
  • the image stabilization strength setting module 440 may set the image stabilization strength to decrease as the predicted blur level increases, and set the image stabilization strength to increase as the predicted blur level decreases.
  • image stabilization is performed based on the image stabilization strength set by the VDIS execution module 450 and the image stabilization strength setting module 440 and the stabilization path (or stabilization trajectory) calculated/generated by the path calculation module 420.
  • VDIS can generate image frames performed.
  • the VDIS performing module 450 may perform shake correction/image stabilization on an actual motion path according to image stabilization strength based on the generated stabilization path.
  • the image stabilization strength setting module 440 may provide information about the set image stabilization strength (or shake correction strength) to the path calculation module 420 .
  • the path calculation module 420 may calculate/generate a stabilization path (or stabilization trajectory) in consideration of the provided image stabilization strength, and information on the calculated/generated stabilization path (or stabilization trajectory) ) can be passed on.
  • the blur prediction module 430 may predict the degree of blur that may occur in image frames by using the received stabilization path (or stabilization trajectory) and the illuminance condition (eg, exposure value), and the blur prediction module 430 ) may provide the predicted degree of blur to the image stabilization strength setting module 440 .
  • operations of the path calculation module 420, the blur prediction module 430, and the image stabilization strength setting module 440 may be cyclical.
  • the motion detection module 410, the path calculation module 420, the blur prediction module 430, the image stabilization strength setting module 440, and the VDIS execution module 450 included in the VDIS module 330 may be implemented in hardware or software, and may operate under the control of the processor 350.
  • FIG. 5 illustrates a flow of an electronic device performing VDIS according to an embodiment.
  • the electronic device may determine an exposure value to be applied when photographing.
  • the electronic device 101 may determine an exposure value applicable to capturing (or acquiring an image frame) under the control of the processor 350 .
  • the electronic device 101 may determine an exposure value based on illuminance when photographing (or acquiring an image frame) under the control of the processor 350 .
  • the electronic device 101 may maintain or change an exposure value determined according to a camera switching event under the control of the processor 350 .
  • the electronic device may acquire image frames based on the exposure value.
  • the electronic device 101 may perform photographing (or acquire image frames) using the determined exposure value under the control of the processor 350 .
  • the electronic device may obtain motion information about a camera or lens assembly while obtaining image frames.
  • the electronic device 101 may obtain motion information of the camera 310 under the control of the processor 350 .
  • the electronic device 101 may obtain motion information about movement/shake of the camera 310 under the control of the processor 350 .
  • the electronic device 101 may obtain motion information about the motion/shake of the electronic device 101 linked to the motion/shake of the camera 310 under the control of the processor 350 .
  • the electronic device 101 may obtain motion information about movement/shake of the electronic device 101 or the camera 310 using the sensor 320 under the control of the processor 350.
  • the sensor 320 may be a gyro sensor.
  • the electronic device 101 may obtain motion information about a lens assembly (eg, the lens assembly 210) under the control of the processor 350. For example, under the control of the processor 350, the electronic device 101 may obtain whether or not the camera 310 has OIS and motion information of the lens assembly 210 when OIS is performed.
  • a lens assembly eg, the lens assembly 210
  • the electronic device 101 may obtain whether or not the camera 310 has OIS and motion information of the lens assembly 210 when OIS is performed.
  • the electronic device 101 may use the camera 310 (or the electronic device 101) or a lens assembly (eg, the lens assembly 210 acquired while acquiring image frames) under the control of the processor 350. )) may normalize motion information.
  • the electronic device 101 transmits motion information about the camera 310 (or the electronic device 101) or a lens assembly (eg, the lens assembly 210) according to the control of the processor 350. Normalization may be performed by classifying according to the amount of shake and/or the speed of movement/shake.
  • the electronic device 101 may classify normalized motion information and store it in a memory (eg, the memory 130) under the control of the processor 350.
  • a memory eg, the memory 130
  • the electronic device may determine a degree of blur that may occur based on the exposure value and motion information.
  • the electronic device 101 determines the first weight for the exposure value and the amount of motion/shake of the camera 310 (or the electronic device 101), and/or A second weight for normalized motion information may be set according to the motion/shaking speed.
  • the electronic device 101 may generate a first blur in image frames based on a first weight for the exposure value and a second weight for the motion information under the control of the processor 350. degree can be judged.
  • the electronic device 101 may determine the second degree of blur using OIS motion information of a lens assembly (eg, the lens assembly 210) under the control of the processor 350. For example, the electronic device 101 sets 0 when the lens assembly 210 does not perform OIS and 100 when the lens assembly 210 performs 100% OIS under the control of the processor 350.
  • a third weight may be set.
  • the electronic device 101 may set the third weight to a level between 0 and 100 according to the degree to which the lens assembly 210 performs OIS under the control of the processor 350 .
  • the electronic device 101 may set the third weight to 50 when the degree to which the lens assembly 210 performs OIS is 50%, which is one It is an example and may not be limited.
  • the electronic device 101 may set the third weight to 0 under the control of the processor 350 even when there is no OIS.
  • the electronic device 101 reflects the third weight on the OIS performance of the lens assembly 210 to the first degree of blur, under the control of the processor 350, thereby generating image frames.
  • a second degree of blur may be calculated/obtained.
  • the electronic device may determine image stabilization strength based on the exposure value and motion information.
  • the electronic device 101 may determine the image stabilization strength corresponding to the exposure value and the degree of motion under the control of the processor 350 .
  • the electronic device 101 uses a degree of blur (eg, a second degree of blur) that may be generated in the calculated/obtained image frames under the control of the processor 350 to generate image frames. It is possible to determine the strength of image stabilization to remove the effect of motion/shaking on the image.
  • a degree of blur eg, a second degree of blur
  • the effect of motion/shaking on the above image frames may be referred to as image jitter.
  • Movement/shake of the electronic device 101 and/or the camera 310 may affect image frames such as light dragging, light spreading, and blurring.
  • image jitter movement/shake of the electronic device 101 and/or the camera 310 may affect image frames such as light dragging, light spreading, and blurring.
  • the above is an example, and may not be limited to the above.
  • the electronic device 101 may determine, under the control of the processor 350, that the image stabilization strength decreases as the predicted blur level increases, and the image stabilization strength increases as the predicted blur level decreases. can decide.
  • the electronic device may perform VDIS on image frames based on the image stabilization strength.
  • the electronic device 101 may perform image stabilization or VDIS on acquired image frames based on the determined image stabilization strength under the control of the processor 350 .
  • image stabilization or VDIS according to operation 550 will be described in more detail with reference to FIGS. 6 and 7 .
  • FIG. 6 illustrates VDIS performed according to image stabilization strength, according to an embodiment
  • FIG. 7 illustrates a specific state in which VDIS is performed according to image stabilization strength, according to an embodiment.
  • VDIS is performed according to image stabilization strength
  • different types of graphs may be obtained.
  • the x-axis of FIG. 6 means acquired image frames
  • the y-axis may indicate motion information of image frames corrected by performing image stabilization (or VDIS) by applying image stabilization strength.
  • the motion information corresponding to the y-axis may be a trajectory/movement of the camera 310, and a unit may be a pixel.
  • line 610 may indicate motion information of image frames for which VDIS has not been performed.
  • lines 620 and 630 may indicate motion information of corrected image frames according to image stabilization strength.
  • line 610 represents motion information of image frames for which VDIS has not been performed
  • line 620 represents motion information of image frames for which VDIS has been performed with 50% image stabilization strength
  • a line 630 represents motion information of image frames for which VDIS has been performed with an image stabilization strength of 100%.
  • lines 630 of FIGS. 6 to 7 may represent motion information of image frames on which VDIS is performed with 100% image stabilization strength, and line 630 in FIGS. 4 to 5 It may correspond to the aforementioned stabilization path (or stabilization trajectory).
  • the above-described lines according to the image stabilization strength are examples, and may not be limited to the above.
  • FIG 8 illustrates a flow in which an electronic device including a plurality of cameras performs VDIS when switching cameras, according to an embodiment.
  • the electronic device may be performing a first VDIS based on a first camera.
  • the first VDIS performed by the electronic device 101 in operation 810 based on the first camera 311 may correspond to the VDIS performing operations according to operations 510 to 550 of FIG. 5 .
  • the operation performed by the electronic device 101 based on the first camera 311 in operation 810 is the operation performed by the electronic device 101 based on the camera 310 in operations 510 to 550.
  • the electronic device 101 may determine an exposure value based on illuminance when photographing (or acquiring an image frame) under the control of the processor 350 .
  • the electronic device 101 may acquire image frames based on the exposure value under the control of the processor 350 .
  • the electronic device 101 may acquire motion information about the first camera 311 or a first lens assembly included in the first camera 311 . Under the control of the processor 350, the electronic device 101 may obtain whether or not the first camera 311 has OIS and motion information of the first lens assembly when OIS is performed.
  • the electronic device 101 may determine a degree of blur that may occur based on the exposure value and motion information, and determine image stabilization strength based on the degree of blur. . Under the control of the processor 350, the electronic device 101 may perform the first VDIS on the image frames based on the image stabilization strength.
  • the operation according to operation 810 of the electronic device 101 is not limited to the above, and descriptions according to operation 510 to operation 550 of FIG. 5 may be applied.
  • the electronic device may detect an event of switching the main camera from the first camera to the second camera.
  • the electronic device 101 may include a plurality of cameras.
  • the electronic device 101 may include at least a first camera 311 and a second camera 312, but is not limited thereto and may include three or more cameras.
  • the main camera switching event may be a touch input, a voice input, or a physical input, and may not be particularly limited.
  • the first camera 311 and the second camera 312 included in the electronic device 101 may have different characteristics/parameters. For example, there may be differences in characteristics/parameters related to a field of view (FOV), performance of an image sensor, existence and performance of OIS, focal length, presence of AF, and zoom operation.
  • FOV field of view
  • OIS optical image stabilization
  • focal length focal length
  • presence of AF zoom operation
  • the electronic device 101 may detect an event of switching the main camera from the first camera 311 to the second camera 312 under the control of the processor 350 .
  • the electronic device 101 may switch the main camera from the first camera 311 to the second camera 312 according to the control of the processor 350 in response to detecting the switching event. Also, a time point at which the electronic device 101 switches the main camera according to the control of the processor 350 may be before the operation of identifying the parameter difference according to operation 830 or after the operation of identifying the parameter difference.
  • the electronic device 101 may maintain or change an exposure value determined according to a camera switching event under the control of the processor 350 .
  • the electronic device may identify a parameter difference between the first camera and the second camera.
  • the electronic device 101 may identify a parameter difference according to the presence or absence of OIS under the control of the processor 350 . For example, when a switch is made from the first camera 311, which does not perform OIS, to the second camera 312, which can perform OIS, the electronic device 101 performs OIS under the control of the processor 350. can identify whether
  • the electronic device 101 may identify a parameter difference according to the performance of the image sensor, the angle of view, or the focal length under the control of the processor 350 .
  • the electronic device 101 may identify sensitivity differences of image sensors respectively included in the first camera 311 and the second camera 312 under the control of the processor 350 .
  • the electronic device 101 may identify a difference in angle of view and/or a difference in focal length between the first camera 311 and the second camera 312 under the control of the processor 350 .
  • the electronic device 101 may identify a parameter difference according to OIS performance under the control of the processor 350 . For example, when both the first camera 311 and the second camera 312 can perform OIS, the electronic device 101 operates the first camera 311 under the control of the processor 350. A difference in performance between the OIS operation performed and the OIS operation performed by the second camera 312 may be identified.
  • the electronic device may obtain image frames based on the parameter difference.
  • the electronic device 101 may acquire image frames based on the switched second camera 312 under the control of the processor 350 .
  • the electronic device 101 may obtain motion information about the second camera 312 or a second lens assembly included in the second camera 312 while acquiring image frames. .
  • the electronic device 101 may obtain motion information of the second camera 312 under the control of the processor 350 .
  • the electronic device 101 may obtain motion information about movement/shake of the second camera 312 under the control of the processor 350 .
  • the electronic device 101 may obtain motion information about the motion/shake of the electronic device 101 linked to the motion/shake of the second camera 312 under the control of the processor 350. there is.
  • the electronic device 101 obtains motion information about motion/shake of the electronic device 101 or the second camera 312 using the sensor 320 under the control of the processor 350.
  • the sensor 320 may be a gyro sensor.
  • the electronic device 101 may obtain motion information about the second lens assembly under the control of the processor 350 .
  • the electronic device 101 may obtain whether or not the second camera 312 has OIS and motion information of the second lens assembly when OIS is performed.
  • the electronic device 101 moves the second camera 312 (or the electronic device 101) or the second lens assembly acquired while acquiring the image frames under the control of the processor 350.
  • Information can be normalized.
  • the electronic device 101 transmits motion information about the second camera 312 (or the electronic device 101) or the second lens assembly according to the amount of motion/shake, and/or the electronic device 101 under the control of the processor 350.
  • normalization may be performed by classifying according to movement/shaking speed.
  • the electronic device 101 may classify normalized motion information and store it in a memory (eg, the memory 130) under the control of the processor 350.
  • a memory eg, the memory 130
  • the electronic device may determine a degree of blur that may occur based on the parameter difference and motion information.
  • the electronic device 101 when switching from the first camera 311 that does not perform OIS to the second camera 312 that can perform OIS occurs, the electronic device 101, under the control of the processor 350, It can be identified that the degree of blur in the absence of OIS is higher than that in the case in which OIS is performed.
  • the electronic device 101 when the sensitivity of the image sensor of the second camera 312 is higher than the sensitivity of the image sensor of the first camera 311, the electronic device 101 operates the second camera according to the control of the processor 350. It may be identified that the degree of blur in the state in which 312 is the main camera is lower than the degree of blur in the state in which the first camera 311 is the main camera.
  • the electronic device 101 when the angle of view of the first camera 311 is greater than the angle of view of the second camera 312, the electronic device 101 performs the control of the processor 350 so that the second camera 312 is the main camera. It may be identified that the degree of blur in the in state is greater than the degree of blur in the state in which the first camera 311 is the main camera.
  • the electronic device 101 controls the processor 350 to set the second camera 312 as the main camera. It may be identified that the degree of blur in the in state is greater than the degree of blur in the state in which the first camera 311 is the main camera.
  • the electronic device 101 when the focal length of the second camera 312 is longer than the focal length of the first camera 311, the electronic device 101, under the control of the processor 350, the second camera 312 It may be identified that the degree of blur in the main camera state is greater than the degree of blur in the state in which the first camera 311 is the main camera.
  • the electronic device 101 when the first camera 311 performs AF and the second camera 312 does not perform AF, the electronic device 101 performs the control of the processor 350, and the second camera 312 ) is the main camera, it can be identified that the degree of blur occurs lower than the degree of blur in the state where the first camera 311 is the main camera.
  • the electronic device may determine image stabilization strength based on the degree of blur.
  • Image stabilization strength when OIS is performed may be determined to be greater than image stabilization strength when OIS is absent.
  • the electronic device 101 when the sensitivity of the image sensor of the second camera 312 is higher than the sensitivity of the image sensor of the first camera 311, the electronic device 101 operates the second camera according to the control of the processor 350.
  • the image stabilization strength in the state in which 312 is the main camera may be determined to be higher than the image stabilization strength in the state in which the first camera 311 is the main camera.
  • the electronic device 101 when the angle of view of the first camera 311 is greater than the angle of view of the second camera 312, the electronic device 101 performs the control of the processor 350 so that the second camera 312 is the main camera.
  • the image stabilization strength in the in state may be determined to be lower than the image stabilization strength in the state in which the first camera 311 is the main camera.
  • the electronic device 101 controls the processor 350 to set the second camera 312 as the main camera.
  • the image stabilization strength in the in state may be determined to be smaller than the image stabilization strength in the state in which the first camera 311 is the main camera.
  • the electronic device 101 when the focal length of the second camera 312 is longer than the focal length of the first camera 311, the electronic device 101, under the control of the processor 350, the second camera 312
  • the image stabilization strength in the main camera state may be determined to be smaller than the image stabilization strength in the state in which the first camera 311 is the main camera.
  • the electronic device 101 when the first camera 311 performs AF and the second camera 312 does not perform AF, the electronic device 101 performs the control of the processor 350, and the second camera 312 ) may be determined to be greater than the image stabilization strength in a state in which the first camera 311 is the main camera.
  • the electronic device 101 moves the second camera 312 (or the electronic device 101) or the second lens assembly acquired while acquiring the image frames under the control of the processor 350. Based on the information (or normalized motion information), it is possible to predict the degree of blur that can occur. Also, under the control of the processor 350, the electronic device 101 may determine a low image stabilization strength when it is determined that the degree of blur that may occur is large. Under the control of the processor 350, the electronic device 101 may determine a high image stabilization strength when it is determined that the degree of blur that may occur is small.
  • the electronic device may perform a second VDIS on image frames based on the image stabilization strength.
  • the electronic device 101 may determine the image stabilization strength based on the degree of possible blur determined based on the parameter difference and motion information under the control of the processor 350 .
  • the electronic device 101 may perform a second VDIS on image frames obtained based on the determined image stabilization strength under the control of the processor 350 .
  • the electronic device 101 may determine image stabilization strength according to a change in magnification during a zoom operation under the control of the processor 350 .
  • the angle of view of the first camera 311 may be wider than the angle of view of the second camera 312 , and the magnification of the first camera 312 may be lower than that of the second camera 312 .
  • the electronic device 101 controls the processor 350 to frame the image as the magnification increases. It can be determined that the degree of blur that can be generated in the fields is high. Under the control of the processor 350, the electronic device 101 may determine a lower image stabilization strength as the magnification increases.
  • the sensitivity of the image sensor of the first camera 311 is higher than that of the second camera 312.
  • the sensitivity of the image sensor is high, it may be identified that the degree of blur in the state where the second camera 312 is the main camera is lower than the degree of blur in the state where the first camera 311 is the main camera.
  • the electronic device 101 According to the control of the processor 350, it may be determined that the degree of blur that may occur is low when the second camera 312 is switched. Under the control of the processor 350, the electronic device 101 may determine the image stabilization strength as the degree of blur that may occur decreases.
  • the electronic device 101 may perform VDIS based on the image stabilization strength determined according to the change in magnification during a zoom operation under the control of the processor 350 .
  • the electronic device 101 may perform shake/motion correction or VDIS according to the determined image stabilization strength under the control of the processor 350 .
  • the electronic device 101 may perform shake/motion correction under the control of the processor 350 .
  • the electronic device 101 provides the degree of blur expected to occur in image frames, an exposure value according to the illuminance, and motion information (eg, shaking and motion of the electronic device 101 or the camera 310). ), and based on the determination, shake/motion correction or VDIS may be performed by adjusting the intensity of the image stabilization strength.
  • the electronic device 101 weakens the image stabilization strength to limit the amount of correction possible in image frames. It is possible to improve the quality of image frames as a whole by not exceeding.
  • the electronic device includes a camera including a lens assembly (eg, the camera 310), the camera (eg, camera 310) or at least one sensor (eg, at least one sensor 320) detecting a movement of the lens assembly, the camera (eg, camera 310) and the at least one sensor (
  • a processor eg, the processor 350
  • the processor includes the camera (eg, the camera 310).
  • Determining an exposure value to be applied when photographing using obtaining image frames based on the exposure value, and using the at least one sensor (eg, at least one sensor 320) while the image frames are acquired. to obtain motion information for the camera (eg, camera 310) or the lens assembly, determine image stabilization strength based on the exposure value and the motion information, and determine the image frame based on the image stabilization strength video digital image stabilization (VDIS) can be performed.
  • the camera eg, camera 310
  • VDIS video digital image stabilization
  • the processor eg, the processor 350 performs a blur that may be generated based on the exposure value and the motion information.
  • the degree of blur may be determined, and image stabilization strength may be determined based on the degree of blur.
  • the motion information includes information about shaking of the camera (eg, camera 310) and shaking of the lens assembly according to OIS performance. may contain information about
  • the processor determines a first weight for the exposure value, and determines a first weight value for the exposure value
  • the camera determines a second weight for the motion information, determines a first blur degree based on the first weight and the second weight, and determines a third weight for the motion information of the lens assembly
  • a degree of blur may be determined based on the first degree of blur and the third weight.
  • the processor determines the image stabilization strength to be lower as the degree of blur that can occur increases. , the image stabilization strength may be determined to be higher as the degree of blur that may occur decreases.
  • the degree of blur may include at least one of an amount of blur, a size of blur, and a probability of occurrence of blur.
  • the processor may perform the acquisition of the motion information, the determination of the degree of blur, and the image stabilization strength.
  • the determination of can be performed cyclically.
  • the electronic device further includes at least one camera
  • the processor eg, the processor ( 350) detects an event of switching the main camera from the camera (eg, the camera 310) to the one or more cameras while performing the VDIS, and in response to detecting the event, the camera (eg, the camera 310). : Identify a parameter difference between the camera 310 and the at least one camera, obtain the image frames based on the parameter difference, and acquire motion information for the at least one camera while acquiring the image frames.
  • a second VDIS may be performed on the image frames.
  • the event may be at least one of a touch input, a voice input, and a physical input.
  • the motion information may include motion information about at least one lens assembly included in the at least one camera.
  • the parameter difference is a field of view (FOV), an image sensor, an OIS, a focal length, AF, and a zoom operation. It may be a parameter difference for at least one.
  • the processor may maintain or change the exposure value in response to the event.
  • the processor eg, the processor 350 performs the movement relative to the camera (eg, the camera 310) or the lens assembly.
  • a stabilization path may be generated using the information, and the image stabilization strength may be determined based on the stabilization path.
  • the at least one sensor may be a gyro sensor.
  • the movement of the electronic device is interlocked with the movement of the camera (eg, the camera 310). It can be.
  • the electronic device includes a first camera (eg, camera 311) including a first lens assembly. ), a second camera (eg, camera 312) including a second lens assembly, the first camera (eg, camera 311), a second camera (eg, camera 312), and the first lens assembly, or at least one sensor (eg, at least one sensor 320) detecting a movement of the second lens assembly, the first camera (eg, camera 311), and a second camera (eg, camera ( 312)), and a processor (eg, processor 350) electrically connected to the at least one sensor (eg, at least one sensor 320), wherein the processor (eg, processor 350) ,
  • the first VDIS is performed based on the first camera (eg, camera 311), and while the first VDIS is performed, the first camera (eg, camera 311) performs the second camera (eg, camera 311).
  • a degree of blur that may occur may be determined, an image stabilization strength may be determined based on the degree of blur, and a second VDIS may be performed on the image frames based on the image stabilization strength.
  • the event may be at least one of a touch input, a voice input, and a physical input.
  • the parameter difference is a field of view (FOV), an image sensor, an OIS, a focal length, AF, and a zoom operation. It may be a parameter difference for at least one.
  • the senor may be a gyro sensor.
  • the processor determines the image stabilization strength to be lower as the degree of blur that can occur increases. , the image stabilization strength may be determined to be higher as the degree of blur that may occur decreases.
  • Electronic devices may be devices of various types.
  • the electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance.
  • a portable communication device eg, a smart phone
  • a computer device e.g., a smart phone
  • a portable multimedia device e.g., a portable medical device
  • a camera e.g., a portable medical device
  • a camera e.g., a portable medical device
  • a camera e.g., a camera
  • a wearable device e.g., a smart bracelet
  • first, second, or first or secondary may simply be used to distinguish that component from other corresponding components, and may refer to that component in other respects (eg, importance or order) is not limited.
  • a (eg, first) component is said to be “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively.”
  • the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.
  • module used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeably interchangeable with terms such as, for example, logic, logic blocks, components, or circuits.
  • a module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions.
  • the module may be implemented in the form of an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • a storage medium eg, internal memory 136 or external memory 138
  • a machine eg, electronic device 101
  • a processor eg, the processor 120
  • a device eg, the electronic device 101
  • the one or more instructions may include code generated by a compiler or code executable by an interpreter.
  • the device-readable storage medium may be provided in the form of a non-transitory storage medium.
  • the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.
  • a signal e.g. electromagnetic wave
  • the method according to various embodiments disclosed in this document may be included and provided in a computer program product.
  • Computer program products may be traded between sellers and buyers as commodities.
  • a computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store TM ) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones.
  • a device e.g. compact disc read only memory (CD-ROM)
  • an application store e.g. Play Store TM
  • It can be distributed (eg downloaded or uploaded) online, directly between smart phones.
  • at least part of the computer program product may be temporarily stored or temporarily created in a storage medium readable by a device such as a manufacturer's server, an application store server, or a relay server's memory.
  • each component (eg, module or program) of the components described above may include a single object or a plurality of objects, and some of the multiple objects may be separately disposed in other components.
  • one or more components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added.
  • a plurality of components eg modules or programs
  • the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. .
  • operations performed by modules, programs, or other components are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
PCT/KR2022/011891 2021-08-10 2022-08-10 이미지 안정화를 수행하는 전자 장치 및 그의 동작 방법 WO2023018201A1 (ko)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP22856198.1A EP4340342A1 (de) 2021-08-10 2022-08-10 Elektronische vorrichtung zur durchführung von bildstabilisierung und betriebsverfahren dafür
US18/075,750 US12015848B2 (en) 2021-08-10 2022-12-06 Electronic device performing image stabilization and operating method thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020210105683A KR20230023506A (ko) 2021-08-10 2021-08-10 이미지 안정화를 수행하는 전자 장치 및 그의 동작 방법
KR10-2021-0105683 2021-08-10

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/075,750 Continuation US12015848B2 (en) 2021-08-10 2022-12-06 Electronic device performing image stabilization and operating method thereof

Publications (1)

Publication Number Publication Date
WO2023018201A1 true WO2023018201A1 (ko) 2023-02-16

Family

ID=85200104

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2022/011891 WO2023018201A1 (ko) 2021-08-10 2022-08-10 이미지 안정화를 수행하는 전자 장치 및 그의 동작 방법

Country Status (4)

Country Link
US (1) US12015848B2 (de)
EP (1) EP4340342A1 (de)
KR (1) KR20230023506A (de)
WO (1) WO2023018201A1 (de)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170131608A (ko) * 2015-08-21 2017-11-29 구글 엘엘씨 움직임 보상을 갖는 이미지 캡처 시스템
US20200137308A1 (en) * 2018-10-30 2020-04-30 Qualcomm Incorporated Optical image stabilization techniques
KR20200048609A (ko) * 2018-10-30 2020-05-08 삼성전자주식회사 인공 신경망을 이용한 이미지 처리 방법 및 이를 지원하는 전자 장치
KR20200098790A (ko) * 2019-02-12 2020-08-21 삼성전자주식회사 이미지 센서 및 이를 포함하는 전자 기기
KR20200101180A (ko) * 2019-02-19 2020-08-27 삼성전자주식회사 이미지 안정화를 위한 전자 장치 및 그의 동작 방법

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI444753B (zh) 2010-11-16 2014-07-11 Altek Corp 影像擷取裝置及其曝光時間調整方法
GB2492529B (en) * 2011-05-31 2018-01-10 Skype Video stabilisation
US20140307950A1 (en) 2013-04-13 2014-10-16 Microsoft Corporation Image deblurring
KR101805629B1 (ko) 2013-05-07 2017-12-07 삼성전자주식회사 영상의 상태에 따라 적응적인 영상 처리를 위한 방법 및 장치
GB201312892D0 (en) 2013-07-18 2013-09-04 Omg Plc Still image capture with exposure control
US9066014B2 (en) * 2013-10-11 2015-06-23 Facebook, Inc. Applying video stabilization to a multimedia clip
US9204048B2 (en) * 2014-03-27 2015-12-01 Facebook, Inc. Stabilization of low-light video
US9674438B2 (en) * 2014-07-06 2017-06-06 Apple Inc. Low light video image stabilization strength modulation
US10097765B2 (en) * 2016-04-20 2018-10-09 Samsung Electronics Co., Ltd. Methodology and apparatus for generating high fidelity zoom for mobile video
CN107820024B (zh) 2017-12-05 2020-09-29 北京小米移动软件有限公司 图像拍摄方法、装置及存储介质
CN108234873A (zh) 2018-01-10 2018-06-29 北京小米移动软件有限公司 一种获取图像的方法和装置
KR102525000B1 (ko) * 2018-08-08 2023-04-24 삼성전자 주식회사 복수의 이미지들이 합성된 이미지를 깊이 정보에 기반하여 흐림 처리하는 전자 장치 및 상기 전자 장치의 구동 방법
US11190689B1 (en) * 2020-07-29 2021-11-30 Google Llc Multi-camera video stabilization

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170131608A (ko) * 2015-08-21 2017-11-29 구글 엘엘씨 움직임 보상을 갖는 이미지 캡처 시스템
US20200137308A1 (en) * 2018-10-30 2020-04-30 Qualcomm Incorporated Optical image stabilization techniques
KR20200048609A (ko) * 2018-10-30 2020-05-08 삼성전자주식회사 인공 신경망을 이용한 이미지 처리 방법 및 이를 지원하는 전자 장치
KR20200098790A (ko) * 2019-02-12 2020-08-21 삼성전자주식회사 이미지 센서 및 이를 포함하는 전자 기기
KR20200101180A (ko) * 2019-02-19 2020-08-27 삼성전자주식회사 이미지 안정화를 위한 전자 장치 및 그의 동작 방법

Also Published As

Publication number Publication date
US12015848B2 (en) 2024-06-18
KR20230023506A (ko) 2023-02-17
US20230101888A1 (en) 2023-03-30
EP4340342A1 (de) 2024-03-20

Similar Documents

Publication Publication Date Title
WO2020091262A1 (ko) 인공 신경망을 이용한 이미지 처리 방법 및 이를 지원하는 전자 장치
WO2022114801A1 (ko) 복수의 카메라를 포함하는 전자 장치 및 그 전자 장치의 제어 방법
WO2022108235A1 (ko) 슬로우 셔터를 제공하기 위한 방법, 장치 및 저장 매체
WO2022039424A1 (ko) 흔들림을 보정하는 방법 및 그 전자 장치
WO2022092706A1 (ko) 복수의 카메라를 이용한 사진 촬영 방법 및 그 장치
WO2022149654A1 (ko) 흔들림 보정을 수행하는 전자 장치 및 그 동작 방법
WO2022149812A1 (ko) 카메라 모듈을 포함하는 전자 장치 및 그 전자 장치의 동작 방법
WO2022260252A1 (ko) 카메라 모듈을 포함하는 전자 장치 및 그 전자 장치의 동작 방법
WO2022244970A1 (ko) 전자 장치의 촬영 방법 및 그 전자 장치
WO2023018201A1 (ko) 이미지 안정화를 수행하는 전자 장치 및 그의 동작 방법
WO2021235747A1 (ko) 카메라 및 마이크를 포함하는 전자 장치 및 그 운용 방법
WO2022154168A1 (ko) 자동 초점 조절이 가능한 전자 장치 및 그 동작 방법
WO2022031050A1 (ko) 이미지를 생성하는 전자 장치 및 방법
WO2023063520A1 (ko) 전자 장치 및 전자 장치에서 촬영 기능을 수행하는 방법
WO2024080730A1 (ko) 슬로우 셔터 기능을 제공하는 전자 장치 및 방법
WO2022203211A1 (ko) 카메라 모듈을 포함하는 전자 장치 및 그 전자 장치의 동작 방법
WO2023075335A1 (ko) 복수의 렌즈들을 구비하는 전자 장치 및 그 제어 방법
WO2022240122A1 (ko) 카메라를 포함하는 전자 장치
WO2024117587A1 (ko) 필터 기능을 제공하는 전자 장치 및 그 동작 방법
WO2023033396A1 (ko) 연속된 촬영 입력을 처리하는 전자 장치 및 그의 동작 방법
WO2023017981A1 (ko) 언더 디스플레이 카메라를 포함하는 전자 장치 및 그 동작 방법
WO2023080459A1 (ko) 이미지를 제공하는 방법 및 이를 지원하는 전자 장치
WO2023038252A1 (ko) 동영상을 촬영하는 전자 장치 및 그 동작 방법
WO2023171939A1 (ko) 이미지 안정화를 수행하는 전자 장치 및 그 동작 방법
WO2022114789A1 (ko) 광량을 획득하는 전자 장치 및 방법

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2022856198

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022856198

Country of ref document: EP

Effective date: 20231215

NENP Non-entry into the national phase

Ref country code: DE