KR102558471B1 - Electronic apparatus and operating method thereof - Google Patents

Abstract

An electronic device and an operating method thereof according to various embodiments may be configured to acquire a first image from a first image sensor, acquire a second image from a second image sensor, compare the first image and the second image, estimate motion in the first image and the second image, and compensate for the motion in the first image and the second image.

Description

Electronic device and its operating method {ELECTRONIC APPARATUS AND OPERATING METHOD THEREOF}

Various embodiments relate to an electronic device and an operating method thereof.

With the recent development of digital technology, various types of electronic devices such as mobile communication terminals, smart phones, tablet PCs (Personal Computers), PDAs (Personal Digital Assistants), electronic notebooks, notebooks, or wearable devices are widely used. BACKGROUND OF THE INVENTION Electronic devices are reaching a stage of mobile convergence that encompasses the functions of other devices.

An electronic device may include a plurality of image sensors. The electronic device may acquire the same images by simultaneously driving a plurality of image sensors.

However, in the electronic device as described above, a Jello phenomenon may occur in images acquired by image sensors due to movement of a subject or a photographer.

Various embodiments may provide an electronic device having a plurality of image sensors and an operating method thereof capable of compensating motion in images acquired by the image sensors.

An electronic device according to various embodiments may include a first image sensor, a second image sensor, and a processor functionally connected to the first image sensor and the second image sensor.

According to various embodiments, the processor may be configured to acquire a first image from the first image sensor, obtain a second image from the second image sensor, compare the first image and the second image, estimate motion in the first image and the second image, and compensate for the motion in the first image and the second image.

An operating method of an electronic device including a plurality of image sensors according to various embodiments may include acquiring a first image from a first image sensor and obtaining a second image from a second image sensor, estimating motion in the first image and second image by comparing the first image with the second image, and compensating for the motion in the first image and the second image.

An electronic device and an operating method thereof according to various embodiments may compensate motion in images acquired by image sensors.

1 shows a block diagram of a network environment system according to various embodiments.
2 shows a block diagram of an electronic device according to various embodiments.
3 shows a block diagram of program modules in accordance with various embodiments.
4 shows a block diagram of an electronic device according to various embodiments.
5A, 5B, and 5C are exemplary diagrams for explaining a scanning method of image sensors in an electronic device according to various embodiments.
6A, 6B, 6C, and 6D are exemplary diagrams for explaining a scanning method of image sensors in an electronic device according to various embodiments.
7 is a block diagram illustrating a detailed configuration of image sensors in an electronic device according to various embodiments.
8 is a block diagram illustrating a detailed configuration of a processor in an electronic device according to various embodiments.
9 is a flowchart illustrating a method of operating an electronic device according to various embodiments.
10A, 10B, and 10C are exemplary diagrams for explaining an operating method of an electronic device according to various embodiments.
11 is a flowchart illustrating a method for acquiring an image in an electronic device according to various embodiments.
12 is a flowchart illustrating a method for estimating motion in an image in an electronic device according to various embodiments.

Hereinafter, various embodiments of this document will be described with reference to the accompanying drawings. Examples and terms used therein are not intended to limit the technology described in this document to specific embodiments, and should be understood to include various modifications, equivalents, and/or substitutes of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like elements. Singular expressions may include plural expressions unless the context clearly dictates otherwise. In this document, expressions such as "A or B" or "at least one of A and/or B" may include all possible combinations of the items listed together. Expressions such as “first,” “second,” “first,” or “second,” may modify corresponding components regardless of order or importance, and are used to distinguish one component from another component, but do not limit the corresponding components. When a (e.g., first) component is referred to as being "(functionally or communicatively) connected" or "connected" to another (e.g., second) component, the component may be directly connected to the other component, or may be connected through another component (e.g., a third component).

In this document, “configured to (or configured to)” may be used interchangeably with “suitable for,” “having the ability to,” “modified to,” “made to,” “capable of,” or “designed to” depending on the situation, for example, hardware or software. In some contexts, the expression "device configured to" can mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase “a processor configured (or configured) to perform A, B, and C” may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations by executing one or more software programs stored in a memory device.

An electronic device according to various embodiments of the present document may include, for example, at least one of a smart phone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a portable multimedia player (PMP), an MP3 player, a medical device, a camera, or a wearable device. 웨어러블 장치는 액세서리형(예: 시계, 반지, 팔찌, 발찌, 목걸이, 안경, 콘택트 렌즈, 또는 머리 착용형 장치(head-mounted-device(HMD)), 직물 또는 의류 일체형(예: 전자 의복), 신체 부착형(예: 스킨 패드 또는 문신), 또는 생체 이식형 회로 중 적어도 하나를 포함할 수 있다. 어떤 실시예들에서, 전자 장치는, 예를 들면, 텔레비전, DVD(digital video disk) 플레이어, 오디오, 냉장고, 에어컨, 청소기, 오븐, 전자레인지, 세탁기, 공기 청정기, 셋톱 박스, 홈 오토매이션 컨트롤 패널, 보안 컨트롤 패널, 미디어 박스(예: 삼성 HomeSync ^TM , 애플TV ^TM , 또는 구글 TV ^TM ), 게임 콘솔(예: Xbox ^TM , PlayStation ^TM ), 전자 사전, 전자 키, 캠코더, 또는 전자 액자 중 적어도 하나를 포함할 수 있다.

In another embodiment, the electronic device may include various types of medical devices (e.g., various portable medical measuring devices (eg, blood glucose meter, heart rate monitor, blood pressure monitor, or body temperature monitor), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), camera, or ultrasound), navigation device, global navigation satellite system (GNSS), event data recorder (EDR), flight data recorder (FDR), and automobile infotainment device. , Electronic equipment for ships (eg, navigation systems for ships, gyrocompasses, etc.), avionics, security equipment, head units for vehicles, industrial or household robots, drones, ATMs in financial institutions, point of sales (POS) in stores, or IoT devices (eg, light bulbs, various sensors, sprinklers, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.). According to some embodiments, the electronic device may include at least one of furniture, a part of a building/structure or a vehicle, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (eg, water, electricity, gas, or radio wave measuring devices, etc.). In various embodiments, the electronic device may be flexible or a combination of two or more of the various devices described above. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (eg, an artificial intelligence electronic device).

Referring to FIG. 1 , an electronic device 101 in a network environment 100 in various embodiments is described. The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input/output interface 150, a display 160, and a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or may additionally include other components. Bus 110 may include circuitry that connects components 110-170 to each other and communicates (eg, control messages or data) between components. The processor 120 may include one or more of a central processing unit, an application processor, or a communication processor (CP). The processor 120 may, for example, execute calculations or data processing related to control and/or communication of at least one other component of the electronic device 101 .

Memory 130 may include volatile and/or non-volatile memory. The memory 130 may store, for example, commands or data related to at least one other component of the electronic device 101 . According to one embodiment, memory 130 may store software and/or programs 140 . The program 140 may include, for example, a kernel 141 , middleware 143 , an application programming interface (API) 145 , and/or an application program (or “application”) 147 . At least part of the kernel 141, middleware 143, or API 145 may be referred to as an operating system. The kernel 141, for example, may control or manage system resources (eg, the bus 110, processor 120, or memory 130, etc.) used to execute operations or functions implemented in other programs (eg, middleware 143, API 145, or application program 147). In addition, the kernel 141 may provide an interface capable of controlling or managing system resources by accessing individual components of the electronic device 101 from the middleware 143, API 145, or application program 147.

The middleware 143 may perform an intermediary role so that, for example, the API 145 or the application program 147 communicates with the kernel 141 to exchange data. Also, the middleware 143 may process one or more task requests received from the application program 147 according to priority. For example, the middleware 143 may assign a priority for using system resources (e.g., the bus 110, the processor 120, or the memory 130, etc.) of the electronic device 101 to at least one of the application programs 147, and process the one or more task requests. The API 145 is an interface for the application 147 to control functions provided by the kernel 141 or the middleware 143, and may include, for example, at least one interface or function (eg, command) for file control, window control, image processing, or text control. The input/output interface 150 may transmit, for example, commands or data input from a user or other external device to other element(s) of the electronic device 101, or may output commands or data received from other element(s) of the electronic device 101 to the user or other external device.

The display 160 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display 160 may display various types of content (eg, text, image, video, icon, and/or symbol) to the user. The display 160 may include a touch screen, and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body. The communication interface 170 may establish communication between the electronic device 101 and an external device (eg, the first external electronic device 102, the second external electronic device 104, or the server 106). For example, the communication interface 170 may be connected to the network 162 through wireless or wired communication to communicate with an external device (eg, the second external electronic device 104 or the server 106).

Wireless communication may include, for example, cellular communication using at least one of LTE, LTE advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM). According to an embodiment, wireless communication may include at least one of, for example, wireless fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), magnetic secure transmission, radio frequency (RF), or body area network (BAN). According to one embodiment, wireless communication may include GNSS. GNSS may be, for example, a global positioning system (GPS), a global navigation satellite system (Glonass), a Beidou Navigation Satellite System (hereinafter “Beidou”) or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS". Wired communication may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS). Network 162 may include at least one of a telecommunication network, for example, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 102 and 104 may be the same as or different from the electronic device 101 . According to various embodiments, all or part of operations executed in the electronic device 101 may be executed in one or more electronic devices (eg, the electronic devices 102 and 104, or the server 106). According to one embodiment, when the electronic device 101 needs to perform a certain function or service automatically or upon request, the electronic device 101, instead of or in addition to executing the function or service itself, performs at least some functions related to the function or service to another device (eg, electronic device 101). A request may be made to the device 102 or 104 or the server 106. Another electronic device (eg, the electronic device 102 or 104 or the server 106) may execute the requested function or additional function and deliver the result to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as it is or additionally. To this end, for example, cloud computing, distributed computing, or client-server Computing technology may be used.

2 is a block diagram of an electronic device 201 according to various embodiments. The electronic device 201 may include all or part of the electronic device 101 shown in FIG. 1 , for example. The electronic device 201 includes one or more processors (eg AP) 210, communication module 220, subscriber identity module 224, memory 230, sensor module 240, input device 250, display 260, interface 270, audio module 280, camera module 291, power management module 295, battery 296, indicator 297, and motor 29 8) can be included. The processor 210 may control a plurality of hardware or software components connected to the processor 210 by driving, for example, an operating system or an application program, and may perform various data processing and calculations. The processor 210 may be implemented as, for example, a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphic processing unit (GPU) and/or an image signal processor. The processor 210 may include at least some of the components shown in FIG. 2 (eg, the cellular module 221). The processor 210 may load and process a command or data received from at least one of other components (eg, non-volatile memory) into a volatile memory, and store resultant data in the non-volatile memory.

The communication module 220 may have the same or similar configuration as the communication interface 170 , for example. The communication module 220 may include, for example, a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GNSS module 227, an NFC module 228, and an RF module 229. The cellular module 221 may provide, for example, a voice call, a video call, a text service, or an Internet service through a communication network. According to one embodiment, the cellular module 221 may identify and authenticate the electronic device 201 within a communication network using the subscriber identity module (eg, SIM card) 224 . According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 may provide. According to one embodiment, the cellular module 221 may include a communication processor (CP). According to some embodiments, at least some (eg, two or more) of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 may be included in one integrated chip (IC) or IC package. The RF module 229 may transmit and receive communication signals (eg, RF signals), for example. The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 transmits and receives an RF signal through a separate RF module. The subscriber identity module 224 may include, for example, a card or embedded SIM including a subscriber identity module, and may include unique identification information (e.g., integrated circuit card identifier (ICCID)) or subscriber information (e.g., international mobile subscriber identity (IMSI)).

The memory 230 (eg, the memory 130) may include, for example, an internal memory 232 or an external memory 234. The built-in memory 232 may include, for example, at least one of volatile memory (eg, DRAM, SRAM, or SDRAM), non-volatile memory (eg, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, flash memory, a hard drive, or a solid state drive (SSD). The external memory 234 may include a flash drive, for example, a compact flash (CF), secure digital), Micro-SD, Mini-SD, extreme digital (xD), multi-media card (MMC), memory stick, etc. The external memory 234 may be functionally or physically connected to the electronic device 201 through various interfaces.

The sensor module 240 may, for example, measure a physical quantity or detect an operating state of the electronic device 201 and convert the measured or sensed information into an electrical signal. The sensor module 240 includes, for example, a gesture sensor 240A, a gyro sensor 240B, a pressure sensor 240C, a magnetic sensor 240D, an acceleration sensor 240E, a grip sensor 240F, a proximity sensor 240G, a color sensor 240H (eg, a red, green, blue (RGB) sensor), a bio sensor 240I, and a temperature/humidity sensor 240. 0J), an illuminance sensor 240K, or an ultra violet (UV) sensor 240M. Additionally or alternatively, sensor module 240 may include, for example, an e-nose sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris sensor, and/or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling one or more sensors included therein. In some embodiments, the electronic device 201 may further include a processor configured to control the sensor module 240, either as part of the processor 210 or separately, to control the sensor module 240 while the processor 210 is in a sleep state.

The input device 250 may include, for example, a touch panel 252 , a (digital) pen sensor 254 , a key 256 , or an ultrasonic input device 258 . The touch panel 252 may use at least one of, for example, a capacitive type, a pressure-sensitive type, an infrared type, or an ultrasonic type. Also, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response to the user. The (digital) pen sensor 254 may be, for example, part of a touch panel or may include a separate recognition sheet. Keys 256 may include, for example, physical buttons, optical keys, or keypads. The ultrasonic input device 258 may detect ultrasonic waves generated from an input tool through a microphone (eg, the microphone 288) and check data corresponding to the detected ultrasonic waves.

The display 260 (eg, the display 160) may include a panel 262, a hologram device 264, a projector 266, and/or a control circuit for controlling them. Panel 262 may be implemented to be flexible, transparent, or wearable, for example. The panel 262 may include the touch panel 252 and one or more modules. According to one embodiment, the panel 262 may include a pressure sensor (or force sensor) capable of measuring the strength of a user's touch. The pressure sensor may be implemented integrally with the touch panel 252 or may be implemented as one or more sensors separate from the touch panel 252 . The hologram device 264 may display a 3D image in the air using interference of light. The projector 266 may display an image by projecting light onto a screen. The screen may be located inside or outside the electronic device 201 , for example. Interface 270 may include, for example, HDMI 272, USB 274, optical interface 276, or D-sub (D-subminiature) 278. Interface 270 may be included in, for example, communication interface 170 shown in FIG. 1 . Additionally or alternatively, the interface 270 may include, for example, a mobile high-definition link (MHL) interface, an SD card/multi-media card (MMC) interface, or an infrared data association (IrDA) standard interface.

The audio module 280 may, for example, convert a sound and an electrical signal in both directions. At least some components of the audio module 280 may be included in the input/output interface 145 shown in FIG. 1 , for example. The audio module 280 may process sound information input or output through, for example, the speaker 282, the receiver 284, the earphone 286, or the microphone 288. The camera module 291 is, for example, a device capable of capturing still images and moving images. According to one embodiment, the camera module 291 may include one or more image sensors (eg, a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (eg, LED or xenon lamp). The power management module 295 may manage power of the electronic device 201 , for example. According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charger IC, or a battery or fuel gauge. A PMIC may have a wired and/or wireless charging method. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier. The battery gauge may measure, for example, the remaining capacity of the battery 296, voltage, current, or temperature during charging. Battery 296 may include, for example, a rechargeable battery and/or a solar cell.

The indicator 297 may indicate a specific state of the electronic device 201 or a part thereof (eg, the processor 210), for example, a booting state, a message state, or a charging state. The motor 298 may convert electrical signals into mechanical vibrations and generate vibrations or haptic effects. The electronic device 201 may include, for example, a mobile TV supporting device (eg, a GPU) capable of processing media data according to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or mediaFlo ^™ . Each of the components described in this document may be composed of one or more components, and the name of the corresponding component may vary depending on the type of electronic device. In various embodiments, an electronic device (e.g., the electronic device 201) is configured as a single entity by omitting some components, further including additional components, or combining some of the components, but performing the same functions of the corresponding components before combining.

3 is a block diagram of a program module according to various embodiments. According to one embodiment, the program module 310 (eg, program 140) may include an operating system that controls resources related to an electronic device (eg, electronic device 101) and/or various applications (eg, application program 147) running on the operating system. The operating system may include, for example, Android ^TM , iOS ^TM , Windows ^TM , Symbian ^TM , Tizen ^TM , or Bada ^TM . Referring to FIG. 3 , the program module 310 may include a kernel 320 (eg, kernel 141), middleware 330 (eg, middleware 143), (API 360 (eg, API 145)), and/or an application 370 (eg, application program 147). At least a portion of the program module 310 may be preloaded on an electronic device or an external electronic device (eg, It can be downloaded from the electronic devices 102 and 104, the server 106, etc.).

The kernel 320 may include, for example, a system resource manager 321 and/or a device driver 323 . The system resource manager 321 may perform control, allocation, or recovery of system resources. According to one embodiment, the system resource manager 321 may include a process management unit, a memory management unit, or a file system management unit. The device driver 323 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication (IPC) driver. The middleware 330 may, for example, provide functions commonly required by the application 370 or provide various functions to the application 370 through the API 360 so that the application 370 can use limited system resources inside the electronic device. According to one embodiment, middleware 330 includes runtime library 335, application manager 341, window manager 342, multimedia manager 343, resource manager 344, power manager 345, database manager 346, package manager 347, connectivity manager 348, notification manager 349, location manager 350, graphics manager 351, or security At least one of the managers 352 may be included.

The runtime library 335 may include, for example, a library module used by a compiler to add new functions through a programming language while the application 370 is being executed. The runtime library 335 may perform input/output management, memory management, or arithmetic function processing. The application manager 341 may manage the life cycle of the application 370 , for example. The window manager 342 may manage GUI resources used in the screen. The multimedia manager 343 may determine a format required for reproducing media files, and encode or decode the media files using a codec suitable for the format. The resource manager 344 may manage a source code of the application 370 or a memory space. The power manager 345 may manage, for example, battery capacity or power, and provide power information necessary for the operation of the electronic device. According to one embodiment, the power manager 345 may interoperate with a basic input/output system (BIOS). The database manager 346 may create, search, or change a database to be used in the application 370 , for example. The package manager 347 may manage installation or update of applications distributed in the form of package files.

The connectivity manager 348 may manage wireless connections, for example. The notification manager 349 may provide a user with an event such as an arrival message, an appointment, or proximity notification. The location manager 350 may manage, for example, location information of an electronic device. The graphic manager 351 may manage, for example, graphic effects to be provided to the user or user interfaces related thereto. Security manager 352 may provide system security or user authentication, for example. According to one embodiment, the middleware 330 may include a telephony manager for managing voice or video call functions of an electronic device or a middleware module capable of forming a combination of functions of the aforementioned components. According to one embodiment, the middleware 330 may provide modules specialized for each type of operating system. The middleware 330 may dynamically delete some existing components or add new components. The API 360 is, for example, a set of API programming functions, and may be provided in different configurations depending on the operating system. For example, in the case of Android or iOS, one API set can be provided for each platform, and in the case of Tizen, two or more API sets can be provided for each platform.

The application 370 includes, for example, a home 371, a dialer 372, an SMS/MMS 373, an instant message (IM) 374, a browser 375, a camera 376, an alarm 377, a contact 378, a voice dial 379, an email 380, a calendar 381, a media player 382, an album 383, and a watch 38. 4), health care (eg, measurement of exercise or blood sugar, etc.), or environmental information (eg, atmospheric pressure, humidity, or temperature information) providing applications. According to one embodiment, the application 370 may include an information exchange application capable of supporting information exchange between an electronic device and an external electronic device. The information exchange application may include, for example, a notification relay application for delivering specific information to an external electronic device or a device management application for managing an external electronic device. For example, a notification delivery application may transfer notification information generated by another application of an electronic device to an external electronic device or may receive notification information from an external electronic device and provide the notification information to a user. The device management application may, for example, install, delete, or update a function of an external electronic device that communicates with the electronic device (e.g., turn-on/turn-off of the external electronic device itself (or some component parts) or control of brightness (or resolution) of a display) or an application running in the external electronic device. According to an embodiment, the application 370 may include an application designated according to the properties of an external electronic device (eg, a health management application of a mobile medical device). According to one embodiment, the application 370 may include an application received from an external electronic device. At least part of the program module 310 may be implemented (eg, executed) in software, firmware, hardware (eg, the processor 210), or a combination of at least two or more of these, and may include a module, program, routine, instruction set, or process for performing one or more functions.

The term "module" used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A “module” may be an integrally constructed component or a minimal unit or part thereof that performs one or more functions. A “module” may be implemented mechanically or electronically and may include, for example, known or future developed application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), or programmable logic devices that perform certain operations.

At least some of devices (eg modules or functions thereof) or methods (eg operations) according to various embodiments may be implemented as instructions stored in a computer-readable storage medium (eg memory 130) in the form of program modules. When the command is executed by a processor (eg, the processor 120), the processor may perform a function corresponding to the command. A computer-readable recording medium may include a hard disk, a floppy disk, a magnetic medium (eg, magnetic tape), an optical recording medium (eg, CD-ROM, DVD, a magneto-optical medium (eg, a floppy disk), a built-in memory, etc. The command may include a code generated by a compiler or a code that can be executed by an interpreter.

A module or program module according to various embodiments may include at least one or more of the aforementioned components, some may be omitted, or other components may be further included. According to various embodiments, operations performed by modules, program modules, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, may be omitted, or other operations may be added.

4 is a block diagram illustrating an electronic device 400 according to various embodiments.

Referring to FIG. 4 , the electronic device 400 may include a processor 410, a memory 420, a plurality of image sensors 430 and 440, a display unit 450, and an input unit 460.

The processor 410 may include the processor 120 of FIG. 1 and the processor 210 of FIG. 2 . The processor 410 may acquire a plurality of images through the image sensors 430 and 440 . To this end, the processor 410 may asymmetrically perform readout with respect to the image sensors 430 and 440 . Also, the processor 410 may compensate for the images by estimating motion in the images.

The memory 420 may include the memory 130 of FIG. 1 and the memory 230 of FIG. 2 . The memory 420 may store instructions that cause the processor 410 to asymmetrically perform readout with respect to the image sensors 430 and 440 . And the memory 420 may store instructions that cause the processor 410 to estimate motion in the images and compensate for the images. For example, memory 420 may store instructions that, when executed, cause processor 410 to compare images, estimate motion in each image, and correct each image based on motion.

The image sensors 430 and 440 may be configured as at least part of the input/output interface 150 of FIG. 1 and the camera module 291 of FIG. 2 . The image sensors 430 and 440 may include a first image sensor 430 and a second image sensor 440 . Each of the first image sensor 430 and the second image sensor 440 may include a row driver, a pixel array, and a column driver. The first image sensor 430 may generate a first image as the access period is controlled by the processor 410 . The second image sensor 440 may generate a second image as the access period is controlled by the processor 410 . The first image sensor 430 and the second image sensor 440 may have the same resolution or may have different resolutions. Resolution can refer to the number of pixels occupied per specific distance or space. For example, the first image sensor 430 and the second image sensor 440 may each generate a pixel array capable of generating any one of high definition (HD), full high definition (FHD), and ultra high definition (UHD) images. It may include a pixel array.

The display unit 450 may include the display 160 of FIG. 1 and the display 260 of FIG. 2 . The display unit 450 may display images acquired by the image sensors 430 and 440 .

The input unit 460 may include at least a part of the input/output interface 150 of FIG. 1 and the input device 250 of FIG. 2 . The input unit 460 may input commands and data for controlling the operation of the electronic device 400 .

According to various embodiments, the processor 410 may obtain images from the first image sensor 430 and the second image sensor 440 using a rolling shutter method. The processor 410 may obtain images by performing asymmetric readout on the first image sensor 430 and the second image sensor 440 . That is, the processor 410 may acquire a plurality of images by performing readout on the first image sensor 430 and the second image sensor 440 in different ways. For example, the first image sensor 430 and the second image sensor 440 may be disposed in the same shape or different shapes. Through this, the processor 410 may obtain line images of the first image sensor 430 and the second image sensor 440 in different order.

For example, if the first image sensor 430 and the second image sensor 440 are disposed in the same form, the processor 410 may address the first image sensor 430 to read out the first line image in the order of the last line image, and address the second image sensor 440 to read out the last line image in the order of the first line image. For example, if the first image sensor 430 and the second image sensor 440 are rotated by 180 degrees from each other, the first image sensor 430 may obtain a normal phase image, and the second image sensor 440 may acquire an anti-phase image rotated by 180 degrees. In this case, the processor 410 may read out the last line image from the second image sensor 440 when reading out the first line image from the first image sensor 430, and address the first line image from the second image sensor 440 when reading out the last line image from the first image sensor 430.

According to various embodiments, the first image sensor 430 or the second image sensor 440 may each have a structure of one of CMOS, CIS, and CCD. Each of the first image sensor 430 and the second image sensor 440 may include a pixel array, a row control block for controlling the pixel array, and a column readout block for reading out from the pixel array. The control block may accumulate charges as an analog signal in a photo-sensing element of the pixel array, for example, a photo diode for a predetermined exposure time after resetting the pixel. The read-out block may convert the analog signal accumulated in the photodetector into digital data. To this end, the readout block may include an analog to digital converter (ADC). Digital data may be output from an internal digital block of each of the first image sensor 430 and the second image sensor 440 to the outside, for example, to the processor 410 through an external interface such as a mobile industry processor interface (MIPI).

However, as the first image sensor 430 and the second image sensor 440 use a rolling shutter method, a jello effect (rolling shutter artifact) may occur in images. The jello phenomenon may refer to a phenomenon in which an image shakes like jelly due to movement of a subject or a photographer. For example, if a moving subject is included in an image or a photographer captures an image while moving, a jello phenomenon may occur in the images.

According to various embodiments, the processor 410 may continuously acquire images through the first image sensor 430 . The processor 410 may compare successive images acquired through the first image sensor 430 to determine whether motion compensation of the image is required. For example, the processor 410 may determine that motion compensation for the images acquired through the first image sensor 430 is necessary when errors in the successive images acquired through the first image sensor 430 are out of a reference range. For example, the processor 410 may obtain an error between the images by comparing the images on a pixel-by-pixel basis. The processor 410 may activate the second image sensor 440 when it is determined that motion compensation of the image acquired through the first image sensor 430 is required.

According to various embodiments, when the processor 410 activates the second image sensor 440 for motion compensation of an image obtained through the first image sensor 430, the processor 410 may set operating parameters of the second image sensor 440. For example, when the readout of the first image sensor 430 and the second image sensor 440 are asymmetrical, the processor 410 may set the operating variable of the second image sensor 440 to be different from the operating variable of the first image sensor 430. Additionally or alternatively, when the image acquired through the second image sensor 440 is used as an image for motion compensation of the image acquired through the first image sensor 430, the processor 410 may set an operating variable of the second image sensor 440 to correspond to motion detection. That is, the processor 410 may set operating parameters of the second image sensor 440 so as to obtain an image without blur without considering noise through the second image sensor 440 . In addition, the processor 410 extracts an operating variable for motion detection corresponding to the operating variable of the first image sensor 430 from a look up table (LUT) stored in the memory 420 and sets it as an operating variable of the second image sensor 440. For example, the operating variable of the image sensor may include at least one of an exposure of the image sensor or a sensor gain (International Organization for Standardization (ISO)).

According to various embodiments, the processor 410 may acquire a first image by reading out the first image sensor 430 as a first attribute, and acquire a second image by reading out the second image sensor 440 as a second attribute. The first property may include at least one of a first direction and a first speed, and the second property may include at least one of a second direction and a second speed. For example, the processor 410 may acquire a first image by reading out the first image sensor 430 in a first direction, and acquire a second image by reading out the second image sensor 440 in a second direction. Also, the processor 410 may estimate each motion in the first image and the second image. Also, the processor 410 may compensate each motion in the first image and the second image. According to an embodiment, the processor 410 may compare the first image and the second image to determine whether motion compensation of the image is required. For example, if the error between the first image and the second image is out of a reference range, the processor 410 may determine that motion compensation for the image is required. When the processor 410 determines that image motion compensation is required, the processor 410 may estimate the respective motions in the first image and the second image.

5A, 5B, and 5C are exemplary diagrams for explaining a scanning method of the image sensors 430 and 440 in the electronic device 400 according to various embodiments. FIG. 5A is a diagram for explaining a scanning method of the first image sensor 430 and FIG. 5B is a diagram for explaining a scanning method of the second image sensor 440 .

The first image sensor 420 and the second image sensor 430 may include a pixel array having a plurality of row lines, and may scan one row line as a unit. The first image sensor 430 and the second image sensor 440 may include the same number of row lines. For example, as shown in FIGS. 5A and 5B , each of the first image sensor 430 and the second image sensor 440 may include N row lines. Alternatively, the first image sensor 430 and the second image sensor 440 may include different numbers of row lines. For example, the first image sensor 430 may include N row lines, and the second image sensor 440 may include M row lines. The first image sensor 430 and the second image sensor 440 may scan row lines in different shapes. For example, the first image sensor 430 may perform exposure and readout in the order from the first row line to the last row line in the first direction, and the second image sensor 440 may perform exposure and readout in the order from the last row line to the first row line in the second direction. As shown in FIGS. 5A , 5B and 5C , the first image sensor 430 and the second image sensor 440 may include pixel arrays having the same resolution.

Referring to FIG. 5A , the processor 410 may perform exposures from the first row line R1 to the Nth row line RN of the first image sensor 430 in order. The processor 410 may start exposing the first row line R1 at a first start point 510 and end exposure of the first row line R1 at a first end point 515 . The processor 410 may start exposing the second row line R2 at the second start time 520 and end exposure of the second row line R2 at the second end time 525 . In this case, the time when the set line delay time has elapsed from the first start time 510 may be the second start time 520 . In this way, the processor 410 may start exposing the N-th row line RN at the N-th starting time point 530 when N line delay times have elapsed from the first starting time point 510, and may end exposure of the N-th row line RN at the N-th ending time point 535. Through this, the processor 410 may accumulate analog signals in the light sensing element while selecting the row lines R1, R2, R3, R4, ?, RN-1, and RN in the order from the first row line R1 to the last row line RN of the pixel array in the first image sensor 430.

Referring to FIG. 5B , the processor 410 may expose in order from the Nth row line RN to the first row line R1 of the second image sensor 440 . The processor 420 may start exposing the N-th row line RN at a first start point 550 and end exposure of the N-th row line RN at a first end point 555 . The processor 410 may start exposing the N-1 row line RN-1 at the second start point 560 and end the exposure of the N-1 row line RN-1 at the second end point 565. In this case, the time when the set line delay time has elapsed from the first start time 550 may be the second start time 560 . In this way, the processor 410 may start exposing the first row line R1 at the Nth starting time point 570 when N line delay times have elapsed from the first starting time point 550, and may end exposure of the first row line R1 at the Nth ending time point 575. Through this, the processor 410 may accumulate analog signals in the light sensing element while selecting the row lines RN, RN-1, RN-2, RN-3, ?, R2, and R1 in the order from the last row line RN to the first row line R1 of the pixel array in the second image sensor 440.

Referring to FIG. 5C , the first image sensor 430 and the second image sensor 440 may activate an exposure time 580 while sequentially selecting row lines in the first frame F1. In addition, the first image sensor 430 and the second image sensor 440 activate the readout time 590 at the end of the exposure time 580, and output analog signals accumulated in the photodetectors of the corresponding row line to the column line. When the readout time 590 ends, the first image sensor 430 and the second image sensor 440 may activate the exposure time 580 in the second frame F2 .

6A, 6B, 6C, and 6D are exemplary diagrams for explaining a scanning method of the image sensors 430 and 440 in the electronic device 400 according to various embodiments.

According to various embodiments, from a real image as shown in FIG. 6A, the processor 410 may obtain a first image by scanning the first image sensor 430 in a first direction, and may obtain a second image by scanning the second image sensor 440 in a second direction opposite to the first direction. Depending on the movement of the subject or the photographer, the first image and the second image may include different Jello phenomena. For example, in the first image, a relatively large jello phenomenon may occur in the second half line images of the pixel array, and in the second image, a relatively large jello phenomenon may occur in the first half line images of the pixel array. The first image and the second image may have the same Jello phenomenon in the middle line images in each pixel array.

According to one embodiment, the first image sensor 430 and the second image sensor 440 may have the same resolution, eg, N row lines, and may be scanned at the same speed, eg, A fps. In this case, the processor 410 may acquire the first image in one frame from the first image sensor 430 as shown in (a) of FIG. 6B, and acquire the second image in one frame from the second image sensor 440 as shown in (b) of FIG. 6B.

For example, the time required to scan each row line, that is, the time difference between row lines may be 1/(N*A) sec. According to this, the scan time T1(n) of the n-th row line in the first image sensor 430 is determined as in [Equation 1] based on the scan time T1(1) of the first row line, and correspondingly, the scan time T2(N-n) of the N-n row line in the second image sensor 440 is determined as [Equation 2] based on the scan time T2(1) of the first row line It can be. Through this, assuming that the scan time (T1 (1)) of the first row line in the first image sensor 430 and the scan time (T2 (1)) of the first row line in the second image sensor 440 are the same, the difference between the scan time (T1 (n)) of the n-th row line in the first image sensor 430 and the scan time (T2 (N-n)) of the N-n row line in the second image sensor 440 [ It can be determined as in Equation 3]. On the other hand, when the scan time T1(N/2) of the middle row line in the first image sensor 430 is taken as a reference, the scan time T1(n) of the nth row line in the first image sensor 430 may differ from the scan time T1(N/2) of the middle row line by T1(N/2)-T1(n). Also, when the scanning time T2(N/2) of the middle row line in the second image sensor 440 is taken as a reference, the scanning time T2(N-n) of the N-nth row line in the second image sensor 440 may differ from the scanning time T2(N/2) of the middle row line by T2(N/2)-T2(N-n).

According to another embodiment, the first image sensor 430 and the second image sensor 440 may have the same resolution, eg, N row lines, but may be scanned at different speeds. The scan speed of the first image sensor 430 may be A fps, and the scan speed of the second image sensor 440 may be B fps. For example, if the processor 410 obtains a first image as a frame in one frame in the first image sensor 430 as shown in FIG. The scan speed of the image sensor 430 may be twice. Alternatively, when the processor 410 acquires the first image with one frame from the first image sensor 430 as shown in (a) of FIG. 6D and the second image with three frames from the second image sensor 440 as shown in (b), (c) and (d) of FIG. .

For example, when the processor 410 acquires a first image with one frame from the first image sensor 430 and a second image with three frames from the second image sensor 440, the time required to scan each row line in the first image sensor 430, that is, the time difference between row lines is 1/(N*A) sec, and the second image sensor 440 scans each row line. The time taken to do this, that is, the time difference between the row lines, may be 1/(N*3A). According to this, the scan time T1(n) of the n-th row line in the first image sensor 430 is determined as in [Equation 4] based on the scan time T1(1) of the first row line, and correspondingly, the scan time T2(N-n) of the N-n row line in the second image sensor 440 is determined based on the scan time T2(1) of the first row line as in [Equation 5]. can be determined individually.

Through this, the processor 410 may estimate each motion in the first image and the second image based on the intermediate line images of the first image and the second image. Also, the processor 410 may compensate each motion in the first image and the second image. To this end, the processor 410 may determine a compensation amount for each row line, that is, a correction parameter, based on a motion vector (p) representing a pixel difference between row lines corresponding to each other in the first image sensor 430 and the second image sensor 440 and the scan timing of the row lines.

7 is a block diagram illustrating a detailed configuration of image sensors 430 and 440 in an electronic device 400 according to various embodiments.

Referring to FIG. 7 , the processor 410 may output information for controlling access between the first image sensor 430 and the second image sensor 440 . The processor 410 may output information for controlling access of the first image sensor 430 and the second image sensor 440 according to operating speeds of the first image sensor 430 and the second image sensor 440, that is, exposure time and readout time, resolution, and operation mode (e.g., binning mode). At this time, the processor 410 may generate a timing control signal capable of matching frame synchronization and line synchronization between the first image sensor 430 and the second image sensor 440 .

The first image sensor 430 may include a first sensor controller 710 , a first row driver 720 , a first pixel array 730 , a first converter 740 and a first column driver 750 . The first sensor controller 710 may control the operation of the first image sensor 430 under the control of the processor 410 . The first pixel array 720 includes a plurality of light sensing elements, and the light sensing elements may be connected to a plurality of row lines and a plurality of column lines. The first row driver 730 may set an exposure time for row lines of the first pixel array 720 and a readout time for reading out analog signals of the row lines to column lines. Through this, the light sensing elements of the first pixel array 720 may detect light for each row line unit during each exposure time and accumulate the analog signal, and output the analog signal to each column line at each readout time. The first converter 740 may convert an analog signal into digital data for each column line. The first column driver 750 may receive digital data from the first converter 740 and transfer the digital data to the first image processing unit included in the processor 410 in the MIPI method.

The second image sensor 440 may include a second sensor controller 715 , a second row driver 725 , a second pixel array 735 , a second converter 745 and a second column driver 755 . The second sensor controller 715 may control the operation of the second image sensor 440 under the control of the processor 410 . The second pixel array 725 includes a plurality of light sensing elements, and the light sensing elements may be connected to a plurality of row lines and a plurality of column lines. The second row driver 735 may set an exposure time for row lines of the second pixel array 725 and a readout time for reading out analog signals of the row lines to column lines. Through this, the photodetectors of the second pixel array 725 may sense light in units of each row line during each exposure time and accumulate the analog signal, and output the analog signal to each column line at each readout time. The second converter 745 may convert analog signals into digital data for each column line. The second column driver 755 may receive digital data from the second converter 745 and transfer the digital data to the second image processor included in the processor 410 in the MIPI method.

8 is a block diagram illustrating a detailed configuration of a processor 410 in an electronic device 400 according to various embodiments.

Referring to FIG. 8 , the processor 410 may include a first image processing unit 810, a second image processing unit 820, a motion estimation unit 830, an image correction unit 840, and an image control unit 850. The first image processor 810 may process digital data of the first image sensor 430 . Through this, the first image processing unit 810 may acquire the first image from the digital data of the first image sensor 430 . The second image processor 820 may process digital data of the second image sensor 440 . Through this, the second image processing unit 820 may acquire the second image from the digital data of the second image sensor 440 . The motion estimator 830 may estimate each motion in the first image and the second image. The image compensator 840 may compensate for each motion in the first image and the second image. The image controller 850 may combine the first image and the second image into one image, that is, a third image. Alternatively, the image controller 850 may select either the first image or the second image. Through this, the image controller 850 can display at least one of the first image, the second image, and the third image. According to an embodiment, the image controller 850 may display either the first image or the second image through a smooth transition between the first image and the second image. According to another embodiment, the image controller 850 may display a third image.

The electronic device 400 according to various embodiments may include a first image sensor 430, a second image sensor 440, and a processor 410 functionally connected to the first image sensor and the second image sensor.

According to various embodiments, the processor 410 may be configured to acquire a first image from the first image sensor 430, obtain a second image from the second image sensor 440, compare the first image and the second image, estimate motion in the first image and the second image, and compensate for the motion in the first image and the second image.

According to various embodiments, the processor 410 may be further configured to acquire the first image by reading out the first image sensor 430 in a first direction, and acquire the second image by reading out the second image sensor 440 in a second direction.

According to various embodiments, the first direction may be for reading out in the order of the first row line to the last row line of the first image sensor 430, and the second direction may be for reading out in the order of the first row line from the last row line of the second image sensor 440.

According to various embodiments, the first image sensor 430 and the second image sensor 440 may be disposed rotated 180 degrees from each other.

According to various embodiments, the processor 410 may be further configured to detect reference regions corresponding to each other in the first image and the second image, and to detect the motion in remaining regions of the first image and the second image based on the reference regions.

According to various embodiments, the processor 410 may be further configured to detect a pixel difference between a middle row line of the first image and a middle row line of the second image, detect a pixel difference between row lines corresponding to each other in the first image and the second image, and determine a correction parameter for each row line based on a difference between the pixel differences.

According to various embodiments, the processor 410 may be further configured to perform correction for each row line based on the correction parameter.

According to various embodiments, the processor 410 may be further configured to acquire the first image by reading out the first image sensor 430 at a first rate and acquire the second image by reading out the second image sensor 440 at a second rate.

According to various embodiments, the first image may have one frame and the second image may have a plurality of frames.

According to various embodiments, the processor 410 may be further configured to compare row lines scanned at the same time point in the first image and the second image.

9 is a flowchart illustrating an operating method of an electronic device 400 according to various embodiments. 10A, 10B, and 10C are exemplary diagrams for explaining an operating method of an electronic device 400 according to various embodiments.

Referring to FIG. 9 , the processor 410 may acquire images 1010 and 1020 in operation 911 . The processor 410 may acquire the first image 1010 by exposing and reading out the first pixel array 720 of the first image sensor 430 in a first direction, and may acquire the second image 1020 by exposing and reading out the second pixel array 725 of the second image sensor 440 in a second direction.

However, the jello phenomenon may occur in the first image 1010 and the second image 1020 . The first image 190 and the second image 1020 may include different Jello phenomena. For example, when a photographer captures the first image 1010 and the second image 1020 while moving from left to right, as shown in (a) and (b) of FIG. Alternatively, if the photographer captures the first image 1010 and the second image 1020 while moving from bottom to top, as shown in (a) and (b) of FIG. In the first image 1010, a relatively large jello phenomenon may occur corresponding to the second half row lines of the first pixel array 720, and in the second image 1020, a relatively large jello phenomenon may occur corresponding to the first half row lines of the second pixel array 725. Alternatively, when the first image 1010 and the second image 1020 including a moving subject are captured, a Jello phenomenon may occur with respect to the position of the subject in the first image 1010 and the second image 1020, as shown in (a) and (b) of FIG. 10C, respectively.

The processor 410 may estimate motion of each of the images 1010 and 1020 in operation 913 . The processor 410 may compare the first image 1010 and the second image 1020 to estimate motions in the first image 1010 and the second image 1020 . The processor 410 may determine reference areas in the first image 1010 and the second image 1020, and estimate respective motions in the first image 1010 and the second image 1020 based on the reference areas.

For example, the processor 410 may compare the first image 1010 and the second image 1020 based on midline images in the first image 1010 and the second image 1020, as shown in (c) of FIGS. 10A and 10B, respectively. To this end, the processor 410 may determine intermediate line images in the first image 1010 and the second image 1020 as reference regions r. The processor 410 may determine a pixel difference between reference regions r in the first image 1010 and the second image 1020 as a parallax correction value. The processor 410 may determine row lines corresponding to each other in the first image 1010 and the second image 1020 as motion detection areas a and b. The processor 410 may determine the first row line of the first image 1010 and the last row line of the second image 1020 as motion detection regions a by corresponding to each other. The processor 410 may determine a pixel difference between motion detection areas a in the first image 1010 and the second image 1020 as a motion detection value. The processor 410 may detect a correction parameter for a corresponding row line based on the parallax correction value and the motion detection value. In this way, the processor 410 may detect correction parameters for each row line in the first image 1010 and the second image 1020 as shown in [Equation 6] below. The correction parameter may have a relatively large value in the second half row lines of the first pixel array 720 in the first image 1010 and a relatively large value in the first half row lines of the second pixel array 725 in the second image 1020. Alternatively, the correction parameter may have a relatively large value in some of the row lines of the first pixel array 720 in the first image 1010 and some of the row lines of the second pixel array 725 in the second image 1020.

The processor 410 may compensate for motion in the images 1010 and 1020 in operation 915 . The processor 410 may correct row lines in the first image 1010 and the second image 1020 based on the correction parameter for each row line. Through this, the processor 410 may correct the first image 1010 and the second image 1020 as a whole, as shown in (d) and (e) of FIGS. 10A and 10B, respectively. Alternatively, the processor 410 may partially correct the first image 1010 and the second image 1020 as shown in (d) and (e) of FIG. 10C, respectively.

The processor 410 may control the images 1010 and 1020 in operation 917 . The processor 410 may combine the first image 1010 and the second image 1020 into one image, that is, a third image. Alternatively, the processor 410 may select one of the first image 1010 and the second image 1020 . Through this, the processor 410 may display at least one of the first image 1010, the second image 1020, and the third image. According to an embodiment, the processor 410 may display either the first image 1010 or the second image 1020 through a smooth transition between the first image 1010 and the second image 1020 . According to another embodiment, the processor 410 may display the third image.

11 is a flowchart illustrating a method for acquiring an image in an electronic device according to various embodiments. The following description describes an operation for acquiring images in operation 911 of FIG. 9 .

Referring to FIG. 11 , in operation 1111, the processor 410 may obtain images having time differences through the first image sensor 430 . The processor 410 may acquire at least two consecutive images by exposing and reading out the first pixel array 720 of the first image sensor 430 in a first direction.

In operation 1113, the processor 410 may compare images continuously acquired through the first image sensor 430 to determine whether motion compensation of the image is required. The processor 410 may determine that image motion compensation is required when pixel errors of images consecutively acquired through the first image sensor 430 are out of a reference range. The processor 410 may determine that image motion compensation is unnecessary when pixel errors of images successively acquired through the first image sensor 430 are within a reference range.

In operation 1115, the processor 410 may activate the second image sensor 440 when it is determined that motion compensation of the image is necessary. The processor 410 may set operating parameters of the second image sensor 440 so that an image for motion compensation of an image obtained through the first image sensor 430 may be acquired through the second image sensor 440 . For example, the processor 410 may extract an operating variable for motion detection corresponding to an operating variable of the first image sensor 430 from the LUT stored in the memory 420 and set it as an operating variable of the second image sensor 440. For example, the motion variable for motion detection may include at least one of an exposure value or a sensor gain capable of obtaining an image without blur without considering the amount of noise included in the image through the second image sensor 440 .

The processor 410 may acquire the first image 1010 and the second image 1020 through the first image sensor 430 and the second image sensor 440 in operation 1117 .

12 is a flowchart illustrating a method for estimating motion in an image in an electronic device according to various embodiments. Hereinafter, operations for estimating motion in an image in operations 911 to 913 of FIG. 9 will be described.

Referring to FIG. 12 , in operation 1211, the processor 410 may obtain a first image 1010 and a second image 1020 through the first image sensor 430 and the second image sensor 440. For example, the processor 410 may acquire the first image 1010 and the second image 1020 through the first image sensor 430 and the second image sensor 440 that perform asymmetric readout, as in operation 911 of FIG. 9 .

In operation 1213, the processor 410 may compare the first image 1010 and the second image 1020 to determine whether motion compensation of the image is required. When the error between the first image 1010 and the second image 1020 is out of a reference range, the processor 410 may determine that image motion compensation is required. When the error between the first image 1010 and the second image 1020 is within the reference range, the processor 410 may determine that image motion compensation is unnecessary.

In operation 1215, the processor 410 may estimate motion in the first image 1010 and the second image 1020 when it is determined that motion compensation of the image is necessary. For example, the processor 410 may detect reference regions corresponding to each other in the first image 1010 and the second image 1020 as in operation 913 of FIG. 9 . The processor 410 may estimate each motion in the first image 1010 and the second image 1020 based on the reference region.

An operating method of an electronic device 400 including a plurality of image sensors 430 and 440 according to various embodiments includes obtaining a first image from a first image sensor 430 and acquiring a second image from a second image sensor 440, estimating motion in the first image and second image by comparing the first image and second image, and compensating for the motion in the first image and the second image. can include

According to various embodiments, the operation of acquiring the first image and the second image may acquire the first image by reading out the first image sensor 430 in a first direction, and acquire the second image by reading out the second image sensor 440 in a second direction.

According to various embodiments, the first direction may be for reading out in the order of the first row line to the last row line of the first image sensor 430, and the second direction may be for reading out in the order of the first row line from the last row line of the second image sensor 440.

According to various embodiments, the first image sensor 430 and the second image sensor 440 may be disposed rotated 180 degrees from each other.

According to various embodiments, the motion estimation operation may include an operation of detecting reference regions corresponding to each other in the first image and the second image, and an operation of detecting the motion in remaining regions of the first image and the second image based on the reference regions.

According to various embodiments, the motion detection may include detecting a pixel difference between a middle row line of the first image and a middle row line of the second image, detecting a pixel difference between row lines corresponding to each other in the first image and the second image, and determining a correction parameter for each row line based on a difference between the pixel differences.

According to various embodiments, the motion compensation method may include performing correction for each row line based on the correction parameter.

According to various embodiments, the operation of obtaining the first image and the second image may acquire the first image by reading out the first image sensor 430 at a first speed, and acquire the second image by reading out the second image sensor 440 at a second speed.

According to various embodiments, the first image may have one frame and the second image may have a plurality of frames.

According to various embodiments, the motion estimation operation may include an operation of comparing row lines scanned at the same time point in the first image and the second image.

The embodiments disclosed in this document are presented for explanation and understanding of the disclosed technical content, and do not limit the scope of various embodiments of the present invention. Therefore, the scope of various embodiments of the present invention should be construed as including all changes or various other embodiments based on the technical spirit of various embodiments of the present invention.

Claims (20)

A method of operating an electronic device including a plurality of image sensors,
acquiring a first image having one frame from a first image sensor and acquiring a second image having N frames from a second image sensor;
comparing the first image with the second image and estimating motion in the first image and the second image; and
Compensating for the motion in the first image and the second image;
The scan speed of the second image sensor is N times the scan speed of the first image sensor,
The operation of obtaining the first image and the second image,
acquiring the first image by reading out the first image sensor; and
acquiring the second image by reading out the second image sensor; including,
The operation of estimating motion in the first image and the second image,
detecting reference regions corresponding to each other in the first image and the second image; and
detecting the motion in remaining regions of the first image and the second image based on the reference regions; Including,
Compensating for the motion in the first image and the second image,
setting a second operating variable associated with the second image sensor to include a different operating variable from the first operating variable associated with the first image sensor, based on the asymmetrical readout of the first image sensor and that of the second image sensor; and
setting the second motion variable to correspond to the motion detection operation based on using the second image acquired through the second image sensor as an image for motion compensation for the first image acquired through the first image sensor; How to include.
According to claim 1,
The first image and the second image obtaining operation,
A method of acquiring the first image by reading out the first image sensor in a first direction and acquiring the second image by reading out the second image sensor in a second direction.
◈Claim 3 was abandoned when the registration fee was paid.◈ According to claim 2,
The first direction is for reading out in order from a first row line to a last row line of the first image sensor;
wherein the second direction is to read out in order of a first row line from a last row line of the second image sensor.
◈Claim 4 was abandoned when the registration fee was paid.◈ According to claim 1,
The first image sensor and the second image sensor are disposed rotated 180 degrees from each other.
delete ◈Claim 6 was abandoned when the registration fee was paid.◈ According to claim 1,
The motion detection operation,
detecting a pixel difference between a middle row line of the first image and a middle row line of the second image;
detecting a pixel difference between row lines corresponding to each other in the first image and the second image;
and determining a correction parameter for each row line based on a difference value of the pixel differences.
◈Claim 7 was abandoned when the registration fee was paid.◈ According to claim 6,
The motion compensation operation,
and performing correction for each row line based on the correction parameter.
◈Claim 8 was abandoned when the registration fee was paid.◈ According to claim 1,
The first image and the second image obtaining operation,
A method of acquiring the first image by reading out the first image sensor at a first rate and acquiring the second image by reading out the second image sensor at a second rate.
delete ◈Claim 10 was abandoned when the registration fee was paid.◈ According to claim 1,
The motion estimation operation,
and comparing row lines scanned at the same time point in the first image and the second image.
In electronic devices,
a first image sensor;
a second image sensor; and
a processor operatively connected to the first image sensor and the second image sensor;
the processor,
Acquiring a first image having one frame from the first image sensor, acquiring a second image having N frames from the second image sensor,
Comparing the first image and the second image to estimate motion in the first image and the second image;
configured to compensate for the motion in the first image and the second image;
read out the first image sensor to obtain the first image, and read out the second image sensor to obtain the second image sensor;
detecting reference regions corresponding to each other in the first image and the second image, and detecting the motion in remaining regions of the first image and the second image based on the reference regions;
Based on the asymmetrical readout of the first image sensor and that of the second image sensor, set a second operating variable associated with the second image sensor to include a different operating variable from the first operating variable associated with the first image sensor,
Based on the use of the second image acquired through the second image sensor as an image for motion compensation for the first image obtained through the first image sensor, the second motion variable is set to correspond to the motion detection operation,
The electronic device, wherein the scan speed of the second image sensor is N times the scan speed of the first image sensor.
According to claim 11,
wherein the processor is further configured to acquire the first image by reading out the first image sensor in a first direction, and acquire the second image by reading out the second image sensor in a second direction.
According to claim 12,
The first direction is for reading out in order from a first row line to a last row line of the first image sensor;
The second direction is for reading out in order of a first row line from a last row line of the second image sensor.
According to claim 11,
The electronic device wherein the first image sensor and the second image sensor are rotated 180 degrees from each other.
delete According to claim 11,
the processor,
detecting a pixel difference between a middle row line of the first image and a middle row line of the second image;
detecting a pixel difference between row lines corresponding to each other in the first image and the second image;
The electronic device further configured to determine a correction parameter for each row line based on a difference value of the pixel differences.
17. The method of claim 16,
The electronic device of claim 1 , wherein the processor is further configured to perform correction for each row line based on the correction parameter.
According to claim 11,
wherein the processor is further configured to acquire the first image by reading out the first image sensor at a first rate, and acquire the second image by reading out the second image sensor at a second rate.
delete According to claim 11,
The processor is further configured to compare row lines scanned at the same time point in the first image and the second image.

Images