WO2021145612A1

WO2021145612A1 - Electronic device for performing image correction, and method for correcting image

Info

Publication number: WO2021145612A1
Application number: PCT/KR2021/000270
Authority: WO
Inventors: 이보희; 송원석; 이세현; 조정환; 허민
Original assignee: 삼성전자 주식회사
Priority date: 2020-01-15
Filing date: 2021-01-08
Publication date: 2021-07-22
Also published as: KR20210092012A

Abstract

An electronic device according to various embodiments comprises: a camera; and at least one processor, wherein the at least one processor can be set to obtain a first image including a plurality of pixels from the camera, and confirm a second image by performing first warping corresponding to lens distortion correction, rolling shutter correction, and video digital image stabilization (VDIS) correction on at least a portion of the plurality of pixels of the first image. Various other embodiments are also possible.

Description

Electronic devices that perform image correction and methods of image correction

Various embodiments relate to an electronic device for performing image correction and a method for correcting an image, and more particularly, to an electronic device for correcting distortion according to lens characteristics of an image and distortion due to camera shake and lens characteristics of an image The present invention relates to a method of correcting distortion caused by camera shake and distortion caused by camera shake.

BACKGROUND Electronic devices for correcting distortion occurring in an image are widespread. The electronic device may correct distortion according to lens characteristics of the image. The distortion according to the lens characteristics of the image may be a distortion in which a straight object appears convexly or concavely. The process of correcting distortion according to the lens characteristics of an image is called lens distortion correction (LDC).

The electronic device may correct distortion caused by camera shake. If the camera shakes when acquiring an image through the camera, the shake may appear in at least some areas of the image. In order to correct distortion due to camera shake, the electronic device may determine a crop region in the image and crop the image. The process of correcting distortion caused by camera shake is called video digital image stabilization (VDIS) correction.

The electronic device may correct rolling shutter distortion. Rolling shutter distortion is distortion that occurs because an image is not acquired as one full screen at a time, but by rapidly scanning the scene horizontally or vertically. The electronic device may correct rolling shutter distortion by performing rolling shutter correction on the image.

The electronic device is configured to perform video digital image stabilization (VDIS) correction on an image having distortion according to lens characteristics. In this case, the crop area in the VDIS correction process changes due to distortion according to the lens characteristics, and as a result, distortion occurs in the corrected image.

Another type of electronic device performs lens distortion correction and VDIS correction independently. In this case, since warping for correcting lens distortion and warping for correcting VDIS are independently performed, a buffer for correcting lens distortion and a buffer for correcting VDIS are separately required. In addition, when warping for correcting lens distortion and warping for correcting VDIS are independently performed, there is a problem in that the time and power consumption required for image correction are large because the warping must be processed twice.

According to an electronic device performing image correction and a method of correcting an image according to various embodiments, the electronic device may correct an image by performing one warping corresponding to lens distortion correction, VDIS correction, and rolling shutter correction. . According to various embodiments, the electronic device may correct the image by performing one warping corresponding to the lens distortion correction and the VDIS correction.

According to various embodiments, an electronic device includes a camera and at least one processor, wherein the at least one processor acquires a first image including a plurality of pixels from the camera, and At least some of the pixels may be configured to perform a first warping corresponding to a lens distortion correction, a rolling shutter correction, and a video digital image stabilization (VDIS) correction to confirm a second image.

According to various embodiments, a method performed by at least one processor of an electronic device includes: acquiring a first image including a plurality of pixels from a camera of the electronic device; At least in part, the method may include performing a first warping corresponding to a lens distortion correction, a rolling shutter correction, and a video digital image stabilization (VDIS) correction to confirm a second image.

According to various embodiments, an electronic device includes a camera configured to perform rolling shutter correction on an acquired image, and at least one processor, wherein the at least one processor includes a plurality of pixels from the camera , acquiring a first image on which rolling shutter correction has been performed, and second warping corresponding to lens distortion correction and video digital image stabilization (VDIS) correction on at least some of a plurality of pixels of the first image may be set to confirm the second image by performing .

According to various embodiments, an electronic device for performing image correction and a method for correcting an image are provided. Accordingly, the electronic device may correct the image by performing one warping corresponding to the lens distortion correction, the VDIS correction, and the rolling shutter correction, or by performing one warping corresponding to the lens distortion correction and the VDIS correction. Accordingly, since the electronic device does not separately include a buffer for correcting lens distortion and a buffer for correcting VDIS, memory resource requirements can be reduced. In addition, since the electronic device performs lens distortion correction, VDIS correction, and rolling shutter correction with one warping, or performs lens distortion correction and VDIS correction, image correction is performed compared to a case where warping is performed several times once for each correction. It is possible to reduce the time and power consumption required for

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;

2A is a block diagram of a camera module according to various embodiments.

2B is a block diagram of an electronic device according to various embodiments.

3 is a flowchart illustrating an operation of an electronic device according to various embodiments of the present disclosure;

4 is a flowchart illustrating an operation of an electronic device according to various embodiments of the present disclosure;

5A, 5B, and 5C are diagrams for explaining lens distortion correction according to various embodiments.

6A and 6B are diagrams for explaining rearrangement of a rolling shutter correction value according to various embodiments of the present disclosure;

7 is a diagram illustrating VDIS correction according to various embodiments.

8A and 8B are diagrams for explaining crop region setting in VDIS correction according to various embodiments.

9 is a diagram for describing motion compensation according to various embodiments.

10A and 10B are diagrams for explaining lens distortion correction, VDIS correction, and rolling shutter correction according to various embodiments.

11 is a flowchart illustrating an operation of an electronic device according to various embodiments of the present disclosure;

12 is a flowchart illustrating an operation of an electronic device according to various embodiments of the present disclosure;

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input device 150 , a sound output device 155 , a display device 160 , an audio module 170 , and a sensor module ( 176 , interface 177 , haptic module 179 , camera module 180 , power management module 188 , battery 189 , communication module 190 , subscriber identification module 196 , or antenna module 197 . ) may be included. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components may be implemented as one integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be loaded into the volatile memory 132 , process commands or data stored in the volatile memory 132 , and store the resulting data in the non-volatile memory 134 . According to an embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and a secondary processor 123 (eg, a graphics processing unit, an image signal processor) that can be operated independently or in conjunction with the main processor 121 . , a sensor hub processor, or a communication processor). Additionally or alternatively, the auxiliary processor 123 may be configured to use less power than the main processor 121 or to be specialized for a designated function. The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 may be, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input device 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output a sound signal to the outside of the electronic device 101 . The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display device 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to an embodiment, the display device 160 may include a touch circuitry configured to sense a touch or a sensor circuit (eg, a pressure sensor) configured to measure the intensity of a force generated by the touch. there is.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input device 150 , or an external electronic device (eg, a sound output device 155 ) connected directly or wirelessly with the electronic device 101 . The sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 199 (eg, a cellular network, the Internet, or It may communicate with an external electronic device via a computer network (eg, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified and authenticated.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as a part of the antenna module 197 .

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the

electronic devices

102 and 104 may be the same or a different type of the electronic device 101 . According to an embodiment, all or part of the operations executed in the electronic device 101 may be executed in one or more of the external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. The one or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the execution result to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

2A is a block diagram 200a illustrating a camera module 180, according to various embodiments. Referring to FIG. 2A , the camera module 180 includes a lens assembly 210a, a flash 220a, and an image. It may include a sensor 230a, an image stabilizer 240a, a memory 250a (eg, a buffer memory), or an image signal processor 260a. The lens assembly 210a may collect light emitted from a subject, which is an image capturing object. The lens assembly 210a may include one or more lenses. According to an embodiment, the camera module 180 may include a plurality of lens assemblies 210a. In this case, the camera module 180 may form, for example, a dual camera, a 360 degree camera, or a spherical camera. Some of the plurality of lens assemblies 210a may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. It may have one or more lens properties different from the lens properties of . The lens assembly 210a may include, for example, a wide-angle lens or a telephoto lens.

The flash 220a may emit light used to enhance light emitted or reflected from the subject. According to an embodiment, the flash 220a may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 230a converts light emitted or reflected from the subject and transmitted through the lens assembly 210a into an electrical signal, thereby acquiring an image corresponding to the subject. According to an embodiment, the image sensor 230a is, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having a property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230a may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

In response to the movement of the camera module 180 or the electronic device 101 including the same, the image stabilizer 240a moves at least one lens or the image sensor 230a included in the lens assembly 210a in a specific direction or Operation characteristics of the image sensor 230a may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken. According to an embodiment, the image stabilizer 240a is, according to an embodiment, the image stabilizer 240a is a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180 . Such a movement of the camera module 180 or the electronic device 101 may be detected using . According to an embodiment, the image stabilizer 240a may be implemented as, for example, an optical image stabilizer. The memory 250a may temporarily store at least a portion of the image acquired through the image sensor 230a for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, Bayer-patterned image or high-resolution image) is stored in the memory 250a and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display device 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a portion of the original image stored in the memory 250a may be obtained and processed by, for example, the image signal processor 260a. According to an embodiment, the memory 250a may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .

The image signal processor 260a may perform one or more image processing on an image acquired through the image sensor 230a or an image stored in the memory 250a. The one or more image processes may include, for example, depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring), sharpening (sharpening), or softening (softening) Additionally or alternatively, the image signal processor 260a may include at least one of the components included in the camera module 180 (eg, an image sensor). 230a)), such as exposure time control, readout timing control, etc. The image processed by the image signal processor 260a is stored back in the memory 250a for further processing. or may be provided as an external component of the camera module 180 (eg, the memory 130, the display device 160, the electronic device 102, the electronic device 104, or the server 108). According to an example, the image signal processor 260a may be configured as at least a part of the processor 120 or may be configured as a separate processor operated independently of the processor 120. The image signal processor 260a may be configured as the processor 120 and a separate processor, the at least one image processed by the image signal processor 260a may be displayed through the display device 160 as it is or after additional image processing is performed by the processor 120 .

According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least the other may be a rear camera.

2B is a block diagram of an electronic device according to various embodiments. According to various embodiments, the electronic device 200b may include a processor 210b, a camera 220b, and a memory 230b. According to various embodiments, the electronic device 200b may be the electronic device 101 illustrated in FIG. 1 . According to various embodiments, the processor 210b may be the processor 120 illustrated in FIG. 1 . According to various embodiments, the processor 210b may be an application processor. According to various embodiments, the camera 220b may be the camera module 180 illustrated in FIG. 1 . According to various embodiments, the memory 230b may be at least one of the memory 130 illustrated in FIG. 1 or a memory included in the processor 120 (eg, an application processor) of FIG. 1 .

According to various embodiments, the camera 220b acquires an image on which lens distortion correction, rolling shutter correction, and video digital image stabilization (VDIS) correction are not performed, and transmits it to the processor 210b as an input image. can The processor 210b may check the output image by performing lens distortion correction, rolling shutter correction, and VDIS correction on at least some of the plurality of pixels of the input image obtained from the camera 220b. According to various embodiments, the processor 210b may perform a first warping for performing lens distortion correction, rolling shutter correction, and VDIS correction on at least some of the plurality of pixels of the input image, and performs the first warping In this case, data may be stored in the memory 230b or referenced to the memory 230b.

According to various embodiments, the camera 220b may acquire an image on which rolling shutter correction has been performed and transmit it to the processor 210b as an input image. The processor 210b may check the output image by performing lens distortion correction and VDIS correction on at least some of the plurality of pixels of the input image obtained from the camera 220b. According to various embodiments, the processor 210b may perform second warping for performing lens distortion correction and VDIS correction on at least some of a plurality of pixels of the input image, and when performing the second warping, the memory 230b ) or store data in the memory 230b.

3 is a flowchart 300 for explaining an operation of an electronic device (eg, the electronic device 200b) according to various embodiments of the present disclosure. The processor (eg, the processor 210b) of the electronic device (eg, the electronic device 200b) may acquire the first image from the camera (eg, the camera 220b) in operation 310 . According to various embodiments, the first image may be an image on which lens distortion correction, rolling shutter correction, and VDIS correction are not performed.

In operation 320 , the processor 210b may confirm the second image by performing the first warping corresponding to the lens distortion correction, the rolling shutter correction, and the VDIS correction on at least some of the plurality of pixels of the first image. According to various embodiments, the first warping may be performed based on a lens distortion correction, a rolling shutter correction value for a rolling shutter correction, a crop region for VDIS correction, and a smoothed motion trajectory of the camera. According to various embodiments, the rolling shutter correction value may be determined based on the lens distortion correction. According to various embodiments, a crop region for VDIS correction may be identified based on lens distortion correction. In operation 320 , lens distortion correction, rolling shutter correction, and VDIS correction may be performed at a time with a single first warping.

According to various embodiments, after operation 320 is performed, the processor 210b displays the identified second image on the electronic device (eg, the electronic device 101) on the display (eg, the display device 160). can be displayed on the

4 is a block diagram 400 illustrating an operation of an electronic device according to various embodiments of the present disclosure. Specifically, the block diagram 400 of FIG. 4 shows information exchange between the memory 401 , the buffer 402 , and the processor 403 , and a process of an operation performed by the processor 403 . 5A-5C, 6A-6B, 7, 8A-8B, and 9 are referenced to describe the process of the operation performed by the processor 403. FIG. 5A, 5B, and 5C are diagrams for explaining lens distortion correction according to various embodiments. 6A and 6B are diagrams for explaining rearrangement of a rolling shutter correction value according to various embodiments of the present disclosure; 7 is a diagram illustrating VDIS correction according to various embodiments. 8A and 8B are diagrams for explaining crop region setting in VDIS correction according to various embodiments. 9 is a diagram for describing motion compensation according to various embodiments.

According to various embodiments, the memory 401 may be a memory and/or a memory 130 included in the processor 120 (eg, an application processor) of FIG. 1 . According to various embodiments, the buffer 402 may be included in the processor 120 (eg, an application processor) of FIG. 1 or the memory 130 . According to various embodiments, the processor 403 may be the processor 210b of FIG. 2B .

In operation 411 , the memory 401 may transmit the N-th frame to the processor 403 as an input image. According to various embodiments, the input image may be a distorted image 421 on which lens distortion correction, rolling shutter correction, and VDIS correction are not performed.

In operation 412 , the processor 403 may enqueue an N-th frame that is an input image in an N-th buffer of the buffer 402 . In operation 413 , the processor 403 may dequeue the previous buffer from the N-th buffer of the buffer 402 by a predetermined number of delays, for example, d. The data dequeued from the N-d-th buffer may be used in the processor 403 in a calculation process of correcting the distortion of the input image.

The processor 403 may check 422 motion information of the camera (eg, the camera 220b). According to various embodiments, the motion information may be information indicating the motion of the camera 220b when the camera 220b captures an input image. According to various embodiments, the motion information may be identified based on at least one of a sensing value obtained from a gyro sensor included in the electronic device (eg, the electronic device 101 ) or an input image. According to various embodiments, the motion information may include a primary rolling shutter correction value and a motion trajectory of the camera. According to various embodiments, the primary rolling shutter correction value may be a rolling shutter correction value calculated based on the movement of the camera with respect to the N-th frame. According to various embodiments, the processor 403 may check the movement trajectory of the camera by accumulating the amount of movement of the camera from the first frame to the N-th frame.

The processor 403 may perform lens distortion correction 423 . Lens distortion correction may be a process for correcting distortion according to lens characteristics. FIG. 5A shows an exemplary image 500a before lens distortion correction is performed, and FIG. 5B shows an exemplary image 500b after lens distortion correction is performed on the exemplary image 500a of FIG. 5A . Fig. 5c shows the relationship between the distortion and the image height of the paraxial chief ray, used for lens distortion correction.

Elevation of the paraxial chief ray

, true ray

The relationship between , and distortion DIY is as follows.

By rearranging Equation 1, the following relationship is established.

In the graph 500c shown in Fig. 5c, the curve 510c is the image height of the paraxial chief ray.

shows the relationship between and distortion. Curve 520c represents curve 510c as the height of the paraxial chief ray

It is an approximate curve in polynomial form for . The image height of the paraxial chief ray, represented by curve 520c

The relationship between and distortion is

According to various embodiments, Equation 3 may be stored in the memory 130 of the electronic device (eg, the electronic device 101 ), and in this case, the processor 403 (eg, the processor 120 ) may perform lens distortion correction with reference to Equation 3 stored in the memory 130 . In another embodiment, a lookup table representing a mapping relationship between exemplary values corresponding to Equation 3 may be stored in the memory 130 of the electronic device 101 , and in this case, the processor 403 stores the memory 130 in the memory 130 . Lens distortion correction may be performed by referring to the stored lookup table.

Coordinates on the input image

and coordinates on the output image obtained through lens distortion correction

The relationship between them can be expressed as

That is, the processor 403 may check the lens distortion correction coordinates by performing the lens distortion correction 423 .

The processor 403 that has performed the lens distortion correction 423 may remap the primary rolling shutter correction value included in the motion information based on the lens distortion correction to check the remapped rolling shutter correction value. In the rolling shutter remapping 424 process, the lens distortion correction map may be referred to. According to various embodiments, the rolling shutter remapping 424 process may be performed by applying a lens distortion correction map to the primary rolling shutter correction value.

6A shows an exemplary image 600a in which a primary rolling shutter correction value is displayed. 6B is an exemplary image 600b in which the remapped rolling shutter correction value is displayed after the primary rolling shutter correction value of the exemplary image 600a of FIG. 6A is remapped through the rolling shutter remapping 424 process. shows Coordinates on the input image

and coordinates on the output image obtained through the remapped rolling shutter correction

The relationship between them can be expressed as

The processor 403 that has performed the lens distortion correction 423 may smooth the motion trajectory of the camera included in the motion information based on the lens distortion correction and check a crop region for the VDIS ( 425 ). 7 is a graph 700 for explaining a process of smoothing a motion trajectory of a camera and checking a crop area for VDIS. A curve 710 indicates a motion trajectory of a camera included in the motion information, and a curve 720 indicates a smoothed motion trajectory of the camera obtained by smoothing the motion trajectory of the camera.

Curves

730 and 740 represent the boundaries of the crop region for VDIS. According to various embodiments, the crop area for the VDIS may be set so that the curve 720 does not deviate between the curve 730 and the curve 740 .

In the process of checking the crop region 425 , the lens distortion correction coordinates confirmed as a result of the lens distortion correction 423 may be referred to. Since the input image is deformed in the lens distortion correction process 423, the boundary of the crop region for VDIS is changed. Accordingly, the processor 403 may identify the crop region such that only valid regions are included in the crop region. Fig. 8A shows an exemplary input image 800a and an exemplary crop region 810a before lens distortion correction 423 is performed, and Fig. 8B shows an exemplary input image 800a of Fig. 8A with lens distortion correction ( 423) is performed and shows an image 800b obtained as a result and

exemplary crop regions

810b, 820b, and 830b. In FIG. 8B ,

exemplary crop regions

810b , 820b , 830b are shown within the effective region of image 800b .

The VDIS correction formula can be expressed as follows.

According to various embodiments, the processor 403 may first perform a crop region check 425 and then perform a rolling shutter remapping 424 . The processor 403 may perform a check as a result of the crop region check 425 . The motion compensation 426 may be performed based on the smoothed motion trajectory of the camera and the remapped rolling shutter compensation value identified as a result of the rolling shutter remapping 424 .

9 is a diagram for explaining exemplary motion correction 426 and lens distortion correction 423 processes. An exemplary process illustrated in FIG. 9 checks coordinates on the input image 920 corresponding to respective coordinates of the output image 910 corresponding to the crop region, and corrects lens distortion, which corresponds to the coordinates on the input image 920 . This is a process of confirming coordinates on the considered image 930 . In FIG. 9 , the coordinate 911 of the output image 910 corresponds to the coordinate 921 on the input image 920 , and the coordinate 921 on the input image 920 is an image 930 in which lens distortion correction is considered again. Corresponding to the coordinates 931 of the image are shown. In addition, in FIG. 9 , the coordinates 912 of the output image 910 correspond to the coordinates 922 on the input image 920 , and the coordinates 921 on the input image 920 are again the image in which the lens distortion correction is considered ( Corresponding to the coordinates 932 on the 930 are shown.

In the example of Figure 9, the coordinates on the output image

and coordinates on the input image

The relationship between them can be expressed as

According to various embodiments, the processor 403 may perform the lens distortion correction 423 with reference to the crop coordinates confirmed through the crop region confirmation 425 and the crop coordinates on which the motion correction 426 is performed. 10A and 10B are diagrams for explaining lens distortion correction, VDIS correction, and rolling shutter correction according to various embodiments. 10A illustrates an example of applying the first rolling shutter correction value 1020a to the image 1010a in which the crop region 1011a in consideration of lens distortion correction is displayed. As a result of the rolling shutter correction, it can be seen that the crop region 1031a in the result image 1030a of FIG. 10A is different from the crop region 1011a in the image 1010a. 10B illustrates an example of applying the remapped rolling shutter correction value 1020b to the image 1010b in which the crop region 1011b in consideration of lens distortion correction is displayed. It can be seen that the crop region 1031b in the result image 1030b of FIG. 10B is different from the crop region 1011b in the image 1010b. Compared with the result image 1030a of FIG. 10A , it can be seen that the crop region 1011b is different from the crop region 1011a in the case of FIG. 10B because the remapped rolling shutter correction value 1020b is used.

The processor 403 determines the coordinates on the output image obtained through the motion compensation 426 .

and coordinates on the input image

Warping 427 for generating an output image may be performed by inputting a pixel value of a coordinate on the input image corresponding to a pixel of each coordinate on the output image based on the correspondence between them. The processor 403 may check the output image generated as a result of performing the warping 427 as the corrected image 428 .

In operation 414 , the memory 401 may transmit the N+1th frame to the processor 403 as an input image.

11 is a flowchart 1100 illustrating an operation of an electronic device (eg, the electronic device 200b) according to various embodiments of the present disclosure. The processor (eg, the processor 210b) of the electronic device (eg, the electronic device 200b) performs the rolling shutter correction from the camera (eg, the camera 220b) in operation 1110 on the first image. can be obtained. According to various embodiments, the first image may be an image on which lens distortion correction is performed and rolling shutter correction and VDIS correction are not performed.

In operation 1120 , the processor 210b may confirm the second image by performing second warping corresponding to lens distortion correction and VDIS correction on at least some of the plurality of pixels of the first image. According to various embodiments, the first warping may be performed based on a lens distortion correction, a crop region for VDIS correction, and a smoothed motion trajectory of the camera. According to various embodiments, a crop region for VDIS correction may be identified based on lens distortion correction. In operation 1120 , lens distortion correction and VDIS correction may be performed at a time with a single second warping.

According to various embodiments, after operation 1120 is performed, the processor 210b displays the identified second image on the electronic device (eg, the electronic device 101) (eg, the display device 160). can be displayed on the

12 is a flowchart 1200 illustrating an operation of an electronic device (eg, the electronic device 200b) according to various embodiments of the present disclosure. Specifically, FIG. 12 shows operations performed to perform operation 1120 of FIG. 11 .

The processor (eg, the processor 210b) of the electronic device (eg, the electronic device 200b) may check motion information of the camera (eg, the camera 220b) in operation 1210 . The details of the process of confirming 422 of the motion information described with reference to 422 of FIG. 4 may be equally applied to operation 1210, and thus will not be repeated herein.

In operation 1220 , the processor 210b may identify a smoothed motion trajectory and a crop region of the camera 220b based on the motion information. The details of the process of confirming the smoothed motion trajectory and the crop region described with reference to 423 and 425 of FIG. 4 may be equally applied to operation 1220, and thus will not be repeated herein.

In operation 1230 , the processor 210b may check the second image by performing second warping based on the lens distortion correction, the smoothed motion trajectory of the camera, and the crop region. Specifically, the processor 210b performs motion correction based on the lens distortion correction, the smoothed motion trajectory of the camera, and the crop region, and coordinates on the output image identified through the motion correction.

and coordinates on the input image

The second warping may be performed based on the correspondence between the two, and an output image generated as a result of performing the second warping may be identified as the second image.

The details of the motion compensation process described with reference to 426 of FIG. 4 may be equally applied to operation 1230, and thus will not be repeated herein. However, unlike the motion compensation described with reference to 426 of FIG. 4 , in motion compensation included in operation 1230, coordinates on the output image

and coordinates on the input image

Equation 7 does not apply to the correspondence between them. In motion compensation included in operation 1230, coordinates on the output image

and coordinates on the input image

The correspondence between them is as follows.

The second warping may be a process of generating an output image by inputting pixel values of coordinates on the input image corresponding to pixels of each coordinate on the output image based on the correspondence obtained based on Equation (8).

According to various embodiments, the electronic device 101 includes a camera 180 and at least one processor 120 , wherein the at least one processor 120: receives a plurality of pixels from the camera 180 . acquire a first image comprising; A second image is identified by performing a first warping on at least a portion of the plurality of pixels of the first image corresponding to lens distortion correction, rolling shutter correction, and video digital image stabilization (VDIS) correction. can be set to

According to various embodiments, the first warping is performed based on the lens distortion correction, the rolling shutter correction value for the rolling shutter correction, the crop region for the VDIS correction, and the smoothed motion trajectory of the camera 180 . can be

According to various embodiments, the at least one processor 120 may be set to check the rolling shutter correction value based on the lens distortion correction.

According to various embodiments, the at least one processor 120 may be configured to check the crop region based on the lens distortion correction.

According to various embodiments, the at least one processor 120 may be further configured to check the motion information of the camera 180 .

According to various embodiments, the electronic device 101 further includes a gyro sensor, and the at least one processor 120 moves the movement based on at least one of a sensing value obtained from the gyro sensor or the first image. It may be further configured to confirm information.

According to various embodiments, the at least one processor 120 is configured to: identify a primary rolling shutter correction value for each of the plurality of pixels of the first image based on the motion information; Based on the primary rolling shutter correction value and the lens distortion correction, the primary rolling shutter correction value may be set to confirm a remapped rolling shutter correction value.

According to various embodiments, the at least one processor 120 is configured to: identify a movement trajectory of the camera 180 based on the movement information; check the smoothed motion trajectory by smoothing the motion trajectory of the camera 180; Based on the lens distortion correction, the smoothed motion trajectory may be set to identify the crop region that does not deviate from the crop region for the VDIS correction.

According to various embodiments, the electronic device 101 may further include a display, and the at least one processor 120 may be further configured to display the second image on the display.

According to various embodiments, the electronic device 101 includes a camera 180 configured to perform rolling shutter correction on an acquired image, and at least one processor 120 , and the at least one processor 120 . obtains, from the camera 180 , a first image including a plurality of pixels on which rolling shutter correction has been performed; The second image may be identified by performing second warping corresponding to lens distortion correction and video digital image stabilization (VDIS) correction on at least some of the plurality of pixels of the first image.

According to various embodiments, the second warping may be performed based on a crop region for correcting the lens distortion and correcting the VDIS.

According to various embodiments, the at least one processor 120 may be configured to identify a crop region for the VDIS correction based on the lens distortion correction.

According to various embodiments, the at least one processor 120 is configured to: identify a movement trajectory of the camera 180 based on the movement information; check the smoothed trajectory by smoothing the movement trajectory of the camera 180; Based on the lens distortion correction, the smoothed trajectory may be set to identify the crop region that does not deviate from the crop region for the VDIS correction.

According to various embodiments, the method performed by the at least one processor 120 of the electronic device 101 includes: acquiring a first image including a plurality of pixels from the camera 180 of the electronic device 101 . movement; and confirming a second image by performing a first warping corresponding to lens distortion correction, rolling shutter correction, and video digital image stabilization (VDIS) correction on at least a portion of a plurality of pixels of the first image It may include an action to

According to various embodiments of the present disclosure, the method may further include checking a rolling shutter correction value for correcting the rolling shutter and a crop region for correcting the VDIS based on the lens distortion correction.

According to various embodiments, the method may include: checking motion information of the camera 180 ; checking a primary rolling shutter correction value for each of the plurality of pixels of the first image based on the motion information; and checking a rolling shutter correction value to which the primary rolling shutter correction value is remapped based on the primary rolling shutter correction value and the lens distortion correction.

According to various embodiments, the method may include: checking motion information of the camera 180 ; checking a motion trajectory of the camera 180 based on the motion information; checking the smoothed trajectory by smoothing the movement trajectory of the camera 180; and checking the crop region so that the smoothed trajectory does not deviate from the crop region for VDIS correction based on the lens distortion correction.

According to various embodiments, the method performed by the at least one processor 120 of the electronic device 101 includes: a camera configured to perform rolling shutter correction on an acquired image included in the electronic device 101 ( 180), the operation of obtaining a first image including a plurality of pixels, the rolling shutter correction is performed; and confirming a second image by performing second warping corresponding to lens distortion correction and video digital image stabilization (VDIS) correction on at least some of the plurality of pixels of the first image. can

According to various embodiments, the crop region for the VDIS correction may be identified based on the lens distortion correction.

According to various embodiments, the method may further include checking motion information of the camera 180 .

According to various embodiments, the checking of the motion information of the camera 180 includes checking the motion information based on at least one of a sensing value obtained from the gyro sensor of the electronic device 101 or the first image. It can include actions.

According to various embodiments, the method may include: identifying a movement trajectory of the camera 180 based on the movement information; checking the smoothed trajectory by smoothing the movement trajectory of the camera 180; and checking the crop region so that the smoothed trajectory does not deviate from the crop region for VDIS correction based on the lens distortion correction.

According to various embodiments, the method may further include displaying the second image on the display of the electronic device 101 .

The various embodiments of this document and the terms used therein are not intended to limit the technology described in this document to a specific embodiment, but it should be understood to cover various modifications, equivalents, and/or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like components. The singular expression may include the plural expression unless the context clearly dictates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as "first", "second", "first" or "second" can modify the corresponding elements regardless of order or importance, and are used only to distinguish one element from another element. The components are not limited. When an (eg, first) component is referred to as being “connected (functionally or communicatively)” or “connected” to another (eg, second) component, that component is It may be directly connected to the component or may be connected through another component (eg, a third component).

As used herein, the term “module” includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present document include instructions stored in a machine-readable storage media (eg, internal memory 136 or external memory 138) that can be read by a machine (eg, a computer). It may be implemented as software (eg, the program 140). The device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, the electronic device 101 ) according to the disclosed embodiments. When the instruction is executed by a processor (eg, the processor 120), the processor may directly or use other components under the control of the processor to perform a function corresponding to the instruction. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

According to an example, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)) or online through an ^{application store (eg Play Store TM ).} In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

Each of the components (eg, a module or a program) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be It may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, parallel, iteratively, or heuristically, or at least some operations are executed in a different order, are omitted, or other operations are added. can be

Claims

In an electronic device,

camera, and

at least one processor;

The at least one processor comprises:

acquiring a first image comprising a plurality of pixels from the camera;

A second image is identified by performing a first warping on at least a portion of the plurality of pixels of the first image corresponding to lens distortion correction, rolling shutter correction, and video digital image stabilization (VDIS) correction.

An electronic device configured to do so.
According to claim 1,

The first warping is performed based on the lens distortion correction, a rolling shutter correction value for correcting the rolling shutter, a crop region for correcting the VDIS, and a smoothed motion trajectory of the camera.
3. The method of claim 2, wherein the at least one processor is

and to check the rolling shutter correction value based on the lens distortion correction.
3. The method of claim 2, wherein the at least one processor is

and to identify the crop region based on the lens distortion correction.
According to claim 1,

The at least one processor is further configured to check motion information of the camera.
6. The method of claim 5,

The electronic device further includes a gyro sensor,

The at least one processor is further configured to identify the motion information based on at least one of a sensing value obtained from the gyro sensor and the first image.
6. The method of claim 5,

The at least one processor comprises:

check a primary rolling shutter correction value for each of the plurality of pixels of the first image based on the motion information;

The electronic device is configured to identify a rolling shutter correction value to which the primary rolling shutter correction value is remapped, based on the primary rolling shutter correction value and the lens distortion correction.
6. The method of claim 5,

At least one processor is:

check a motion trajectory of the camera based on the motion information;

check the smoothed motion trajectory by smoothing the motion trajectory of the camera;

and to identify the crop region that does not deviate from the crop region for the VDIS correction by the smoothed motion trajectory based on the lens distortion correction.
According to claim 1,

The electronic device further includes a display,

and the at least one processor is further configured to display the second image on the display.
A method performed by at least one processor of an electronic device, comprising:

acquiring a first image including a plurality of pixels from a camera of the electronic device; and

Confirming a second image by performing a first warping corresponding to at least some of the plurality of pixels of the first image, lens distortion correction, rolling shutter correction, and video digital image stabilization (VDIS) correction movement

How to include.
11. The method of claim 10, wherein the method

The method further comprising the operation of identifying a rolling shutter correction value for correcting the rolling shutter and a crop area for correcting the VDIS based on the lens distortion correction.
In an electronic device,

a camera set to perform rolling shutter correction on the acquired image, and

at least one processor;

The at least one processor comprises:

acquiring, from the camera, a first image on which rolling shutter correction has been performed, including a plurality of pixels;

A second image is identified by performing second warping corresponding to lens distortion correction and video digital image stabilization (VDIS) correction on at least some of the plurality of pixels of the first image

An electronic device configured to do so.
13. The method of claim 12,

The second warping is performed based on a crop region for the lens distortion correction and the VDIS correction.
14. The method of claim 13, wherein the at least one processor comprises:

The electronic device is configured to identify a crop region for the VDIS correction based on the lens distortion correction.
13. The method of claim 12,

The at least one processor is further configured to check motion information of the camera.