EP3970111A1 - Image processing device and method - Google Patents

Image processing device and method

Info

Publication number
EP3970111A1
EP3970111A1 EP19728376.5A EP19728376A EP3970111A1 EP 3970111 A1 EP3970111 A1 EP 3970111A1 EP 19728376 A EP19728376 A EP 19728376A EP 3970111 A1 EP3970111 A1 EP 3970111A1
Authority
EP
European Patent Office
Prior art keywords
image
pixels
current
distortion
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19728376.5A
Other languages
German (de)
French (fr)
Inventor
Stephen Busch
Cedric RODRIGUES
Youness NOURI
Eric Badi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of EP3970111A1 publication Critical patent/EP3970111A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually

Definitions

  • the present invention relates to the field of image processing. More specifically, the present invention relates to an image processing device and method for compensating image distortions.
  • ISP image signal processor
  • SoC System on a Chip
  • the ISP can process an electronic image obtained by the camera in order to compensate, i.e. reduce or remove image distortions caused, for instance, by the camera itself, such as the rolling shutter effect, or by a movement of the camera, while capturing the image. Because of high camera resolution these electronic images are usually very large so that the image processor can only process respective portions of the image stored in a local cache memory one after the other, while the whole image is stored in a high latency main memory, in particular a DRAM of the electronic device.
  • the invention relates to an image processing device, such as a smartphone or a component of a smartphone.
  • the image processing device comprises a main memory, in particular a DRAM, for storing a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels.
  • the image processing device comprises a processing circuitry configured to generate a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image.
  • the processing circuitry is configured to apply the second mapping subsequently to, i.e. after the first mapping for generating the distortion- compensated image.
  • the image processing device further comprises a fast-access i.e. low latency, local cache memory for storing, i.e. caching a plurality of non-overlapping, rectangular portions of the current image in a corresponding plurality of sectors of the cache memory, wherein each sector comprises a subset of the plurality of pixels stored in the main memory.
  • a fast-access i.e. low latency, local cache memory for storing, i.e. caching a plurality of non-overlapping, rectangular portions of the current image in a corresponding plurality of sectors of the cache memory, wherein each sector comprises a subset of the plurality of pixels stored in the main memory.
  • the processing circuitry is further configured to determine on the basis of the image distortion model one or more positions of one or more pixels of the first set of support pixels and/or one or more pixels of the second set of support pixels in the current image and to select the plurality of portions of the current image for storing in the corresponding plurality of sectors of the cache memory on the basis of the one or more positions of the one or more pixels of the first set of support pixels and/or the one or more pixels of the second set of support pixels in the current image.
  • the first mapping of the image distortion model is configured to compensate one or more distortions caused by an image rotation and/or the second mapping of the image distortion model is configured to
  • a current full-integer pixel in the current image lies within a global cell defined by a plurality of, in particular four current pixels of the first set of support pixels or the second set of support pixels in the current image
  • the processing circuitry is configured to generate the corresponding current sub integer pixel in the distortion-compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory by: (i) determining the respective sub-integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image;
  • the processing circuitry is configured to determine the sub-integer position of the corresponding current pixel in the distortion-compensated image on the basis of the image distortion model and the respective sub-integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image using interpolation, in particular bi-linear interpolation.
  • the processing circuitry is further configured to determine a pixel value of a current full-integer pixel in the distortion- compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory by:
  • the processing circuitry is configured to determine the pixel value of the current full-integer pixel in the distortion- compensated image on the basis of the pixel values and the sub-integer positions of the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image using interpolation, in particular bi-linear interpolation.
  • the image processing device further comprises an image capturing device, in particular a camera for obtaining the current image.
  • the first set of support pixels and the second set of support pixels are equal.
  • the first set of support pixels is a subset of the second set of support pixels or the second set of support pixels is a subset of the first set of support pixels.
  • the respective support grids defined by the first set of support pixels and the second set of support pixels can have different resolutions.
  • the invention relates to a corresponding image processing method comprising the steps of: providing a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and generating a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image and wherein the second mapping is applied subsequently to, i.e. after the first mapping for generating the distortion-compensated image.
  • the image processing method according to the second aspect of the invention can be performed by the image processing device according to the first aspect of the invention. Further features of the image processing method according to the second aspect of the invention result directly from the functionality of the image processing device according to the first aspect of the invention and its different implementation forms described above and below.
  • the invention relates to a computer program with a program code for performing an image processing method according to the second aspect of the invention when the computer program runs on a computer.
  • the invention relates to a computer readable storage medium comprising computer program code instructions, being executable by a computer, for performing an image processing method according to the second aspect of the invention when the computer program code instructions run on a computer.
  • Embodiments of the invention can be implemented in hardware, in software or in a combination of hardware and software.
  • Fig. 1 is a schematic diagram illustrating the architecture of an image processing device according to an embodiment of the invention
  • Fig. 2 is a schematic diagram illustrating different processing steps implemented by an image processing device and an image processing method according to an embodiment of the invention
  • Fig. 3 is a schematic diagram illustrating a current image, the distortion-compensated current image and a plurality of support pixels defining a support grid as used by an image processing device according to an embodiment of the invention
  • Fig. 4 is a schematic diagram illustrating an image processing device according to a further embodiment of the invention.
  • Fig. 5 is a schematic diagram illustrating further details of several processing steps implemented by an image processing device and an image processing method according to an embodiment of the invention
  • Fig. 6 is a schematic diagram illustrating a mapping for compensating image distortions caused by an image rotation as implemented by an image processing device and an image processing method according to an embodiment of the invention
  • Fig. 7 is a schematic diagram illustrating a mapping for compensating image distortions caused by the rolling shutter effect as implemented by an image processing device and an image processing method according to an embodiment of the invention.
  • Fig. 8 is a flow diagram illustrating an image processing method according to an embodiment of the invention.
  • a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa.
  • a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures.
  • a specific apparatus is described based on one or a plurality of units, e.g.
  • a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other, unless specifically noted otherwise.
  • Figure 1 is a schematic diagram showing an image processing device 100 configured to process a current image into a distortion-compensated image, i.e. an image having less distortions than the current image.
  • the image processing device 100 is a smartphone, a laptop, a tablet or a component of these or other similar devices and can comprise an image capturing device, in particular a camera for capturing the current image.
  • the image processing device 100 may comprise a communication interface configured to receive the current image, for instance, from an application server.
  • the current image can be a single image or one of a plurality of images, for instance, an image of a video frame.
  • the image processing device 100 comprises a main memory 101 , in particular a DRAM 101 for storing the current image.
  • the current image comprises a plurality of pixels including a first set of support pixels defining a first support grid and a second set of support pixels defining a second support grid, as will be described in more detail below, in particular, in the context of figures 3 and 5.
  • the image processing device 100 comprises processing circuitry 103 (herein also referred to as“WARP” unit 103) configured to generate a distortion-compensated image, i.e. a processed version of the current image having less distortions than the original current image, on the basis of the current image using an image distortion model.
  • processing circuitry 103 herein also referred to as“WARP” unit 103 configured to generate a distortion-compensated image, i.e. a processed version of the current image having less distortions than the original current image, on the basis of the current image using an image distortion model.
  • the main memory, in particular DRAM 101 is connected with the processing circuitry, i.e. the WARP unit 103 by means of an SOC interconnect 102.
  • the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image.
  • the processing circuitry 103 is configured to apply the second mapping subsequently to, i.e. after the first mapping for generating the distortion-compensated image.
  • the processing stage is configured to apply in a first processing stage the first mapping for compensating image distortions of a first type and, subsequently, in a second processing state the second mapping for compensating image distortions of a second type.
  • FIG. 2 is a schematic diagram illustrating different processing steps, i.e. the processing chain implemented by the image processing device 100 according to an embodiment of the invention.
  • a 2D image 203 of a 3D scene 201 is provided to the image processing device 100.
  • the image processing device 100 comprises a camera for obtaining the 2D image 203.
  • the processing circuitry 103 of the image processing device 100 is configured to compensate image distortions of the current 2D image 203 caused by the lens of the camera of the image processing device 100 using the first mapping.
  • the processing circuitry 103 of the image processing device 100 is configured to compensate distortions caused by the rolling shutter effect using the second mapping.
  • FIG. 2 refers to the successive compensation of image distortions caused by the lens of the camera of the image processing device 100 and the rolling shutter effect, it will be appreciated that other types of image distortions can be compensated successively as well, such as image distortions caused by a rotation of the camera, perspective correction, translation (stripe processing or stabilization) and the like.
  • the image processing device 100 further comprises a cache memory 103a for storing a plurality of portions of the current image in a corresponding plurality of sectors of the cache memory 103a, wherein each sector comprises a subset of the plurality of pixels stored in the main memory 101.
  • the local cache memory 103 can be implemented as a component of the WARP unit 103.
  • the processing circuitry i.e. WARP unit 103 comprises several components, whose function will be described in more detail in the following.
  • An “Input image Pixel coordinate calculator” 103c can be configured to provide sub-integer pixel positions/coordinates in the distortion-compensated image on the basis of full-integer pixel positions/coordinates 103b in the current image making use of or providing pixel
  • a refill engine 103e and a pixels kernel fetch unit 103f can be configured to manage the accesses to the main memory, in particular DRAM 101 and/or the local cache memory 103a, in particular to check whether any required pixels are available in the local cache memory 103a or have to be requested from the main memory 101.
  • a pixel interpolator 103g can be configured to provide the pixel values of full-integer pixels in the current image on the basis of the data retrieved from the local cache memory 103a and/or the main memory 101 for providing the final output image 103h, which can be further processed by an ISP pipe 105.
  • the processing circuitry 103 is further configured to determine on the basis of the image distortion model one or more positions of one or more pixels of the first set of support pixels and/or one or more pixels of the second set of support pixels in the current image and to select the plurality of portions of the current image for storing in the corresponding plurality of sectors of the cache memory 103a on the basis of the one or more positions of the one or more pixels of the first set of support pixels and/or the one or more pixels of the second set of support pixels in the current image.
  • each of the plurality of horizontal rows of pixels defined by the grid of first or second support pixels in the distortion-compensated image defines a respective substantially horizontal, but possibly curved line of pixels in the current image.
  • the exemplary grid of support pixels shown in figure 3 could have, for instance, 27x21 grid nodes, wherein each grid node is associated with a dx and dy displacement, i.e. a mapping from its position in the current image to its position in the distortion-compensated image, as illustrated in figure 3.
  • this exemplary support grid may require at least (27x21x8x120) multiplications/s and (27x21x2x120) additions/s just for the application of the rotation/projection matrix.
  • the full support grid can be computed by the processing circuitry 103 of the image processing device 100 for each new frame or new stripe of frame (when the image is too big and processed in several stripes) whatever the transformation changes are.
  • the deformation caused by the lens grid (21x27) is not modified from frame to frame.
  • FIG 4 is a schematic diagram illustrating a further embodiment of the image processing device 100.
  • the image processing device 100 comprises in addition to the DRAM 101 and the processing circuitry 103 already shown in figure 1 a camera sensor (or short camera) 401 configured to provide the current image(s) to be distortion-compensated, a serial interface 405 connecting the camera sensor 401 with the processing circuitry 103, an ISP controller 407 configured to manage the ISP, an application processor 409 configured to execute an application triggering the capturing of the current image and a DRAM controller 41 1 configured to schedule the accesses of the processing circuitry 103 to the DRAM 101.
  • a camera sensor or short camera
  • serial interface 405 connecting the camera sensor 401 with the processing circuitry 103
  • an ISP controller 407 configured to manage the ISP
  • an application processor 409 configured to execute an application triggering the capturing of the current image
  • a DRAM controller 41 1 configured to schedule the accesses of the processing circuitry 103 to the DRAM 101.
  • FIG 5 is a schematic diagram illustrating further details of several processing steps implemented by the image processing device 100 according to an embodiment of the invention.
  • the image processing device 100 is configured to compensate image distortions by successively applying the first mapping and the second mapping. This allows to minimize the grid composition calculation by the processing circuitry 103 as well as how often it is necessary to reload the support grids at each image stripe processing from the DRAM 101.
  • each support grid defines a mapping, which transforms the current pixel coordinate of the distortion-compensated (output) image (coordinate from the pixel generator or from the precedent grid output) into the coordinate of the pixel in the current (input) image (which can be considered as a kind of back tracking process) or vice versa.
  • the different support grids corresponding to the different mappings e.g. the first mapping and the second mapping, can be applied in the same order as the transformations of the image in the image processing.
  • the first transformation i.e. mapping is for compensating the camera motion caused by the non-static hand of the user.
  • the rotation support grid and the translation are going to be applied on the image coordinates first.
  • the grid combination process consists in transforming an input pixel coordinate into an output pixel coordinate and each input pixel coordinate is either the output of the pixel generator for the first grid application or the output of the precedent grid.
  • the nodes of the grid contain the node displacements.
  • the application of a grid on an input pixel can comprise interpolating a pixel displacement from the surrounding nodes of the grid, as will be described in more detail in the following.
  • a current full-integer pixel in the current image lies within a global cell defined by a plurality of, in particular four current pixels of the first set of support pixels or the second set of support pixels in the current image and the processing circuitry 103, in particular the“Input image Pixel coordinate calculator” 103c shown in figure 1 is configured to generate the corresponding current sub-integer pixel in the distortion- compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory 103a by the following steps: (i) determining the respective positions of the four current pixels of the first set of support pixels or the second set of support pixels in the current image, which are generally sub integer positions;
  • the current sub-integer pixel in the distortion-compensated image is assigned the same pixel value as the corresponding full-integer pixel in the current image.
  • the processing circuitry 103 is configured to repeat this process for all full-integer pixels of the current image.
  • the processing circuitry 103 in particular the pixel interpolator 103g shown in figure 1 is further configured to determine the pixel value of a current full- integer pixel in the distortion-compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory 103a by:
  • Figure 6 is a schematic diagram illustrating a mapping, i.e. the first or second mapping for compensating image distortions caused by an image rotation as implemented by the image processing device 100 according to an embodiment of the invention.
  • the grid of support pixels which often due to camera motions will change from frame to frame, can be defined by the first or second set of support pixels comprising only 4 pixels, i.e. displacements nodes.
  • the exemplary support grid shown in figure 6 also enables to compensate projections by controlling the displacement at each node.
  • the rotation center can be defined by positioning the rotation grid using a grid offset that locates the output image, i.e. the distortion-compensated image in the transformation grid referentially.
  • the output image, i.e. the distortion-compensated image can be a crop of the current input image to make sure that all the pixels obtained by rotation are fetched from the current input image.
  • the position of the crop can be defined by a parameter l_offset (offset of the output image in the input image), as illustrated in figure 6.
  • the parameter l_offset can also be used to compensate image distortions caused by a translation of the camera (VSTAB).
  • Figure 7 is a schematic diagram illustrating a mapping, i.e. the first or second mapping for compensating image distortions caused by the rolling shutter effect as implemented by the image processing device 100 according to an embodiment of the invention.
  • the grid of support pixels can be defined by a first or second set of support pixels comprising 16 pixels, i.e. displacement nodes.
  • FIG. 8 is a flow diagram illustrating a corresponding image processing method 800 according to an embodiment of the invention.
  • the image processing method 800 comprises the steps of: providing 801 a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and generating 803 a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its position in the current image and wherein the second mapping is applied subsequently to, i.e. after the first mapping for generating the distortion-compensated image.
  • embodiments of the invention provide in particular the following advantages: complex SW computations and data reloading are avoided as much as possible; the needs of CPU computation are decreased; complex transformation is not computed by software to program a distortion-compensation function each time an elementary transformation parameter is changing; the distortion-compensation function implemented by the processing circuitry 103 of the image processing device 100 can be controlled by a small CPU without L2 cache and being far from the DRAM 101 ; the need to transfer parameters from the DRAM to the distortion-compensation function implemented by the processing circuitry 103 is decreased; the support grid, which can change from grid to grid, preferably comprises only a couple of nodes, i.e. support pixels.
  • the image processing device 100 allows compensating complex image distortions by applying elementary transformations, i.e. mappings sequentially. Each elementary transformation can be provided separately to the distortion-compensation function implemented by the processing circuitry 103 of the image processing device 100 with only the required parameters as signal overhead.
  • An offset can be defined to compensate the camera translation, which increases the stability of the implementation.
  • Rotation or projection can be defined by just 4 pixels.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to an image processing device (100) comprising: a main memory, in particular a DRAM, (101) for storing a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and processing circuitry (103) configured to generate a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image. The processing circuitry (103) is configured to apply the second mapping subsequently to the first mapping for generating the distortion-compensated image. Moreover, the invention relates to a corresponding image processing method.

Description

DESCRIPTION
Image processing device and method
TECHNICAL FIELD
In general, the present invention relates to the field of image processing. More specifically, the present invention relates to an image processing device and method for compensating image distortions.
BACKGROUND
Many modern electronic devices, such as smartphones, are capable of capturing and processing electronic images or videos, such as by means of a camera and an image signal processor (ISP), which are often implemented as a System on a Chip (SoC). The ISP can process an electronic image obtained by the camera in order to compensate, i.e. reduce or remove image distortions caused, for instance, by the camera itself, such as the rolling shutter effect, or by a movement of the camera, while capturing the image. Because of high camera resolution these electronic images are usually very large so that the image processor can only process respective portions of the image stored in a local cache memory one after the other, while the whole image is stored in a high latency main memory, in particular a DRAM of the electronic device.
For an efficient processing of an image it is desirable to minimize the accesses of the ISP to the main memory for obtaining pixels from the main memory not stored in the cache memory.
Thus, there is a need for an improved image processing device and method allowing to compensate image distortions more efficiently.
SUMMARY
It is an object of the invention to provide an improved image processing device and method allowing to compensate image distortions more efficiently.
The foregoing and other objects are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.
According to a first aspect, the invention relates to an image processing device, such as a smartphone or a component of a smartphone. The image processing device comprises a main memory, in particular a DRAM, for storing a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels. Moreover, the image processing device comprises a processing circuitry configured to generate a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image. The processing circuitry is configured to apply the second mapping subsequently to, i.e. after the first mapping for generating the distortion- compensated image.
In a further possible implementation form of the first aspect, the image processing device further comprises a fast-access i.e. low latency, local cache memory for storing, i.e. caching a plurality of non-overlapping, rectangular portions of the current image in a corresponding plurality of sectors of the cache memory, wherein each sector comprises a subset of the plurality of pixels stored in the main memory.
In a further possible implementation form of the first aspect, the processing circuitry is further configured to determine on the basis of the image distortion model one or more positions of one or more pixels of the first set of support pixels and/or one or more pixels of the second set of support pixels in the current image and to select the plurality of portions of the current image for storing in the corresponding plurality of sectors of the cache memory on the basis of the one or more positions of the one or more pixels of the first set of support pixels and/or the one or more pixels of the second set of support pixels in the current image.
In a further possible implementation form of the first aspect, the first mapping of the image distortion model is configured to compensate one or more distortions caused by an image rotation and/or the second mapping of the image distortion model is configured to
compensate one or more distortions caused by an image deformation.
In a further possible implementation form of the first aspect, a current full-integer pixel in the current image lies within a global cell defined by a plurality of, in particular four current pixels of the first set of support pixels or the second set of support pixels in the current image, wherein the processing circuitry is configured to generate the corresponding current sub integer pixel in the distortion-compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory by: (i) determining the respective sub-integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image;
(ii) determining the sub-integer position of the corresponding current pixel in the distortion- compensated image on the basis of the image distortion model and the respective sub integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels.
In a further possible implementation form of the first aspect, the processing circuitry is configured to determine the sub-integer position of the corresponding current pixel in the distortion-compensated image on the basis of the image distortion model and the respective sub-integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image using interpolation, in particular bi-linear interpolation.
In a further possible implementation form of the first aspect, the processing circuitry is further configured to determine a pixel value of a current full-integer pixel in the distortion- compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory by:
(i) determining the sub-integer positions of a plurality of neighboring interpolation pixels in the distortion-compensated image on the basis of the image distortion model, wherein the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image define a respective local cell and wherein the current full-integer pixel in the distortion- compensated image is located in the local cell; and
(ii) determining the pixel value of the current full-integer pixel in the distortion-compensated image on the basis of the pixel values and the sub-integer positions of the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image.
In a further possible implementation form of the first aspect, the processing circuitry is configured to determine the pixel value of the current full-integer pixel in the distortion- compensated image on the basis of the pixel values and the sub-integer positions of the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image using interpolation, in particular bi-linear interpolation. In a further possible implementation form of the first aspect, the image processing device further comprises an image capturing device, in particular a camera for obtaining the current image.
In a further possible implementation form of the first aspect, the first set of support pixels and the second set of support pixels are equal.
In a further possible implementation form of the first aspect, the first set of support pixels is a subset of the second set of support pixels or the second set of support pixels is a subset of the first set of support pixels. In other words the respective support grids defined by the first set of support pixels and the second set of support pixels can have different resolutions.
According to a second aspect the invention relates to a corresponding image processing method comprising the steps of: providing a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and generating a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image and wherein the second mapping is applied subsequently to, i.e. after the first mapping for generating the distortion-compensated image.
The image processing method according to the second aspect of the invention can be performed by the image processing device according to the first aspect of the invention. Further features of the image processing method according to the second aspect of the invention result directly from the functionality of the image processing device according to the first aspect of the invention and its different implementation forms described above and below.
According to a third aspect the invention relates to a computer program with a program code for performing an image processing method according to the second aspect of the invention when the computer program runs on a computer.
According to a fourth aspect the invention relates to a computer readable storage medium comprising computer program code instructions, being executable by a computer, for performing an image processing method according to the second aspect of the invention when the computer program code instructions run on a computer.
Embodiments of the invention can be implemented in hardware, in software or in a combination of hardware and software.
Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.
BRIEF DESCRIPTION OF THE DRAWINGS
In the following embodiments of the invention are described in more detail with reference to the attached figures and drawings, in which:
Fig. 1 is a schematic diagram illustrating the architecture of an image processing device according to an embodiment of the invention;
Fig. 2 is a schematic diagram illustrating different processing steps implemented by an image processing device and an image processing method according to an embodiment of the invention;
Fig. 3 is a schematic diagram illustrating a current image, the distortion-compensated current image and a plurality of support pixels defining a support grid as used by an image processing device according to an embodiment of the invention;
Fig. 4 is a schematic diagram illustrating an image processing device according to a further embodiment of the invention;
Fig. 5 is a schematic diagram illustrating further details of several processing steps implemented by an image processing device and an image processing method according to an embodiment of the invention;
Fig. 6 is a schematic diagram illustrating a mapping for compensating image distortions caused by an image rotation as implemented by an image processing device and an image processing method according to an embodiment of the invention; Fig. 7 is a schematic diagram illustrating a mapping for compensating image distortions caused by the rolling shutter effect as implemented by an image processing device and an image processing method according to an embodiment of the invention; and
Fig. 8 is a flow diagram illustrating an image processing method according to an embodiment of the invention.
In the following identical reference signs refer to identical or at least functionally equivalent features.
DETAILED DESCRIPTION OF THE EMBODIMENTS
In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the invention or specific aspects in which embodiments of the present invention may be used. It is understood that embodiments of the invention may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.
For instance, it is understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other, unless specifically noted otherwise.
Figure 1 is a schematic diagram showing an image processing device 100 configured to process a current image into a distortion-compensated image, i.e. an image having less distortions than the current image. According to an embodiment, the image processing device 100 is a smartphone, a laptop, a tablet or a component of these or other similar devices and can comprise an image capturing device, in particular a camera for capturing the current image. According to a further embodiment, the image processing device 100 may comprise a communication interface configured to receive the current image, for instance, from an application server. The current image can be a single image or one of a plurality of images, for instance, an image of a video frame.
As illustrated in figure 1 , the image processing device 100 comprises a main memory 101 , in particular a DRAM 101 for storing the current image. The current image comprises a plurality of pixels including a first set of support pixels defining a first support grid and a second set of support pixels defining a second support grid, as will be described in more detail below, in particular, in the context of figures 3 and 5.
Furthermore, the image processing device 100 comprises processing circuitry 103 (herein also referred to as“WARP” unit 103) configured to generate a distortion-compensated image, i.e. a processed version of the current image having less distortions than the original current image, on the basis of the current image using an image distortion model. In the embodiment shown in figure 1 , the main memory, in particular DRAM 101 is connected with the processing circuitry, i.e. the WARP unit 103 by means of an SOC interconnect 102.
As will be described in more detail below, the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its corresponding position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its corresponding position in the current image. The processing circuitry 103 is configured to apply the second mapping subsequently to, i.e. after the first mapping for generating the distortion-compensated image. In other words, the processing stage is configured to apply in a first processing stage the first mapping for compensating image distortions of a first type and, subsequently, in a second processing state the second mapping for compensating image distortions of a second type.
Figure 2 is a schematic diagram illustrating different processing steps, i.e. the processing chain implemented by the image processing device 100 according to an embodiment of the invention. A 2D image 203 of a 3D scene 201 is provided to the image processing device 100. As already mentioned above, according to an embodiment the image processing device 100 comprises a camera for obtaining the 2D image 203. In a first distortion compensation stage 205 the processing circuitry 103 of the image processing device 100 is configured to compensate image distortions of the current 2D image 203 caused by the lens of the camera of the image processing device 100 using the first mapping. In a second distortion compensation stage 207 the processing circuitry 103 of the image processing device 100 is configured to compensate distortions caused by the rolling shutter effect using the second mapping. Further image processing steps can be performed in a stage 209 for providing the distortion-compensated image 211. Although the exemplary processing chain illustrated in figure 2 refers to the successive compensation of image distortions caused by the lens of the camera of the image processing device 100 and the rolling shutter effect, it will be appreciated that other types of image distortions can be compensated successively as well, such as image distortions caused by a rotation of the camera, perspective correction, translation (stripe processing or stabilization) and the like.
According to an embodiment, the image processing device 100 further comprises a cache memory 103a for storing a plurality of portions of the current image in a corresponding plurality of sectors of the cache memory 103a, wherein each sector comprises a subset of the plurality of pixels stored in the main memory 101. As can be taken from figure 1 , according to an embodiment the local cache memory 103 can be implemented as a component of the WARP unit 103.
In the embodiment shown in figure 1 , the processing circuitry, i.e. WARP unit 103 comprises several components, whose function will be described in more detail in the following. An “Input image Pixel coordinate calculator” 103c can be configured to provide sub-integer pixel positions/coordinates in the distortion-compensated image on the basis of full-integer pixel positions/coordinates 103b in the current image making use of or providing pixel
displacement tables 103d, which reflect the image distortion model. A refill engine 103e and a pixels kernel fetch unit 103f can be configured to manage the accesses to the main memory, in particular DRAM 101 and/or the local cache memory 103a, in particular to check whether any required pixels are available in the local cache memory 103a or have to be requested from the main memory 101. A pixel interpolator 103g can be configured to provide the pixel values of full-integer pixels in the current image on the basis of the data retrieved from the local cache memory 103a and/or the main memory 101 for providing the final output image 103h, which can be further processed by an ISP pipe 105.
According to an embodiment, the processing circuitry 103 is further configured to determine on the basis of the image distortion model one or more positions of one or more pixels of the first set of support pixels and/or one or more pixels of the second set of support pixels in the current image and to select the plurality of portions of the current image for storing in the corresponding plurality of sectors of the cache memory 103a on the basis of the one or more positions of the one or more pixels of the first set of support pixels and/or the one or more pixels of the second set of support pixels in the current image.
As can be taken from figure 3, which schematically illustrates the current image, the distortion-compensated current image, the support grid defined by the first or second set of support pixels in the current image and the support grid defined by the first or second set of support pixels in the distortion-compensated image, each of the plurality of horizontal rows of pixels defined by the grid of first or second support pixels in the distortion-compensated image defines a respective substantially horizontal, but possibly curved line of pixels in the current image. The exemplary grid of support pixels shown in figure 3 could have, for instance, 27x21 grid nodes, wherein each grid node is associated with a dx and dy displacement, i.e. a mapping from its position in the current image to its position in the distortion-compensated image, as illustrated in figure 3. For a simple rotation this exemplary support grid may require at least (27x21x8x120) multiplications/s and (27x21x2x120) additions/s just for the application of the rotation/projection matrix. According to an embodiment, the full support grid can be computed by the processing circuitry 103 of the image processing device 100 for each new frame or new stripe of frame (when the image is too big and processed in several stripes) whatever the transformation changes are. However, the deformation caused by the lens grid (21x27) is not modified from frame to frame. On the other hand, it may be necessary to update the rotation (rotational stabilization) for each frame as well as for the translation information (video stabilization) and the rolling shutter information.
Figure 4 is a schematic diagram illustrating a further embodiment of the image processing device 100. In the embodiment shown in figure 4 the image processing device 100 comprises in addition to the DRAM 101 and the processing circuitry 103 already shown in figure 1 a camera sensor (or short camera) 401 configured to provide the current image(s) to be distortion-compensated, a serial interface 405 connecting the camera sensor 401 with the processing circuitry 103, an ISP controller 407 configured to manage the ISP, an application processor 409 configured to execute an application triggering the capturing of the current image and a DRAM controller 41 1 configured to schedule the accesses of the processing circuitry 103 to the DRAM 101. As illustrated in figure 4, the serial interface 405, the processing circuitry 103, the ISP controller 407, the application processor 409 and the DRAM controller 411 could be implemented as a system on a chip 403. Figure 5 is a schematic diagram illustrating further details of several processing steps implemented by the image processing device 100 according to an embodiment of the invention. As already described above, the image processing device 100 is configured to compensate image distortions by successively applying the first mapping and the second mapping. This allows to minimize the grid composition calculation by the processing circuitry 103 as well as how often it is necessary to reload the support grids at each image stripe processing from the DRAM 101.
As already described above, each support grid defines a mapping, which transforms the current pixel coordinate of the distortion-compensated (output) image (coordinate from the pixel generator or from the precedent grid output) into the coordinate of the pixel in the current (input) image (which can be considered as a kind of back tracking process) or vice versa. The different support grids corresponding to the different mappings, e.g. the first mapping and the second mapping, can be applied in the same order as the transformations of the image in the image processing.
In the exemplary scenario shown in figure 5, the first transformation, i.e. mapping is for compensating the camera motion caused by the non-static hand of the user. In other words, in the exemplary scenario shown in figure 5 the rotation support grid and the translation are going to be applied on the image coordinates first. As illustrated in figure 5, the grid combination process consists in transforming an input pixel coordinate into an output pixel coordinate and each input pixel coordinate is either the output of the pixel generator for the first grid application or the output of the precedent grid. In an embodiment, the nodes of the grid contain the node displacements. The application of a grid on an input pixel can comprise interpolating a pixel displacement from the surrounding nodes of the grid, as will be described in more detail in the following.
Thus, according to an embodiment, a current full-integer pixel in the current image lies within a global cell defined by a plurality of, in particular four current pixels of the first set of support pixels or the second set of support pixels in the current image and the processing circuitry 103, in particular the“Input image Pixel coordinate calculator” 103c shown in figure 1 is configured to generate the corresponding current sub-integer pixel in the distortion- compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory 103a by the following steps: (i) determining the respective positions of the four current pixels of the first set of support pixels or the second set of support pixels in the current image, which are generally sub integer positions;
(ii) determining the sub-integer position of the corresponding current pixel in the distortion- compensated image on the basis of the image distortion mode and the respective sub integer positions of the four current pixels of the first set of support pixels or the second set of support pixels in the current image, for instance, by using interpolation, in particular bi-linear interpolation.
The current sub-integer pixel in the distortion-compensated image is assigned the same pixel value as the corresponding full-integer pixel in the current image. In an embodiment, the processing circuitry 103 is configured to repeat this process for all full-integer pixels of the current image.
According to an embodiment, the processing circuitry 103, in particular the pixel interpolator 103g shown in figure 1 is further configured to determine the pixel value of a current full- integer pixel in the distortion-compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory 103a by:
(i) determining, as described above, the sub-integer positions and pixel values of a plurality of, in particular four neighboring sub-integer interpolation pixels in the distortion- compensated image on the basis of the image distortion model, wherein the plurality of neighboring interpolation pixels in the distortion-compensated image define a respective local cell and wherein the current full-integer pixel in the distortion-compensated image is located in the local cell; and
(ii) determining the pixel value of the current full-integer pixel in the distortion-compensated image on the basis of the pixel values and the sub-integer positions of the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image, for instance, by using interpolation, in particular bi-linear interpolation.
Figure 6 is a schematic diagram illustrating a mapping, i.e. the first or second mapping for compensating image distortions caused by an image rotation as implemented by the image processing device 100 according to an embodiment of the invention. As can be taken from figure 6 and as already described above, according to an embodiment the grid of support pixels, which often due to camera motions will change from frame to frame, can be defined by the first or second set of support pixels comprising only 4 pixels, i.e. displacements nodes. According to an embodiment, the exemplary support grid shown in figure 6 also enables to compensate projections by controlling the displacement at each node.
According to an embodiment, the rotation center can be defined by positioning the rotation grid using a grid offset that locates the output image, i.e. the distortion-compensated image in the transformation grid referentially. According to an embodiment, the output image, i.e. the distortion-compensated image can be a crop of the current input image to make sure that all the pixels obtained by rotation are fetched from the current input image. According to an embodiment, the position of the crop can be defined by a parameter l_offset (offset of the output image in the input image), as illustrated in figure 6. According to an embodiment, the parameter l_offset can also be used to compensate image distortions caused by a translation of the camera (VSTAB).
Figure 7 is a schematic diagram illustrating a mapping, i.e. the first or second mapping for compensating image distortions caused by the rolling shutter effect as implemented by the image processing device 100 according to an embodiment of the invention. For this exemplary mapping the grid of support pixels can be defined by a first or second set of support pixels comprising 16 pixels, i.e. displacement nodes.
Figure 8 is a flow diagram illustrating a corresponding image processing method 800 according to an embodiment of the invention. The image processing method 800 comprises the steps of: providing 801 a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and generating 803 a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its position in the current image and wherein the second mapping is applied subsequently to, i.e. after the first mapping for generating the distortion-compensated image.
Thus, embodiments of the invention provide in particular the following advantages: complex SW computations and data reloading are avoided as much as possible; the needs of CPU computation are decreased; complex transformation is not computed by software to program a distortion-compensation function each time an elementary transformation parameter is changing; the distortion-compensation function implemented by the processing circuitry 103 of the image processing device 100 can be controlled by a small CPU without L2 cache and being far from the DRAM 101 ; the need to transfer parameters from the DRAM to the distortion-compensation function implemented by the processing circuitry 103 is decreased; the support grid, which can change from grid to grid, preferably comprises only a couple of nodes, i.e. support pixels.
The image processing device 100 allows compensating complex image distortions by applying elementary transformations, i.e. mappings sequentially. Each elementary transformation can be provided separately to the distortion-compensation function implemented by the processing circuitry 103 of the image processing device 100 with only the required parameters as signal overhead. An offset can be defined to compensate the camera translation, which increases the stability of the implementation. Rotation or projection can be defined by just 4 pixels.
While a particular feature or aspect of the disclosure may have been disclosed with respect to only one of several implementations, such feature or aspect may be combined with one or more other features or aspects of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include", "have", "with", or other variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term "comprise". Also, the terms "exemplary", "for example" and "e.g." are merely meant as an example, rather than the best or optimal. The terms“coupled” and“connected”, along with derivatives may have been used. It should be understood that these terms may have been used to indicate that two elements cooperate or interact with each other regardless whether they are in direct physical or electrical contact, or they are not in direct contact with each other.
Although specific aspects have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and/or equivalent
implementations may be substituted for the specific aspects shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific aspects discussed herein.
Although the elements in the following claims are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.
Many alternatives, modifications, and variations will be apparent to those skilled in the art in light of the above teachings. Of course, those skilled in the art readily recognize that there are numerous applications of the invention beyond those described herein. While the present invention has been described with reference to one or more particular embodiments, those skilled in the art recognize that many changes may be made thereto without departing from the scope of the present invention. It is therefore to be understood that within the scope of the appended claims and their equivalents, the invention may be practiced otherwise than as specifically described herein.

Claims

1. An image processing device (100) comprising: a main memory, in particular a DRAM, (101 ) for storing a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and processing circuitry (103) configured to generate a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its position in the current image and wherein the processing circuitry (103) is configured to apply the second mapping subsequently to the first mapping for generating the distortion-compensated image.
2. The image processing device (100) of claim 1 , wherein the image processing device (100) further comprises a cache memory (103a) for storing a plurality of portions of the current image in a corresponding plurality of sectors of the cache memory (103a), wherein each sector comprises a subset of the plurality of pixels stored in the main memory (101 ).
3. The image processing device (100) of claim 2, wherein the processing circuitry (103) is further configured to determine on the basis of the image distortion model one or more positions of one or more pixels of the first set of support pixels and/or one or more pixels of the second set of support pixels in the current image and to select the plurality of portions of the current image for storing in the corresponding plurality of sectors of the cache memory (103a) on the basis of the one or more positions of the one or more pixels of the first set of support pixels and/or the one or more pixels of the second set of support pixels in the current image.
4. The image processing device (100) of claims 2 or 3, wherein a current full-integer pixel in the current image lies within a global cell defined by a plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image and wherein the processing circuitry (103) is configured to generate the corresponding current sub-integer pixel in the distortion-compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory (103a) by:
(i) determining the respective sub-integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image;
(ii) determining the sub-integer position of the corresponding current pixel in the distortion- compensated image on the basis of the image distortion model and the respective sub integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image.
5. The image processing device (100) of claim 4, wherein the processing circuitry (103) is configured to determine the sub-integer position of the corresponding current pixel in the distortion-compensated image on the basis of the image distortion model and the respective sub-integer positions of the plurality of current pixels of the first set of support pixels or the second set of support pixels in the current image using interpolation, in particular bi-linear interpolation.
6. The image processing device (100) of claim 4 or 5, wherein the processing circuitry (103) is further configured to determine a pixel value of a current full-integer pixel in the distortion-compensated image on the basis of the plurality of portions of the current image stored in the corresponding plurality of sectors of the cache memory (103a) by:
(i) determining the sub-integer positions of a plurality of neighboring interpolation pixels in the distortion-compensated image on the basis of the image distortion model, wherein the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image define a respective local cell and wherein the current full-integer pixel in the distortion- compensated image is located in the local cell; and
(ii) determining the pixel value of the current full-integer pixel in the distortion-compensated image on the basis of the pixel values and the sub-integer positions of the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image.
7. The image processing device (100) of claim 6, wherein the processing circuitry (103) is configured to determine the pixel value of the current full-integer pixel in the distortion- compensated image on the basis of the pixel values and the sub-integer positions of the plurality of neighboring sub-integer interpolation pixels in the distortion-compensated image using interpolation, in particular bi-linear interpolation.
8. The image processing device (100) of any one of the preceding claims, wherein the first mapping of the image distortion model is configured to compensate one or more distortions caused by an image rotation and/or the second mapping of the image distortion model is configured to compensate one or more distortions caused by an image deformation.
9. The image processing device (100) of any one of the preceding claims, wherein the image processing device (100) further comprises an image capturing device (401 ), in particular a camera for obtaining the current image.
10. The image processing device (100) of any one of the preceding claims, wherein the first set of support pixels and the second set of support pixels are equal.
1 1. The image processing device (100) of any one of claims 1 to 9, wherein the first set of support pixels is a subset of the second set of support pixels or the second set of support pixels is a subset of the first set of support pixels.
12. An image processing method (800) comprising: providing (801 ) a current image, wherein the current image comprises a plurality of pixels, including a first set of support pixels and a second set of support pixels; and generating (803) a distortion-compensated image on the basis of the current image using an image distortion model, wherein the image distortion model defines for each pixel of the first set of support pixels a first mapping between its position in the current image and its position in the distortion-compensated image and for each pixel of the second set of support pixels a second mapping between its position in the current image and its position in the current image and wherein the second mapping is applied subsequently to the first mapping for generating the distortion-compensated image.
13. A computer program with a program code for performing an image processing method (800) according to claim 12 when the computer program runs on a computer.
14. A computer readable storage medium comprising computer program code
instructions, being executable by a computer, for performing an image processing method (800) according to claim 12 when the computer program code instructions run on a computer.
EP19728376.5A 2019-05-29 2019-05-29 Image processing device and method Pending EP3970111A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2019/063900 WO2020239212A1 (en) 2019-05-29 2019-05-29 Image processing device and method

Publications (1)

Publication Number Publication Date
EP3970111A1 true EP3970111A1 (en) 2022-03-23

Family

ID=66752082

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19728376.5A Pending EP3970111A1 (en) 2019-05-29 2019-05-29 Image processing device and method

Country Status (2)

Country Link
EP (1) EP3970111A1 (en)
WO (1) WO2020239212A1 (en)

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7126616B2 (en) * 2001-06-12 2006-10-24 Silicon Optix Inc. Method and system for processing a non-linear two dimensional spatial transformation
US9280810B2 (en) * 2012-07-03 2016-03-08 Fotonation Limited Method and system for correcting a distorted input image
US10275863B2 (en) * 2015-04-03 2019-04-30 Cognex Corporation Homography rectification

Also Published As

Publication number Publication date
WO2020239212A1 (en) 2020-12-03

Similar Documents

Publication Publication Date Title
US9262807B2 (en) Method and system for correcting a distorted input image
US9280810B2 (en) Method and system for correcting a distorted input image
WO2021115136A1 (en) Anti-shake method and apparatus for video image, electronic device, and storage medium
CN111710049B (en) Method and device for determining ambient illumination in AR scene
JP2012105246A (en) Image correction method and related image correction system
CN111402139B (en) Image processing method, apparatus, electronic device, and computer-readable storage medium
WO2013151873A1 (en) Joint video stabilization and rolling shutter correction on a generic platform
CN114586337B (en) Video anti-shake optimization processing method and device and electronic equipment
Gunturk et al. High-resolution image reconstruction from multiple differently exposed images
US11699218B2 (en) Method controlling image sensor parameters
US8994837B2 (en) Image processing devices and image processing methods of moving objects
CN111462021B (en) Image processing method, apparatus, electronic device, and computer-readable storage medium
EP3970111A1 (en) Image processing device and method
US10846837B2 (en) Image processing apparatus, image processing method, and storage medium
US9979908B2 (en) Image processing devices and image processing methods with interpolation for improving image resolution
JP2007257042A (en) Image magnifying device and image magnifying method
CN114463213A (en) Video processing method, video processing device, terminal and storage medium
JP2010232710A (en) Image processing apparatus, and image processing method
US20130315502A1 (en) Image processing apparatus and image processing method
JP2010033507A (en) Memory controller and image processing apparatus
US9917972B2 (en) Image processor, image-processing method and program
US9454801B2 (en) Image processing apparatus, method for processing image, and program
CN117173027A (en) Template diagram coordinate transformation circuit and target matching device
WO2020211935A1 (en) An electronic device and a method of operating such the electronic device
US8179474B2 (en) Fast iterative motion estimation method on gradually changing images

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20211214

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)