CN112669200A - Image deformation method, device and system - Google Patents

Abstract

The present disclosure provides an image deformation method, device and system, wherein the method comprises: according to the size of the output image and the original transformation relation from the output image coordinate to the input image coordinate, carrying out block processing on the output image to obtain a plurality of corresponding sub-blocks; fitting a simple transformation relation in each sub-block to approximately replace the original transformation relation from the sub-block coordinates to the input image coordinates; for each subblock, fitting the input image according to the simple transformation relation to obtain at least two subblock ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges; and reordering the sub-block ranges and the corresponding fitting parameters by using a nearest neighbor first calculation principle.

Description

Image deformation method, device and system

Technical Field

The present disclosure relates to image deformation processing, and in particular, to an image deformation method, apparatus, and system.

Background

Due to the improvement of the resolution of the camera and the development of deep learning, more and more information can be extracted from the image, and the information is more and more accurate. Therefore, image processing techniques are currently widely used in various industries.

Due to camera lens distortion and interference from factors such as camera angle, object angle, and rain, the captured original image is often distorted. Therefore, the image processing program usually restores the distorted original image by using an image deformation operation. In addition, image deformation is also widely applied to enhancement of data in deep learning training. However, most of the conventional image deformation operations are complex, including complex mathematical operations such as exponents, logarithms and the like, and are often floating point operations, which not only reduces the speed of the whole image processing program, but also wastes a large amount of computing resources. The two points are more obvious on edge type equipment which has certain requirements on real-time performance and has less storage resources and operation resources.

Disclosure of Invention

In view of this, an aspect of the present disclosure provides an image deformation method, including: according to the size of the output image and the original transformation relation from the output image coordinate to the input image coordinate, the output image is subjected to block processing to obtain a plurality of corresponding sub-blocks; fitting a simple transformation relation in each sub-block to approximately substitute the original transformation relation from the coordinates of the sub-block to the coordinates of the input image; for each subblock, fitting the input image according to the simple transformation relation to obtain at least two subblock ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges; and reordering the sub-block ranges and the corresponding fitting parameters by using the nearest neighbor first calculation principle.

Optionally, the method further includes: dividing an output image into a plurality of sub-blocks identical to the input image; obtaining the range of the subblocks of each output image mapped into the input image according to the simple transformation relation to obtain the corresponding subblock range in the input image; reordering the subblocks of the output image according to the order of the subblock range arrangement; searching a parameter table for each sub-block of the output image for a fitting parameter quantized to an integer; and performing image deformation operation according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the converted coordinates.

Optionally, quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges, including: and multiplying the original floating point number by the n power of 2 to amplify, rounding the amplified floating point number to obtain a fixed point number, and compressing and storing the fixed point number in the parameter table.

Optionally, the reordering of the sub-block ranges and the corresponding fitting parameters by using the nearest neighbor first calculation principle includes: the sorting strategy takes the smallest ordinate value in the range of the subblocks mapped by the output image into the input image as a sorting key, and sorts the subblock range of the input image from small to large so that the subblock with the smallest ordinate closest to the ordinate is always adjacent.

Alternatively, a simple transformation relationship contains only multiplications and additions; each simple transformation relation has a corresponding local area, and the simple transformation relation can approximately replace the original transformation relation in the corresponding local area so as to improve the operation efficiency.

Optionally, performing an image deformation operation according to the fitting parameter quantized into an integer to obtain a pixel position corresponding to the converted coordinate, including: carrying out graph deformation operation according to the fitting parameters quantized into integers, outputting a calculation result, dividing the calculation result by the nth power of 2, and restoring the result into fixed point numbers with n decimal places to obtain converted coordinates; and obtaining pixel values adjacent to the converted coordinates for interpolation to obtain approximate pixel values of the converted coordinates, and storing the approximate pixel values to pixel positions corresponding to the coordinates of the output image.

Optionally, in the operation of fitting the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters for each sub-block, fitting is performed by using a greedy algorithm and a least square method to obtain the sub-block ranges and the corresponding fitting parameters.

Another aspect of the present disclosure provides an image morphing apparatus including:

the input image caching module is used for receiving the instruction of the fixed point calculating module, reading the input image data from the memory according to the instruction and sending the input image data to the fixed point calculating module, receiving the converted coordinates sent by the fixed point calculating module and sending the corresponding approximate pixel values of the converted coordinates in the input image to the output image caching module; the input image data comprises pixels of each point of the input image and corresponding coordinates thereof;

the output image caching module is used for receiving the instruction of the fixed point calculation module, reading the output image data from the memory according to the instruction and sending the output image data to the fixed point calculation module; receiving an approximate pixel value corresponding to the converted coordinates sent by the input image caching module, and writing the approximate pixel value into a memory; the output image data comprises the size of an output image, pixels of each point of the output image and corresponding coordinates of the pixels;

the index cache module is used for reading the first data and the second data in the parameter table from the memory and sending the first data and the second data to the fixed point calculation module; wherein the first data comprises an original transformation relation from an output image to an input image; the second data is obtained by calculation of the fixed point calculation module according to the output image data, the input image data and the first data, and comprises sub-block range sorting and fitting parameters quantized into integers;

the fixed point calculation module is used for receiving the output image data, the input image data and the first data and executing the following operations according to the output image data, the input image data and the first data: according to the size of the output image and the original transformation relation from the output image coordinate to the input image coordinate, the output image is subjected to block processing to obtain a plurality of corresponding sub-blocks; fitting a simple transformation relation in each sub-block to approximately substitute the original transformation relation from the coordinates of the sub-block to the coordinates of the input image; for each subblock, fitting the input image according to the simple transformation relation to obtain at least two subblock ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges; and reordering the sub-block ranges and the corresponding fitting parameters by using the nearest neighbor first calculation principle.

Optionally, the fixed point calculation module is further configured to receive the output image data, the input image data, and the second data, and perform the following operations according to the output image data, the input image data, and the second data: obtaining the range of the subblocks of each output image mapped into the input image according to the simple transformation relation to obtain the corresponding subblock range in the input image; reordering the subblocks of the output image according to the order of the subblock range arrangement; searching a parameter table for each sub-block of the output image for a fitting parameter quantized to an integer; and performing image deformation operation according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the converted coordinates.

Yet another aspect of the present disclosure provides an image warping system, including: the image transformation device is used for realizing each step of the method so as to obtain the pixel position corresponding to the converted coordinates.

The image deformation method, the image deformation device and the image deformation system have the beneficial effects that:

on one hand, after the fitting parameters are obtained in the training stage, the fitting parameters are quantized into integer storage (namely fixed point numbers are used for replacing floating point numbers for storage), so that fixed point operation can be performed in the reasoning stage, the resource utilization rate of a fixed point calculation module is improved, and the operation speed is accelerated.

On the other hand, the sub-block parameters in the lookup table are reordered according to the nearest neighbor first calculation principle and are calculated according to the new sequence of the sub-blocks in the inference stage, so that the fixed point calculation module can utilize the data in the cache to carry out calculation to the maximum extent during the calculation in the inference stage. Therefore, the fixed-point calculation module can be prevented from directly reading data from the memory, and the operation efficiency of the whole image deformation can be greatly improved.

Drawings

FIG. 1 illustrates an image before and after image deformation in the prior art; wherein, fig. 1a is an image before mapping, and fig. 1b is an image after mapping;

FIG. 2A schematically illustrates a flow diagram of a training phase of an image warping method in an embodiment of the present disclosure;

FIG. 2B schematically illustrates a flow diagram of an inference phase of an image warping method in an embodiment of the disclosure;

FIG. 3 is a schematic diagram illustrating a process of inverse mapping from an output image to an input image according to a simple transformation relationship in an embodiment of the present disclosure;

fig. 4 schematically shows a block diagram of an image warping device in an embodiment of the present disclosure;

fig. 5 schematically illustrates a block diagram of an image warping system of an embodiment of the present disclosure.

Description of the reference numerals

1. 2, 3-dividing the sub-blocks in the output image;

1 ', 2 ', 3 ' -are inversely mapped to corresponding sub-blocks in the input image by the sub-block 1, the sub-block 2 and the sub-block 3 in the output image;

P₁、P₂、P₃、P₄-outputting coordinate points in the image;

P₁’、P₂’、P₃' -from a coordinate point P in the output image₁、P₂、P₃Inversely mapping to corresponding coordinate points in the input image;

400-image morphing means; 410-an input image caching module; 420-index cache module; 430-output image buffer module; 440-fixed point calculation module; 500-an image warping system; 510-memory.

Detailed Description

For a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.

Fig. 1 shows an image before and after image deformation in the related art. As shown in fig. 1, in the image deformation process, each pixel in the input image is not defined in the output image, i.e. the two pixels are not coordinate transformation in one-to-one correspondence, so that if the input image is transformed to the output image, much noise is generated, i.e. image deformation occurs. At present, most of image deformation methods use the coordinates of an output image as input, and the corresponding coordinates of the output image in the input image are inversely deduced by the inverse transformation of the original transformation, and finally, the output image is filled. This is also done in the present disclosure to avoid noise.

As described in the background art, most of the image deformation operations used in the prior art for restoring a distorted image are complex, including complex mathematical operations such as exponents, logarithms, and the like, and are often floating point operations, which not only reduces the speed of the whole image processing program, but also wastes a large amount of computing resources. The two points are more obvious on edge type equipment which has certain requirements on real-time performance and has less storage resources and operation resources.

Therefore, if the image deformation module can be designed in an accelerated manner, on one hand, the image deformation operation speed can be increased, the requirements of more applications with requirements on real-time performance are met, on the other hand, the computing resources can be saved, and more and larger images can be processed when the computing resources are limited.

Based on this, the present disclosure proposes an image morphing method, an image morphing apparatus, and a system for solving at least one of the above-mentioned problems.

It can be understood that, since the interaction between the logic operation module and the memory is 100x times slower than the interaction with the cache, the present disclosure adds the corresponding cache module when designing the image morphing device, and after the logic operation module reads a data point from the memory, the data of 32 adjacent bytes of the data point is read into the cache module together for storage, which does not increase extra reading time. When the logic operation module needs to use the data, the logic operation module can directly read the data from the cache module without reading the data from the memory.

Fig. 2A and 2B schematically show a flowchart of a training phase and an inference phase, respectively, of an image warping method in an embodiment of the present disclosure.

As shown in fig. 2A, the training phase of the image warping method in the present disclosure includes operations S101 to S105.

In operation S101, the output image is partitioned according to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates, so as to obtain a plurality of corresponding sub-blocks.

The output image is divided into a plurality of sub-blocks according to the known size of the output image and the transformation relationship f from the output image coordinates to the input image coordinates (the transformation relationship refers to the transformation relationship from the output image coordinates to the input image coordinates). Each sub-block may be, for example, a rectangle with a size of m × n, and m and n may be specifically defined according to actual situations.

In operation S102, a simple transformation relationship is fitted within each sub-block to approximate the original transformation relationship of the coordinates of the substitute sub-block to the coordinates of the input image.

Specifically, in operation S102, a simple transformation relation f 'is fitted to each sub-block using a least square method instead of the original transformation relation f, wherein the simple transformation relation f' may be a simple binary polynomial, for example. According to an embodiment of the present disclosure, the simple transformation relationship f' only includes multiplication and addition, so that the arithmetic logic unit (e.g., fixed point calculation unit) operates fast. Each simple transformation relation f 'has a corresponding local area, and the simple transformation relation f' can approximately replace the transformation relation in the corresponding local area, so that the operation efficiency is improved.

In operation S103, for each subblock, fitting the input image according to the simple transformation relationship to obtain at least two subblock ranges and corresponding fitting parameters.

Specifically, in this step, a greedy algorithm is used to search whether there are subblocks around a subblock that can be merged. Specifically, the subblock is merged with its neighboring subblocks, and the simple transformation relationship f ' is re-fitted (for example, the simple transformation relationship f ' may be represented by a binary polynomial), after the fitting parameters are obtained, the difference between the fitting parameter transformation and the original simple transformation relationship f ' is calculated, and if the difference is smaller than the preset parameter α (where α may be self-defined according to the actual situation), the two subblocks are merged. And obtaining the final sub-block range and the corresponding fitting parameters until all the sub-blocks which can be combined are searched.

In operation S104, the fitting parameters are quantized into integers, and the quantized fitting parameters and subblock ranges are compressed and stored.

In the prior art, the image deformation operation usually adopts floating point operation, which not only reduces the speed of the whole image processing program, but also wastes a large amount of computing resources. Specifically, according to the embodiment of the present disclosure, quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and subblock ranges, includes: multiplying the original floating point number by the n power of 2 to amplify the parameter to enlarge the parameter by n times, rounding the amplified floating point number to obtain a fixed point number, and finally compressing and storing the fixed point number in a parameter table. And the fixed point number is used for replacing the floating point number for compression and storage, so that the subsequent reasoning stage can use the fixed point number for calculation, and the calculation result is inversely quantized by using right shift operation. Since the added right-shift operation is almost time consuming, it amounts to an unproductive replacement of a large number of floating-point calculations with fixed-point operations in the inference phase.

In operation S105, the sub-block ranges and the corresponding fitting parameters are reordered according to the nearest neighbor first calculation rule.

According to the embodiment of the disclosure, the reordering of the sub-block ranges and the corresponding fitting parameters by the principle of nearest neighbor first calculation includes: the sorting strategy takes the smallest ordinate value in the range of the subblocks mapped by the output image into the input image as a sorting key, and sorts the subblock range of the input image from small to large so that the subblock with the smallest ordinate closest to the ordinate is always adjacent.

Specifically, in operation S105, the subblock ranges and the corresponding fitting parameters stored in the parameter table are reordered according to the nearest neighbor first calculation principle, and the purpose of doing so is mainly to maximize the interaction between the logic operation unit (e.g., the fixed point calculation unit shown in fig. 5) and the cache module.

It is understood that when the logic operation unit (e.g. the fixed point calculation unit shown in fig. 5) fetches data from the memory, it will not only read the currently required data, but also continuously read 32 bytes or 64 bytes of data and temporarily store the data in the cache module. When a logic operation unit (e.g., a fixed point computing unit) needs to fetch data, it checks whether there is needed data in the cache module first, and if not, it sends a data request to the memory. Due to the timeliness of the data in the cache module, in order to improve the hit rate of the data read by the logic operation unit (e.g., the fixed point calculation unit shown in fig. 5), the sequence of the sub-block range is reordered in operation S105 so that the currently calculated sub-block is always closest to the last calculated sub-block, which can greatly reduce the data transmission time.

Fig. 3 is a schematic diagram illustrating a process of inverse mapping from an output image to an input image according to a simple transformation relation in an embodiment of the present disclosure. The process of operation S105 in the embodiment of the present disclosure will be described in detail below with reference to what is shown in fig. 3. It should be noted that the number of divisions of the sub-block of the output image and the input image in the embodiment of the present disclosure is not limited to 3, and the number of divisions of the sub-block may be defined according to actual situations. The number of sub-blocks shown in fig. 3 is only for the convenience of understanding the present disclosure and is not intended to limit what is protected by the present disclosure.

As shown in fig. 3, in the image warping operation, the adjacent regular rectangles in the output image become irregular patterns when inversely mapped into the input image, and the relative positions change (for example, sub-block 1 in the output image shown in fig. 3 is inversely mapped to sub-block 1' in the input image). On the premise that the image deformation operation process is to perform inverse mapping from the output image to the input image, it can be known that if the sub-blocks are calculated according to the adjacent order of the output image, the reading of the input image in the operation process is easily discontinuous. If we finish calculating sub-block 1 and then calculate sub-block 2, the logic unit (e.g. the fixed point calculation unit shown in fig. 5) has to read the data of the row of sub-block 2 from the memory and empty the row of data of sub-block 1 stored in the cache memory because sub-block 1 and sub-block 2 are not consecutive in the row. As can be seen from fig. 3, there are several rows of sub-block 1 and sub-block 3 located on the same row, which means that the logical operation unit (e.g. the fixed point calculation unit shown in fig. 5) is likely to have read the data to be used by sub-block 3 on the 3 rows into the cache (since the read data of the logical operation unit (e.g. the fixed point calculation unit shown in fig. 5) is usually read by row). A reasonable solution is to first calculate sub-block 3 which overlaps more with sub-block 1 row.

Referring to fig. 3, each item in the parameter table is composed of two parts, i.e., a coordinate value and a simple function parameter, wherein the first part of the coordinate values includes an upper left corner coordinate value and a lower right corner coordinate value of each output image sub-block, e.g., the coordinate value (1, x) of the output image sub-block 1_min，y_min，，x_max，y_max) From the coordinate point P at the upper left corner in sub-block 1₁' coordinate value (where P₁The coordinate of' is (x)_min，y_min) And the coordinate point P of the lower right corner₄’(x_max，y_max) The coordinate values of (a). The second part is the parameters of the simple function obtained by fitting. In the sorting process, all the subblocks of the output image are mapped to the subblock range in the input image through a simple function, then the smallest ordinate of each mapped subblock in the input image is found, and the subblocks with the smallest ordinate closest to the ordinate in the input image are sorted together, so that the subblocks with the smallest ordinate closest to the ordinate are always adjacent. The reordered subblocks can be obtained through the operation.

As shown in fig. 2B, the image warping method inference phase in the present disclosure includes operations S106 to S110.

In operation S106, the output image is divided into the same plurality of sub-blocks as the input image.

Specifically, in the inference stage, the output image is divided into a plurality of sub-blocks identical to the input image, which may refer to fig. 3, as shown in fig. 3, for example, the output image may be divided into 3 sub-blocks including sub-block 1, sub-block 2, and sub-block 3. In the embodiment of the present disclosure, the number of divisions between the output image and the input image sub-block is not limited to 3, and may be defined according to actual situations. The number of sub-blocks shown in fig. 3 is only for the convenience of understanding the present disclosure and is not intended to limit what is protected by the present disclosure.

In operation S107, a range in which the subblock of each output image is mapped into the input image is obtained according to the simple transformation relationship, and a corresponding subblock range in the input image is obtained.

As shown in fig. 3, solving the inverse mapping of the sub-blocks (e.g. sub-block 1, sub-block 2, and sub-block 3) of each output image to the range in the input image according to the simple transformation relationship f 'results in the corresponding sub-block range (e.g. sub-block 1', sub-block 2 ', and sub-block 3') in the input image.

In operation S108, the subblocks of the output image are reordered in the order of the arrangement of the subblock ranges.

As described above, in the image morphing operation, adjacent regular rectangles in the output image become irregular patterns when mapped into the input image, and the relative positions change. On the premise that the image deformation operation is inverse mapping from the output image to the input image, it can be understood that if the sub-blocks are calculated according to the adjacent order of the output image, the reading of the input image is easily discontinuous. Therefore, in operation S108, the sub-blocks of the output image are reordered according to the order of the sub-block range arrangement in the parameter table obtained in the preceding training stage so that the sub-block with the closest smallest ordinate is always adjacent. This is advantageous for the logic operation unit (such as the fixed point computation module shown in fig. 5) to continuously read data in the subsequent operation process, thereby greatly reducing the data transmission time.

In operation S109, fitting parameters quantized to integers are looked up in the parameter table for each sub-block of the output image.

The sub-blocks of each output image look up fitting parameters (namely fixed point number parameters) quantized into integers in a parameter table, and then participate in image deformation operation by using the fixed point number.

In operation S110, an image deformation operation is performed according to the fitting parameters quantized to integers to obtain pixel positions corresponding to the transformed coordinates.

According to the embodiment of the present disclosure, in operation S110, the method specifically further includes:

in operation S1101, an image deformation operation is performed according to the fitting parameters quantized into integers stored in the parameter table, and the calculation result is output, divided by the nth power of 2, and restored to a fixed-point number having n decimal places to obtain converted coordinates (i.e., coordinates inversely mapped from the output image to the input image).

In operation S1102, pixel values adjacent to the converted coordinates are obtained for interpolation, so as to obtain an approximate pixel value of the converted coordinates, and the approximate pixel value is stored in a pixel position corresponding to the coordinates of the output image.

The embodiment of the disclosure provides an image deformation method, which quantizes fitting parameters into integer storage (that is, fixed point number instead of floating point number storage) after the fitting parameters are obtained in a training stage, so that fixed point operation can be performed in an inference stage, the resource utilization rate of a logic operation unit (for example, a fixed point calculation module shown in fig. 5) is improved, and the operation speed is increased. On the other hand, the method also reorders the subblock parameters in the lookup table according to the nearest neighbor first calculation principle and calculates according to the new subblock sequence in the inference stage, so that a logic operation unit (such as a fixed point calculation module shown in fig. 5) can utilize the data in the cache to calculate to the maximum extent during the operation in the inference stage, thereby avoiding directly reading the data from the memory and greatly improving the operation efficiency of the whole image deformation operation.

Fig. 4 schematically shows a block diagram of an image warping apparatus 400 in an embodiment of the present disclosure. As shown in fig. 4, the image morphing apparatus 400 includes: an input image buffer module 410, an index buffer module 420, an output image buffer module 430, and a fixed point calculation module 440.

The input image caching module 410 is configured to receive an instruction of the fixed-point calculating module 440, read input image data from a memory (shown in fig. 5) according to the instruction, send the input image data to the fixed-point calculating module 440, receive the converted coordinates sent by the fixed-point calculating module 440, and send the corresponding approximate pixel values of the converted coordinates in the input image to the output image caching module 430; the input image data comprises pixels of each point of the input image and corresponding coordinates of the pixels.

The output image caching module 430 is configured to receive an instruction of the fixed-point calculating module 440, read output image data from a memory (shown in fig. 5) according to the instruction, and send the output image data to the fixed-point calculating module 440; receiving the approximate pixel value corresponding to the converted coordinates sent by the input image buffer module 410, and writing the approximate pixel value into a memory (shown in fig. 5); the output image data comprises the size of the output image, each point pixel of the output image and the corresponding coordinate thereof.

The index cache module 420 is configured to read the first data and the second data in the parameter table from a memory (shown in fig. 5) and send the first data and the second data to the fixed point calculation module 440; wherein the first data comprises an original transformation relation from an output image to an input image; the second data is calculated by the fixed point calculation module 440 according to the output image data, the input image data, and the first data, and includes a fitting parameter that is an integer and is quantized into a sub-block range ordering.

The fixed point calculation module 440 is configured to receive the output image data, the input image data, and the first data, and perform the following operations according to the output image data, the input image data, and the first data: according to the size of the output image and the original transformation relation from the output image coordinate to the input image coordinate, the output image is subjected to block processing to obtain a plurality of corresponding sub-blocks; fitting a simple transformation relation in each sub-block to approximately substitute the original transformation relation from the coordinates of the sub-block to the coordinates of the input image; for each subblock, fitting the input image according to the simple transformation relation to obtain at least two subblock ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges; and reordering the sub-block ranges and the corresponding fitting parameters by using the nearest neighbor first calculation principle.

According to an embodiment of the present disclosure, the fixed point calculation module 440 is further configured to receive the output image data, the input image data, and the second data, and perform the following operations according to the output image data, the input image data, and the second data: dividing an output image into a plurality of sub-blocks identical to the input image; obtaining the range of the subblocks of each output image mapped into the input image according to the simple transformation relation to obtain the corresponding subblock range in the input image; reordering the subblocks of the output image according to the order of the subblock range arrangement; searching a parameter table for each sub-block of the output image for a fitting parameter quantized to an integer; and performing image deformation operation according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the converted coordinates.

The image deformation device provided by the disclosure can improve the image deformation operation speed, meet the requirements of more applications with requirements on real-time performance, save computing resources and process more and larger images when the computing resources are limited.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any number of the input image caching module 410, the index caching module 420, the output image caching module 430, and the fixed point calculation module 440 may be combined in one module to be implemented, or any one of them may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the input image buffer module 410, the index buffer module 420, the output image buffer module 430, and the fixed point calculation module 440 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented in any one of three implementations of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the input image caching module 410, the index caching module 420, the output image caching module 430 and the fixed point calculation module 440 may be at least partially implemented as a computer program module that, when executed, may perform a corresponding function.

Fig. 5 schematically illustrates a system block diagram 500 suitable for implementing the image warping method described above, according to an embodiment of the present disclosure. The system illustrated in fig. 5 is only an example and should not impose any limitations on the functionality or scope of use of embodiments of the disclosure.

As shown in fig. 5, the system 500 includes: the storage 510, the image warping apparatus 400, wherein the storage 510 is used for storing the input image, the output image and the parameter table, and the image warping apparatus 400 is used for implementing the above steps of the method to obtain the pixel position corresponding to the transformed coordinates.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

While the disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined not only by the appended claims, but also by equivalents thereof.

Claims (10)

1. An image warping method, comprising:

according to the size of the output image and the original transformation relation from the output image coordinate to the input image coordinate, carrying out block processing on the output image to obtain a plurality of corresponding sub-blocks;

fitting a simple transformation relation in each sub-block to approximately replace the original transformation relation from the sub-block coordinates to the input image coordinates;

for each subblock, fitting the input image according to the simple transformation relation to obtain at least two subblock ranges and corresponding fitting parameters;

quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges;

and reordering the sub-block ranges and the corresponding fitting parameters by using a nearest neighbor first calculation principle.

2. The method of claim 1, wherein the method further comprises:

dividing the output image into a plurality of sub-blocks identical to the input image;

obtaining the range of each subblock of the output image mapped into the input image according to the simple transformation relation to obtain the corresponding subblock range of the input image;

reordering the sub-blocks of the output image in the order in which the sub-block ranges are arranged;

looking up fitting parameters quantized to integers in a parameter table for the sub-blocks of each of the output images;

and performing image deformation operation according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the converted coordinates.

3. The method of claim 1, wherein quantizing the fitting parameters to integers and storing the quantized fitting parameters and the sub-block ranges in a compressed manner comprises:

and multiplying the fitting parameters which are originally floating point numbers by the nth power of 2 for amplification, then rounding the amplified floating point numbers to obtain fixed point numbers, and compressing and storing the fixed point numbers in a parameter table.

4. The method of claim 1, wherein said reordering said sub-block ranges and said corresponding fitting parameters on a nearest neighbor first calculation basis comprises:

and the sorting strategy takes the smallest ordinate value in the range of the subblocks mapped by the output image into the input image as a sorting key, and sorts the subblock range of the input image from small to large so as to ensure that the subblocks with the smallest ordinate closest to the ordinate are always adjacent.

5. A method according to claim 1 or 2, wherein the simple transformation relationship comprises only multiplication and addition; each simple transformation relation has a corresponding local area, and the simple transformation relation can approximately replace the original transformation relation in the corresponding local area so as to improve the operation efficiency.

6. The method of claim 2, wherein performing an image warping operation based on the fitting parameters quantized to integers to obtain pixel locations corresponding to the transformed coordinates comprises:

carrying out graph deformation operation according to the fitting parameters quantized into integers and outputting a calculation result, dividing the calculation result by the nth power of 2, and restoring the result into fixed point numbers with n decimal places to obtain the converted coordinates;

and obtaining pixel values adjacent to the converted coordinates for interpolation to obtain approximate pixel values of the converted coordinates, and storing the approximate pixel values to pixel positions corresponding to the coordinates of the output image.

7. The method according to claim 1, wherein in the operation of fitting the input image according to the simple transformation relationship for each of the subblocks to obtain at least two subblock ranges and corresponding fitting parameters, fitting is performed using a greedy algorithm and a least squares method to obtain the subblock ranges and the corresponding fitting parameters.

8. An image morphing apparatus, characterized by comprising:

the input image caching module is used for receiving an instruction of the fixed point calculating module, reading input image data from a memory according to the instruction, sending the input image data to the fixed point calculating module, receiving the converted coordinates sent by the fixed point calculating module, and sending the corresponding approximate pixel values of the converted coordinates in the input image to the output image caching module; the input image data comprises pixels of each point of the input image and corresponding coordinates of the pixels;

the output image caching module is used for receiving the instruction of the fixed point calculation module, reading output image data from the memory according to the instruction and sending the output image data to the fixed point calculation module; receiving the approximate pixel value corresponding to the converted coordinate sent by the input image caching module and writing the approximate pixel value into the memory; the output image data comprises the size of the output image, each pixel of the output image and corresponding coordinates thereof;

the index cache module is used for reading first data and second data in the parameter table from the memory and sending the first data and the second data to the fixed point calculation module; wherein the first data comprises an original transformation relationship of the output image to the input image; the second data is obtained by the fixed point calculation module according to the output image data, the input image data and the first data, and comprises the sub-block range ordering and the fitting parameters quantized into integers;

the fixed point calculation module is configured to receive the output image data, the input image data, and the first data, and perform the following operations according to the output image data, the input image data, and the first data:

according to the size of the output image and the original transformation relation from the output image coordinate to the input image coordinate, carrying out block processing on the output image to obtain a plurality of corresponding sub-blocks;

fitting a simple transformation relation in each sub-block to approximately replace the original transformation relation from the sub-block coordinates to the input image coordinates;

for each subblock, fitting the input image according to the simple transformation relation to obtain at least two subblock ranges and corresponding fitting parameters;

quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the subblock ranges;

and reordering the sub-block ranges and the corresponding fitting parameters by using a nearest neighbor first calculation principle.

9. The apparatus of claim 8, wherein the fixed point computation module is further configured to receive the output image data, the input image data, and the second data, and to perform the following operations based on the output image data, the input image data, and the second data:

dividing the output image into a plurality of sub-blocks identical to the input image;

obtaining the range of each subblock of the output image mapped into the input image according to the simple transformation relation to obtain the corresponding subblock range of the input image;

reordering the sub-blocks of the output image in the order in which the sub-block ranges are arranged;

looking up fitting parameters quantized to integers in a parameter table for the sub-blocks of each of the output images;

and performing image deformation operation according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the converted coordinates.

10. An image warping system, comprising: the image transformation device is used for carrying out image transformation operation according to the method of any one of claims 1 to 7 so as to obtain the pixel position corresponding to the transformed coordinate.

Images