WO2018195819A1 - 图像处理方法和设备 - Google Patents
图像处理方法和设备 Download PDFInfo
- Publication number
- WO2018195819A1 WO2018195819A1 PCT/CN2017/082026 CN2017082026W WO2018195819A1 WO 2018195819 A1 WO2018195819 A1 WO 2018195819A1 CN 2017082026 W CN2017082026 W CN 2017082026W WO 2018195819 A1 WO2018195819 A1 WO 2018195819A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- compressed
- layer
- pixel
- tracking
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
Definitions
- the present application relates to the field of image processing and, more particularly, to an image processing method and apparatus.
- Image pyramid is a kind of multi-scale representation of images. It is an effective but conceptually simple structure for interpreting images with multiple resolutions.
- the image pyramid of the image pyramid is very versatile, for example, image tracking can be performed. Since the pyramid includes multi-layer image layering, when the pyramid data is stored, it takes a large storage space, and the pyramid data can generally be stored in the off-chip system.
- the embodiment of the present application provides an image processing method and device, which can improve system performance.
- an image processing method comprising: constructing an image pyramid of a target image, the image pyramid comprising a plurality of image layers; and compressing pixel points of at least one of the plurality of image layers; A compressed pixel point is stored in the storage device; at least a portion of the compressed pixel point is read from the storage device; the read pixel point is decompressed to obtain a decompressed pixel point.
- an image processing apparatus comprising: a construction unit for constructing an image pyramid of a target image, the image pyramid comprising a plurality of image layers; and a compression unit for compressing the plurality of image layers a pixel of the at least one image layer; a storage unit for storing the compressed pixel point in the storage device; a reading unit for reading the compressed at least part of the pixel point from the storage device; and a decompression unit Used to decompress the read pixel points to get the decompressed pixel points.
- At least one image included in the image pyramid constructed for the target image is layer-compressed and stored in the storage device, and when processed to the at least one image layer, may be from the storage device Read the compressed pixels and decompress them. Since the amount of compressed data is less than the amount of data before compression, the bandwidth required for reading is small and can be saved. Time, improve the processing efficiency of the system, which can improve system performance.
- FIG. 1 is a schematic diagram of an image pyramid in accordance with an embodiment of the present application.
- FIG. 2 is a schematic block diagram of a feature tracking system in accordance with an embodiment of the present application.
- FIG. 3 is a schematic flowchart of an image processing method according to an embodiment of the present application.
- FIGS 4a-4c are schematic diagrams of compressed image blocks in accordance with an embodiment of the present application.
- FIG. 5a is a schematic diagram of pixel points that need to be read, in accordance with an embodiment of the present application.
- FIG. 5b is a schematic diagram of a compressed block corresponding to the pixel point to be read shown in FIG. 5a according to an embodiment of the present application.
- FIG. 6 is a schematic diagram of a manner 400 of feature point tracking.
- FIG. 7 is a schematic block diagram of an image processing apparatus according to an embodiment of the present application.
- FIG. 8 is a schematic block diagram of an image processing apparatus according to an embodiment of the present application.
- FIG. 9 is a schematic block diagram of a mobile device in accordance with an embodiment of the present application.
- Image pyramid is a kind of multi-scale representation of images. It is an effective but conceptually simple structure for interpreting images with multiple resolutions. Image pyramids can be obtained by smoothing and/or downsampling.
- the image pyramid may include multiple image layers (which may also be referred to as image layers, layers, pyramid layers, etc.), and the upper layer of the image pyramid has a lower resolution than the lower layer.
- Image pyramids are very versatile, for example, can be applied to feature point tracking and the like.
- feature point tracking an image pyramid of the first image and an image pyramid of the second image may be established, wherein the position of the feature point at the first image is known, and the position of the partial pixel in the second image may be utilized And/or pixel values for feature point tracking.
- FIG. 1 illustrates an image pyramid of a first image (ie, image I) and a second image (image J) in accordance with an embodiment of the present application.
- image I a first image
- image J a second image
- the image pyramid 102 of the image I and the image pyramid 104 of the image J have m+1 image layers, where m is an integer not less than 0.
- the bottom layer of the image pyramid is image layer 0, and image layer 0 has the highest resolution of image I and image J.
- the highest resolution of image I and image J may be the highest resolution of the original image of image I and image J, respectively.
- image layer 0 of image I and image J includes pixel values of some or all of the pixel locations of image I and image J, respectively.
- the pixel values may be gray values of image I and image J.
- the pixel value may also include the brightness of the pixel location.
- the pixel values of the pixel positions in the image I and the image J may be represented as I(x, y) and J(x, y), respectively, where x and y are pixel coordinates of the pixel position.
- the pixel coordinate vector of the upper left corner can be represented as [0 0] T
- the width and height of image I and image J are the same, and can be expressed as n x and n y (all are not less than An integer of 1).
- the pixel coordinate vector in the lower right corner can be expressed as [n x-1 n y-1 ] T .
- image I and image J may not have the same height and width, and it is assumed herein that the same height and width are merely for convenience of description.
- the image pyramids of image I and image J can be established in a recursive manner.
- Image layering 1 is calculated based on image layering
- image layering 2 is calculated based on image layering 1, and so on.
- the image layering of the pyramid can be established from bottom to top in a smooth or downsampled manner.
- n x L-1 and n y L-1 are the width and height of the I L-1 layer, respectively.
- the image layer I L can be obtained according to the following formula 1:
- the value of the 3x3 kernal weight matrix sampled by Equation 1 may be changed, or a weight matrix of 5x5, 7x7 may also be used.
- layer L may comprise approximately 1/4 of the pixel position of layer L-1.
- the image layers I 1 , I 2 , I 3 and I 4 are 960x 512, 480x 256, 240x 128 and 120x64, respectively.
- the images J 1 , J 2 , J 3 and J 4 have the same size, respectively.
- Image recursions of image I and image J are formed by recursive processing.
- the image pyramid can include 2-10 layers. In Figure 1, the image pyramids 102 and 104 have the same number of layers, including a layer 0 112, 114 of the layer 1, 2, and L m at the top layer 118 of the layer 116.
- the pixel locations of the feature points at different layers may be determined based on a recursive method of establishing an image pyramid of the image. For example, based on equation (1), u 1 160 can be identified at layer 1 and u 2 170 at layer 2 until the pixel locations of feature points at all layers are obtained.
- the formula (1) shows that the image pyramid is established by using a downsampling method, and the image pyramid may be established by using the other downsampling method and the smoothing method in the embodiment of the present application.
- a Gaussian pyramid, a Laplacian pyramid, and a controllable pyramid can be established.
- Embodiments of the present application do not define the manner in which the pyramids of image I and/or image J are established.
- feature tracking can be performed. For example, feature tracking can start from the topmost level of image I and image J, and the topmost result can include optical flow information. This result can be used for feature tracking of the next layer.
- the manner in which the image pyramid is established according to the embodiment of the present application has been described above with reference to FIG. 1.
- the feature tracking is taken as an example, and the feature tracking system shown in FIG. 2 is used to describe how to store and use the created image pyramid.
- FIG. 2 illustrates a feature tracking system 200 in accordance with an embodiment of the present application.
- the feature tracking system 200 includes an external storage device 210 and an electronic device 220.
- the electronic device 220 includes an internal storage device 222, a feature point control unit 224, and an image pyramid processing unit 226.
- Each or combination of internal storage device 222, and processing device 224 can be implemented by at least one electronic circuit.
- the electronic device 220 can be implemented using a general purpose processor.
- the electronic device 220 can be implemented by one or more application-specific integrated circuits (ASICs) and field programmable gate arrays (FPGAs).
- ASICs application-specific integrated circuits
- FPGAs field programmable gate arrays
- Electronic device 220 may be referred to as a visual/graphic accelerator, or may simply be referred to as an accelerator.
- Electronic device 220 is coupled to external storage device 210.
- the external storage device 210 may store an image pyramid (also referred to as image pyramid data) for each image in at least one image.
- External storage device 210 can be large enough to store multiple image pyramids.
- the following is an example of image tracking from the first image to the second image.
- the first image may also be referred to as image I
- the second image may also be referred to as image J.
- image I image
- image J image
- embodiments of the present application are used to identify matching pixel points of a plurality of feature points in image J.
- the embodiments of the present application can also be applied to other tracking scenes instead of feature tracking from image I to image J.
- embodiments of the present application can be applied to feature tracking from a first image to a series of other images, or from a video frame. Tracking to one or more other video frames.
- External storage device 210 can be coupled or in communication with internal storage device 222, which can include a smaller storage device than external storage device 210 for feature tracking and storage of partial pyramid data from external storage device 210.
- internal storage device 222 can include a smaller storage device than external storage device 210 for feature tracking and storage of partial pyramid data from external storage device 210.
- one or more layers of the image pyramid data may not be stored in the external storage device 210, but stored directly in the internal storage device 222.
- Internal storage device 224 can store data from different image pyramids at different locations in internal memory 130.
- the external storage device 210 and the internal storage device 222 may include one or more of a variety of dynamic random-access memories (DRAMs), such as double data rate synchronous DRAM (DDR DRAM, or simply DDR). Single data rate (single data Rate, SDR) SDRAM, static RAM (SRAM), persistent mass storage devices (eg, flash memory, disk, etc.), persistent storage (such as phase change memory (PCM), memristor, Spin-transfer torque (STT) RAM.
- DRAMs dynamic random-access memories
- DDR DRAM double data rate synchronous DRAM
- SDR single data rate SDRAM
- SRAM static RAM
- persistent mass storage devices eg, flash memory, disk, etc.
- PCM phase change memory
- memristor spin-transfer torque
- external storage device 210 includes DDR SDRAM to store pyramid data.
- the internal storage device 222 includes a cache to store image pyramid data (eg, partial image pyramid data acquired from the external storage device 210, or one or more layers of image pyramids when the image pyramid is generated.
- image pyramid data eg, partial image pyramid data acquired from the external storage device 210, or one or more layers of image pyramids when the image pyramid is generated.
- the external storage device 210 may be shown external to the electronic device 210, the external storage device 210 may also be implemented in the electronic device 210.
- the electronic device 220 can also be implemented on a chip system including the external storage device 210.
- Processing device 224 can perform an optical flow tracking algorithm.
- processing device 224 may perform an optical flow tracking algorithm using the Kanade-Lucas-Tomasi (KLT) algorithm.
- KLT Kanade-Lucas-Tomasi
- processing device 224 can determine pixel point information that needs to be read and can be provided to internal storage device 222. Processing device 224 can provide the correct image pyramid data to internal storage device 222 based on the pixel point information.
- processing device 224 may also provide pixel point information to internal storage device 222 so that internal storage device 222 may request and retrieve the correct image pyramid data from external storage device 210 based on the pixel point information.
- the iterative calculation of the image pyramid data of the first image and the second image can be performed, and the feature point tracking system can acquire the matching points in the image J with the plurality of feature points in the image I.
- the electronic device 220 reads the image pyramid data from the external storage device 210, since the data amount of the pyramid data is large, a large bandwidth is required, and a long reading time is required.
- the embodiment of the present application provides an image processing method, so as to solve the problem as much as possible. It should be understood that the image processing method of the embodiment of the present application can be applied to optical flow tracking, and can also be applied to other scenarios.
- FIG. 3 is a schematic flowchart of an image processing method 300 according to an embodiment of the present application.
- the method 300 includes at least a portion of the following.
- an image pyramid of the target image is constructed, the image pyramid comprising a plurality of image layers.
- the manner in which the image pyramid is constructed may be, but is not limited to, the configuration shown in FIG. 1.
- pixel points of at least one of the plurality of image layers are compressed.
- At least a portion of the image hierarchy in the image hierarchy included in the constructed image pyramid may be compressed.
- At least part of the image layer of the image pyramid of the image I and at least part of the image layer of the image pyramid of the compressed image J may be compressed; or only at least part of the image I may be compressed
- the image is layered, or only at least part of the image layer of the image J is compressed.
- the image layer to be compressed may be determined according to at least one of system available bandwidth and pre-configured image layered information that needs to be compressed.
- the bandwidth required to read the pyramid data from the external storage device may be estimated according to the available bandwidth of the system, and according to the estimated bandwidth, and the resolution of each image layer and/or the number of pixel locations, etc. Determine which image layers need to be compressed and compress.
- the pre-configured information of the image hierarchy that needs to be compressed includes the resolution and/or number of image layers that need to be compressed.
- the resolution and/or the number of layers of the image to be compressed may be pre-configured, and the image to be compressed may be layered and compressed according to the pre-configured information.
- the pre-configured resolution range of the image to be compressed may directly compress all the images in the range, or may combine the current available bandwidth of the system, and select partial image layer to compress from the range. .
- the pre-configured image layer to be compressed has a resolution of 960x 512 and 480x256
- the images with resolutions of 960x 512 and 480x 256 can be directly compressed, or they can be based on the current available bandwidth of the system. In the case, choose to compress.
- the number of layers of the image that can be pre-configured to be compressed may be combined with the current available bandwidth of the system to select a certain number of images (belonging to the preset number range) for layer compression.
- the pixel points of the at least one image layer having the highest resolution among the plurality of image layers may be compressed.
- At least one image layer with the highest resolution can be compressed due to the bandwidth occupied by the higher resolution image layering and the time taken for reading to be larger than the lower resolution image layering.
- the image pyramid includes four image layers, and the resolutions of the four image layers are respectively 960x 512, 480x 256, 240x 128 and 120x 64, you can compress image resolutions of 960x 512, 480x 256.
- the at least one image layer may be compressed in an image block including a plurality of pixel points to obtain a plurality of compressed blocks.
- the image block may comprise a plurality of pixel points, which may be adjacent pixel points, for example, as shown in Figure 4a, may be each of the image layers, or, for example, as shown in Figure 4b , may be each column of the image pyramid, of course, may also be a block of N x M, where N is greater than 1, and M is greater than 1, for example, as shown in FIG. 4c, may be a 4x4 image block in the image layer .
- the manner in which the pixel points are read may be determined according to the storage manner of the pixel points.
- a pixel is compressed for storage in a row, it can be read and decompressed in rows when reading.
- a pixel is compressed for storage in columns, it can be read and decompressed in columns when reading.
- the compression mode of the pixel point may be determined according to the manner in which the pixel is read.
- a pixel is read in rows, it can be compressed in rows when compressed.
- a pixel is read in columns, it can be compressed in rows when compressed.
- determining the number of pixel points included in the image block according to at least one of a preset number of pixels included in the image block and a number of pixel points included in the to-be-processed window according to a system available bandwidth;
- the image block including the number of pixels is layered and compressed for the at least one image.
- the to-be-processed window includes at least one pixel to be processed, and the pixel to be processed can be used for feature tracking.
- the pending window may alternatively be referred to as a neighborhood of feature points.
- the size of the image block that needs to be compressed may be determined based on the available bandwidth of the system and/or the size of the window to be processed.
- the preset number of pixels included in the image block may be a specific value, and may be compressed according to an image block having the number of pixels.
- the preset number of pixels included in the image block may be a range, and a quantity may be selected from the range according to the available bandwidth of the system and/or the size of the window to be processed, and the quantity may be The image block of the pixel is compressed.
- the size of the data readable from the external storage device may be determined according to the available bandwidth of the system, and the compressed image block may be determined according to the data size before compression and the readable data size. Compression ratio.
- the compression ratio can be a specific value that can be determined as the compression ratio of the compressed image block.
- the pre-configured compression ratio may be a range, and may be used directly from the range to compress the compression ratio of the image block, or may be combined with the current available bandwidth of the system, and select a compressed image from the range.
- the compression ratio of the block may be a range, and may be used directly from the range to compress the compression ratio of the image block, or may be combined with the current available bandwidth of the system, and select a compressed image from the range. The compression ratio of the block.
- each image layer is layered compared to other image layers, and the number of pixels included in the image block to be compressed may be the same or different.
- the compressed pixel points are stored in a storage device.
- the first image layer and the second image layer are cached in different storage devices.
- the different storage devices have different controllers.
- the storage device is located in an off-chip system.
- At 340 at least a portion of the compressed pixel points are read from the storage device.
- At least one compressed block is read from the storage device.
- the compressed block to be read is determined according to the position of the window to be processed in the image layer, and the compressed block to be read is read from the storage device.
- the to-be-processed window includes at least one pixel to be processed, and the pixel to be processed can be used for feature tracking.
- the pending window may alternatively be referred to as a neighborhood of feature points.
- the obtained compressed block can be as shown in FIG. 5b.
- the corresponding compressed block is 2-1, if it needs to be read.
- the row where the data is located is 6-8, the column is 6-8, and the compression mode is the compression mode shown in Figure 4c.
- the compressed block to be read is 3-3,3- 4, 4-3 and 4-4. Thereby the compressed block can be read in an external storage device.
- the read pixel points are decompressed to obtain decompressed pixel points.
- At least the image pyramid constructed for the target image includes at least An image is layered and compressed, and stored in a storage device.
- the compressed pixel points can be read from the storage device and decompressed, since the compressed data volume is smaller than the compression. The amount of data before, when reading, requires less bandwidth, and can save time and improve efficiency.
- the image processing method in the embodiment of the present application may be used for optical flow tracking.
- image pyramids of image I and image J may be constructed, at least part of the image pyramid of image I may be layered and compressed, and/or at least part of the image pyramid of image J may be segmented
- the layer is compressed and stored, wherein the tracking pixel point of the optical flow tracking is known at the position of the first image, and the position and gray level of the tracking pixel at the first image is used to determine the tracking pixel point at the A position of the second image, and/or a displacement vector of the tracking pixel between the first image and the second image.
- the optical flow tracking is implemented by a KLT algorithm.
- the target image mentioned in method 300 includes a first image and a second image, the image pyramid comprising a first image pyramid of the first image and a second image pyramid of the second image, the at least one image
- the first gray level gradient of the partial pixel point of the decompressed first pixel point in the first direction and the second gray level gradient in the second direction may be determined;
- One Calculating a gradient matrix according to the grayscale gradient and the second grayscale gradient; and according to the gradient matrix, and decompressing the at least part of the pixel points in the first pixel and the at least part of the pixels in the second pixel The pixel value and position determine the displacement vector of the tracking pixel between the first image layer and the second image layer in an intra-layer iteration manner.
- optical flow tracking will be described below in conjunction with the pseudo code shown in FIG. 6.
- FIG. 6 shows a schematic diagram of a manner 400 of feature point tracking.
- the purpose of mode 400 is to find the corresponding location in image J for a single point u in image I.
- an image pyramid of image I and image J is established.
- the image pyramid of image I and image J can be established using the method described in FIG.
- the image pyramid of image I and image J can be expressed as with .
- an image pyramid of image I and an image pyramid storing J wherein at least a portion of the image layer in image pyramid of image I can be compressed prior to storage, and/or at least a portion of image pyramid of image J Image layering can be compressed before being stored.
- the initial optical flow estimate for the pyramid is initialized according to Equation 2:
- the outer loop begins. In the outer loop, starting from the highest level, iteratively processing in each layer in descending order until the bottom layer. In each iterative process, the pixel position of point u at the corresponding layer can be determined.
- the pixel location of point u in image layer I L can be obtained.
- the image I L is the Lth layer of the image pyramid of the image I, and in the image I L , the position of the point u can be determined according to the method described in FIG. 2.
- the position of the window to be processed can be determined, and the pixel to be read is determined according to the position of the window to be processed, and the pixel is read from the storage device.
- Point if the read pixel is a compressed pixel, the compressed pixel can be decompressed to obtain the pixel value I L (x, y).
- the size of the window to be processed may be (2 ⁇ x +1) ⁇ (2 ⁇ y +1).
- a derivative of the pixel value of the read pixel point relative to x can be determined.
- the derivative of x can be calculated according to the following formula (4).
- a derivative of the pixel value of the read pixel point relative to y can be determined.
- the derivative of y can be calculated according to the following formula (5).
- the derivative of the pixel values relative to x and y can be used to calculate the spatial gradient matrix G.
- the matrix G can be calculated according to the following formula 6.
- an optical loop can be performed using an iterative KLT optical flow tracking algorithm through an inner loop.
- the initial pixel displacement is assumed according to Equation 7:
- an inner loop calculation can be performed to obtain the optical flow of point u at layer L, and the optical flow estimate at layer L-1.
- the inner loop can be executed a predetermined number of times K or until the calculated optical flow Small enough.
- the image difference is calculated for a given k within [1K]. For a given k, it can be calculated according to the following formula 8:
- the to-be-processed window of the image J may be determined, and the pixel points included in the to-be-processed window are read from the storage device, and if the read pixel point is a compressed pixel point, the compressed pixel point may be decompressed. To get the pixel value J L (x, y) of the pixel.
- a mismatch vector of the image can be calculated.
- an image mismatch vector may be calculated according to Equation 9 below.
- optical flow can be calculated using the KLT algorithm Alternatively, the calculation can be performed according to the following formula 10.
- the operation for k is ended. If the inner loop condition is not satisfied, the next iteration is performed, that is, for the operation of k+1, the operation at 422-428 is repeated, wherein the optical flow is used. Used for k+1 iterations.
- the optical flow estimator for k can be calculated according to the following formula:
- the end of the inner loop of layer L is determined.
- both optical flow tracking at image layer L can be determined.
- the optical flow of layer L can be obtained using the following formula:
- an optical flow estimate for the L-1 layer can be determined.
- the optical flow estimator at layer L-1 can be obtained using the following formula.
- the outer loop at layer L is completed.
- the operation of the next layer (L-1 layer) is continued, that is, the operations at 310-334 are repeatedly performed repeatedly.
- the final optical flow vector d at layer 0 can be acquired based on the optical flow estimate at layer 0 and the optical flow at layer 0.
- the final optical flow vector d can be obtained using the following formula:
- a matching point of feature point u in image I in image J can be determined based on the final optical flow vector.
- the pixel position of the matching point u can be obtained using the following formula:
- the matching point v of the feature point u in the image I in the image J can be determined. It should be understood that the pseudo code shown in FIG. 6 is only one implementation manner of the present application, and other implementation manners may be implemented in the present application. For brevity, details are not described herein again.
- the image processing method has been described above in connection with the methods 200 and 300, which can improve the performance of the system and improve the processing efficiency of the system.
- the embodiment of the present application may have other methods for achieving the purpose.
- the image I and the image I are respectively stored in different memories, and may further have different Controller.
- a high speed external caching device can be used without the need to compress the image pyramids of image I and image J.
- pyramid data is stored in an on-chip system.
- multiple KLT processing units can be used for parallel processing.
- FIG. 7 is a schematic block diagram of an image processing apparatus 500 in accordance with an embodiment of the present application.
- the apparatus 500 includes a construction unit 510, a compression unit 520, a storage unit 530, a reading unit 540, and a decompression unit 550.
- the constructing unit 510 is configured to: construct an image pyramid of the target image, the image pyramid including a plurality of image layers;
- the compressing unit 520 is configured to: compress pixel points of at least one of the plurality of image layers;
- the storage unit 530 is configured to: store the compressed pixel points in the storage device;
- the reading unit 540 is configured to: read at least part of the compressed pixel points from the storage device;
- the decompression unit 550 is configured to decompress the read pixel points to obtain decompressed pixel points.
- the compression unit 520 is further configured to:
- the at least one image layer is compressed by an image block including a plurality of pixels to obtain a plurality of compressed blocks.
- the reading unit 540 is further configured to:
- At least one of the compressed blocks is read.
- the reading unit 540 is further configured to:
- the compressed block to be read is read.
- the compression unit 520 is further configured to:
- the at least one image layer is compressed in the image block including the number of pixels.
- the compression unit 520 is further configured to:
- the image block is compressed according to the determined compression ratio to obtain the compressed block.
- the compression unit 520 is further configured to:
- At least one of information based on system available bandwidth and pre-configured image layering that needs to be compressed Determining the layer of the image to be compressed
- the image layer to be compressed is compressed.
- the pre-configured information of the image layer to be compressed includes a resolution of the image layer to be compressed.
- the compression unit 520 is further configured to:
- Pixel pixels of the at least one image layer having the highest resolution among the plurality of image layers are compressed.
- the image processing device 500 is configured to track optical flow
- the target image includes a first image and/or a second image
- a tracking pixel of the optical flow tracking is at a position of the first image Knowing that the position and gray level of the tracking pixel at the first image are used to determine a position of the tracking pixel at the second image, and/or determining the tracking pixel point in the first A displacement vector between an image and the second image.
- the optical flow tracking is implemented by a music-Lucas-tomasi KLT algorithm.
- the target image includes a first image and a second image, the image pyramid including a first image pyramid of the first image and a second image pyramid of the second image, the at least one image segment
- the layer includes a first image layer belonging to the first image pyramid and a second image layer belonging to the second image pyramid, the first image layer and the second image layer belong to the same hierarchical level Where the image layering of the same hierarchical level has the same resolution;
- the reading unit 540 is further configured to:
- the decompression unit 550 is further configured to:
- the device 500 further includes a determining unit 560, configured to:
- Determining the tracking pixel by means of intra-layer convergence iteration according to at least part of the decompressed first pixel point and at least part of the decompressed second pixel point Pointing a displacement vector between the first image layer and the second layer;
- the displacement vector is used as the tracking pixel point in the first image layer and the second image
- An initial displacement vector of the next hierarchical level like layering, and/or an initial displacement vector of the displacement vector is a previous layer of the tracking pixel point in the first image layer and the second image layer
- the displacement vector of the hierarchy is used as the tracking pixel point in the first image layer and the second image.
- the determining unit 560 is further configured to:
- the first image layer and the second image layer are cached in different storage devices.
- the different storage devices have different controllers.
- the storage device is located in an off-chip system.
- image processing apparatus 500 can implement the method 300 or 400, and for brevity, no further details are provided herein.
- FIG. 8 is a schematic block diagram of an image processing apparatus 600 according to an embodiment of the present application.
- the image processing device 600 may comprise a plurality of different components, which may be integrated circuits (ICs), or part of an integrated circuit, discrete electronic devices, or other suitable for a circuit board (such as a motherboard) Modules, or additional boards, may also be incorporated as part of a computer system.
- ICs integrated circuits
- circuit board such as a motherboard
- Modules or additional boards, may also be incorporated as part of a computer system.
- the image processing device can include a processor 610 and a storage medium 620 coupled to the processor 610.
- Processor 610 may include one or more general purpose processors, such as a central processing unit (CPU), or a processing device or the like. Specifically, the processor 610 may be a complex instruction set computing (CISC) microprocessor, a very long instruction word (VLIW) microprocessor, and implements micro-processing of multiple instruction set combinations. Device. The processor may also be one or more dedicated processors, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and a digital signal processor. , DSP).
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- DSP digital signal processor
- Processor 610 can be in communication with storage medium 620.
- the storage medium 620 can be a magnetic disk, an optical disk, a read only memory (ROM), a flash memory, or a phase change memory.
- the storage medium 620 can store instructions stored by the processor, and/or can cache some information stored from an external storage device, such as image layered pixel information of a pyramid read from an external storage device.
- the image processing apparatus may include a display controller and/or display device unit 630, a transceiver 640, a video input output unit 650, an audio input output unit 660, and other input and output units 670.
- a display controller and/or display device unit 630 may include a display controller and/or display device unit 630, a transceiver 640, a video input output unit 650, an audio input output unit 660, and other input and output units 670.
- These components included in image processing device 600 may be interconnected by a bus or internal connection.
- the transceiver 640 can be a wired transceiver or a wireless transceiver, such as a WIFI transceiver, a satellite transceiver, a Bluetooth transceiver, a wireless cellular telephone transceiver, or combinations thereof.
- a wireless transceiver such as a WIFI transceiver, a satellite transceiver, a Bluetooth transceiver, a wireless cellular telephone transceiver, or combinations thereof.
- the video input and output unit 650 may include an image processing subsystem such as a video camera including a photo sensor, a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) light. Sensor for use in shooting functions.
- a video camera including a photo sensor, a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) light. Sensor for use in shooting functions.
- CCD charge coupled device
- CMOS complementary metal-oxide semiconductor
- the audio input and output unit 660 may include a speaker, a microphone, an earpiece, and the like.
- other input and output devices 670 may include a storage device, a universal serial bus (USB) port, a serial port, a parallel port, a printer, a network interface, and the like.
- USB universal serial bus
- the image processing device 600 can perform the operations shown in the method 300 or 400.
- the image processing device 600 can perform the operations shown in the method 300 or 400.
- details are not described herein again.
- the image processing device 500 or 600 may be located in a mobile device.
- the mobile device can be moved in any suitable environment, for example, in the air (eg, a fixed-wing aircraft, a rotorcraft, or an aircraft with neither a fixed wing nor a rotor), in water (eg, a ship or submarine), on land. (for example, a car or train), space (for example, a space plane, satellite or detector), and any combination of the above.
- the mobile device can be an aircraft, such as an Unmanned Aerial Vehicle (UAV).
- UAV Unmanned Aerial Vehicle
- the mobile device can carry a living being, such as a person or an animal.
- FIG. 9 is a schematic block diagram of a removable device 700 in accordance with an embodiment of the present application.
- the removable device 700 includes a carrier 710 and a load 720.
- the description of the mobile device in Figure 9 as a drone is for illustrative purposes only.
- the load 720 may not be connected to the removable device via the carrier 710.
- the mobile device 700 can also include a power system 730, a sensing system 740, and a communication system 750.
- Image processing device 760 can also include a power system 730, a sensing system 740, and a communication system 750.
- Power system 730 can include an electronic governor (referred to as an ESC), one or more propellers, and one or more electric machines corresponding to one or more propellers.
- the motor and the propeller are disposed on the corresponding arm; the electronic governor is configured to receive a driving signal generated by the flight controller, and provide a driving current to the motor according to the driving signal to control the rotation speed and/or steering of the motor.
- the motor is used to drive the propeller to rotate to power the UAV's flight, which enables the UAV to achieve one or more degrees of freedom of motion.
- the UAV can be rotated about one or more axes of rotation.
- the above-described rotating shaft may include a roll axis, a pan axis, and a pitch axis.
- the motor can be a DC motor or an AC motor.
- the motor can be a brushless motor or a brush motor.
- the sensing system 740 is used to measure the attitude information of the UAV, that is, the position information and state information of the UAV in space, for example, three-dimensional position, three-dimensional angle, three-dimensional speed, three-dimensional acceleration, and three-dimensional angular velocity.
- the sensing system may include, for example, a gyroscope, an electronic compass, an Inertial Measurement Unit ("IMU"), a vision sensor, a Global Positioning System (GPS), and a barometer. At least one of them.
- the flight controller is used to control the flight of the UAV, for example, the UAV flight can be controlled based on the attitude information measured by the sensing system. It should be understood that the flight controller may control the UAV in accordance with pre-programmed program instructions, or may control the UAV in response to one or more control commands from the operating device.
- Communication system 750 can communicate with a terminal device 780 having communication system 770 via wireless signal 790.
- Communication system 750 and communication system 770 can include a plurality of transmitters, receivers, and/or transceivers for wireless communication.
- the wireless communication herein may be one-way communication, for example, only the mobile device 700 may transmit data to the terminal device 780.
- the wireless communication may be two-way communication, and the data may be transmitted from the mobile device 700 to the terminal device 780 or may be transmitted by the terminal device 780 to the mobile device 700.
- terminal device 780 can provide control data for one or more of removable device 700, carrier 710, and load 720, and can receive information transmitted by mobile device 700, carrier 710, and load 720.
- the control data provided by the terminal device 780 can be used to control the status of one or more of the mobile device 700, the carrier 710, and the load 720.
- a carrier 710 and a load 720 include a communication module for communicating with the terminal device 780.
- image processing device 660 of the mobile device illustrated in FIG. 9 can perform the methods 300 and 400, which are not described herein for brevity.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Compression Of Band Width Or Redundancy In Fax (AREA)
Abstract
提供了一种图像处理方法和设备,可以提升系统性能。该方法包括:构造目标图像的图像金字塔,所述图像金字塔包括多个图像分层;压缩所述多个图像分层中的至少一个图像分层的像素点;在存储设备中存储压缩的像素点;从所述存储设备中读取压缩的至少部分像素点;解压缩读取的像素点,以得到解压缩的像素点。
Description
本申请涉及图像处理领域,并且更具体地,涉及一种图像处理方法和设备。
图像金字塔是图像多尺度表达的一种,是一种以多分辨率来解释图像的有效但概念简单的结构。
图像金字塔的图像金字塔的用途非常广泛,例如,可以进行图像跟踪。由于金字塔包括多层图像分层,在进行金字塔数据的存储时,需要占用较大的存储空间,一般可以将金字塔数据存储在片外系统。
然而,在从片外系统读取金字塔数据时,需要占用较大的系统带宽,并且读取时间较长,导致系统性能较差。
发明内容
本申请实施例提供了一种图像处理方法和设备,可以提升系统性能。
一方面,提供了一种图像处理方法,包括:构造目标图像的图像金字塔,所述图像金字塔包括多个图像分层;压缩所述多个图像分层中的至少一个图像分层的像素点;在存储设备中存储压缩的像素点;从所述存储设备中读取压缩的至少部分像素点;解压缩读取的像素点,以得到解压缩的像素点。
另一方面,提供了一种图像处理设备,包括:构造单元,用于构造目标图像的图像金字塔,所述图像金字塔包括多个图像分层;压缩单元,用于压缩所述多个图像分层中的至少一个图像分层的像素点;存储单元,用于在存储设备中存储压缩的像素点;读取单元,用于从所述存储设备中读取压缩的至少部分像素点;解压缩单元,用于解压缩读取的像素点,以得到解压缩的像素点。
因此,在本申请实施例中,将为目标图像构造的图像金字塔包括的至少一个图像分层进行压缩,并存储在存储设备中,当处理到该至少一个图像分层时,可以从存储设备中读取压缩的像素点,并进行解压缩,由于压缩后的数据量小于压缩前的数据量,在读取时,需要占用的带宽较小,且可以节省
时间,提高系统的处理效率,从而可以提升系统性能。
为了更清楚地说明本申请实施例的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。
图1是根据本申请实施例的图像金字塔的示意性图。
图2是根据本申请实施例特征跟踪系统的示意性框图。
图3是根据本申请实施例的图像处理方法的示意性流程图。
图4a-4c是根据本申请实施例的被压缩图像块的示意性图。
图5a是根据本申请实施例的需要读取的像素点的示意性图。
图5b是根据本申请实施例的图5a所示的需要读取的像素点对应的压缩块的示意性图。
图6是一种特征点跟踪的方式400的示意性图。
图7是根据本申请实施例的图像处理设备的示意性框图。
图8是根据本申请实施例的图像处理设备的示意性框图。
图9是根据本申请实施例的可移动设备的示意性框图。
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。
图像金字塔是图像多尺度表达的一种,是一种以多分辨率来解释图像的有效但概念简单的结构。可以通过平滑和/或降采样的方式,获取图像金字塔。图像金字塔可以包括多个图像分层(也可以称为图像层,层,金字塔层等),图像金字塔的上层的分辨率低于下层的分辨率。
图像金字塔的用途非常广泛,例如,可以应用于特征点跟踪等。在特征点跟踪中,可以建立第一图像的图像金字塔和第二图像的图像金字塔,其中,特征点在第一图像的位置是已知的,可以利用第二图像中部分像素点的位置
和/或像素值进行特征点跟踪。
本申请虽然以特征点跟踪进行描述,但并不代表本申请实施例的图像处理方法不应用于其他场景。
以下将以场景为特征跟踪为例,结合图1描述图像金字塔的建立方式,但应理解,图1所示的金字塔建立方式仅仅是一种实现方式,不应对本申请造成限定。
图1示出了根据本申请实施例的第一图像(即,图像I)和第二图像(图像J)的图像金字塔。如图1所示,图像I的图像金字塔102和图像J的图像金字塔104具有m+1个图像分层,其中,m为不小于0的整数。
在112处,图像金字塔的底层是图像分层0,图像分层0具有图像I和图像J的最高分辨率。例如,图像I和图像J的最高分辨率可以分别是图像I和图像J的原始图像的最高分辨率。
可选地,图像I和图像J的图像分层0分别包括图像I和图像J的部分或全部像素位置的像素值。可选地,像素值可以是图像I和图像J的灰度值。可选地,像素值也可以包括像素位置的亮度。
可选地,图像I和图像J中的像素位置的像素值可以分别表示为I(x,y)以及J(x,y),其中,x和y是像素位置的像素坐标。
例如,对于图像分层0,左上角的像素坐标向量可以表示为[0 0]T,图像I和图像J的宽度和高度是相同的,并且可以表示为nx和ny(均为不小于1的整数)。右下角的像素坐标向量可以表示为[nx-1ny-1]T。应理解,图像I和图像J可以不具有相同的高度和宽度,此处假设具有相同的高度和宽度仅仅是为了描述的方便。
可选地,可以以递归的方式建立图像I和图像J的图像金字塔。图像分层1基于图像分层0计算得到,图像分层2基于图像分层1计算得到,以此类推。可选地,可以采用平滑或降采样的方式,从下到上建立金字塔的图像分层。
例如,令L=1,2,...代表金字塔的层数,IL-1是第L-1层的图像。nx
L-1和ny
L-1分别是IL-1层的宽度和高度。其中,图像分层IL可以按照以下公式1求得:
应理解,上述式1所示的图像分层的确定方法仅仅是一种实现方式,本申请实施例还可以有其他的实现方式。
例如,可以对式1采样的3x3的kernal权重矩阵的取值进行改变,或者也可以采用5x5,7x7的权重矩阵。
通过这样的平滑和/或降采样,层L可以包括层L-1的大约1/4的像素位置。例如,对于尺寸为1920x 1024的图像(I0具有相同的尺寸),图像分层I1,I2,I3和I4的尺寸分别为960x 512,480x 256,240x 128和120x64。其中,对于尺寸为1920x 1024的图像J,图像J1,J2,J3和J4分别具有相同的尺寸。通过递归处理,形成图像I和图像J的图像金字塔。图像金字塔可以包括2-10层。在图1中,图像金字塔102和104具有相同的层数,包括在112处的层0,在114处的层1,在116处的层2,和在118处的顶层Lm。
可选地,可以基于建立图像的图像金字塔的递归方法,可以确定特征点在不同层的像素位置。例如,基于公式(1),可以在层1识别出u1 160,在层2识别出u2 170,直到得到特征点在所有层处的像素位置。
应理解,公式(1)示出利用一种降采样的方法建立图像金字塔,本申请实施例也可以采用其他降采样方法和平滑方法建立图像金字塔。例如,可以建立高斯金字塔,拉普拉斯金字塔,可控金字塔。本申请的实施例不限定建立图像I和/或图像J的金字塔的方式。
一旦建立了图像I和图像J的图像金字塔,可以执行特征跟踪。例如,特征跟踪可以从图像I和图像J的最顶层开始,最顶层的结果可以包括光流信息。该结果可以用于下一层的特征跟踪。递归处理继续直到得到在层0处的光流值d=[dx dy]T,因此,如图1所示,特征点在图像J中被识别为v=u+d。
以上已结合图1描述了根据本申请实施例的图像金字塔的建立方式,以下将以特征跟踪为例,结合图2所示的特征跟踪系统描述如何存储和使用建立的图像金字塔。
图2示出了根据本申请实施例的特征跟踪系统200。
如图2所示,该特征跟踪系统200包括外部存储设备210和电子设备220。该电子设备220包括内部存储设备222,特征点控制单元224和图像金字塔处理单元226。内部存储设备222,和处理设备224的每一个或组合可以由至少一个电子电路实现。可选地,电子设备220可以通过使用通用处理器实现。可选地,电子设备220可以由一个或多个应用型专用集成电路(application-specific integrated circuit,ASIC)和现场可编程门阵列(field programmable gate array,FPGA)实现。电子设备220可以被称为视觉/图形加速器,或可以简单地称为加速器。
电子设备220耦合至外部存储设备210。外部存储设备210可以存储至少一个图像中每个图像的图像金字塔(也可以称为图像金字塔数据)。外部存储设备210可以足够大以用于存储多个图像金字塔。
为了简单的描述,以下以从第一图像到第二图像的图像跟踪为例进行说明。其中,第一图像也可以被称为图像I,以及第二图像也可以称为图像J。对于在图像J中,对图像I中的多个特征点进行特征跟踪,本申请的实施例用于在图像J中识别多个特征点的匹配像素点。本申请实施例也可以应用于其它跟踪场景,而不是从图像I到图像J的特征跟踪,例如,本申请实施例可以应用于从第一图像到一系列其他图像的特征跟踪,或从视频帧到一个或多个其他视频帧的跟踪。
外部存储设备210可以与内部存储设备222耦合或通信,内部存储设备222可以包括一个比外部存储设备210小的存储设备,用于特征跟踪以及存储来自外部存储设备210的部分金字塔数据。可选地,一旦生成图像金字塔,图像金字塔数据的一层或多层可以不存储在外部存储设备210中,而是直接存储在内部存储设备222中。内部存储设备224可以将来自不同图像金字塔的数据存储在内部存储器130中的不同位置。
外部存储设备210和内部存储设备222可以包括多种动态随机存储器(dynamic random-access memory,DRAM)中的一种或多种,诸如,双数据速率同步DRAM(DDR DRAM,或简称为DDR),单数据速率(single data
rate,SDR)SDRAM,静态RAM(static RAM,SRAM),持久性海量存储设备(例如,闪存,磁盘等),持久性存储器(诸如,相变存储器(Phase Change Memory,PCM),忆阻器,自旋转移矩(spin-transfer torque,STT)RAM。
可选地,外部存储设备210包括DDR SDRAM,以存储金字塔数据。
可选地,内部存储设备222包括缓存,以存储图像金字塔数据(例如,从外部存储设备210获取的部分图像金字塔数据,或图像金字塔生成时的一层或多层图像金字塔。
虽然外部存储设备210可以示出在电子设备210的外部,但是外部存储设备210也可以在电子设备210中实现。例如电子设备220也可以在包括外部存储设备210的芯片系统上实现。
内部存储设备222中的部分图像金字塔数据可以提供给处理设备224。处理设备224可以执行光流跟踪算法。可选地,处理设备224可以使用奏-卢卡斯-托马西(Kanade–Lucas–Tomasi,KLT)算法进行光流跟踪算法的实现。
可选地,处理设备224可以确定需要读取的像素点信息,并可以提供给内部存储设备222。处理设备224可以基于该像素点信息,提供正确的图像金字塔数据给内部存储设备222。
可选地,处理设备224也可以将像素点信息提供给内部存储设备222,以便于内部存储设备222可以基于该像素点信息从外部存储设备210请求以及获取正确的图像金字塔数据。
从而,可以通过第一图像和第二图像的图像金字塔数据的迭代计算,特征点跟踪系统可以获取图像J中,与图像I中的多个特征点的匹配点。
然而,电子设备220在从外部存储设备210中读取图像金字塔数据时,由于金字塔数据的数据量较大,需要较大的带宽,以及较长的读取时间。
因此,本申请实施例提供一种图像处理方法,以便于尽量解决该问题。应理解,本申请实施例的图像处理方法,可以应用于光流跟踪中,也可以应用于其他场景。
图3是根据本申请实施例的图像处理方法300的示意性流程图。该方法300包括以下内容中的至少部分内容。
在310中,构造目标图像的图像金字塔,该图像金字塔包括多个图像分层。
可选地,构造图像金字塔的方式可以采用但不限于图1所示的构造方式。
在320中,压缩该多个图像分层中的至少一个图像分层的像素点。
可选地,可以压缩构造的图像金字塔包括的图像分层中的至少部分图像分层。
可选地,在该方法300用于特征跟踪时,可以压缩图像I的图像金字塔的至少部分图像分层以及压缩图像J的图像金字塔的至少部分图像分层;也可以仅压缩图像I的至少部分图像分层,或仅压缩图像J的至少部分图像分层。
可选地,可以根据系统可用带宽和预配置的需要压缩的图像分层的信息中的至少一种,确定待压缩的图像分层。
具体地说,可以根据系统可用带宽,估计从外部存储设备读取金字塔数据时所需要占用的带宽,并依据该估计的带宽,以及每个图像分层的分辨和/或像素位置的数量等,确定需要压缩哪些图像分层,并进行压缩。
可选地,该预配置的需要压缩的图像分层的信息包括需要压缩的图像分层的分辨率和/或数量。
具体地说,可以预配置需要压缩的图像分层的分辨率和/或数量等,可以按照该预配置的信息,得到需要压缩的图像分层并进行压缩。
其中,可以预配置的需要压缩的图像分层的分辨率范围,可以直接将该范围内的所有图像分层进行压缩,也可以结合系统当前可用带宽,从该范围内选择部分图像分层进行压缩。
例如,假设预配置的需要压缩的图像分层的分辨率为960x 512和480x256,则可以直接将分辨率为960x 512和480x 256的图像分层进行压缩,或者,也可以根据系统当前可用带宽的情况,择一进行压缩。
或者,可以预配置的需要压缩的图像分层的数量范围,可以结合系统当前可用带宽,选择一定数量(属于预设的数量范围)的图像分层进行压缩。
可选地,可以压缩该多个图像分层中的分辨率最高的该至少一个图像分层的像素点。
由于分辨率较高的图像分层在读取时占用的带宽,以及读取时所花费的时间要大于分辨率较低的图像分层,因此可以压缩分辨率最高的至少一个图像分层。
例如,图像金字塔包括四个图像分层,该四个图像分层的分辨率分别为
960x 512,480x 256,240x 128和120x 64,则可以压缩分辨率为960x 512,480x 256的图像分层。
可选地,在本申请实施例中,可以按包括多个像素点的图像块,对该至少一个图像分层进行压缩,以得到多个压缩块。
该图像块可以包括多个像素点,该多个像素点可以是相邻的像素点,例如,如图4a所示,可以是图像分层中的每一行,或者,例如,如图4b所示,可以是图像金字塔的每一列,当然,也可以是N x M的块,其中,N大于1,且M大于1,例如,如图4c所示,可以是图像分层中的4x4的图像块。
可选地,可以根据像素点的存储方式,确定像素点的读取方式。
例如,如果像素点是按行进行压缩存储,则在读取时,可以按行进行读取并进行解压缩。
例如,如果像素点是按列进行压缩存储,则在读取时,可以按列进行读取并进行解压缩。
或者,也可以根据像素点的读取方式,确定像素点的压缩方式。
例如,如果像素点是按行读取的,则在压缩时,可以按行进行压缩。
例如,如果像素点是按列读取的,则在压缩时,可以按行进行压缩。
可选地,根据系统可用带宽,预设的该图像块包括的像素点的数量和该待处理窗口包括的像素点的数量中的至少一种,确定该图像块包括的像素点的数量;按包括该数量的像素点的该图像块,对该至少一个图像分层进行压缩。其中,待处理窗口包括至少一个待处理的像素点,该待处理的像素点可以用于特征跟踪。该待处理窗口可选地可以称为特征点的邻域。
在一种实现方式中,可以根据系统可用带宽和/或待处理窗口大小,确定在需要压缩的图像块的大小。
在一种实现方式中,预设的该图像块包括的像素点的数量可以是一个具体数值,可以按照具有该数量的像素点的图像块进行压缩。
在一种实现方式中,预设的该图像块包括的像素点的数量可以是一个范围,可以根据系统可用带宽和/或待处理窗口大小,从该范围内选择一个数量,可以按照具有该数量的像素点的图像块进行压缩。
可选地,根据系统可用带宽和预设的压缩率中的至少一种,确定对该图像块进行压缩的压缩率;按照确定的该压缩率,对该图像块进行压缩,以得到该压缩块。
在一种实现方式中,可以根据系统可用带宽,确定在从外部存储设备可读取的数据大小,并可以根据压缩前的数据大小,以及可读取的数据大小,确定用于压缩图像块的压缩率。
在一种实现方式中,压缩率可以是一个具体数值,可以将该数值的压缩率确定为用于压缩的图像块的压缩率。
在一种实现方式中,预配置的压缩率可以是一个范围,可以直接从该范围内随机用于压缩图像块的压缩率,也可以结合系统当前可用带宽,从该范围内选择用于压缩图像块的压缩率。
应理解,在本申请实施例中,在压缩多个图像分层时,每个图像分层相比于其他图像分层,待压缩的图像块包括的像素点的数量可以相同,也可以不相同
还应理解,在本申请实施例中,在对图像分层的像素点进行压缩时,还可以仅压缩部分的像素点。
在330中,在存储设备中存储压缩的像素点。
可选地,该第一图像分层和该第二图像分层缓存在不同的存储设备中。
可选地,该不同的存储设备具有不同的控制器。
可选地,该存储设备位于片外系统。
在340中,从该存储设备中读取压缩的至少部分像素点。
可选地,在本申请实施例中,从该存储设备中,读取至少一个该压缩块。
在一种实现方式中,根据待处理窗口在该每个图像分层中的位置,确定待读取的该压缩块,从该存储设备中,读取待读取的该压缩块。其中,待处理窗口包括至少一个待处理的像素点,该待处理的像素点可以用于特征跟踪。该待处理窗口可选地可以称为特征点的邻域。
例如,假设按照图4c的方式对某一图像分层的像素点进行压缩,则对于图5a所示的图像分层,得到压缩块可以如图5b所示。如图5a所示,如果需要读取的数据所处的列是1-2,所处的行是3-4,则按照压缩方式,可知对应的压缩块为2-1,如果需要读取的数据所处的行是6-8,所处的列是6-8,压缩方式为图4c所示的压缩方式,则按照该压缩方式,可知需要读取的压缩块为3-3,3-4,4-3和4-4。从而可以在外部存储设备中读取该压缩块。
在350中,解压缩读取的像素点,以得到解压缩的像素点。
因此,在本申请实施例中,将为目标图像构造的图像金字塔包括的至少
一个图像分层进行压缩,并存储在存储设备中,当处理到该至少一个图像分层时,可以从存储设备中读取压缩的像素点,并进行解压缩,由于压缩后的数据量小于压缩前的数据量,在读取时,需要占用的带宽较小,且可以节省时间,提高效率。
可选地,本申请实施例中的图像处理方法可以用于光流跟踪。在进行光流跟踪时,可以构造图像I和图像J的图像金字塔,可以将图像I的图像金字塔的至少部分图像分层进行压缩并存储,和/或将图像J的图像金字塔的至少部分图像分层进行压缩并存储,其中,光流跟踪的跟踪像素点在该第一图像的位置是已知的,该跟踪像素点在该第一图像的位置和灰度用于确定该跟踪像素点在该第二图像的位置,和/或,确定该跟踪像素点在该第一图像和该第二图像之间的位移矢量。
可选地,该光流跟踪通过KLT算法实现。
可选地,在方法300中提到的目标图像包括第一图像和第二图像,该图像金字塔包括该第一图像的第一图像金字塔和该第二图像的第二图像金字塔,该至少一个图像分层包括属于该第一图像金字塔的第一图像分层和属于该第二图像金字塔的第二图像分层,该第一图像分层与该第二图像分层属于同一分层层级,其中,同一分层层级的图像分层具有相同的分辨率;从该存储设备中读取压缩的至少部分像素点,可以包括:根据该第一图像的第一待处理窗口在该第一图像分层的位置,从该存储设备中读取压缩的第一像素点,以及,根据该第二图像的第二待处理窗口在该第二图像分层的位置,从该存储设备中读取压缩的第二像素点;该解压缩读取的像素点,以得到解压缩的像素点,包括:对压缩的该第一像素点进行解压缩,得到解压缩的该第一像素点,以及对压缩的该第二像素点进行解压缩,得到解压缩的该第二像素点;该方法300还包括:根据解压缩的该第一像素点中的至少部分像素点和解压缩的该第二像素点中的至少部分像素点的位置和灰度,利用层内收敛迭代的方式,确定该跟踪像素点在该第一图像分层与该第二分层之间的位移矢量;其中,该位移矢量作为该跟踪像素点在该第一图像分层和第二图像分层的下一分层层级的初始位移矢量,和/或该位移矢量的初始位移矢量为该跟踪像素点在该第一图像分层和第二图像分层的上一分层层级的位移矢量。
可选地,在本申请实施例中,可以确定解压缩的第一像素点的该部分像素点在第一方向的第一灰度梯度和在第二方向的第二灰度梯度;根据该第一
灰度梯度和该第二灰度梯度,计算梯度矩阵;根据该梯度矩阵,以及解压缩的该第一像素点中的该至少部分像素点和该第二像素点中的该至少部分像素点的像素值和位置,以层内迭代的方式,确定该跟踪像素点在该第一图像分层与该第二图像分层之间的位移矢量。
为了更加清楚地理解本申请,以下将结合图6所示的伪代码对光流跟踪进行描述。
图6示出了一种特征点跟踪的方式400的示意性图。方式400的目的是为图像I中的单个点u找到在图像J中的对应位置。
在404中,存储图像I的图像金字塔和存储J的图像金字塔,其中,图像I的图像金字塔中的至少部分图像分层在存储之前可以进行压缩,和/或图像J的图像金字塔中的至少部分图像分层在存储之前可以进行压缩。
在406中,按照公式2初始化金字塔的初始光流估计量:
以下将介绍如何在各个层处更新初始光流估计值。
在408处,外循环开始。在外循环中,从最高层开始,以递减方式在每层进行迭代处理直到最底层。在每次迭代处理中,可以确定点u在对应层的像素位置。
在410处,可以获取图像分层IL中点u的像素位置。图像IL是图像I的图像金字塔的第L层,以及图像IL中,点u的位置可以根据图2中描述的方法确定。可选地,在图像IL中,点u的像素位置可以使用以下公式3得到,其中,u0=u:
uL=[px py]T=u/2L 式(3)
在412处,可以根据点u的像素位置以及待处理窗口大小,可以确定待处理窗口所处的位置,根据待处理窗口所处的位置,确定待读取的像素点,从存储设备读取像素点,如果读取的像素点是压缩的像素点,则可以对该压缩像素点进行解压缩,以获取像素值IL(x,y)。其中,待处理窗口的大小可以是(2ωx+1)×(2ωy+1)。
在414处,可以确定读取的像素点的像素值相对x的导数。可选地,可以按照以下公式(4)计算相对x的导数。
在416处,可以确定读取的像素点的像素值相对y的导数。可选地,可以按照以下公式(5)计算y的导数。
在418处,像素值相对x和y的导数可以用于计算空间梯度矩阵G。可选地,可以按照以下公式6计算该矩阵G。
对于给定的层L,可以通过内循环,可以使用迭代的KLT光流跟踪算法进行光流跟踪。其中,在开始该内循环之前,按照式7假设初始像素位移:
在422处,对于[1K]内的给定k,计算图像差。对于给定k,可以按照以下公式8进行计算:
其中,在422之前,可以确定图像J的待处理窗口,从存储设备读取待处理窗口包括的像素点,如果读取的像素点是压缩的像素点,则可以对该压缩像素点进行解压缩,以得到像素点的像素值JL(x,y)。
在424处,基于该图像差,可以计算图像的不匹配向量。在一种实施例中,对于给定k,可以按照以下公式9计算图像不匹配向量。
在430处,确定层L的内循环的结束。
在432处,一旦内循环条件满足,既可以确定在图像分层L的光流跟踪。可选地,可以使用以下公式获取层L的光流:
在434处,可以确定L-1层的光流估计量。可选地,可以使用以下公式获取层L-1处的光流估计量。
在436处,完成层L处的外循环。继续执行下一层(L-1层)的操作,即重复重复执行在310-334处的操作。
在438处,一旦在308处的外循环条件得到满足,这意味最底层,即层0处的计算已经完成。可以基于在层0处的光流估计值以及在层0处的光流,可以获取在层0处的最终的光流向量d。可选地,可以使用以下公式获取最终的光流向量d:
d=go+do 式(14)
在440处,可以基于最终的光流向量,确定在图像I中的特征点u在图像J中的匹配点。可选地,可以使用以下公式获取匹配点u的像素位置:
v=u+d 式(15)
因此,通过402-440的操作,可以确定在图像I中的特征点u在图像J中的匹配点v。应理解,图6所示的伪代码仅仅是本申请的一种实现方式,本申请还可以有其它实现方式,为了简洁,在此不再赘述。
以上已将结合方法200和300描述了一种图像处理方法,可以提升系统的性能,提高系统的处理效率。但是,本申请实施例还可以有其它方法实现此目的,例如,在不对图像I和图像J的图像金字塔进行压缩,而是将图像I和图像I分别存储在不同的存储器,进一步可以具有不同的控制器。例如,可以使用高速外部缓存设备,无需对图像I和图像J的图像金字塔进行压缩。例如,将金字塔数据存储在片内系统。例如,可以采用多个KLT处理单元并行处理。
图7是根据本申请实施例的图像处理设备500的示意性框图。如图7所示,该设备500包括:构造单元510、压缩单元520、存储单元530、读取单元540和解压缩单元550。
构造单元510用于:构造目标图像的图像金字塔,所述图像金字塔包括多个图像分层;
压缩单元520用于:压缩所述多个图像分层中的至少一个图像分层的像素点;
存储单元530用于:在存储设备中存储压缩的像素点;
读取单元540用于:从所述存储设备中读取压缩的至少部分像素点;
解压缩单元550用于:解压缩读取的像素点,以得到解压缩的像素点。
可选地,所述压缩单元520进一步用于:
按包括多个像素点的图像块,对所述至少一个图像分层进行压缩,以得到多个压缩块。
可选地,所述读取单元540进一步用于:
从所述存储设备中,读取至少一个所述压缩块。
可选地,所述读取单元540进一步用于:
根据待处理窗口在所述每个图像分层中的位置,确定待读取的所述压缩块;
从所述存储设备中,读取待读取的所述压缩块。
可选地,所述压缩单元520进一步用于:
根据系统可用带宽,预设的所述图像块包括的像素点的数量和所述待处理窗口包括的像素点的数量中的至少一种,确定所述图像块包括的像素点的数量;
按包括所述数量的像素点的所述图像块,对所述至少一个图像分层进行压缩。
可选地,所述压缩单元520进一步用于:
根据系统可用带宽和预设的压缩率中的至少一种,确定对所述图像块进行压缩的压缩率;
按照确定的所述压缩率,对所述图像块进行压缩,以得到所述压缩块。
可选地,所述压缩单元520进一步用于:
根据系统可用带宽和预配置的需要压缩的图像分层的信息中的至少一
种,确定待压缩的图像分层;
压缩所述待压缩的图像分层。
可选地,所述预配置的需要压缩的图像分层的信息包括需要压缩的图像分层的分辨率。
可选地,所述压缩单元520进一步用于:
压缩所述多个图像分层中的分辨率最高的所述至少一个图像分层的像素点。
可选地,所述图像处理设备500用于光流跟踪,所述目标图像包括第一图像和/或第二图像,其中,光流跟踪的跟踪像素点在所述第一图像的位置是已知的,所述跟踪像素点在所述第一图像的位置和灰度用于确定所述跟踪像素点在所述第二图像的位置,和/或,确定所述跟踪像素点在所述第一图像和所述第二图像之间的位移矢量。
可选地,所述光流跟踪通过奏-卢卡斯-托马西KLT算法实现。
可选地,所述目标图像包括第一图像和第二图像,所述图像金字塔包括所述第一图像的第一图像金字塔和所述第二图像的第二图像金字塔,所述至少一个图像分层包括属于所述第一图像金字塔的第一图像分层和属于所述第二图像金字塔的第二图像分层,所述第一图像分层与所述第二图像分层属于同一分层层级,其中,同一分层层级的图像分层具有相同的分辨率;
所述读取单元540进一步用于:
根据所述第一图像的第一待处理窗口在所述第一图像分层的位置,从所述存储设备中读取压缩的第一像素点,以及,根据所述第二图像的第二待处理窗口在所述第二图像分层的位置,从所述存储设备中读取压缩的第二像素点;
所述解压缩单元550进一步用于:
对压缩的所述第一像素点进行解压缩,得到解压缩的所述第一像素点,以及对压缩的所述第二像素点进行解压缩,得到解压缩的所述第二像素点;
可选地,如图6所示,所述设备500还包括确定单元560,用于:
根据解压缩的所述第一像素点中的至少部分像素点和解压缩的所述第二像素点中的至少部分像素点的位置和灰度,利用层内收敛迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二分层之间的位移矢量;
其中,所述位移矢量作为所述跟踪像素点在所述第一图像分层和第二图
像分层的下一分层层级的初始位移矢量,和/或所述位移矢量的初始位移矢量为所述跟踪像素点在所述第一图像分层和第二图像分层的上一分层层级的位移矢量。
可选地,所述确定单元560进一步用于:
确定解压缩的第一像素点的所述部分像素点在第一方向的第一灰度梯度和在第二方向的第二灰度梯度;
根据所述第一灰度梯度和所述第二灰度梯度,计算梯度矩阵;
根据所述梯度矩阵,以及解压缩的所述第一像素点中的所述至少部分像素点和所述第二像素点中的所述至少部分像素点的像素值和位置,以层内迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二图像分层之间的位移矢量。
可选地,所述第一图像分层和所述第二图像分层缓存在不同的存储设备中。
可选地,所述不同的存储设备具有不同的控制器。
可选地,所述存储设备位于片外系统。
应理解,该图像处理设备500可以实现方法300或400,为了简洁,在此不再赘述。
图8是根据本申请实施例的图像处理设备600的示意性框图
可选地,该图像处理设备600可以包括多个不同的部件,这些部件可以作为集成电路(integrated circuits,ICs),或集成电路的部分,离散的电子设备,或其它适用于电路板(诸如主板,或附加板)的模块,也可以作为并入计算机系统的部件。
可选地,该图像处理设备可以包括处理器610和与处理器610耦合的存储介质620。
处理器610可以包括一个或多个通用处理器,诸如中央处理单元(central processing unit,CPU),或处理设备等。具体地,该处理器610可以是复杂指令集处理(complex instruction set computing,CISC)微处理器,超长指令字(very long instruction word,VLIW)微处理器,实现多个指令集组合的微处理器。该处理器也可以是一个或多个专用处理器,诸如应用专用集成电路(application specific integrated circuit,ASIC),现场可编程门阵列(field programmable gate array,FPGA),数字信号处理器(digital signal processor,
DSP)。
处理器610可以与存储介质620通信。该存储介质620可以为磁盘、光盘、只读存储器(read only memory,ROM),闪存,相变存储器。该存储介质620可以存储有处理器存储的指令,和/或,可以缓存一些从外部存储设备存储的信息,例如,从外部存储设备读取的金字塔的图像分层的像素信息。
可选地,除了处理器620和存储介质620,图像处理设备可以包括显示控制器和/或显示设备单元630,收发器640,视频输入输出单元650,音频输入输出单元660,其他输入输出单元670。图像处理设备600包括的这些部件可以通过总线或内部连接互联。
可选地,该收发器640可以是有线收发器或无线收发器,诸如,WIFI收发器,卫星收发器,蓝牙收发器,无线蜂窝电话收发器或其组合等。
可选地,视频输入输出单元650可以包括诸如摄像机的图像处理子系统,其包括光传感器,电荷耦合器件(charged coupled device,CCD)或互补金属氧化物半导体(complementary metal-oxide semiconductor,CMOS)光传感器,以用于实现拍摄功能。
可选地,该音频输入输出单元660可以包括扬声器,话筒,听筒等。
可选地,其他输入输出设备670可以包括存储设备,universal serial bus(USB)端口,串行端口,并行端口,打印机,网络接口等。
可选地,该图像处理设备600可以执行方法300或400所示的操作,为了简洁,在此不再赘述。
可选地,图像处理设备500或600可以位于可移动设备中。可移动设备可以在任何合适的环境下移动,例如,空气中(例如,定翼飞机、旋翼飞机,或既没有定翼也没有旋翼的飞机)、水中(例如,轮船或潜水艇)、陆地上(例如,汽车或火车)、太空(例如,太空飞机、卫星或探测器),以及以上各种环境的任何组合。可移动设备可以是飞机,例如无人机(Unmanned Aerial Vehicle,简称为“UAV”)。在一些实施例中,可移动设备可以承载生命体,例如,人或动物。
图9是根据本申请实施例的可移动设备700的示意性框图。如图9所示,可移动设备700包括载体710和负载720。图9中将可移动设备描述为无人机仅仅是为了描述方面。负载720可以不通过载体710连接到可移动设备上。可移动设备700还可以包括动力系统730、传感系统740和通信系统750和
图像处理设备760。
动力系统730可以包括电子调速器(简称为电调)、一个或多个螺旋桨以及与一个或多个螺旋桨相对应的一个或多个电机。电机和螺旋桨设置在对应的机臂上;电子调速器用于接收飞行控制器产生的驱动信号,并根据驱动信号提供驱动电流给电机,以控制电机的转速和/或转向。电机用于驱动螺旋桨旋转,从而为UAV的飞行提供动力,该动力使得UAV能够实现一个或多个自由度的运动。在某些实施例中,UAV可以围绕一个或多个旋转轴旋转。例如,上述旋转轴可以包括横滚轴、平移轴和俯仰轴。应理解,电机可以是直流电机,也可以交流电机。另外,电机可以是无刷电机,也可以有刷电机。
传感系统740用于测量UAV的姿态信息,即UAV在空间的位置信息和状态信息,例如,三维位置、三维角度、三维速度、三维加速度和三维角速度等。传感系统例如可以包括陀螺仪、电子罗盘、惯性测量单元(Inertial Measurement Unit,简称为“IMU”)、视觉传感器、全球定位系统(Global Positioning System,简称为“GPS”)和气压计等传感器中的至少一种。飞行控制器用于控制UAV的飞行,例如,可以根据传感系统测量的姿态信息控制UAV的飞行。应理解,飞行控制器可以按照预先编好的程序指令对UAV进行控制,也可以通过响应来自操纵设备的一个或多个控制指令对UAV进行控制。
通信系统750能够与一个具有通信系统770的终端设备780通过无线信号790进行通信。通信系统750和通信系统770可以包括多个用于无线通信的发射机、接收机和/或收发机。这里的无线通信可以是单向通信,例如,只能是可移动设备700向终端设备780发送数据。或者无线通信可以是双向通信,数据即可以从可移动设备700发送给终端设备780,也可以由终端设备780发送给可移动设备700。
可选地,终端设备780能够提供针对于一个或多个可移动设备700、载体710和负载720的控制数据,并能接收可移动设备700、载体710和负载720发送的信息。终端设备780提供的控制数据能够用于控制一个或多个可移动设备700、载体710和负载720的状态。可选地,载体710和负载720中包括用于与终端设备780进行通信的通信模块。
可以理解的是,图9所示出的可移动设备包括的图像处理设备660能够执行方法300和400,为了简洁,在此不再赘述。
以上所述,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本申请揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应所述以权利要求的保护范围为准。
Claims (32)
- 一种图像处理方法,其特征在于,包括:构造目标图像的图像金字塔,所述图像金字塔包括多个图像分层;压缩所述多个图像分层中的至少一个图像分层的像素点;在存储设备中存储压缩的像素点;从所述存储设备中读取压缩的至少部分像素点;解压缩读取的像素点,以得到解压缩的像素点。
- 根据权利要求1所述的方法,其特征在于,所述压缩所述多个图像分层中的至少一个图像分层的像素点,包括:按包括多个像素点的图像块,对所述至少一个图像分层进行压缩,以得到多个压缩块。
- 根据权利要求2所述的方法,其特征在于,所述从所述存储设备中读取压缩的至少部分像素点,包括:从所述存储设备中,读取至少一个所述压缩块。
- 根据权利要求3所述的方法,其特征在于,所述从所述存储设备中,读取至少一个所述压缩块,包括:根据待处理窗口在所述每个图像分层中的位置,确定待读取的所述压缩块;从所述存储设备中,读取待读取的所述压缩块。
- 根据权利要求2至4中任一项所述的方法,其特征在于,所述按包括多个像素点的图像块,对所述至少一个图像分层进行压缩,以得到多个压缩块,包括:根据系统可用带宽,预设的所述图像块包括的像素点的数量和所述待处理窗口包括的像素点的数量中的至少一种,确定所述图像块包括的像素点的数量;按包括所述数量的像素点的所述图像块,对所述至少一个图像分层进行压缩。
- 根据权利要求2至5中任一项所述的方法,其特征在于,所述按包括多个像素点的图像块,对所述至少一个图像分层进行压缩,以得到多个压缩块,包括:根据系统可用带宽和预设的压缩率中的至少一种,确定对所述图像块进 行压缩的压缩率;按照确定的所述压缩率,对所述图像块进行压缩,以得到所述压缩块。
- 根据权利要求1至6中任一项所述的方法,其特征在于,所述压缩所述多个图像分层中的至少一个图像分层的像素点,包括:根据系统可用带宽和预配置的需要压缩的图像分层的信息中的至少一种,确定待压缩的图像分层;压缩所述待压缩的图像分层。
- 根据权利要求7所述的方法,其特征在于,所述预配置的需要压缩的图像分层的信息包括需要压缩的图像分层的分辨率。
- 根据权利要求1至8中任一项所述的方法,其特征在于,所述压缩所述多个图像分层中的至少一个图像分层的像素点,包括:压缩所述多个图像分层中的分辨率最高的所述至少一个图像分层的像素点。
- 根据权利要求1至9中任一项所述的方法,其特征在于,所述图像处理方法用于光流跟踪,所述目标图像包括第一图像和/或第二图像,其中,光流跟踪的跟踪像素点在所述第一图像的位置是已知的,所述跟踪像素点在所述第一图像的位置和灰度用于确定所述跟踪像素点在所述第二图像的位置,和/或,确定所述跟踪像素点在所述第一图像和所述第二图像之间的位移矢量。
- 根据权利要求10所述的方法,其特征在于,所述光流跟踪通过奏-卢卡斯-托马西KLT算法实现。
- 根据权利要求10或11所述的方法,其特征在于,所述目标图像包括第一图像和第二图像,所述图像金字塔包括所述第一图像的第一图像金字塔和所述第二图像的第二图像金字塔,所述至少一个图像分层包括属于所述第一图像金字塔的第一图像分层和属于所述第二图像金字塔的第二图像分层,所述第一图像分层与所述第二图像分层属于同一分层层级,其中,同一分层层级的图像分层具有相同的分辨率;所述从所述存储设备中读取压缩的至少部分像素点,包括:根据所述第一图像的第一待处理窗口在所述第一图像分层的位置,从所述存储设备中读取压缩的第一像素点,以及,根据所述第二图像的第二待处理窗口在所述第二图像分层的位置,从所述存储设备中读取压缩的第二像素 点;所述解压缩读取的像素点,以得到解压缩的像素点,包括:对压缩的所述第一像素点进行解压缩,得到解压缩的所述第一像素点,以及对压缩的所述第二像素点进行解压缩,得到解压缩的所述第二像素点;所述方法还包括:根据解压缩的所述第一像素点中的至少部分像素点和解压缩的所述第二像素点中的至少部分像素点的位置和灰度,利用层内收敛迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二分层之间的位移矢量;其中,所述位移矢量作为所述跟踪像素点在所述第一图像分层和第二图像分层的下一分层层级的初始位移矢量,和/或所述位移矢量的初始位移矢量为所述跟踪像素点在所述第一图像分层和第二图像分层的上一分层层级的位移矢量。
- 根据权利要求12所述的方法,其特征在于,所述根据解压缩的所述第一像素点中的至少部分像素点和解压缩的所述第二像素点中的至少部分像素点的位置和灰度,利用层内收敛迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二分层之间的位移矢量,包括:确定解压缩的第一像素点的所述部分像素点在第一方向的第一灰度梯度和在第二方向的第二灰度梯度;根据所述第一灰度梯度和所述第二灰度梯度,计算梯度矩阵;根据所述梯度矩阵,以及解压缩的所述第一像素点中的所述至少部分像素点和所述第二像素点中的所述至少部分像素点的像素值和位置,以层内迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二图像分层之间的位移矢量。
- 根据权利要求12或13所述的方法,所述第一图像分层和所述第二图像分层缓存在不同的存储设备中。
- 根据权利要求14所述的方法,其特征在于,所述不同的存储设备具有不同的控制器。
- 根据权利要求1至15中任一项所述的方法,其特征在于,所述存储设备位于片外系统。
- 一种图像处理设备,其特征在于,包括:构造单元,用于构造目标图像的图像金字塔,所述图像金字塔包括多个 图像分层;压缩单元,用于压缩所述多个图像分层中的至少一个图像分层的像素点;存储单元,用于在存储设备中存储压缩的像素点;读取单元,用于从所述存储设备中读取压缩的至少部分像素点;解压缩单元,用于解压缩读取的像素点,以得到解压缩的像素点。
- 根据权利要求17所述的设备,其特征在于,所述压缩单元进一步用于:按包括多个像素点的图像块,对所述至少一个图像分层进行压缩,以得到多个压缩块。
- 根据权利要求18所述的设备,其特征在于,所述读取单元进一步用于:从所述存储设备中,读取至少一个所述压缩块。
- 根据权利要求19所述的设备,其特征在于,所述读取单元进一步用于:根据待处理窗口在所述每个图像分层中的位置,确定待读取的所述压缩块;从所述存储设备中,读取待读取的所述压缩块。
- 根据权利要求18至20中任一项所述的设备,其特征在于,所述压缩单元进一步用于:根据系统可用带宽,预设的所述图像块包括的像素点的数量和所述待处理窗口包括的像素点的数量中的至少一种,确定所述图像块包括的像素点的数量;按包括所述数量的像素点的所述图像块,对所述至少一个图像分层进行压缩。
- 根据权利要求18至21中任一项所述的设备,其特征在于,所述压缩单元进一步用于:根据系统可用带宽和预设的压缩率中的至少一种,确定对所述图像块进行压缩的压缩率;按照确定的所述压缩率,对所述图像块进行压缩,以得到所述压缩块。
- 根据权利要求17至22中任一项所述的设备,其特征在于,所述压 缩单元进一步用于:根据系统可用带宽和预配置的需要压缩的图像分层的信息中的至少一种,确定待压缩的图像分层;压缩所述待压缩的图像分层。
- 根据权利要求23所述的设备,其特征在于,所述预配置的需要压缩的图像分层的信息包括需要压缩的图像分层的分辨率。
- 根据权利要求17至24中任一项所述的设备,其特征在于,所述压缩单元进一步用于:压缩所述多个图像分层中的分辨率最高的所述至少一个图像分层的像素点。
- 根据权利要求17至25中任一项所述的设备,其特征在于,所述图像处理设备用于光流跟踪,所述目标图像包括第一图像和/或第二图像,其中,光流跟踪的跟踪像素点在所述第一图像的位置是已知的,所述跟踪像素点在所述第一图像的位置和灰度用于确定所述跟踪像素点在所述第二图像的位置,和/或,确定所述跟踪像素点在所述第一图像和所述第二图像之间的位移矢量。
- 根据权利要求26所述的设备,其特征在于,所述光流跟踪通过奏-卢卡斯-托马西KLT算法实现。
- 根据权利要求26或27所述的设备,其特征在于,所述目标图像包括第一图像和第二图像,所述图像金字塔包括所述第一图像的第一图像金字塔和所述第二图像的第二图像金字塔,所述至少一个图像分层包括属于所述第一图像金字塔的第一图像分层和属于所述第二图像金字塔的第二图像分层,所述第一图像分层与所述第二图像分层属于同一分层层级,其中,同一分层层级的图像分层具有相同的分辨率;所述读取单元进一步用于:根据所述第一图像的第一待处理窗口在所述第一图像分层的位置,从所述存储设备中读取压缩的第一像素点,以及,根据所述第二图像的第二待处理窗口在所述第二图像分层的位置,从所述存储设备中读取压缩的第二像素点;所述解压缩单元进一步用于:对压缩的所述第一像素点进行解压缩,得到解压缩的所述第一像素点, 以及对压缩的所述第二像素点进行解压缩,得到解压缩的所述第二像素点;所述设备还包括确定单元,用于:根据解压缩的所述第一像素点中的至少部分像素点和解压缩的所述第二像素点中的至少部分像素点的位置和灰度,利用层内收敛迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二分层之间的位移矢量;其中,所述位移矢量作为所述跟踪像素点在所述第一图像分层和第二图像分层的下一分层层级的初始位移矢量,和/或所述位移矢量的初始位移矢量为所述跟踪像素点在所述第一图像分层和第二图像分层的上一分层层级的位移矢量。
- 根据权利要求28所述的设备,其特征在于,所述确定单元进一步用于:确定解压缩的第一像素点的所述部分像素点在第一方向的第一灰度梯度和在第二方向的第二灰度梯度;根据所述第一灰度梯度和所述第二灰度梯度,计算梯度矩阵;根据所述梯度矩阵,以及解压缩的所述第一像素点中的所述至少部分像素点和所述第二像素点中的所述至少部分像素点的像素值和位置,以层内迭代的方式,确定所述跟踪像素点在所述第一图像分层与所述第二图像分层之间的位移矢量。
- 根据权利要求28或29所述的设备,所述第一图像分层和所述第二图像分层缓存在不同的存储设备中。
- 根据权利要求30所述的设备,其特征在于,所述不同的存储设备具有不同的控制器。
- 根据权利要求17至31中任一项所述的设备,其特征在于,所述存储设备位于片外系统。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2017/082026 WO2018195819A1 (zh) | 2017-04-26 | 2017-04-26 | 图像处理方法和设备 |
CN201780004578.6A CN108476321A (zh) | 2017-04-26 | 2017-04-26 | 图像处理方法和设备 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2017/082026 WO2018195819A1 (zh) | 2017-04-26 | 2017-04-26 | 图像处理方法和设备 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018195819A1 true WO2018195819A1 (zh) | 2018-11-01 |
Family
ID=63266506
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2017/082026 WO2018195819A1 (zh) | 2017-04-26 | 2017-04-26 | 图像处理方法和设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN108476321A (zh) |
WO (1) | WO2018195819A1 (zh) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102646269A (zh) * | 2012-02-29 | 2012-08-22 | 中山大学 | 一种基于拉普拉斯金字塔的图像处理方法及其装置 |
CN102800094A (zh) * | 2012-07-13 | 2012-11-28 | 南京邮电大学 | 一种快速彩色图像分割方法 |
CN102890949A (zh) * | 2011-07-21 | 2013-01-23 | 索尼公司 | 标记信息记录装置和方法、标记信息呈现装置和方法 |
CN104699826A (zh) * | 2014-06-10 | 2015-06-10 | 霍亮 | 一种影像数据的金字塔层式存储方法及空间数据库系统 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN100385453C (zh) * | 2003-12-16 | 2008-04-30 | 富可视公司 | 处理图像数据的方法 |
CN102186067B (zh) * | 2011-03-31 | 2013-10-09 | 深圳超多维光电子有限公司 | 一种图像帧的传输方法和装置、显示方法和系统 |
US9014421B2 (en) * | 2011-09-28 | 2015-04-21 | Qualcomm Incorporated | Framework for reference-free drift-corrected planar tracking using Lucas-Kanade optical flow |
CN103177455B (zh) * | 2013-03-20 | 2016-04-20 | 南京理工大学 | 基于多核dsp的klt运动目标跟踪算法的实现方法 |
CN103426184B (zh) * | 2013-08-01 | 2016-08-10 | 华为技术有限公司 | 一种光流跟踪方法和装置 |
CN104978728B (zh) * | 2014-04-08 | 2017-11-14 | 南京理工大学 | 一种光流法的图像匹配系统 |
CN105528460A (zh) * | 2016-01-12 | 2016-04-27 | 中国测绘科学研究院 | 一种瓦片金字塔模型的建立方法及瓦片读取方法 |
CN105787901B (zh) * | 2016-03-21 | 2018-07-24 | 昆明理工大学 | 一种用于太阳高分辨图像序列中相邻两帧间的多尺度速度场测量方法 |
-
2017
- 2017-04-26 CN CN201780004578.6A patent/CN108476321A/zh active Pending
- 2017-04-26 WO PCT/CN2017/082026 patent/WO2018195819A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102890949A (zh) * | 2011-07-21 | 2013-01-23 | 索尼公司 | 标记信息记录装置和方法、标记信息呈现装置和方法 |
CN102646269A (zh) * | 2012-02-29 | 2012-08-22 | 中山大学 | 一种基于拉普拉斯金字塔的图像处理方法及其装置 |
CN102800094A (zh) * | 2012-07-13 | 2012-11-28 | 南京邮电大学 | 一种快速彩色图像分割方法 |
CN104699826A (zh) * | 2014-06-10 | 2015-06-10 | 霍亮 | 一种影像数据的金字塔层式存储方法及空间数据库系统 |
Also Published As
Publication number | Publication date |
---|---|
CN108476321A (zh) | 2018-08-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11024005B2 (en) | Optical flow tracking device and method | |
US20200389595A1 (en) | Systems and methods for rolling shutter correction | |
US11100652B2 (en) | Method and system for feature tracking using image pyramids | |
US11057604B2 (en) | Image processing method and device | |
WO2018090308A1 (en) | Enhanced localization method and apparatus | |
WO2019104571A1 (zh) | 图像处理方法和设备 | |
CN110296717B (zh) | 一种事件数据流的处理方法及计算设备 | |
CN112789655A (zh) | 用于标定惯性测试单元和相机的系统和方法 | |
GB2585645A (en) | Computer vision method and system | |
CN107749069B (zh) | 图像处理方法、电子设备和图像处理系统 | |
WO2021217371A1 (zh) | 可移动平台的控制方法和装置 | |
WO2021081774A1 (zh) | 一种参数优化方法、装置及控制设备、飞行器 | |
JP2023505987A (ja) | 人間関節を用いた無人航空機上のカメラの校正 | |
CN109792484A (zh) | 无人自主飞行器中的图像处理 | |
KR20220078283A (ko) | 뉴럴 네트워크 프로세서를 구비하는 이미지 처리 장치 및 이의 동작 방법 | |
WO2019019172A1 (en) | ADAPTIVE IMAGE PROCESSING IN A ROBOTIC VEHICLE | |
CN115668374A (zh) | 存储器中的运动传感器 | |
CN110800023A (zh) | 图像处理方法和设备、摄像装置以及无人机 | |
WO2018195819A1 (zh) | 图像处理方法和设备 | |
WO2021056503A1 (zh) | 可移动平台的定位方法、装置、可移动平台及存储介质 | |
CN115294280A (zh) | 三维重建方法、装置、设备、存储介质和程序产品 | |
CN108701348A (zh) | 处理图像的方法、集成电路、处理器、系统和可移动设备 | |
US11138052B2 (en) | System and method for supporting data communication in a movable platform | |
WO2021035749A1 (zh) | 三维重建模型的优化方法、设备和可移动平台 | |
WO2021077427A1 (zh) | 图像处理方法、设备和可移动平台 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17907370 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 17907370 Country of ref document: EP Kind code of ref document: A1 |