WO2023272652A1

WO2023272652A1 - Image preprocessing method and apparatus, computer device, and storage medium

Info

Publication number: WO2023272652A1
Application number: PCT/CN2021/103831
Authority: WO
Inventors: 卢浪平
Original assignee: 东莞市小精灵教育软件有限公司
Priority date: 2021-06-30
Filing date: 2021-06-30
Publication date: 2023-01-05

Abstract

Disclosed in embodiments of the present application is an image preprocessing method. The method comprises: obtaining an image to be processed, the format of the image to be processed being YUV data; performing, by using a libyuv library, transform processing on the image to be processed to obtain a transformed image in a format of YUV data; and performing format conversion on the transformed image by using a preset computing engine of an Android bottom layer to obtain a target image in a bitmap format. By using the libyuv library and the computing engine which are better in performance and higher in processing speed, the speed of converting the YUV data into the bitmap required by a service is greatly increased, such that the response speed of image preprocessing is increased, and the efficiency of image preprocessing is improved. In addition, further provided are an image preprocessing apparatus, a computer device, and a storage medium.

Description

Image preprocessing method, device, computer equipment and storage medium

technical field

The present application relates to the field of computer technology, and in particular to an image preprocessing method, device, computer equipment and storage medium.

Background technique

With the continuous development of science and technology, some electronic devices such as point readers, tutoring machines, etc. have been widely used. However, for the use of electronic devices such as point readers, image processing will be involved. For example, the camera preprocesses the preview frame image data. The traditional method is to use the OpenCV algorithm to convert YUV data (a color coding method, also known as YCrCb) Converting to the bitmap (bitmap file) required by the business, that is, to process the data of each pixel one by one for the YUV data. The amount of processed data is large and takes about 400ms, which affects the processing speed, resulting in long processing time and reducing the image quality. The speed of preprocessing affects user experience.

application content

Based on this, it is necessary to propose an image preprocessing method, device, computer equipment, and storage medium capable of accelerating image processing to address the above problems.

An image preprocessing method, the method comprising:

Obtain the image to be processed, the format of the image to be processed is YUV data;

Utilize libyuv storehouse to carry out transformation processing to described image to be processed, obtain the transformed image that format is YUV data;

Using the preset calculation engine at the bottom of the android to perform format conversion on the transformed image to obtain a target image in bitmap format.

An image preprocessing device, the device comprising:

An acquisition module, configured to acquire an image to be processed, the format of the image to be processed is YUV data;

Transformation module, for utilizing libyuv storehouse to carry out transformation processing to described image to be processed, obtains the transformed image that format is YUV data;

The conversion module is used to convert the format of the converted image by using the preset calculation engine at the bottom of android to obtain the target image in bitmap format.

A computer device comprising a memory and a processor, the memory stores computer-readable instructions, the computer-readable instructions, when executed by the processor, cause the processor to perform the following steps:

A computer-readable medium, storing computer-readable instructions, which, when executed by a processor, cause the processor to perform the following steps:

The above-mentioned image preprocessing method, system, computer equipment and storage medium obtain the image to be processed, the format of the image to be processed is YUV data; the image to be processed is transformed using the libyuv library, and the format obtained is the transformation of YUV data Image: using the preset computing engine at the bottom of the android to convert the format of the converted image to obtain a target image in bitmap format. By using the libyuv library and computing engine with better performance and higher processing speed, the speed of converting YUV data into bitmaps required by business is greatly improved, thereby improving the response speed of image preprocessing and improving the efficiency of image preprocessing.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

in:

Fig. 1 is a flowchart of an image preprocessing method in an embodiment;

Fig. 2 is a flowchart of a method for transforming and processing an image to be processed in an embodiment;

Fig. 3 is a flowchart of a method for transforming and processing an image to be processed in another embodiment;

Fig. 4 is a structural block diagram of an image preprocessing device in an embodiment;

Figure 5 is a block diagram of a computer device in one embodiment.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

As shown in FIG. 1 , in one embodiment, an image preprocessing method is provided, and the image preprocessing method can be applied to a terminal or a server, and this embodiment is described by taking the application to a server as an example. The image preprocessing method specifically includes the following steps:

Step 102, acquire the image to be processed, the format of the image to be processed is YUV data.

Wherein, the image to be processed refers to an image that requires format conversion, for example, a preview of an image to be processed by a user in a point reader. Specifically, the image to be processed can be obtained by shooting and collecting with a camera, and the format of the image to be processed is YUV data, and in a specific embodiment, the YUV data is in yuvi420 format.

Step 104, using the libyuv library to transform the image to be processed to obtain a transformed image in YUV data format.

Among them, the libyuv library is an open source library of Google (Google) that realizes mutual conversion, rotation, and scaling between various YUV and RGB. It is cross-platform, can be compiled and run on Windows, Linux, Mac, Android and other operating systems, x86, x64, arm architecture, and supports SSE, AVX, NEON and other simd instruction acceleration. Specifically, use the image transformation algorithm that comes with the libyuv library, such as rotation algorithm, compression algorithm, or mirroring algorithm, to perform rapid rotation, mirroring, and cropping on the YUV data corresponding to the image to be processed, and generate a transformed image in the format of YUV data. Understandably, since the libyuv library is the source code of libyuv, the image to be processed can be directly transformed. Compared with the traditional OPENCV transformation process, there is no need to compile the source code or use a binary library, which improves the transformation processing efficiency of the image to be processed. Furthermore, the libyuv library also supports simd instruction acceleration. Therefore, the simd instruction in the ibyuv library can also be used to accelerate the process of transformation processing, further speeding up the transformation processing speed of the image to be processed.

Step 106, using the preset computing engine at the bottom of the android to convert the format of the converted image to obtain a target image in bitmap format.

Among them, the calculation engine at the bottom of android is a data processing framework for calculating and processing data and returning appropriate calculation results according to requirements, for example, the RenderScript calculation engine. Specifically, using the preset calculation engine at the bottom of android, through the conversion format of YUV data and RGB data, the converted image in YUV format can be quickly converted into the target image in bitmap format, and then quickly displayed on the electronic device, thereby improving the user experience. Understandably, since the computing engine at the bottom of android has higher computing performance, using the preset computing engine to convert the format of the converted image greatly improves the format conversion speed of the image, thereby improving the response speed of image preprocessing and improving improve the efficiency of image preprocessing.

In a specific embodiment, it takes 30 ms to convert an image to be processed whose format is YUV data into a target image whose format is bitmap.

The above image preprocessing method, by obtaining the image to be processed, the format of the image to be processed is YUV data; using the libyuv library to transform the image to be processed, to obtain a transformed image in the format of YUV data; using the preset calculation engine at the bottom of android to convert The format of the image is converted to obtain the target image in the format of bitmap. By using the libyuv library and computing engine with better performance and higher processing speed, the speed of converting YUV data into the bitmap required by the business is greatly improved, thereby improving the efficiency of image preprocessing The response speed improves the efficiency of image preprocessing.

As shown in Figure 2, in one embodiment, the image to be processed is transformed using the libyuv library to obtain a transformed image formatted as YUV data, including:

Step 104A, dividing the YUV data of the image to be processed into multiple groups of data;

In step 104B, use the simd command in the libyuv library to perform parallel transformation processing on each group of data to obtain a transformed image.

In this embodiment, the YUV data of an image to be processed is divided into multiple groups of data, for example, the image to be processed can be divided into blocks, each group of data corresponds to an area of an image to be processed, and then the simd command is used to simultaneously Each group of data is transformed separately, and then each group of transformed data is combined according to the division method to obtain a transformed image formatted as YUV data. It can be understood that the accelerated processing of the image to be processed is realized by using the simd instruction, and the transformation processing speed of the image to be processed is greatly improved through parallel processing.

As shown in Figure 3, in one embodiment, each group of data is transformed and processed in parallel using the simd instruction in the libyuv library to obtain a transformed image, including:

Step 104B1, using the image rotation algorithm in the libyuv library to perform parallel rotation processing on each group of data to obtain a rotated image;

Step 104B2, using the image compression algorithm in the libyuv library to compress the rotated image to obtain a compressed image;

Step 104B3, using the mirror transformation algorithm in the libyuv library to perform mirror transformation on the compressed image to generate a transformed image in the form of YUV data.

In this embodiment, firstly, through the image rotation algorithm in the libyuv library, each group of data is calculated according to the calculation formula of the image rotation algorithm at the same time, and each group of data after rotation processing is obtained, and a rotated image is formed based on each group of data, and then, Use the image compression algorithm in the libyuv library to calculate the compressed image according to the calculation formula of the image compression algorithm to obtain a compressed image. Finally, use the mirror transformation algorithm in the libyuv library to calculate the compressed image according to the calculation formula of the mirror transformation algorithm to obtain YUV A transformed image of the data. It is worth noting that one compression process can be directly performed on a rotated image or compressed image, or the rotated image or compressed image can be divided, and each group of divided data can be calculated in parallel. There are no restrictions here. It can be understood that in this embodiment, by using the simd command in the libyuv library to perform conversion processing on each group of data of the image to be processed, the YUV data conversion speed of the image to be processed is improved.

In one embodiment, using the preset computing engine at the bottom of android to convert the format of the transformed image to obtain a target image in the bitmap format includes: using the computing framework in the RenderScript computing engine to perform parallel computing on the YUV data of the transformed image to obtain the bitmap target image.

Among them, the RenderScript computing engine is an efficient computing framework that comes with Android, which can automatically use CPU, GPU, and DSP for parallel computing, and can provide efficient computing capabilities in scenarios such as image processing and mathematical model calculations. The syntax is similar to C/C++, but it is compiled at runtime and is a cross-platform framework for running computationally intensive tasks. In this embodiment, the YUV data of the converted image is calculated in parallel by the calculation framework in the RenderScript calculation engine, making full use of the high-performance computing capability of the RenderScript calculation engine, and realizing accelerated processing of format conversion for the converted image.

In one embodiment, the calculation framework in the RenderScript calculation engine is used to calculate the YUV data of the converted image in parallel to obtain the target image of the bitmap, including: dividing the YUV data of the converted image into multiple groups of matrices; using the calculation in the RenderScript calculation engine The framework performs parallel calculations on each group of matrices to obtain the target image of the bitmap.

In this embodiment, the YUV data of the converted image is divided into multiple groups of matrices, that is, the matrix formed by the YUV data corresponding to the converted image is divided into multiple groups of matrices, and the calculation framework in the RenderScript calculation engine is used to perform parallel calculations on each group of matrices. Get the target image of the bitmap. Utilizing the high performance of the parallel computing of the RenderScript computing engine improves the speed of the format conversion processing of the transformed image.

In one embodiment, the calculation framework in the RenderScript calculation engine is used to perform parallel calculations on each group of matrices to obtain the target image of the bitmap, including: obtaining the brightness value, color value and saturation value in each group of matrices; using the calculation framework, Based on the floating-point matrix multiplication formula, the red channel value, green channel value and blue channel value of each group of matrices are calculated in parallel; the target image is generated according to the red channel value, green channel value and blue channel value of each group of matrices.

Among them, the floating-point matrix multiplication formula is as follows:

In the formula, Y represents the brightness value, U represents the color value, V represents the saturation value, R represents the red channel value, G represents the green channel value, and B represents the blue channel value. Specifically, the corresponding brightness value, color value and saturation value are obtained according to each group of matrices; using the calculation framework, the above-mentioned floating-point matrix multiplication formula is used to calculate the red channel value, green channel value and blue channel value of each group of matrices in parallel ;Generate the target image according to the calculated red channel value, green channel value and blue channel value with each group of matrices, so as to obtain the RGB image, that is, the target image in bitmap format. It can be understood that in this embodiment, the parallel conversion of each group of matrices of the transformed image is realized by using the floating-point matrix multiplication formula, and the transformed image in the YUV format is quickly converted into an RGB data image, which improves the response speed of the image preprocessing , so as to solve the problem that the user browses images for a long time, and improve the user experience.

In one embodiment, after using the preset computing engine at the bottom of android to convert the format of the converted image to obtain the target image in bitmap format, it also includes: performing video image compression encoding on the target image to obtain a compressed code stream for use in network transmission.

In this embodiment, in order to store or transmit the target image conveniently, the target image can be compressed and coded to obtain a compressed code stream, so as to improve the processing speed of the target image and further improve the user experience.

As shown in Figure 4, in one embodiment, an image preprocessing device is proposed, the device includes:

An acquisition module 402, configured to acquire an image to be processed, the format of the image to be processed is YUV data;

The transformation module 404 is used to transform the image to be processed by using the libyuv library to obtain a transformed image whose format is YUV data;

The conversion module 406 is configured to convert the format of the converted image by using the preset calculation engine at the bottom of the android to obtain a target image in bitmap format.

In one embodiment, the transformation module includes:

A blocking unit, configured to divide the YUV data of the image to be processed into multiple groups of data;

The transformation unit is used to use the simd instruction in the libyuv library to perform parallel transformation processing on each group of data to obtain the transformed image.

In one embodiment, the transformation unit includes:

The rotation subunit is used to use the image rotation algorithm in the libyuv library to perform parallel rotation processing on each set of data to obtain a rotated image;

A compression subunit, configured to use the image compression algorithm in the libyuv library to compress the rotated image to obtain a compressed image;

The mirroring subunit is configured to use the mirroring transformation algorithm in the libyuv library to perform mirroring transformation on the compressed image to generate the transformed image in the form of YUV data.

In one embodiment, the conversion module includes: a conversion sub-module, configured to use the calculation framework in the RenderScript calculation engine to perform parallel calculations on the YUV data of the transformed image to obtain the target image of the bitmap.

In one embodiment, the conversion sub-module includes:

A division unit, configured to divide the YUV data of the transformed image into multiple groups of matrices;

A calculation unit, configured to use the calculation framework in the RenderScript calculation engine to perform parallel calculations on each group of the matrices to obtain the target image of the bitmap.

In one embodiment, the computing unit includes:

Obtaining subunits, used to obtain brightness values, color values and saturation values in each group of matrices;

Calculation subunit, for utilizing described calculation framework, calculate the red channel value, green channel value and blue channel value of each group matrix in parallel based on floating-point matrix multiplication formula;

The sound field subunit is configured to generate the target image according to the red channel value, the green channel value and the blue channel value of each matrix.

In one embodiment, the image preprocessing device further includes: a compression encoding module, configured to perform video image compression encoding on the target image to obtain a compressed code stream for network transmission.

Figure 5 shows a diagram of the internal structure of a computer device in one embodiment. Specifically, the computer device may be a server, and the server includes but is not limited to a high-performance computer and a cluster of high-performance computers. As shown in FIG. 5, the computer device includes a processor, a memory, and a network interface connected through a system bus. Wherein, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and may also store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may implement the image preprocessing method. Computer-readable instructions may also be stored in the internal memory, and when the computer-readable instructions are executed by the processor, the processor may execute the image preprocessing method. Those skilled in the art can understand that the structure shown in Figure 5 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation to the computer equipment on which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, the image preprocessing method provided in the present application can be implemented in the form of a computer-readable instruction, and the computer-readable instruction can be run on a computer device as shown in FIG. 5 . Each program template constituting the image preprocessing device can be stored in the memory of the computer equipment. For example, an acquisition module 402 , a transformation module 404 , and a conversion module 406 .

A computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and operable on the processor, when the processor executes the computer-readable instructions, the following steps are implemented: obtaining the Image processing, the format of the image to be processed is YUV data; the libyuv library is used to transform the image to be processed to obtain a transformed image with a format of YUV data; the transformed image is processed using the preset calculation engine at the bottom of android Format conversion to get the target image in bitmap format.

In one embodiment, the converting the image to be processed by using the libyuv library to obtain the converted image in the form of YUV data includes: dividing the YUV data of the image to be processed into multiple groups of data; using the The simd command in the libyuv library transforms each group of data in parallel to obtain the transformed image.

In one embodiment, the parallel transformation processing of each group of data by using the simd instruction in the libyuv library to obtain the transformed image includes: using the image rotation algorithm in the libyuv library to perform the transformation on each group of data Parallel rotation processing to obtain a rotated image; Utilize the image compression algorithm in the libyuv library to compress the rotated image to obtain a compressed image; Utilize the mirror image transformation algorithm in the libyuv library to perform mirror transformation on the compressed image, The transformed image is generated as YUV data.

In one embodiment, the format conversion of the converted image by using the preset calculation engine at the bottom of android to obtain the target image in bitmap format includes: using the calculation framework in the RenderScript calculation engine to convert the YUV of the converted image The data is calculated in parallel to obtain the target image of the bitmap.

In one embodiment, the parallel computing of the YUV data of the converted image by using the computing framework in the RenderScript computing engine to obtain the target image of the bitmap includes: dividing the YUV data of the transformed image into multiple A group of matrices; using the computing framework in the RenderScript computing engine to perform parallel computing on each group of the matrices to obtain the target image of the bitmap.

In one embodiment, the parallel calculation of each set of matrices by using the calculation framework in the RenderScript calculation engine to obtain the target image of the bitmap includes: obtaining the brightness value, color value and Saturation value; using the calculation framework, calculate the red channel value, green channel value and blue channel value of each group of matrices in parallel based on the floating-point matrix multiplication formula; according to the red channel value, green channel value and blue channel value of each group of matrices channel values to generate the target image.

In one embodiment, after converting the format of the converted image using the preset calculation engine at the bottom of android to obtain the target image in bitmap format, it further includes: performing video image compression encoding on the target image to obtain Compressed code stream for network transmission.

A computer-readable storage medium, the computer-readable storage medium stores computer-readable instructions, and is characterized in that, when the computer-readable instructions are executed by a processor, the following steps are implemented: acquiring an image to be processed, the image to be processed The format of the image is YUV data; utilize the libyuv library to transform the image to be processed, and obtain a transformed image whose format is YUV data; utilize the preset calculation engine at the bottom of android to perform format conversion on the transformed image, and obtain a format of bitmap target image.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer-readable instructions to instruct related hardware, and the program can be stored in a non-volatile computer-readable In the storage medium, when the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims

An image preprocessing method, characterized in that the method comprises:

Obtain the image to be processed, the format of the image to be processed is YUV data;

Utilize libyuv storehouse to carry out transformation processing to described image to be processed, obtain the transformed image that format is YUV data;

Using the preset calculation engine at the bottom of the android to perform format conversion on the transformed image to obtain a target image in bitmap format.
The image preprocessing method according to claim 1, wherein said utilizing the libyuv library to transform the image to be processed to obtain a transformed image whose format is YUV data comprises:

Divide the YUV data of the image to be processed into multiple groups of data;

Using the simd command in the libyuv library to perform parallel transformation processing on each group of data to obtain the transformed image.
The image preprocessing method according to claim 1, wherein said utilizing the simd instruction in the libyuv storehouse to perform parallel transformation processing on each group of data to obtain the transformed image comprises:

Utilize the image rotation algorithm in the libyuv storehouse to carry out parallel rotation processing to each group of data to obtain a rotated image;

Using the image compression algorithm in the libyuv library to compress the rotated image to obtain a compressed image;

Using the mirror transformation algorithm in the libyuv library to perform mirror transformation on the compressed image to generate the transformed image in the form of YUV data.
The image preprocessing method according to claim 1, wherein the format conversion of the transformed image is carried out by using the preset computing engine at the bottom of android to obtain a target image whose format is a bitmap, including:

The YUV data of the transformed image is calculated in parallel by using the calculation framework in the RenderScript calculation engine to obtain the target image of the bitmap.
The image preprocessing method according to claim 4, wherein the parallel calculation of the YUV data of the converted image by using the calculation framework in the RenderScript calculation engine to obtain the target image of the bitmap includes:

Dividing the YUV data of the transformed image into multiple groups of matrices;

Using the calculation framework in the RenderScript calculation engine to perform parallel calculations on each group of the matrices to obtain the target image of the bitmap.
The image preprocessing method according to claim 5, wherein said utilizing the computing framework in said RenderScript computing engine to perform parallel computing on each group of said matrices to obtain the target image of said bitmap, comprising:

Obtain the brightness value, color value and saturation value in each group of matrices;

Using the calculation framework, the red channel value, the green channel value and the blue channel value of each group of matrices are calculated in parallel based on the floating-point matrix multiplication formula;

The target image is generated according to the red channel value, the green channel value and the blue channel value of each group of matrices.
The image preprocessing method according to claim 1, wherein, after said utilizing the preset calculation engine at the android bottom layer to perform format conversion on said transformed image, after obtaining the target image whose format is a bitmap, it also includes:

Perform video image compression encoding on the target image to obtain a compressed code stream for network transmission.
An image preprocessing device, characterized in that the image preprocessing device includes:

An acquisition module, configured to acquire an image to be processed, the format of the image to be processed is YUV data;

Transformation module, for utilizing libyuv storehouse to carry out transformation processing to described image to be processed, obtains the transformed image that format is YUV data;

The conversion module is used to convert the format of the converted image by using the preset calculation engine at the bottom of android to obtain the target image in bitmap format.
A computer device, characterized in that it includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable The steps of the image preprocessing method described in any one of claims 1 to 7 are implemented when the instructions are given.
A computer-readable storage medium, the computer-readable storage medium stores computer-readable instructions, characterized in that, when the computer-readable instructions are executed by a processor, the image described in any one of claims 1 to 7 is realized The steps of the preprocessing method.