WO2019072140A1

WO2019072140A1 - Image hardware encoding processing method and device

Info

Publication number: WO2019072140A1
Application number: PCT/CN2018/109280
Authority: WO
Inventors: 张剑
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2017-10-13
Filing date: 2018-10-08
Publication date: 2019-04-18
Also published as: CN109672884A; TW201923703A; CN109672884B

Abstract

Provided in the embodiments of the present application are an image hardware encoding processing method and device, the method comprising: acquiring image data; processing the image data at a framework layer of an operating system according to at least part of configuration information, the configuration information comprising: at least one of a size conversion instruction, a format conversion instruction and an algorithm processing instruction; and sending the processed image data and a hardware encoding instruction to hardware for encoding. In the image hardware encoding processing method provided in the embodiments of the present invention, since at least one of size conversion, format conversion and algorithm processing may be carried out at the framework layer, image data of various sizes and formats may be provided and converted in the framework layer into data formats that the hardware may process for hardware encoding, which avoids multiple data interactions between an application program and bottom layer hardware in existing technology, improves the encoding efficiency, and also improves the flexibility of processing.

Description

Image hardware coding processing method and device

The present application claims priority to Chinese Patent Application Serial No. No. No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No.

Technical field

The present application relates to the field of data processing technologies, and in particular, to an image hardware coding processing method and apparatus.

Background technique

For the encoding of images, the image compression standard is used to encode and compress images. Taking JPEG image as an example, JPEG is an international image compression standard. JPEG image compression algorithm can provide good compression performance and has better reconstruction quality. It is widely used in image and video processing.

Image coding compression can be done by software or by hardware coding. Hardware coding generally uses a proprietary chip, which is superior to software coding in terms of performance and power consumption, especially the speed advantage of hardware coding. It is the main reason for selecting hardware coding for large-size images.

In an electronic device, there are an underlying hardware layer, an intermediate frame layer (Framework), and an upper layer application layer. For example, in the operating system, the bottom layer includes a camera hardware device, etc.; the middle layer's Framework frame layer includes a camera framework application input image hard coding module; and the upper layer application includes a camera application. In hardware coding, image data needs to be passed from an upper application (such as a camera application) to an underlying (such as a hardware module).

In the existing scheme, the underlying hardware cannot handle image coding flexibly. For example, because the underlying hardware can only recognize image data in a specific format, the application layer can only provide image data in a fixed format. In the case of data in other formats obtained by the application from outside, the application needs to convert the format, call the system interface for processing, and then send it to the underlying hardware for encoding. For another example, when the user wants to perform image processing on the image data before hardware coding, such as noise reduction, zooming, etc., the image data must be returned to the application layer for processing, otherwise processing cannot be performed. Therefore, the inability of the underlying hardware to handle image encoding flexibly will result in multiple data interactions between the underlying hardware and the upper application, affecting the coding efficiency.

Summary of the invention

One of the objects of the present invention is to provide an image hardware coding processing method and processing apparatus to solve the problem that the underlying hardware of the prior art cannot flexibly process image coding and affect coding efficiency.

In order to solve the above problem, the embodiment of the present application discloses an image hardware coding processing method, including:

Obtain image data;

And performing data processing on the image data at a frame layer of the operating system according to the partial information of the configuration information, where the configuration information includes: at least one of a size conversion instruction, a format conversion instruction, and an algorithm processing instruction;

The data processed image data and the hardware coded instructions are sent to the hardware for encoding.

The embodiment of the present application further discloses an image hardware encoding processing device, which is applied to an operating system of an electronic device, and the device includes:

An image data acquiring module, configured to acquire image data;

a data processing module, configured to perform data processing on the image data at a framework layer of an operating system according to at least part of the information of the configuration information, where the configuration information includes: a size conversion instruction, a format conversion instruction, and an algorithm processing instruction. one;

The sending module is configured to send the data processed image data and the hardware encoded instruction to the hardware for encoding.

The embodiment of the present application further discloses a terminal device, including: one or more processors; and one or more machine-readable media having instructions stored thereon, when executed by the one or more processors, The terminal device is caused to perform the method as described in one or more of the embodiments of the present application.

The embodiment of the present application further discloses one or more machine readable mediums having stored thereon instructions that, when executed by one or more processors, cause the terminal device to perform one or more of the embodiments as described in the embodiments of the present application. method.

Compared with the prior art, the image hardware coding processing apparatus proposed by the embodiment of the present application at least includes the following advantages:

In the image hardware coding processing method proposed by the embodiment of the present invention, since the frame layer can process at least one of size conversion, format conversion, and algorithm processing, image data of various sizes and formats can be provided, and converted into frame layers to The form of data that the hardware can handle for hardware coding; or the algorithmic processing of image data within the framework layer. This adds the form of image data that can be provided, avoiding multiple attempts between the prior art application and the underlying hardware. Data interaction improves the efficiency of coding and increases the flexibility of processing.

DRAWINGS

1 is a schematic view of an embodiment of the present application;

2 is a flowchart of an image hardware encoding processing method according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an implementable module of the method;

FIG. 4 is a schematic structural diagram of another implementable module of the method;

FIG. 5 is a flowchart of an image hardware encoding processing method according to another embodiment of the present application;

Figure 6 is a flow chart showing the sub-steps of step S202 in Figure 5;

7 is a structural block diagram of an embodiment of an image hardware coding processing apparatus of the present application;

8 is a structural block diagram of another embodiment of an image hardware coding processing apparatus of the present application;

FIG. 9 is a schematic structural diagram of hardware of a terminal device according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of hardware of a terminal device according to another embodiment of the present disclosure.

Detailed ways

The above described objects, features and advantages of the present application will become more apparent and understood.

1 is a schematic diagram of an embodiment of the present application, embodying the concept of the present invention. As described above, from the perspective of data transmission and function implementation, there are an underlying hardware layer, an intermediate frame layer, and an upper application layer in the electronic device. The aforementioned and later frameworks refer to the architecture between the application and the underlying hardware in the operating system, providing a framework for software development, making development more engineering, simple, and stable. As shown in FIG. 1, the application 11 refers to a plurality of applications in the application layer, and the application 11 performs data interaction with the framework layer 12 through an interface provided by the framework layer 12. The hardware 13 refers to various hardware in the hardware layer, and the hardware 13 also performs data interaction with the framework layer 12 through the interface provided by the framework layer 12. In addition, the application 11 and the hardware 13 also have data interaction, and will not be described again.

The configuration information 14 carries various instructions such as a format conversion instruction, a size conversion instruction, an algorithm processing instruction, and the like for image data. At the same time, the configuration information 14 also carries a hardware coding instruction, which is used for hardware coding. After the frame layer 12 obtains the configuration information 14 and the image data 15, the frame layer 12 can perform data processing such as format conversion, algorithm processing, size conversion, etc. on the image data 15 by using various instructions in the configuration information 14, after the data processing. It is sent to the hardware 13 and encoded according to the hardware coded instructions. In this way, the provided image data 15 can be data of multiple formats and sizes, avoiding the limitations of the prior art, and making the image hardware encoding process more flexible.

The following is specifically described by way of a plurality of embodiments.

First embodiment

FIG. 2 is a flowchart of an image hardware encoding processing method according to an embodiment of the present invention, and FIG. 3 is a schematic diagram of an executable module structure of the method. 4 is a schematic diagram of another implementable module structure of the method. Description will be made with reference to Figs. 2, 3 and 4. The method can be applied to an operating system of an electronic device such as a mobile phone, a computer, a server, or the like. The above image hardware encoding processing method may include the following steps.

S102. Acquire image data. In this step, the image data may be provided by an application, or may be acquired by other means, for example, image data that can be extracted according to the extracted address of the image data in the configuration information, which will be separately described below.

In the first case, as shown in connection with FIG. 3, both the image data 14 and the configuration information 15 are provided by the application 11. Both may be provided at the same time or may be provided one after the other, and the present invention is not limited thereto. That is, step S101 and step S102 can be performed simultaneously or sequentially in this case, and the present invention is not limited to the order. After the information is provided to the frame layer 12, an input may be created from the image data 14, as indicated by the number 121, the externally input data is encapsulated as an input data structure of the frame layer; and the image is then imaged according to the configuration information 122 sent to the frame layer 12. The data 15 is processed, such as the size conversion, format conversion, algorithm processing, etc. described above, as indicated by reference numeral 124. When the algorithm processing is performed, the algorithm library 123 embedded in the framework layer can be called. After processing, image data 125 of the input hardware can be generated.

Moreover, in an alternative embodiment, the configuration information 15 sent to the framework layer may also have a memory application instruction. The memory application instruction can be used to apply for a memory buffer for storing image data after storage hardware encoding. Therefore, in the frame layer 12, the memory configuration 126 can be output according to the configuration information 122. When the encoded image data is returned from the hardware 13 to the frame layer, the output buffer queue 127 can be used to open the memory buffer area to store the hardware coded Image data.

In the second case, as shown in FIG. 4, the address of the extracted image data can be obtained by the configuration information 15 provided by the application 11, and the extracted address of the image data can be the memory address of the image data storage. Image data 14 may be provided by an image signal processor (ISP) 16. The image signal processor 16 is hardware in the hardware layer, and the provided image data 14 may be a multi-path image as shown in FIG. 4 or one-way image data. After the multiplexed image data is provided, the image data can be selected using the selector 129, or the input can be directly created with the multiplexed data without using a selector. There is no limit here.

In FIG. 3 and FIG. 4, the main configuration information of the configuration information 122 is from the configuration information 15, but the configuration information 122 also includes some state information of the internal layer of the framework layer, that is, the configuration information 15 set by the application program is converted into the frame layer 14 The configured configuration information 122 is therefore identified by a different number.

Further, the configuration information 14 may further include a photographing instruction and a photographing parameter. The camera command can drive the underlying camera hardware to take a photo. Before the step of acquiring image information in step S102, the method may further comprise the step of driving the camera to take a picture according to the photographing instruction. The photographing parameters include at least one of a photographing frame number, an image parameter, a composite image frame number, and a composite image data selection. The raw image data obtained by the photographing may be output by the image signal processor in the hardware layer, and the photographed image data may be extracted from the memory address output by the image signal processor according to the extracted address after the photographing is completed. In an optional embodiment, if the number of shooting frames in the camera parameter is multiple frames, the corresponding image parameter may be an image parameter corresponding to each frame. The image parameters include, for example, the size, format, zoom ratio, and the like of each photograph.

S103, performing data processing on the image data in a framework layer of an operating system according to at least part information of the configuration information, where the configuration information includes: at least one of a size conversion instruction, a format conversion instruction, and an algorithm processing instruction;

In this step, the image data can be processed correspondingly in the framework layer according to various instructions in the configuration information.

As described above, the configuration information may be provided by an application or may be obtained by other means. In an embodiment, when encoding for image data is required, an application in an application layer (App layer) of the electronic device can provide configuration information to a framework layer (Framework layer) of the electronic device. As described above, there is a data interface between the framework layer and the application layer. When an application in the application layer provides data to a module in the framework layer, data can be provided through the data interface, and the module is specified. receive. For example, an application of the "camera" operating system provides image encoding related configuration information and sends it to the data interface of the framework layer, which is specified by the encoding module of the framework layer. The configuration information includes at least one of a size conversion instruction, a format conversion instruction, and an algorithm processing instruction for invoking related processing within the framework layer. In addition, the configuration information may further include hardware coding instructions for hardware coding, for example, including at least one of an encoded image size, an image rotation angle, an image compression ratio, an output image frame number, and a format conversion of a hardware input to a hardware output, The invention is not particularly limited. In the operations corresponding to the various instructions described above, the size conversion converts the size of the image data into a specific size required by the underlying hardware. Due to the variety of hardware, each hardware may have different requirements for size alignment. Here, the image data can be converted to a size that meets the hardware requirements according to the requirements of the hardware.

Format conversion is the conversion of the original image data format to the specific format required by the underlying hardware. For example, in some cases, if the application configuration needs, the image data can be converted to a specific format in the hardware capability set specified by the application. For example, the format requirement of a certain hardware must be the NV21 format of YUV, so here In the format conversion, the image format can be converted to a format that meets the hardware requirements.

The algorithm processing uses the algorithm embedded in the framework layer or the framework layer to process the image data. The algorithm processing includes, for example, the illustrated HDR algorithm, digital zoom multi-frame optimization, multi-frame noise reduction algorithm, and full focus processing algorithm. The embedded algorithm can be a plugin that can be used after it has been configured.

Each instruction in the configuration information may carry the transmitted target module, and thus may be distributed to each module according to the target in the framework layer, for example, distributing algorithm selection information to the algorithm library module, converting information distributed to the format conversion module, and providing the memory to the memory. Allocate memory information, etc. distributed by the module. In the operation, the variables of each module may be initialized to the information specified by the configuration, and the image data is processed according to the initialized configuration data when the image data passes through the modules, and details are not described herein.

S104: Send the data processed image data and the hardware coded instruction to the hardware for encoding.

In this step, the framework layer may send the processed image data and the hardware coded instructions in the configuration information to the hardware, and implement hardware coding using the hardware-encapsulated encoding program. For example, the hardware-encapsulated encoding program is, for example, a JPEG hardware encoding program that generates an image in JPEG format after encoding. However, it will be apparent to those skilled in the art that the encoding process is not limited to JPEG encoding, but can cover any encoding method that can be implemented in hardware.

In an optional embodiment, the configuration information may further include a package instruction for encapsulating the image data after the frame layer processing and the hardware coded instruction, and transmitting the whole to the hardware for encoding. For the purpose of ease of use, the image data and the hardware coded instructions may be sent as a whole after being packaged, but it is obvious to those skilled in the art that the solution proposed by the embodiment of the present invention may not include the steps of encapsulation.

The method may further include: before the step of acquiring the image data, the method may further include:

S101. Acquire configuration information of an image hardware code.

In this step, the configuration information may be provided by the application, or may be obtained by other means, for example, downloading from the server according to the address provided by the application, or being pushed by the set server, or the system default configuration, Or provide a setting interface set by the user, etc., and will not be described here. In an optional embodiment, the configuration information includes hardware coded instructions, that is, the hardware coded instructions in step S104 may be obtained from configuration information. Similarly, the hardware coded instruction can also be obtained by other means, for example, downloading from the server according to the address provided by the application, or pushing by the set server, or the system default configuration, or providing the setting interface by the user, etc., without limitation Obtained from the configuration information.

In an optional embodiment, after the hardware coding is implemented, the foregoing method may further include:

S105. Store the hardware encoded image; and

S106. Send a storage address of the hardware encoded image.

In this step, the hardware can store the encoded image in the memory buffer and send the address of the memory buffer to the framework layer, for example, can be sent to the data interface of the framework layer and the application, so that the application can The address of the memory buffer is retrieved from the encoded image.

The application of the image hardware coding processing method proposed by the first embodiment will be described below through several specific application examples.

In one scenario, the user needs to perform a format conversion process on the existing image. The user provides image data 14 and configuration information 15 through the application 11. Some or all of the content included in the configuration information 15 may be defined by the user himself or may be preset. For example, in the interface of the application, there are several options for the user to select. These options may include: selecting a template image, selecting a picture from an album, taking a photo, and the like, and may also include conversion size, conversion format, algorithm processing. instruction. These choices of the user generate configuration information in the application and send it to the framework layer. As shown in FIG. 3, when format conversion is required, the format conversion instruction is carried in the configuration information 15 in the application 11 according to the user's selection. In the framework layer, the image data 14 is processed using a format conversion instruction, which is then sent to the hardware for encoding along with the hardware encoding instructions.

In another case, the user needs to perform HDR algorithm processing on the newly taken photo, that is, High-Dynamic Range (HDR). For example, the HDR algorithm requires a frame layer to send a multi-frame photo original image request, and the exposure value of each frame image is different, and a wider brightness domain image is obtained by the HDR algorithm to avoid overexposure or underexposure. In the process, as shown in FIG. 4, the user can select among the above options, for example, select "photographing" and "HDR algorithm processing". According to the user's selection, the configuration information 15 sent by the application 11 to the framework layer includes a photographing instruction and an HDR algorithm processing instruction. In the frame layer, the image data 14 is acquired by the above instruction and the image data 14 is processed. After the process, the picture and the hardware coded instruction are sent together to the hardware for encoding.

It can be seen from the above that the image hardware coding processing method proposed by the embodiment of the present invention has at least the following technical effects:

In the image hardware encoding processing method proposed by the embodiment of the present invention, since the framework layer can process at least one of size conversion, format conversion, algorithm processing, etc., image data of various sizes and formats can be provided, and converted in the framework layer. A form of data that can be processed by the hardware for hardware coding; or algorithmic processing of image data within the framework layer. Such an operation increases the form of image data that can be provided, avoids multiple data interactions between the prior art application and the underlying hardware, improves coding efficiency, and increases processing flexibility.

Second embodiment

FIG. 5 is a flowchart of an image hardware encoding processing method according to another embodiment of the present application. As shown in FIG. 5, the image hardware coding processing method proposed by the embodiment of the present application may include the following steps:

S202. Acquire image data.

S203, performing data processing on the image data in a framework layer of an operating system according to the configuration information, where the data processing includes at least one of format conversion and algorithm processing;

The above two steps may be the same as or similar to the steps S102 and S103 of the first embodiment, and details are not described herein again.

S205. Send the data processed image data and the hardware coded instruction to the hardware for encoding.

This step may be the same as step S103 of the first embodiment, and details are not described herein again.

Figure 6 is a flow chart showing the sub-steps of step S202 in Figure 5. As shown in FIG. 6, in the embodiment, the image data of step S02 may be provided by an application, or may be called from a storage area, such as a memory, according to an extraction instruction in the configuration information. For example, the step of acquiring image data in step S202 may include the following sub-steps:

S202b, obtaining an extracted address of the image data from the configuration information;

S202c: Extract image data from the extracted address.

In the above two steps, for example, although the application does not provide image data, it provides an extraction address for obtaining image data from a specific storage address, and an image can be extracted from a storage area (for example, a memory) according to the extraction address in the frame layer. The data is processed.

In this embodiment, in the above step S202b, the extraction address of the image data may be the address of the camera data in the frame layer. As shown in FIG. 4, the image data 14 may be provided by an image signal processor (ISP, image). Signal processor)16 is provided. The image signal processor 16 is hardware in the hardware layer, and the address of the image data outputted therefrom can be used as an extraction address of the image data to facilitate the frame layer to extract image data from the address.

In this embodiment, the image data acquired in step S202c may be multiplexed image data, and the step of acquiring image data in step S202 may further include:

S202d: Select at least one image data from the multiplexed image data.

In this step, according to FIG. 4, the supplied image data 14 may be data output from an image signal processor of the hardware layer, which may be multiplexed. After the multiplexed image data is provided, the image data can be selected using the selector 129, or the input can be directly created with the multiplexed data without using a selector. There is no limit here.

In practical applications, the underlying camera hardware outputs multiple channels of data simultaneously under normal conditions. One of the multiple channels of data is selected here. Each channel of data is processed differently and can be selected as needed. For example, multi-channel data mainly includes preview data, callback data, photo data stream, and possibly video data. These data size formats can be different, the processed image signal processing algorithm is different, and the field of view may be different. When selecting, the index of the data can be specified by the application layer, or can be specified by the application. An image of what conditions are needed. For example, the conversion needs to use a large-size image. The camera's data index is: preview 1, photo 2, callback 3; at this time, the configuration information specifies the data that needs index 2, and the frame layer will go to the camera system to extract the data of the photo 2, or the application designation. Use the largest size data in the camera data. According to the selection conditions, the image size of different data can be automatically compared, and the largest one, such as the data of the photograph 2, is selected as the input source, and the selection condition here is "size". The selection condition may also be other image parameters, such as zoom magnification, format, etc., and will not be described herein.

In this embodiment, the configuration information further includes a photographing instruction and a photographing parameter. Before the step of acquiring the extracted address of the image data from the configuration information, the step of acquiring the image data includes:

S202a, driving the camera to take a photo according to the photographing instruction and the photographing parameter.

The photographing parameter may include, for example, at least one of a photographing frame number, an image parameter, a composite image frame number, and a composite image data source selection.

The image parameter may be, for example, the size of the image, and when it is necessary to synthesize one frame using a plurality of frame images, the number of frames of the composite image may be defined in the photographing parameter. When it is necessary to use one frame of the main picture in combination with other picture synthesis images, the synthetic image data source selection mode can be defined, for example, selecting the highest resolution image as the main picture, or selecting the image data captured most of the way.

In this embodiment, after the step of transmitting the data processed image data and the hardware encoding instruction to the hardware for encoding, the method may further include:

S206. Store the hardware encoded image; and

S208. Send the hardware to send a stored address of the encoded image.

The above steps are the same as steps S105 and S106 in the first embodiment, and are not described herein again.

In this embodiment, it is also possible to apply for memory for the encoded image according to the memory application instruction. The memory application instruction is obtained, for example, from the encoded information, or obtained from other sources, and is not particularly limited.

In an embodiment, the configuration information further includes a memory application instruction, and the method may further include:

S204, providing a memory buffer according to the memory application instruction, for storing the hardware encoded image;

Then, step S206, that is, the step of storing the hardware-encoded image includes:

The encoded image is stored to the memory buffer.

It should be noted that the execution of the above steps S204 and S205 has no sequential relationship, that is, the step S204 may be performed before, after or simultaneously with the step S205, and the present invention is not particularly limited.

In step S204, a memory application instruction may be carried in the configuration information, and a memory buffer is provided in the framework layer according to the instruction. This memory buffer is used to store the encoded image. The size of the memory buffer can be defined by the user, for example, it can be set to store one or more frames of encoded image data. In the prior art, image data is directly sent from an application to an underlying hardware layer for encoding processing, and a memory buffer reserved for the encoded image is fixed. However, in the solution of the present invention, the memory buffer area can be directly configured through the framework layer. When the image data to be processed is a multi-frame picture, the memory buffer area can have enough space to buffer the multi-frame picture, and the multi-frame picture is simultaneously processed. The foundation.

In this embodiment, the algorithm corresponding to the algorithm processing instruction includes, for example, at least one of an HDR algorithm, a digital zoom multi-frame optimization, a multi-frame noise reduction algorithm, and a full focus processing algorithm.

The HDR algorithm, for example, requires a frame layer to send a multi-frame photo original image request (yuv or rgb format output), and the exposure value of each frame image is different, and a wider brightness domain image is obtained by the HDR algorithm to avoid overexposure or underexposure. The other steps are the same as ordinary photos.

The multi-frame noise reduction algorithm, for example, triggers in the case where the flash is not turned on in the dark condition, that is, photographs of the same parameters as the multi-frame are combined to improve the brightness and noise reduction under the dark light, and finally synthesized into a frame of photos. Others are the same as ordinary photos.

The full-focus processing algorithm, for example, is to issue a plurality of photographing commands (yuv or rgb format output) at the frame layer, and the focal length parameters of each photographing command are different, and the focusing distance is different, thereby synthesizing a panoramic image with a clear panorama.

Digital zoom multi-frame optimization algorithm, for example, configuration information configures the request to take two frames of photos, one frame enlarges N times zoom magnification, one frame does not zoom normally, and then selects the digital zoom multi-frame optimization algorithm in the algorithm layer of the frame layer Combine two frames to optimize the final zoom effect, and finally output one frame of optimized image, or output an unoptimized one-frame N-time magnification (using only one of the N-magnified original images of the requested photo) and The optimized N times magnification is used as the output.

In an embodiment, the hardware coding instruction includes at least one of an encoded image size, an image rotation angle, an image compression ratio, and an output image frame number.

In an embodiment, after step S206, that is, after storing the hardware-encoded image, the method further includes:

S207. Store the plurality of hardware-encoded images to consecutive storage addresses.

Since the hardware can process multiple image data at the same time, the storage address of each processed image data may be different, that is, the memory address is not continuous. In this step, the memory splicing method can copy the data of one piece of memory to the end address of another piece of data, so that the addresses of the two are continuous. Subsequent applications can continuously read from the starting address to obtain the synthesized encoded image.

In an embodiment, between step S203 and step S205, that is, between performing data processing and transmitting to hardware for encoding, the method may further include:

Encapsulating the image data and hardware coded instructions.

In this step, image data and hardware coded instructions transmitted by the application or obtained in other modules in the framework layer may be encapsulated into image data that can be recognized by the underlying hardware to be transmitted to the underlying hardware.

In addition to the above effects, the image hardware encoding processing method proposed by the embodiment of the present invention further has the following technical effects:

In the image hardware coding processing method provided by the present invention, when the application layer does not need data to be processed by itself, the frame layer may be required to acquire image data by itself, for example, image data may be acquired through a storage address of camera data of the camera system. Avoid cross-layer data transfer and communication synchronization overhead, improving performance and unnecessary application development work.

In addition, a plurality of functional modules for data processing may be disposed in the framework layer. In the solution provided by the embodiment of the present invention, a part of the processing may be selected by using an option in the application. For example, the user can select whether to enable HDR algorithm processing by using an option. For other processing such as encapsulation, options may be provided without being provided by the framework layer.

Furthermore, the image hardware encoding processing method provided by the present invention can establish an algorithm channel at the framework layer for image processing. For example, the framework layer can access the camera's data, which in turn can be cached and processed at the framework layer to achieve a zero-latency photo effect. Different algorithms can be set in the framework layer in the application, correspondingly setting different algorithm processing options in the application. For example, an image of different exposures specified by an application generates an HDR image, an image of a different definition is specified by an application, and an image of the highest definition is synthesized, thereby avoiding the limitation of providing an image by the underlying hardware due to coding unity, and increasing The function of the application.

It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the embodiments of the present application are not limited by the described action sequence, because In accordance with embodiments of the present application, certain steps may be performed in other sequences or concurrently. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required in the embodiments of the present application.

Third embodiment

FIG. 7 is a structural block diagram of an image hardware coding processing apparatus according to an embodiment of the present application. As shown in FIG. 7, the image hardware coding processing apparatus proposed by the embodiment of the present application may include:

An image data obtaining module 502, configured to acquire image data;

The data processing module 503 is configured to perform data processing on the image data at a frame layer of the operating system according to at least part of the information of the configuration information, where the configuration information includes: at least a size conversion instruction, a format conversion instruction, and an algorithm processing instruction. One;

The sending module 504 is configured to send the data processed image data and the hardware encoded instruction to the hardware for encoding.

It can be seen from the above that the image hardware coding processing apparatus proposed by the embodiment of the present invention has at least the following technical effects:

In the image hardware coding processing apparatus proposed by the embodiment of the present invention, since the framework layer can handle operations such as size conversion, format conversion, algorithm processing, etc., the application program can provide image data of multiple formats and convert it into hardware capable of processing in the framework layer. Format data for hardware coding. Such an operation avoids multiple data interactions between the prior art application and the underlying hardware, improves coding efficiency, and increases processing flexibility.

Fourth embodiment

FIG. 8 is a flowchart of an image hardware coding processing apparatus according to another embodiment of the present application. As shown in FIG. 8, the image hardware encoding processing apparatus proposed by the embodiment of the present application may include:

An image data obtaining module 602, configured to acquire image data;

The data processing module 603 is configured to perform data processing on the image data at a framework layer of the operating system according to at least part of the information of the configuration information, where the configuration information includes: at least a size conversion instruction, a format conversion instruction, and an algorithm processing instruction. One;

The sending module 605 is configured to send the data processed image data and the hardware encoded instruction to the hardware for encoding.

In an embodiment of the image hardware encoding processing apparatus of the present invention, the image data processing module is configured to receive image data sent by an application.

In an embodiment of the image hardware coding processing apparatus of the present invention, the configuration information further includes an extraction address of the image data, and the image data acquisition module includes:

An address obtaining submodule 6022, configured to acquire an extracted address of the image data from the configuration information;

An extraction submodule 6023 is configured to extract image data from the extracted address.

In an embodiment of the image hardware encoding processing apparatus of the present invention, the extracted address of the image data is a storage address of camera data in a frame layer, and the camera data includes: an image signal processor output from a hardware layer The data.

In an embodiment of the image hardware coding processing apparatus of the present invention, the image data is multiplexed image data, and the image data acquisition module is configured to:

The selecting module 6024 is configured to select at least one image data from the acquired multi-path image data.

In an embodiment of the image hardware coding processing apparatus of the present invention, the configuration information further includes a photographing instruction and a photographing parameter, and the image data module further includes:

The photo sub-module 6021 is configured to drive the camera to take a photo according to the photographing instruction and the photographing parameter.

In an embodiment of the image hardware encoding processing apparatus of the present invention, the photographing parameter includes a photographing frame number and an image parameter, the photographing frame number is a plurality of frames, and the image parameter is an image corresponding to the multi-frame. parameter.

In an embodiment of the image hardware coding processing apparatus of the present invention, the apparatus further includes:

a storage module 606, configured to store the hardware encoded image;

The sending module 608 is configured to send a storage address of the hardware encoded image.

The memory providing module 604 is configured to provide a memory buffer area according to the memory application instruction, for storing the hardware encoded image;

The storage module is configured to store the encoded image to the memory buffer.

The mobile module 607 is configured to store the plurality of encoded images to consecutive storage addresses.

In an embodiment of the image hardware coding processing apparatus of the present invention, the algorithm corresponding to the algorithm processing instruction includes at least one of an HDR algorithm, a digital zoom multi-frame optimization, a multi-frame noise reduction algorithm, and a full focus processing algorithm.

In an embodiment of the image hardware encoding processing device of the present invention, the hardware encoding instruction includes at least one of an encoded image size, an image rotation angle, an image compression ratio, and an output image frame number.

In an embodiment of the image hardware encoding processing device of the present invention, the device may further include a configuration information acquiring module, configured to acquire configuration information. The above hardware coded instructions and/or memory application instructions may be stored in the configuration information, obtained from the configuration information during operation, or may be obtained by other means, for example, downloading from the server according to the address provided by the application, or by setting Server push, or system default configuration, or provide settings interface by user settings, etc.

In addition to the above effects, the image hardware coding processing apparatus proposed by the embodiment of the present invention further has the following technical effects:

In the image hardware coding processing apparatus provided by the present invention, when the application layer does not need data to be processed by itself, the frame layer may be required to acquire image data by itself, for example, image data may be acquired through a storage address of the camera system. Avoid cross-layer data transfer and communication synchronization overhead, improving performance and unnecessary application development work.

In addition, a plurality of functions of the data processing may be set in the framework layer. In the solution provided by the embodiment of the present invention, a part of the processing may be selected by using an option in the application. For example, the user can select whether to enable HDR algorithm processing by using an option. For other processing such as encapsulation, options may be provided without being provided by the framework layer.

Furthermore, the image hardware coding processing apparatus provided by the present invention can establish an algorithm channel at the framework layer for image processing. For example, the framework layer can access the camera's data, which in turn can be cached and processed at the framework layer to achieve a zero-latency photo effect. Different algorithms can be set in the framework layer in the application, correspondingly setting different algorithm processing options in the application. For example, an image of different exposures specified by an application generates an HDR image, an image of a different definition is specified by an application, and an image of the highest definition is synthesized, thereby avoiding the limitation of providing an image by the underlying hardware due to coding unity, and increasing The function of the application.

For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The embodiment of the present application further provides one or more machine readable mediums, on which instructions are stored, when executed by one or more processors, causing the terminal device to execute one or more image hardware as described in the embodiments of the present application. Encoding processing method. An exemplary machine readable medium is a non-volatile readable storage medium having stored therein one or more programs that can be made when the one or more modules are applied to a terminal device The device executes the instructions of the method steps in the embodiment of the present application.

FIG. 9 is a schematic structural diagram of hardware of a terminal device according to an embodiment of the present disclosure. As shown in FIG. 9, the terminal device may include an input device 90, a processor 91, an output device 92, a memory 93, and at least one communication bus 94. Communication bus 94 is used to implement a communication connection between components. Memory 93 may include high speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk memory, in which various programs may be stored for performing various processing functions and implementing the method steps of the present embodiments.

Optionally, the processor 91 may be, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), and programmable logic. A device (PLD), field programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic component is implemented that is coupled to the input device 90 and the output device 92 by a wired or wireless connection.

Optionally, the input device 90 may include multiple input devices, for example, at least one of a user-oriented user interface, a device-oriented device interface, a software programmable interface, a camera, and a sensor. Optionally, the device-oriented device interface may be a wired interface for data transmission between the device and the device, or may be a hardware insertion interface (for example, a USB interface, a serial port, etc.) for data transmission between the device and the device. Optionally, the user-oriented user interface may be, for example, a user-oriented control button, a voice input device for receiving voice input, and a touch-sensing device for receiving a user's touch input (eg, a touch screen with touch sensing function, touch Optionally, the programmable interface of the software may be, for example, an input for the user to edit or modify the program, such as an input pin interface or an input interface of the chip; optionally, the transceiver may have Radio frequency transceiver chip, baseband processing chip, and transceiver antenna for communication functions. An audio input device such as a microphone can receive voice data. Output device 92 can include output devices such as displays, stereos, and the like.

In this embodiment, the processor of the terminal device includes functions for executing the modules of the data processing device in each device. The specific functions and technical effects may be referred to the foregoing embodiments, and details are not described herein again.

FIG. 10 is a schematic structural diagram of hardware of a terminal device according to another embodiment of the present disclosure. Figure 10 is a specific embodiment of the implementation of Figure 9. As shown in FIG. 10, the terminal device of this embodiment includes a processor 101 and a memory 102.

The processor 101 executes the computer program code stored in the memory 102 to implement the image hardware encoding processing method of FIGS. 2 to 5 in the above embodiment.

The memory 102 is configured to store various types of data to support operation at the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, such as messages, pictures, videos, and the like. The memory 102 may include a random access memory (RAM), and may also include a non-volatile memory such as at least one disk storage.

Optionally, the processor 101 is disposed in the processing component 100. The terminal device may also include a communication component 103, a power component 104, a multimedia component 105, an audio component 106, an input/output interface 107, and/or a sensor component 108. The components and the like included in the terminal device are set according to actual requirements, which is not limited in this embodiment.

Processing component 100 typically controls the overall operation of the terminal device. Processing component 100 may include one or more processors 101 to execute instructions to perform all or part of the steps of the methods of Figures 2 through 1 above. Moreover, processing component 100 can include one or more modules to facilitate interaction between component 100 and other components. For example, processing component 100 can include a multimedia module to facilitate interaction between multimedia component 105 and processing component 100.

Power component 104 provides power to various components of the terminal device. The power component 104 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the terminal devices.

The multimedia component 105 includes a display screen between the terminal device and the user that provides an output interface. In some embodiments, the display screen can include a liquid crystal display (LCD) and a touch panel (TP). If the display includes a touch panel, the display can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may sense not only the boundary of the touch or sliding action, but also the duration and pressure associated with the touch or slide operation.

The audio component 106 is configured to output and/or input an audio signal. For example, the audio component 106 includes a microphone (MIC) that is configured to receive an external audio signal when the terminal device is in an operational mode, such as a voice recognition mode. The received audio signal may be further stored in memory 102 or transmitted via communication component 103. In some embodiments, the audio component 106 also includes a speaker for outputting an audio signal.

The input/output interface 107 provides an interface between the processing component 100 and the peripheral interface module, which may be a click wheel, a button, or the like. These buttons may include, but are not limited to, a volume button, a start button, and a lock button.

Sensor assembly 108 includes one or more sensors for providing status assessment of various aspects to the terminal device. For example, sensor component 108 can detect the on/off state of the terminal device, the relative positioning of the components, and the presence or absence of contact of the user with the terminal device. The sensor assembly 108 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, the sensor assembly 108 can also include a camera or the like.

The communication component 103 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In an embodiment, the terminal device may include a SIM card slot for inserting the SIM card, so that the terminal device can log in to the GPRS network and establish communication with the server through the Internet.

It can be seen from the above that the communication component 103, the audio component 106, the input/output interface 107, and the sensor component 108 involved in the embodiment of FIG. 10 can be implemented as an input device in the embodiment of FIG.

An embodiment of the present application provides a terminal device, including: one or more processors; and one or more machine-readable media having instructions stored thereon, when executed by the one or more processors, The terminal device performs an image hardware coding processing method as described in one or more of the embodiments of the present application.

The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments can be referred to each other.

Those skilled in the art will appreciate that embodiments of the embodiments of the present application can be provided as a method, apparatus, or computer program product. Therefore, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, embodiments of the present application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium. Computer readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-persistent computer readable media, such as modulated data signals and carrier waves.

Embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal device to produce a machine such that instructions are executed by a processor of a computer or other programmable data processing terminal device Means are provided for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The instruction device implements the functions specified in one or more blocks of the flow or in a flow or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device such that a series of operational steps are performed on the computer or other programmable terminal device to produce computer-implemented processing, such that the computer or other programmable terminal device The instructions executed above provide steps for implementing the functions specified in one or more blocks of the flowchart or in a block or blocks of the flowchart.

While a preferred embodiment of the embodiments of the present application has been described, those skilled in the art can make further changes and modifications to the embodiments once they are aware of the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the modifications and the modifications

Finally, it should also be noted that in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. There is any such actual relationship or order between operations. Furthermore, the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, or terminal device that includes a plurality of elements includes not only those elements but also Other elements that are included, or include elements inherent to such a process, method, article, or terminal device. An element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that comprises the element, without further limitation.

The foregoing describes an image hardware coding processing method and apparatus provided by the present application in detail. The principles and implementation manners of the present application are described in the specific examples. The description of the above embodiments is only used to help understand the present text. The method of application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be understood. To limit the application.

Claims

An image hardware coding processing method, the method comprising:

Obtain image data;

Performing, at least one of: a size conversion instruction, a format conversion instruction, and an algorithm processing instruction, at least one of a size conversion instruction, a format conversion instruction, and an algorithm processing instruction, according to at least part of the information of the configuration information;

The data processed image data and the hardware coded instructions are sent to the hardware for encoding.
The method of claim 1 wherein said step of acquiring image data comprises:

Receive image data sent by the application.
The method according to claim 1, wherein the configuration information further comprises an extraction address of the image data, and the step of acquiring the image data comprises:

Obtaining an extracted address of the image data from the configuration information;

Image data is extracted from the extracted address.
The method according to claim 3, wherein the extracted address of the image data is a storage address of camera data within a frame layer, and the camera data comprises data output from an image signal processor of the hardware layer.
The method according to claim 3, wherein the image data is multiplexed image data, and the step of acquiring image data comprises:

At least one image data is selected from the acquired multiplexed image data.
The method according to claim 3, wherein the configuration information further includes a photographing instruction and a photographing parameter, and before the step of obtaining the extracted address of the image data from the configuration information, the method further includes:

The camera is driven to take photos according to the camera instruction and the camera parameters.
The method according to claim 6, wherein the photographing parameters include a number of photographing frames and image parameters.
The method according to claim 1, wherein after the step of transmitting the data processed image data and the hardware encoding instructions to the hardware for encoding, the method further comprises:

Storing the hardware encoded image; and

Sending the storage address of the hardware encoded image.
The method of claim 8 further comprising:

Providing a memory buffer according to the memory application instruction for storing the hardware-encoded image;

The step of storing the hardware encoded image includes:

The hardware encoded image is stored to the memory buffer.
The method according to claim 8, wherein after the step of transmitting the data processed image data and the hardware encoding instructions to the hardware for encoding, the method further comprises:

Store multiple encoded images to consecutive storage addresses.
The method according to claim 1, wherein the algorithm corresponding to the algorithm processing instruction instruction comprises:

At least one of an HDR algorithm, a digital zoom multi-frame optimization, a multi-frame noise reduction algorithm, and a full focus processing algorithm.
The method according to claim 1, wherein the hardware coded instruction comprises at least one of an encoded image size, an image rotation angle, an image compression ratio, and an output image frame number.
An image hardware coding processing device, characterized in that the device comprises:

An image data acquiring module, configured to acquire image data;

a data processing module, configured to perform data processing on the image data at a framework layer of the operating system according to at least part of the information of the configuration information, where the configuration information includes: at least one of a size conversion instruction, a format conversion instruction, and an algorithm processing instruction ;

The sending module is configured to send the data processed image data and the hardware encoded instruction to the hardware for encoding.
The apparatus according to claim 13, wherein said image data processing module is operative to receive image data transmitted by an application.
The device according to claim 13, wherein the configuration information further includes an extraction address of the image data, and the image data acquisition module comprises:

An address obtaining submodule, configured to obtain an extracted address of the image data from the configuration information;

An extraction submodule for extracting image data from the extracted address.
The apparatus according to claim 15, wherein the extracted address of the image data is a storage address of camera data within a frame layer, and the camera data includes data output from an image signal processor of the hardware layer.
The device according to claim 15, wherein the image data is multiplexed image data, and the image data acquiring module is configured to:

And a selection module, configured to select at least one image data from the acquired multi-path image data.
The device according to claim 15, wherein the configuration information further includes a photographing instruction and a photographing parameter, and the image data module further comprises:

The photo sub-module is configured to drive the camera to take a photo according to the photographing instruction and the photographing parameter.
The apparatus according to claim 18, wherein said photographing parameters include a number of photographing frames and image parameters.
The device according to claim 13, wherein the device further comprises:

a storage module, configured to store the hardware encoded image;

And a sending module, configured to send a storage address of the hardware encoded image.
The device of claim 20, wherein the device further comprises:

a memory providing module, configured to provide a memory buffer area according to the memory application instruction, for storing the hardware encoded image;

The storage module is configured to store the encoded image to the memory buffer.
The device of claim 20, wherein the device further comprises:

A mobile module for storing a plurality of encoded images to consecutive storage addresses.
The apparatus according to claim 13, wherein the algorithm corresponding to the algorithm processing instruction comprises at least one of an HDR algorithm, a digital zoom multi-frame optimization, a multi-frame noise reduction algorithm, and a full focus processing algorithm.
The apparatus according to claim 13, wherein the hardware coded instruction comprises at least one of an encoded image size, an image rotation angle, an image compression ratio, and an output image frame number.
A terminal device, comprising:

One or more processors; and

One or more machine-readable media having stored thereon instructions that, when executed by the one or more processors, cause the terminal device to perform the method of one or more of claims 1-12.
One or more machine-readable medium having stored thereon instructions that, when executed by one or more processors, cause the terminal device to perform the method of one or more of claims 1-12.