CN114661613A

CN114661613A - Image processing method and device, electronic device and storage medium

Info

Publication number: CN114661613A
Application number: CN202210369667.3A
Authority: CN
Inventors: 李树析; 吴佳飞; 张广程
Original assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Priority date: 2022-04-08
Filing date: 2022-04-08
Publication date: 2022-06-24

Abstract

The present disclosure relates to an image processing method and apparatus, an electronic device, and a storage medium, the method including: obtaining a simulation library according to an interface provided by at least one function of the hardware resource; processing image coding information of an image to be processed acquired by hardware resources through a simulation library to acquire an image processing result; and storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information. According to the image processing method disclosed by the embodiment of the disclosure, the hardware resources can be simulated through the functional interfaces of the hardware resources, and the simulation library of the hardware resources is obtained, so that the functions of the hardware resources of the end-side equipment can be simulated on the general computing platform, the functions of the hardware resources and other business logics can be developed and debugged conveniently according to the image processing result, and the development and debugging efficiency can be improved.

Description

Image processing method and device, electronic device and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

At present, the artificial intelligence technique is widely used in the fields of security protection, monitoring and the like, and along with the continuous preposition of chip computing power to the end side, as the typical representative of the end side, the computing power of an intelligent camera is stronger and stronger, and the use is also wider and wider. For example, a smart camera is a commonly-used end-side device, and the smart camera generally adopts an embedded system, and a chip manufacturer provides an SDK (software development kit) corresponding to the embedded system, and a user writes code based on an interface provided by the chip SDK to perform a corresponding function.

In the above end-side devices such as smart cameras, the development of embedded system debugging is very complicated, and the devices themselves are expensive and limited in number. In addition, embedded development often has the problem of development efficiency (cross compilation, board debugging and few tools for assisting in troubleshooting), which is not as convenient as general computing platforms such as x86, and the problem of the general computing platforms is that the hardware capability of end-side equipment cannot be provided.

Disclosure of Invention

The disclosure provides an image processing method and device, an electronic device and a storage medium.

According to an aspect of the present disclosure, there is provided an image processing method including: obtaining a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function, wherein the simulation library comprises a function module for simulating the at least one function of the hardware resource; processing the image coding information of the image to be processed acquired by the hardware resource through the simulation library to acquire an image processing result; and storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

According to the image processing method disclosed by the embodiment of the disclosure, the hardware resources can be simulated through the functional interfaces of the hardware resources to obtain the simulation library of the hardware resources, so that the functions of the hardware resources of the end-side equipment can be simulated on the general computing platform to correspondingly process the image, and the image processing result can be obtained.

In one possible implementation, obtaining a simulation library of at least one function of a hardware resource of a peer device according to an interface of the at least one function includes: reading the interface to obtain the statement information of at least one function; performing null implementation on the declaration information to obtain a null implementation result of at least one function; determining a target function among the at least one function according to the null-realization result; and establishing a functional module in the simulation library according to the statement information of the target function, wherein the functional module is used for realizing the target function.

By the method, the target function needing to be simulated can be determined through empty implementation, simulation is carried out based on the statement information, the function module is obtained, and the simulation library is constructed, so that the general computing platform can simulate the function of the end-side equipment, and development and debugging are facilitated.

In one possible implementation, the method further includes: and decompressing the image to be processed through the simulation library to obtain the image coding information of the image to be processed.

By the mode, the image coding information convenient for image processing can be obtained through the functional module in the simulation library, and the decompression efficiency is improved.

In one possible implementation, the method further includes: and storing the image processing result in the header information corresponding to the image identifier of the image to be processed.

In this way, the convenience of searching and calling can be improved by storing the image processing result in the unified header information.

In one possible implementation, the method further includes: and searching an image processing result corresponding to the function in the result coding information according to the name of the function.

In one possible implementation, the method further includes: coding the result coding information to obtain push information; and pushing the push information to terminal equipment for displaying.

In this way, the push information which is convenient for the user to obtain can be obtained, and the user can conveniently use the result coding information.

In one possible implementation, the preset field of the image coding information includes a tail field of the image coding information.

By the method, the image processing result is uniformly stored in the tail field and has a preset naming mode, so that the image processing result can be searched based on the name of the processed or functional image, and the convenience of searching and calling is improved.

According to an aspect of the present disclosure, there is provided an image processing apparatus including: a simulation library obtaining module, configured to obtain a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function, where the simulation library includes a function module for simulating the at least one function of the hardware resource; the processing module is used for processing the image coding information of the image to be processed acquired by the hardware resource through the simulation library to acquire an image processing result; and the first storage module is used for storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

In one possible implementation, the simulation library obtaining module is further configured to: reading the interface to obtain the statement information of at least one function; performing null implementation on the declaration information to obtain a null implementation result of at least one function; determining a target function among the at least one function according to the null-realization result; and establishing a functional module in the simulation library according to the statement information of the target function, wherein the functional module is used for realizing the target function.

In one possible implementation, the apparatus further includes: and the decompression module is used for decompressing the image to be processed through the simulation library to obtain the image coding information of the image to be processed.

In one possible implementation, the apparatus further includes: and the second storage module is used for storing the image processing result in the header information corresponding to the image identifier of the image to be processed.

In one possible implementation, the apparatus further includes: and the searching module is used for searching the image processing result corresponding to the function in the result coding information according to the name of the function.

In one possible implementation, the apparatus further includes: the pushing module is used for coding the result coding information to obtain pushing information; and pushing the push information to terminal equipment for displaying.

According to an aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.

According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

FIG. 1 shows a flow diagram of an image processing method according to an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a simulation library according to an embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of header information according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram illustrating a preset information format of a tail field of image coding information according to an embodiment of the present disclosure;

fig. 5 shows an application diagram of an image processing method according to an embodiment of the present disclosure;

fig. 6 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 7 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure;

FIG. 8 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

Fig. 1 illustrates a flowchart of an image processing method according to an embodiment of the present disclosure, which includes, as illustrated in fig. 1:

in step S11, obtaining a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function, wherein the simulation library includes a function module for simulating the at least one function of the hardware resource;

in step S12, processing, by the simulation library, image coding information of the to-be-processed image acquired by the hardware resource to obtain an image processing result;

in step S13, the image processing result is stored in a preset field of the image coding information according to a preset information format, and result coding information is obtained.

In a possible implementation manner, the image processing method may be executed by an electronic device such as a terminal device or a server, and the electronic device may serve as the computing platform. The terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, etc., and the method may be implemented by a processor invoking computer readable instructions stored in a memory. Alternatively, the method may be performed by a server.

In one possible implementation, the hardware resources may include hardware resources of an end-side device, such as a smart camera, which may be emulated by a general purpose computing platform (e.g., x86 platform), such as GPU/NPU/DSP/NNIE, and so on. The relevant code of the hardware resource of the end-side device such as a smart camera often occupies only a small part (e.g. typically less than 5%) of the total code, and the pure CPU code unrelated to the hardware resource often occupies a significant part (e.g. typically more than 95%) of the total code. Most complex service logic codes occur in pure CPU codes, and the codes have the characteristics of complex logic, easy error, easy code corrosion, reconstruction and the like, so that the development and debugging through an embedded system of the end-side equipment are very complex, the development and debugging difficulty can be greatly reduced if a general computing platform is used for development and debugging, and the efficiency is improved. Therefore, functions related to hardware resources of the end-side device (for example, functions realized by simulating the small part of code) are simulated through the computing platform, so that all functions of the end-side device can be realized on the general computing platform, and the method has an important meaning for improving development and debugging efficiency.

Fig. 2 illustrates a schematic diagram of a simulation library according to an embodiment of the present disclosure, as shown in fig. 2, in an end-side device (e.g., a smart camera), an embedded system is typically used, a chip vendor of the embedded system may typically provide an SDK library (e.g., libchipsdk. so library), a developer may develop, e.g., write code, based on an interface of the SDK in the SDK library, and a developed function (e.g., application software) may call a function or an operator in the SDK through the interface, thereby calling a hardware resource of the end-side device to perform an operation. However, as described above, the development and debugging process is very complicated, and therefore, a simulation library (for example, lib mocksdk. so library) can be obtained by simulating the above SDK library on a general-purpose computing platform (for example, x86 platform), so that various functions can be developed and debugged on the general-purpose computing platform, and the developed functions (for example, application software) can call an interface in the simulation library, so that a CPU of the general-purpose computing platform is called to perform operation. Therefore, the development and debugging purposes on the general computing platform can be realized, the development and debugging difficulty is reduced, and the development and debugging efficiency is improved.

In one possible implementation manner, in step S11, a simulation library of the hardware resource may be obtained according to an interface provided by at least one function of the hardware resource, for example, a function (e.g., the above-mentioned SDK) capable of calling the hardware resource of the end-side device is simulated to obtain a function module capable of simulating the function, and a library formed by function modules corresponding to a plurality of functions is determined as the simulation library, which may be used to simulate an SDK library formed by a plurality of SDKs of the end-side device. The function module in the simulation library can simulate the function of the SDK, but the calculation resource which can be called by the function module is the CPU of the general computing platform, but not the hardware resource of the end-side equipment which can be called by the SDK, so that the purpose of developing and debugging various functions (such as application software) developed based on the function module on the general computing platform is realized, the difficulty of development and debugging is reduced, and the efficiency of development and debugging is improved.

In one possible implementation, step S11 may include: reading the interface to obtain the statement information of at least one function; performing null implementation on the declaration information to obtain a null implementation result of at least one function; determining a target function among the at least one function according to the null-realization result; and establishing a functional module in the simulation library according to the statement information of the target function, wherein the functional module is used for realizing the target function.

In one possible implementation, the function modules in the simulation library can be generated, so as to obtain the simulation library. In generating the functional module, it may be generated based on the interface of the corresponding SDK. As described above, the interface of the SDK is used to call a function or an operator in the SDK, so that development can be completed by using the function of the SDK.

In a possible implementation manner, an interface of the SDK may be read by using an information reading tool, for example, declaration information of a function or an operator in the SDK may be read, for example, functions, logics, involved variables, and the like of the function or the operator may be included in the declaration information, and the disclosure does not limit what the declaration information includes. In an example, readelf information reading tools can be used to read the interface of the SDK and obtain declaration information of functions or operators in the SDK.

In one possible implementation, after obtaining the declaration information, a null implementation may be performed based on the declaration information. For example, a constructor without parameters may be generated, the object in the constructor is empty, and the function of the constructor is mainly to return an empty implementation result based on the declaration information, for example, the empty implementation result is returned based on the information of the function, the property, the category, and the like described in the declaration function. For example, if the function output of the SDK is a pointer, the NULL implementation result returned by the NULL implementation is NULL, if the function output of the SDK is integer data, the NULL implementation result returned by the NULL implementation is 0, and if the function of the SDK is a typeless function (void function), the NULL implementation result returned by the NULL implementation is NULL, or no result is returned. In an example, the null implementation can be performed by an open source parser (ANTLR) and a null implementation tool obtained from a version-specific function syntax (e.g., grammar version grammar-v 4 of the c language) to obtain a null implementation result. After the null implementation is carried out, a constructor is obtained in the general computing platform, so that the compiling can be successfully carried out in the general computing platform.

In one possible implementation, the target function that needs to be simulated may be selected based on the null implementation results of various functions (e.g., SDKs). In an example, if the SDK is used to create a resource or close a resource, it does not need to use a hardware resource of the end-side device, and therefore, it does not need to simulate it, and by the null implementation process described above, a build function whose object is null can be obtained to simulate the function of the function. If the SDK needs to use the hardware resource of the end-side device, it needs to be simulated, and the null implementation result returned by the null implementation of this function is usually not null, so that the function (SDK) whose null implementation result is not null can be determined as the target function, and the target function is simulated to obtain the function module in the simulation library.

In one possible implementation, a function consistent with the function may be called in a function library of the general-purpose computing platform based on declaration information of the target function, for example, in an SDK with an image processing function, a function module corresponding thereto may be obtained based on a function in a libjpeg library, the function module may be capable of executing a function consistent with the target function, and may also be capable of providing various interfaces, for example, an interface consistent with a function interface of the end-side device.

In a possible implementation manner, taking a smart camera as an example, the image processing process generally includes image acquisition (i.e., capturing an image or a video frame), decompression, and a series of detection, recognition, and other processes, which can all be implemented by the above simulation library. In the end-side device (for example, a smart camera), a developer may utilize interfaces of multiple functions of the end-side device to develop a function for processing an image, for example, to write a function that may call the multiple functions of the end-side device through the interfaces of the multiple functions of the end-side device, so as to utilize hardware resources of the end-side device to decompress, detect, and recognize the image.

In one possible implementation, the method further includes: and decompressing the image to be processed through the simulation library to obtain the image coding information of the image to be processed. In an example, the image to be processed is an image captured by an end-side device (e.g., a smart camera), and the image to be processed may be a single captured image or a video frame in a video stream. In an example, the image to be processed may be a jpeg format image, which may be decompressed by a corresponding functional module in the simulation library to obtain image coding information, e.g., YUV information.

In one possible implementation, in step S12, the image encoding information may be further processed based on the simulation library, for example, detection, identification, and the like as described above. Step S12 may include: and performing at least one of target detection, target attribute extraction, target feature extraction and target feature comparison on the image coding information through the simulation library to obtain the image processing result.

In a possible implementation manner, the simulation library may include a target detection module, an attribute extraction module, a feature comparison module, and other functional modules, which may be respectively used to execute target detection, target attribute extraction, target feature extraction, and target feature comparison processing on a general computing platform. The interface of the functional module can be called, so as to call at least one of the functional modules, so as to utilize the CPU of the general-purpose computing platform to perform at least one of target detection, target attribute extraction, target feature extraction, and target feature comparison processing on the image, so as to respectively obtain image processing results of various processing, for example, at least one of image processing results such as obtaining a target detection result, obtaining an attribute extraction result, a feature extraction result, and a feature comparison result.

In a possible implementation manner, the multiple image processing results may be saved, the functional module that obtains the image processing result may be stored in correspondence with the corresponding image processing result, and when storing, the functional module may also be in correspondence with an image identifier (e.g., a frame number, etc.) of the image to be processed. For example, the image processing result may be stored in a format of "frame number — function module name". In an example, 100_ detect. data may represent a target detection result obtained by the target detection module performing the target detection process on the image coding information of the 100 th frame image. Data may represent an attribute extraction result obtained by the attribute extraction module performing target attribute extraction processing on the image coding information of the 200 th frame image, and the like. In the example, each image does not need to be processed by all the functional modules, and each functional module does not need to process all the images, for example, if a certain frame of image is subjected to the target detection processing to obtain the target detection result, but if the attribute extraction processing is not performed, the frame of image does not have the attribute extraction result.

In one possible implementation, the method further includes: and storing the image processing result in the header information corresponding to the image identifier of the image to be processed. Various image processing results obtained by each frame of image can be stored in the header information corresponding to the frame of image, for example, if the image coding information of 100 frames of image is subjected to two processes of target detection and target attribute extraction, the obtained target detection result 100_ detect and attribute extraction result 100_ attribute can be stored in the header information jpeg 1 corresponding to the 100 th frame of image, and the disclosure does not limit the type of the header information.

Fig. 3 illustrates a schematic diagram of header information according to an embodiment of the present disclosure, and in an example, the image processing result of each frame image may be stored in the header information jpeg 1 corresponding to the image identifier (e.g., frame number) of each frame in the form illustrated in fig. 3. Thus, a canonical image processing result can be obtained in a form to facilitate the search and call of the image processing result.

In this way, the image processing result can be stored in the unified header information, and the convenience of searching and calling can be improved.

In a possible implementation manner, as the processing procedure progresses, the header information corresponding to each frame may store the image processing result, for example, the header information jpeg 1 corresponding to the 100 th frame image stores the target detection result 100_ detect.data and the attribute extraction result 100_ attribute.data, and the header information jpeg 1 corresponding to the other frame images may also store a plurality of image processing results. After the processing process is completed, for example, after all the images of a certain batch have been processed by the image processing method, all the image processing results to be obtained corresponding to the images can be obtained and stored in the header information corresponding to each frame. In step S13, the image processing result may be stored in a preset field of the image encoding information in a preset information format. For example, the image processing result stored in the header information corresponding to each frame is stored in a preset field of the image coding information of each frame, and the result coding information, that is, the image coding information in which the image processing result is stored is obtained.

In one possible implementation, the preset field of the image coding information includes a tail field of the image coding information. Therefore, the image processing result stored in the header information corresponding to each frame can be shifted to the preset field of the image coding information of each frame according to the preset information format, and the result coding information can be obtained.

Fig. 4 is a schematic diagram illustrating a preset information format of a tail field of image coding information according to an embodiment of the present disclosure, and as shown in fig. 4, the preset information format may be consistent with a format stored in header information, and thus, when storing an image processing result to the tail field of the image coding information (e.g., YUV), it is only necessary to shift to the tail field without changing the storage format.

In one possible implementation, after obtaining the result encoding information, push information, i.e., information that the user can use (e.g., can display, or can perform further calculations), can also be obtained based on the result encoding information, and thus can be pushed to the user. The method further comprises the following steps: coding the result coding information to obtain push information; and pushing the push information to terminal equipment for displaying. In an example, the result coding information is image coding information (e.g., YUV) whose tail field stores the image processing result, the result coding information may be further encoded, e.g., JPEG encoded, to obtain push information in an image format, e.g., a frame number may be used as a name of the push information, such as 100.JPEG, 101.JPEG, etc., to respectively represent the push information of the 100 th frame and the push information of the 101 th frame. The push information in the image format may represent the image processing result, for example, the image processing result of a certain frame of image may include a target detection result of a target object in the image, and then a detection result of a target detection frame or a segmentation frame may be included in the push information in the image format. The present disclosure does not limit the format of the push information.

In a possible implementation manner, according to the name of the function, the image processing result corresponding to the function is searched in the result coding information. When the image processing result is queried, the result coding information may be queried, where the result coding information includes image processing results obtained by various functional modules, and may include at least one of an object detection result obtained by the object detection module, an attribute extraction result obtained by the attribute extraction module, a feature extraction result obtained by the feature extraction module, a feature comparison result obtained by the feature comparison module, and other image processing results. When inquiring the image processing result of a certain frame of image, inquiring the result coding information thereof can be carried out, and the inquiring mode can comprise inquiring the target detection result obtained by the target detection module through the keyword of detect in the result coding information of the 100 th frame of image, for example, when inquiring the target detection result of the 100 th frame of image, and information such as the position coordinate of the detection frame can be obtained. The present disclosure does not limit the query method.

According to the image processing method disclosed by the embodiment of the disclosure, the target function to be simulated can be determined through empty implementation, simulation is performed based on the declaration information, the function module is obtained, and the simulation library is constructed, so that the function of simulating hardware resources on the general computing platform can be used for performing corresponding processing on the image, an image processing result is obtained, the general computing platform can be used for simulating the hardware resources, development and debugging of the functions of the hardware resources and other business logics can be facilitated according to the image processing result, and the development and debugging efficiency can be improved.

Fig. 5 shows an application schematic diagram of the image processing method according to the embodiment of the present disclosure, as shown in fig. 5, the end-side device may include a smart camera, and the smart camera may include multiple functions, and in the process of developing and debugging application software based on these functions, the difficulty of developing and debugging on the smart camera is large, and the efficiency is low, so that some functions on the smart camera, which need to call hardware resources of the smart camera, may be simulated on a general-purpose computing platform such as x86, to obtain a simulation library.

As shown in fig. 5, the simulation library may include a target detection module, an attribute extraction module, a feature extraction module, and a feature comparison module, that is, a module for intelligent operation, which may respectively simulate a function corresponding to the smart camera to process image coding information (YUV) of an image, and may invoke CPU resources of the general computing platform to process the YUV in the process of processing, so as to obtain an image processing result, for example, a target detection result obtained by the target detection module, an attribute extraction result obtained by the attribute extraction module, a feature extraction result obtained by the feature extraction module, and a feature comparison result obtained by the feature comparison module. And stores the obtained image processing results in header information jpeg 1 corresponding to the frame numbers of the images, respectively.

In a possible implementation manner, after the processing process is completed, the image processing result stored in the header information may be shifted to a tail field of image coding information (YUV) to obtain result coding information, and the result coding information may be coded to obtain push information in an image format, so that the push information may be pushed to a terminal of a user for display or further processing, so that the user may debug and develop a new function based on the push information, and the like.

In a possible implementation manner, the image processing method may be used to simulate a specific function in an end-side device such as a smart camera on a general computing platform, so as to simulate a hardware resource of the end-side device through a CPU resource of the general computing platform to process the specific function, so that development and debugging of application software of the end-side device can be performed on the general computing platform, thereby reducing difficulty in development and debugging, and improving efficiency of development and debugging. The present disclosure does not limit the application field of the image processing method.

It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.

In addition, the present disclosure also provides an image processing apparatus, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any one of the image processing methods provided by the present disclosure, and the descriptions and corresponding descriptions of the corresponding technical solutions and the corresponding descriptions in the methods section are omitted for brevity.

Fig. 6 illustrates a block diagram of an image processing apparatus according to an embodiment of the present disclosure, which includes, as illustrated in fig. 6: a simulation library obtaining module 11, configured to obtain a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function, where the simulation library includes a function module for simulating the at least one function of the hardware resource; the processing module 12 is configured to process, through the simulation library, image coding information of an image to be processed, which is obtained by the hardware resource, to obtain an image processing result; the first storage module 13 is configured to store the image processing result in a preset field of the image coding information according to a preset information format, so as to obtain result coding information.

The method has specific technical relevance with the internal structure of the computer system, and can solve the technical problems of how to improve the hardware operation efficiency or the execution effect (including reducing data storage capacity, reducing data transmission capacity, improving hardware processing speed and the like), thereby obtaining the technical effect of improving the internal performance of the computer system according with the natural law.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-mentioned method. The computer readable storage medium may be a volatile or non-volatile computer readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.

The disclosed embodiments also provide a computer program product comprising computer readable code or a non-transitory computer readable storage medium carrying computer readable code, which when run in a processor of an electronic device, the processor in the electronic device performs the above method.

The electronic device may be provided as a terminal, server, or other form of device.

Fig. 7 illustrates a block diagram of an electronic device 800 in accordance with an embodiment of the disclosure. For example, the electronic device 800 may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or other terminal device.

Referring to fig. 7, electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile and non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G), a long term evolution of universal mobile communication technology (LTE), a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like. The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

Fig. 8 illustrates a block diagram of an electronic device 1900 in accordance with an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server or terminal device. Referring to fig. 8, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system, such as a Microsoft Server operating system (Windows Server), stored in the memory 1932^TM) Apple Inc. of the present application based on the graphic user interface operating System (Mac OS X)^TM) Multi-user, multi-process computer operating system (Unix)^TM) Free and open native code Unix-like operating System (Linux)^TM) Open native code Unix-like operating System (FreeBSD)^TM) Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK) or the like.

The foregoing description of the various embodiments is intended to highlight various differences between the embodiments, and the same or similar parts may be referred to each other, and for brevity, will not be described again herein.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

If the technical scheme of the application relates to personal information, a product applying the technical scheme of the application clearly informs personal information processing rules before processing the personal information, and obtains personal independent consent. If the technical scheme of the application relates to sensitive personal information, a product applying the technical scheme of the application obtains individual consent before processing the sensitive personal information, and simultaneously meets the requirement of 'express consent'. For example, at a personal information collection device such as a camera, a clear and significant identifier is set to inform that the personal information collection range is entered, the personal information is collected, and if the person voluntarily enters the collection range, the person is considered as agreeing to collect the personal information; or on the device for processing the personal information, under the condition of informing the personal information processing rule by using obvious identification/information, obtaining personal authorization by modes of popping window information or asking a person to upload personal information of the person by himself, and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing method, and a type of personal information to be processed.

Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or improvements to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, comprising:

obtaining a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function, wherein the simulation library comprises a function module for simulating the at least one function of the hardware resource;

processing the image coding information of the image to be processed acquired by the hardware resource through the simulation library to acquire an image processing result;

and storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

2. The method of claim 1, wherein obtaining a simulation library of at least one function of a hardware resource of a peer device according to an interface of the at least one function comprises:

reading the interface to obtain the statement information of at least one function;

performing null implementation on the declaration information to obtain a null implementation result of at least one function;

determining a target function among the at least one function according to the null-realization result;

and establishing a functional module in the simulation library according to the statement information of the target function, wherein the functional module is used for realizing the target function.

3. The method of claim 1, further comprising:

and decompressing the image to be processed through the simulation library to obtain the image coding information of the image to be processed.

4. The method of claim 1, further comprising:

and storing the image processing result in the header information corresponding to the image identifier of the image to be processed.

5. The method of claim 1, further comprising:

and searching an image processing result corresponding to the function in the result coding information according to the name of the function.

6. The method of claim 1, further comprising:

coding the result coding information to obtain push information;

and pushing the push information to terminal equipment for displaying.

7. The method of claim 1, wherein the predetermined field of the image coding information comprises a tail field of the image coding information.

8. An image processing apparatus characterized by comprising:

a simulation library obtaining module, configured to obtain a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function, where the simulation library includes a function module for simulating the at least one function of the hardware resource;

the processing module is used for processing the image coding information of the image to be processed, which is acquired by the hardware resource, through the simulation library to acquire an image processing result;

and the first storage module is used for storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

9. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to invoke the memory-stored instructions to perform the method of any of claims 1 to 7.

10. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 7.