WO2020207200A1 - 图像处理方法、图像处理器、拍摄装置和电子设备 - Google Patents

图像处理方法、图像处理器、拍摄装置和电子设备 Download PDF

Info

Publication number
WO2020207200A1
WO2020207200A1 PCT/CN2020/079534 CN2020079534W WO2020207200A1 WO 2020207200 A1 WO2020207200 A1 WO 2020207200A1 CN 2020079534 W CN2020079534 W CN 2020079534W WO 2020207200 A1 WO2020207200 A1 WO 2020207200A1
Authority
WO
WIPO (PCT)
Prior art keywords
module
data
processing
application
post
Prior art date
Application number
PCT/CN2020/079534
Other languages
English (en)
French (fr)
Inventor
何新兰
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2020207200A1 publication Critical patent/WO2020207200A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/617—Upgrading or updating of programs or applications for camera control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof

Definitions

  • This application relates to the field of image processing technology, and more specifically, to an image processing method, image processor, photographing device, and electronic equipment.
  • the embodiments of the present application provide an image processing method, an image processor, a photographing device, and electronic equipment.
  • the image processing method of the embodiment of the present application includes: an application program module requests original data from a camera service module; after receiving the request, the camera service module sends the request to the hardware abstraction module; the hardware abstraction module After receiving the request, send the original data to the application module via the camera service module; after receiving the original data, the application module sends the original data to the algorithm for post-processing Module; and the algorithm post-processing module performs post-processing on the original data and sends the processed result data back to the application module.
  • the image processor of the embodiment of the present application includes an application program module, a camera service module, a hardware abstraction module, and an algorithm post-processing module.
  • the application module is used to request original data from the camera service module.
  • the camera service module is configured to send the request to the hardware abstraction module after receiving the request.
  • the hardware abstraction module is configured to send the original data to the application program module via the camera service module after receiving the request.
  • the application module is further configured to send the original data to the algorithm post-processing module after receiving the original data.
  • the algorithm post-processing module is used for post-processing the original data by taking pictures and transmitting the processed result data back to the application program module.
  • the photographing device in the embodiment of the present application includes an image processor and an image sensor, and the image sensor is connected to the image processor.
  • the image processor includes an application program module, a camera service module, a hardware abstraction module and an algorithm post-processing module.
  • the application module is used to request original data from the camera service module.
  • the camera service module is configured to send the request to the hardware abstraction module after receiving the request.
  • the hardware abstraction module is configured to send the original data to the application program module via the camera service module after receiving the request.
  • the application module is further configured to send the original data to the algorithm post-processing module after receiving the original data.
  • the algorithm post-processing module is used for post-processing the original data by taking pictures and transmitting the processed result data back to the application program module.
  • the electronic device of the embodiment of the present application includes a photographing device and a housing, and the photographing device is combined with the housing.
  • the photographing device includes an image processor and an image sensor, and the image sensor is connected to the image processor.
  • the image processor includes an application program module, a camera service module, a hardware abstraction module and an algorithm post-processing module.
  • the application module is used to request original data from the camera service module.
  • the camera service module is configured to send the request to the hardware abstraction module after receiving the request.
  • the hardware abstraction module is configured to send the original data to the application program module via the camera service module after receiving the request.
  • the application module is further configured to send the original data to the algorithm post-processing module after receiving the original data.
  • the algorithm post-processing module is used for post-processing the original data by taking pictures and transmitting the processed result data back to the application program module.
  • FIG. 1 is a schematic diagram of modules of an image processor in some embodiments of the present application.
  • FIG. 2 is a schematic flowchart of an image processing method according to some embodiments of the present application.
  • FIG. 3 is a schematic flowchart of an image processing method according to some embodiments of the present application.
  • FIG. 4 is a schematic flowchart of an image processing method according to some embodiments of the present application.
  • FIG. 5 is a schematic flowchart of an image processing method according to some embodiments of the present application.
  • FIG. 6 is a schematic flowchart of an image processing method according to some embodiments of the present application.
  • FIG. 7 is a schematic flowchart of an image processing method according to some embodiments of the present application.
  • FIG. 8 is a schematic diagram of modules of a photographing device according to some embodiments of the present application.
  • FIG. 9 is a schematic diagram of an algorithm post-processing module of some embodiments of the present application.
  • FIG. 10 is a schematic diagram of the structure of an electronic device according to some embodiments of the present application.
  • FIG. 11 is a schematic diagram of the structure of an electronic device according to some embodiments of the present application.
  • the image processing method of the embodiment of the present application includes:
  • the application module 14 requests original data from the camera service module 18;
  • the camera service module 18 After receiving the request, the camera service module 18 sends the request to the hardware abstraction module 12;
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 via the camera service module 18;
  • the algorithm post-processing module 16 performs photographing post-processing on the original data and transmits the processed result data back to the application program module 14.
  • the original data requested by the application module 14 from the camera service module 18 is multiple frames.
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 (ie 03) via the camera service module 18, including:
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sequentially sends multiple frames of original data to the application module 14 via the camera service module 18;
  • the application module 14 After receiving the original data, the application module 14 sends the original data to the algorithm post-processing module 16 (ie 04), including:
  • the application module 14 After receiving the multi-frame original data, the application module 14 packs the multi-frame original data and sends it to the algorithm post-processing module 16.
  • the application module 14 can close or exit the application interface during the post-processing of the original data by the algorithm post-processing module 16.
  • the image processing method further includes:
  • the camera service module 18 sends suggestions on the number of frames of original data to the application module 14;
  • the application module 14 requests the camera service module 18 for raw data (ie 01), including:
  • the application module 14 requests the camera service module 18 for raw data according to the number of frames.
  • the image processing method further includes:
  • the camera service module 18 sends an algorithm proposal for post-processing the raw data to the application module 14;
  • the application module 14 sets a predetermined algorithm according to the algorithm suggestion and sends the predetermined algorithm to the hardware abstraction module 12;
  • the algorithm post-processing module 16 takes pictures and post-processes the original data and sends the processed result data back to the application program module 14 (ie 05), including:
  • the algorithm post-processing module 16 performs photographing post-processing on the original data according to a predetermined algorithm, and transmits the processed result data back to the application program module 14.
  • the original data includes to-be-processed data and metadata.
  • the application module 14 requests the camera service module 18 for raw data (ie 01), including:
  • the application module 14 requests the camera service module 18 for data to be processed
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 (ie 03) via the camera service module 18, including:
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the data to be processed and the metadata corresponding to the data to be processed to the application module 14 via the camera service module 18;
  • the application module 14 After receiving the original data, the application module 14 sends the original data to the algorithm post-processing module 16 (ie 04), including:
  • the application module 14 After receiving the data and metadata to be processed, the application module 14 sends the data and metadata to be processed to the algorithm post-processing module 16;
  • the algorithm post-processing module 16 takes pictures and post-processes the original data and sends the processed result data back to the application program module 14 (ie 05), including:
  • the algorithm post-processing module 16 performs photographing post-processing on the data to be processed according to the metadata and sends the processed result data back to the application program module 14.
  • the data in the hardware abstraction module 12 is in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is also data in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is data in the second image format, and the image processing method further includes:
  • the hardware abstraction module 12 converts the data in the first image format into data in the second image format.
  • the image processor 10 of the embodiment of the present application includes an application program module 14, a camera service module 18, a hardware abstraction module 12 and an algorithm post-processing module 16.
  • the application module 14 is used to request raw data from the camera service module 18.
  • the camera service module 18 is configured to send the request to the hardware abstraction module 12 after receiving the request.
  • the hardware abstraction module 12 is used to send the original data to the application module 14 via the camera service module 18 after receiving the request.
  • the application module 14 is also used to send the original data to the algorithm post-processing module 16 after receiving the original data.
  • the algorithm post-processing module 16 is used to perform post-processing on the original data and send the processed result data back to the application module 14.
  • the original data requested by the application module 14 from the camera service module 18 is multiple frames.
  • the hardware abstraction module 12 can be used to execute the method in 032, and the application module 14 can be used to execute the method in 042.
  • the hardware abstraction module 12 is used to sequentially send multiple frames of raw data to the application module 14 via the camera service module 18 after receiving the request.
  • the application module 14 is used to package and send the multi-frame original data to the algorithm post-processing module 16 after receiving the multi-frame original data.
  • the application module 14 can close or exit the application interface during the post-processing of the original data by the algorithm post-processing module 16.
  • the camera service module 18 is also used to send suggestions on the number of frames of raw data to the application module 14.
  • the application module 14 is also used to request raw data from the camera service module 18 according to the frame number recommendation.
  • the camera service module 18 is also used to send to the application program module 14 algorithm suggestions for post-processing the raw data.
  • the application module 14 is also used to set a predetermined algorithm according to the algorithm suggestion and send the predetermined algorithm to the hardware abstraction module 12.
  • the algorithm post-processing module 16 is also used for post-processing the raw data according to a predetermined algorithm, and transmits the processed result data back to the application module 14.
  • the original data includes to-be-processed data and metadata.
  • the application program module 14 is used to request the data to be processed from the camera service module 18.
  • the hardware abstraction module 12 is configured to send the data to be processed and the metadata corresponding to the data to be processed to the application module 14 via the camera service module 18 after receiving the request.
  • the application module 14 is configured to send the data and metadata to be processed to the algorithm post-processing module 16 after receiving the data and metadata to be processed.
  • the algorithm post-processing module 16 is used to perform post-photographing and post-processing of the data to be processed according to the metadata and send the processed result data back to the application module 14.
  • the data in the hardware abstraction module 12 is in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is also data in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is data in the second image format.
  • the hardware abstraction module 12 is also used to convert data in the first image format into data in the second image format.
  • the photographing device 100 of the embodiment of the present application includes the image processor 10 and the image sensor 20 of any of the above embodiments.
  • the image processor 10 and the image sensor 20 are connected.
  • the electronic device 1000 of the embodiment of the present application includes the photographing device 100 and the housing 200 of any of the above embodiments, and the photographing device 100 is combined with the housing 200.
  • the image processing method of the embodiment of the present application includes:
  • the application module 14 requests original data from the camera service module 18;
  • the camera service module 18 After receiving the request, the camera service module 18 sends the request to the hardware abstraction module 12;
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 via the camera service module 18;
  • the algorithm post-processing module 16 performs photographing post-processing on the original data and transmits the processed result data back to the application program module 14.
  • the image processor 10 of the embodiment of the present application includes an application program module (APP) 14, a camera service module (Camera Service) 18, a hardware abstraction layer (HAL) 12, and an algorithm post-processing module (Algo Process Service, APS) 16.
  • the image processing method of the embodiment of the present application can be implemented by the image processor 10 of the embodiment of the present application.
  • the application module 14 can be used to execute the methods in 01 and 04
  • the camera service module 18 can be used to execute the methods in 02
  • the hardware abstraction module 12 can be used to execute the methods in 03
  • the algorithm post-processing module 16 can be used to execute the methods in 05. Methods.
  • the application module 14 can be used to request raw data from the camera service module 18.
  • the camera service module 18 may be used to send the request to the hardware abstraction module 12 after receiving the request.
  • the hardware abstraction module 12 can be used to send the original data to the application module 14 via the camera service module 18 after receiving the request.
  • the application module 14 may also be used to send the original data to the algorithm post-processing module 16 after receiving the original data.
  • the algorithm post-processing module 16 can be used to perform post-processing on the original data and send the processed result data back to the application module 14.
  • the image processing method and image processor 10 of the embodiment of the present application implement data shunting through the algorithm post-processing module 16, which can reduce the load of the hardware abstraction module 12, so that the system runs smoothly without stuttering, and the algorithm post-processing module 16
  • the post-processing of the photo can be run in the background without the user having to wait for a long time, which greatly improves the photo experience.
  • the application module 14 requests the camera service module 18 for raw data, the raw data is image data, the raw data can be RAW format data or YUV format data; the camera service module 18 sends the request after receiving the request To the hardware abstraction module 12; after receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 via the camera service module 18, that is, the hardware abstraction module 12 first sends the original data to the camera service module 18. The camera service module 18 and then send the original data to the application module 14. After receiving the original data, the application module 14 sends the original data to the algorithm post-processing module 16.
  • the algorithm post-processing module 16 stores at least one image processing algorithm.
  • Processing algorithms include multi-frame processing algorithms, beauty processing algorithms, blur processing algorithms, High-Dynamic Range (HDR) processing algorithms, filter processing algorithms, rotation processing algorithms, watermark processing algorithms, etc.; post-processing algorithms
  • the module 16 takes pictures and post-processes the raw data and sends the processed result data back to the application module 14.
  • the algorithm post-processing module 16 uses the above-mentioned image processing algorithm to process data in RAW format or data in YUV format, and The processed result data is sent back to the application module 14.
  • the original data requested by the application module 14 from the camera service module 18 is multiple frames.
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 (ie 03) via the camera service module 18, including:
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sequentially sends multiple frames of original data to the application module 14 via the camera service module 18;
  • the application module 14 After receiving the original data, the application module 14 sends the original data to the algorithm post-processing module 16 (ie 04), including:
  • the application module 14 After receiving the multi-frame original data, the application module 14 packs the multi-frame original data and sends it to the algorithm post-processing module 16.
  • the original data requested by the application module 14 from the camera service module 18 is multiple frames.
  • the hardware abstraction module 12 can be used to execute the method in 032, and the application module 14 can be used to execute the method in 042.
  • the hardware abstraction module 12 can be used to sequentially send multiple frames of raw data to the application module 14 via the camera service module 18 after receiving the request.
  • the application module 14 may be used to package and send the multi-frame original data to the algorithm post-processing module 16 after receiving the multi-frame original data.
  • the original data requested by the application module 14 from the camera service module 18 is 4 frames.
  • the camera service module 18 sends the request to the hardware abstraction module 12; after the hardware abstraction module 12 receives the request, first, The hardware abstraction module 12 sends the original data of the first frame to the camera service module 18, and the camera service module 18 sends the original data of the first frame to the application module 14; secondly, the hardware abstraction module 12 sends the original data of the second frame to the camera service Module 18, the camera service module 18 sends the original data of the second frame to the application module 14; then, the hardware abstraction module 12 sends the original data of the third frame to the camera service module 18, and the camera service module 18 sends the original data of the third frame To the application module 14; finally, the hardware abstraction module 12 sends the raw data of the fourth frame to the camera service module 18, and the camera service module 18 sends the raw data of the fourth frame to the application module 14.
  • the hardware abstraction module 12 sends the original data of the previous frame to the camera service module 18, it can start to send the original data of the next frame to the camera service module 18 without waiting for the camera service module 18 to send the previous
  • the raw data of the frame is sent to the application module 14 and then the raw data of the next frame is sent to the camera service module 18.
  • the total time required for sending multiple frames of raw data to the application module 14 via the camera service module 18 can be saved, and the user can avoid waiting for a long time, which is beneficial to improve the photographing experience.
  • the application module 14 After the application module 14 receives the original data of the first frame, the original data of the second frame, the original data of the third frame, and the original data of the fourth frame, the original data of the first frame, the original data of the second frame, and the original data of the third frame And the original data of the fourth frame are packaged and sent to the algorithm post-processing module 16. It can be understood that when the original data requested by the application module 14 from the camera service module 18 is multiple frames, the algorithm post-processing module 16 generally needs to post-process the multiple frames of raw data, and if the application module 14 sequentially performs multiple frames The original data is sent to the algorithm post-processing module 16.
  • the algorithm post-processing module 16 receives only the first frame of original data, or only the first frame of original data and the second frame of original data, or only the first frame of original data In the case of the original data of the second frame and the original data of the third frame, the algorithm post-processing module 16 cannot complete the post-photographing processing of the multi-frame original data.
  • the application module 14 packages and sends multiple frames of original data to the algorithm post-processing module 16, which can efficiently use transmission resources.
  • the application module 14 packages and sends multiple frames of original data to the algorithm post-processing module 16. After that, the algorithm post-processing module 16 can perform post-photographing and post-processing of the multiple frames of original data, and send the processed result data back to the application module 14.
  • the camera service module 18 may also implement the function of packaging multiple frames of original data.
  • the hardware abstraction module 12 sends multiple frames of raw data to the camera service module 18 in turn; the camera service module 18 packages and sends the multiple frames of raw data to the application module 14; After receiving the packed multi-frame original data, the packed multi-frame original data is sent to the algorithm post-processing module 16.
  • the application module 14 can close or exit the application interface during the post-processing of the original data by the algorithm post-processing module 16.
  • the user can selectively control the application module 14 to close or exit the application interface. That is to say, when the user still needs to use the application interface of the application module 14, the user can also keep the application interface of the application module 14 open, and when the user does not need to use the application interface of the application module 14, due to the algorithm post-processing
  • the process of post-processing the raw data by the module 16 does not involve the work of the application module 14.
  • the application module 14 can close or exit the application interface according to the user's control. The user does not need to wait for a long time on the application interface of the application module 14, resulting in unable to perform other operations.
  • the image processing method further includes:
  • the camera service module 18 sends suggestions on the number of frames of original data to the application module 14;
  • the application module 14 requests the camera service module 18 for raw data (ie 01), including:
  • the application module 14 requests the camera service module 18 for raw data according to the number of frames.
  • the camera service module 18 can be used to execute the method in 06, and the application module 14 can be used to execute the method in 012. That is to say, the camera service module 18 can also be used to send suggestions on the number of frames of raw data to the application module 14. The application module 14 can also be used to request raw data from the camera service module 18 according to the frame number recommendation.
  • the number of frames is recommended to indicate a specific value of the number of frames, such as 1 frame, 2 frames, 3 frames, 4 frames, and so on. If the camera service module 18 is recommended to send 4 frames of original data to the application module 14, the application module 14 requests the camera service module 18 for 4 frames of original data; if the camera service module 18 sends the application module 14 The suggested number of frames of raw data is 5 frames, and the application module 14 requests the camera service module 18 for 5 frames of raw data.
  • the hardware abstraction module 12 can obtain the sensitivity (ISO value) information, the jitter of the gyroscope, the AR scene detection results (detection scene types, such as people, animals, scenery, etc.), etc.
  • the hardware abstraction module 12 sends these information to the camera service Module 18, the camera service module 18 sends suggestions on the number of frames of the original data to the application module 14 based on the information. For example, when the jitter is large, the number of frames suggested by the camera service module 18 to the application module 14 can be relatively larger, so as to better realize the post-photographing processing; when there is almost no jitter, the camera service module 18 may report to the application module 14 14 The recommended number of frames can be relatively small to reduce the amount of data transmission.
  • the number of frames recommended by the camera service module 18 to the application module 14 is positively correlated with the jitter degree of the gyroscope.
  • the hardware abstraction module 12 can also directly give suggestions on the number of frames of the original data based on this information, and the camera service module 18 then forwards the suggestions on the number of frames of the original data to the application module 14.
  • the image processing method further includes:
  • the camera service module 18 sends an algorithm proposal for post-processing the raw data to the application module 14;
  • the application module 14 sets a predetermined algorithm according to the algorithm suggestion and sends the predetermined algorithm to the hardware abstraction module 12;
  • the algorithm post-processing module 16 takes pictures and post-processes the original data and sends the processed result data back to the application program module 14 (ie 05), including:
  • the algorithm post-processing module 16 performs photographing post-processing on the original data according to a predetermined algorithm, and transmits the processed result data back to the application program module 14.
  • the camera service module 18 can be used to execute the method in 07
  • the application module 14 can be used to execute the method in 08
  • the algorithm post-processing module 16 can be used to execute the method in 052.
  • the camera service module 18 can also be used to send algorithm suggestions for post-processing the raw data to the application module 14.
  • the application module 14 may also be used to set a predetermined algorithm according to the algorithm suggestion and send the predetermined algorithm to the hardware abstraction module 12.
  • the algorithm post-processing module 16 may also be used to perform post-processing on the original data according to a predetermined algorithm, and send the processed result data back to the application module 14.
  • both the algorithm proposal and the predetermined algorithm may be one or more of the aforementioned multi-frame processing algorithm, beauty processing algorithm, blur processing algorithm, HDR processing algorithm, filter processing algorithm, rotation processing algorithm, and watermark processing algorithm.
  • the algorithm for post-processing the original data sent by the camera service module 18 to the application module 14 is suggested to be a multi-frame processing algorithm
  • the application module 14 sets a multi-frame processing algorithm according to the algorithm recommendation, and sets the multi-frame processing algorithm Send to the hardware abstraction module 12
  • the algorithm post-processing module 16 performs post-processing on the original data according to the multi-frame processing algorithm, and sends the processed result data back to the application module 14;
  • the camera service module 18 sends the data to the application module 14
  • the sent algorithm recommendations for post-processing the raw data are the beauty processing algorithm and the HDR processing algorithm
  • the application module 14 sets the beauty processing algorithm and the HDR processing algorithm according to the algorithm recommendations, and combines the beauty processing algorithm and the HDR processing algorithm It is sent to the hardware abstraction module 12, and the algorithm
  • the hardware abstraction module 12 can obtain the sensitivity (ISO value) information, the jitter of the gyroscope, the AR scene detection results (detection scene types, such as people, animals, scenery, etc.), etc.
  • the hardware abstraction module 12 sends these information to the camera service Module 18, the camera service module 18 sends to the application module 14 algorithm suggestions for post-processing the raw data based on the information.
  • the camera service module 18 recommends a multi-frame processing algorithm to eliminate the jitter according to multi-frame processing; when the scene type detected by the AR scene detection result is a person, the camera service module 18 recommends beauty The color processing algorithm is used to beautify the person; when the scene type detected by the AR scene detection result is a landscape, the camera service module 18 suggests an HDR processing algorithm to form a landscape image with a high dynamic range.
  • the hardware abstraction module 12 can also directly give algorithm suggestions for post-processing the raw data based on this information, and the camera service module 18 forwards the algorithm suggestions for post-processing the raw data to the application module 14.
  • the image processing method further includes: the camera service module 18 sends to the application module 14 a proposal on the number of frames of the original data and an algorithm proposal for post-processing the original data; the application module 14 according to the frame
  • the data suggestion requests raw data from the camera service module 18; the application module 14 sets up a predetermined algorithm according to the algorithm suggestion and sends the predetermined algorithm to the hardware abstraction module 12; the algorithm post-processing module 16 performs photographing and post-processing of the raw data according to the predetermined algorithm, and The processed result data is sent back to the application module 14.
  • the original data includes to-be-processed data and metadata.
  • the application module 14 requests the camera service module 18 for raw data (ie 01), including:
  • the application module 14 requests the camera service module 18 for data to be processed
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the original data to the application module 14 (ie 03) via the camera service module 18, including:
  • the hardware abstraction module 12 After receiving the request, the hardware abstraction module 12 sends the data to be processed and the metadata corresponding to the data to be processed to the application module 14 via the camera service module 18;
  • the application module 14 After receiving the original data, the application module 14 sends the original data to the algorithm post-processing module 16 (ie 04), including:
  • the application module 14 After receiving the data and metadata to be processed, the application module 14 sends the data and metadata to be processed to the algorithm post-processing module 16;
  • the algorithm post-processing module 16 takes pictures and post-processes the original data and sends the processed result data back to the application program module 14 (ie 05), including:
  • the algorithm post-processing module 16 performs photographing post-processing on the data to be processed according to the metadata and sends the processed result data back to the application program module 14.
  • the original data includes to-be-processed data and metadata.
  • the application module 14 can be used to execute the method in 014, the hardware abstraction module 12 can be used to execute the method in 034, the application module 14 can be used to execute the method in 044, and the algorithm post-processing module 16 can be used to execute the method in 054.
  • the application module 14 can be used to request the data to be processed from the camera service module 18.
  • the hardware abstraction module 12 may be used to send the to-be-processed data and metadata corresponding to the to-be-processed data to the application module 14 via the camera service module 18 after receiving the request.
  • the application module 14 may be used to send the data and metadata to be processed to the algorithm post-processing module 16 after receiving the data and metadata to be processed.
  • the algorithm post-processing module 16 can be used to perform post-photographic post-processing of the data to be processed according to metadata and send the processed result data back to the application module 14.
  • Metadata corresponds to the data to be processed.
  • Metadata includes information about the data to be processed, such as 3a (automatic exposure control AE, automatic focus control AF, automatic white balance control AWB) information, picture information (width and height of the picture), exposure parameters (aperture size, shutter speed, and Sensitivity aperture value) etc.
  • the algorithm post-processing module 16 can implement post-photographing processing of the data to be processed according to the metadata. For example, the algorithm post-processing module 16 adjusts the image brightness corresponding to the data to be processed according to the ISO value in the metadata.
  • the application module 14 When the original data includes data to be processed and metadata, the application module 14 requests the data to be processed from the camera service module 18; after receiving the request, the camera service module 18 sends the request to the hardware abstraction module 12; the hardware abstraction module 12 After receiving the request, the data to be processed and the metadata corresponding to the data to be processed are sent to the application module 14 via the camera service module 18; after receiving the data and metadata, the application module 14 combines the data to be processed and the metadata.
  • the metadata is sent to the algorithm post-processing module 16; the algorithm post-processing module 16 performs photographing post-processing on the data to be processed according to the metadata and sends the processed result data back to the application program module 14.
  • the camera service module 18 sends the request to the hardware abstraction module 12 after receiving the request; the hardware abstraction module 12 receives the request ,
  • the multi-frame data to be processed and the multi-frame metadata corresponding to the multi-frame data to be processed are sent to the application module 14 via the camera service module 18; the application module 14 receives the multi-frame data to be processed and the multi-frame element After the data, the multi-frame data to be processed and the multi-frame metadata are packaged and sent to the algorithm post-processing module 16.
  • the algorithm post-processing module 16 takes pictures and post-processes the multi-frame data to be processed according to the multi-frame metadata and processes the result data Pass back to application module 14.
  • the application module 14 requests 4 frames of data to be processed from the camera service module 18.
  • the camera service module 18 sends the request to the hardware abstraction module 12; after the hardware abstraction module 12 receives the request, first, the hardware The abstraction module 12 sends the first frame of to-be-processed data (ie to-be-processed data 1 in Figure 1) and the first frame of metadata (ie, the metadata 1 in Figure 1) to the camera service module 18, and the camera service module 18 sends the 1 frame of to-be-processed data and the first frame of metadata are sent to the application module 14; secondly, the hardware abstraction module 12 sends the second frame of to-be-processed data (ie to-be-processed data 2 in Figure 1) and the second frame of metadata (ie The metadata 2) in Figure 1 is sent to the camera service module 18.
  • the hardware abstraction module 12 sends the second frame of to-be-processed data (ie to-be-processed data 2 in Figure 1) and the second frame of metadata (ie The metadata 2) in Figure 1 is sent to
  • the camera service module 18 sends the second frame to be processed data and the second frame metadata to the application module 14; then, the hardware abstraction module 12 sends the third frame to be processed
  • the processed data that is, the data to be processed 3 in Figure 1
  • the third frame of metadata that is, the metadata 3 in Figure 1
  • the camera service module 18 sends the data to be processed in the third frame and the third frame of metadata.
  • the frame metadata is sent to the application module 14; finally, the hardware abstraction module 12 combines the fourth frame of data to be processed (ie, the data to be processed 4 in Figure 1) and the fourth frame of metadata (ie, the metadata 4 in Figure 1) Send to the camera service module 18, and the camera service module 18 sends the 4th frame to-be-processed data and the 4th frame metadata to the application module 14.
  • the hardware abstraction module 12 sends the previous frame of pending data and metadata to the camera service module 18, it can start to send the next frame of pending data and metadata to the camera service module 18 without Wait for the camera service module 18 to send the previous frame of pending data and metadata to the application module 14, and then start to send the next frame of pending data and metadata to the camera service module 18. This can save the total time required to send multiple frames of to-be-processed data and metadata to the application module 14 via the camera service module 18, avoid long waits for the user, and improve the photographing experience.
  • the application module 14 receives the first frame to be processed data and the first frame metadata, the second frame to be processed data and the second frame metadata, the third frame to be processed data and the third frame metadata, and the fourth frame to be processed. After processing the data and the fourth frame metadata, the first frame to be processed data and the first frame metadata, the second frame to be processed data and the second frame metadata, the third frame to be processed data and the third frame metadata, The 4th frame to-be-processed data and the 4th frame metadata are packaged and sent to the algorithm post-processing module 16.
  • the algorithm post-processing module 16 generally needs to take pictures and post-process the multiple frames of the to-be-processed data, and if the application module 14 sequentially The multi-frame data to be processed and the corresponding metadata are sent to the algorithm post-processing module 16.
  • the algorithm post-processing module 16 receives only the first frame of data to be processed and the first frame of metadata, or only the first frame of data to be processed And the first frame metadata, the second frame to-be-processed data, and the second frame metadata, or only the first frame-to-be-processed data and the first frame metadata, the second frame to-be-processed data and the second frame metadata are received, In the case of the third frame of data to be processed and the third frame of metadata, the algorithm post-processing module 16 cannot complete the post-processing of multiple frames of to-be-processed data. In the embodiment of this application, the application module 14 will And the corresponding metadata are packaged and sent to the algorithm post-processing module 16, which can efficiently use transmission resources.
  • the algorithm post-processing The module 16 can perform photographing and post-processing of the multi-frame data to be processed according to the multi-frame metadata, and send the processed result data back to the application module 14.
  • the data in the hardware abstraction module 12 is in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is also data in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is data in the second image format, and the image processing method further includes:
  • the hardware abstraction module 12 converts the data in the first image format into data in the second image format.
  • the data in the hardware abstraction module 12 is in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is also data in the first image format.
  • the original data requested by the application module 14 from the camera service module 18 is data in the second image format.
  • the hardware abstraction module 12 can be used to execute the method in 09. In other words, the hardware abstraction module 12 can also be used to convert data in the first image format into data in the second image format.
  • the first image format may be a RAW format
  • the second image format may be a YUV format.
  • the data in the hardware abstraction module 12 is the first image format, that is, the RAW format.
  • the original data requested by the application module 14 from the camera service module 18 is also in RAW format.
  • the data in the hardware abstraction module 12 is the same as the original data format requested by the application module 14 from the camera service module 18.
  • the hardware abstraction module 12 does not need to perform format conversion on the RAW format data.
  • the camera service module 18 sends the request to the hardware abstraction module 12 after receiving the request; the hardware abstraction module 12 sequentially receives the request Send the data in the multi-frame RAW format to the application module 14 via the camera service module 18; after the application module 14 receives the data in the multi-frame RAW format, the data in the multi-frame RAW format is packaged and sent to the algorithm post-processing module 16; The algorithm post-processing module 16 performs photographing post-processing on the multi-frame RAW format data and sends the processed result data back to the application module 14.
  • the application module 14 requests the camera service module 18 for data in RAW format; the camera service module 18 sends the request to the hardware abstraction module 12 after receiving the request; the hardware abstraction module 12 After receiving the request, the RAW format data and the corresponding metadata are sent to the application module 14 via the camera service module 18.
  • the application module 14 receives the RAW format data and the corresponding metadata, and then sends the RAW format data to the application module 14
  • the data and corresponding metadata are sent to the algorithm post-processing module 16; the algorithm post-processing module 16 performs post-processing on the RAW format data according to the metadata and sends the processed result data back to the application module 14.
  • the original data requested by the application module 14 from the camera service module 18 is in the YUV format.
  • the data in the hardware abstraction module 12 is different from the original data format requested by the application module 14 from the camera service module 18.
  • the hardware abstraction module 12 needs to convert data in the RAW format into data in the YUV format.
  • the camera service module 18 sends the request to the hardware abstraction module 12 after receiving the request; the hardware abstraction module 12 first receives the request Convert multi-frame RAW format data into multi-frame YUV format data, and then sequentially send multi-frame YUV format data to application module 14 via camera service module 18; application module 14 is receiving multi-frame YUV format data After that, the data in the multi-frame YUV format is packaged and sent to the algorithm post-processing module 16; the post-algorithm processing module 16 takes pictures and post-processes the data in the multi-frame YUV format and sends the processed result data back to the application module 14.
  • the application module 14 When the original data includes to-be-processed data and metadata, the application module 14 requests the data in the YUV format from the camera service module 18; after receiving the request, the camera service module 18 sends the request to the hardware abstraction module 12; the hardware abstraction module 12 After receiving the request, first convert the RAW format data to YUV format data, and then send the YUV format data and corresponding metadata to the application module 14 via the camera service module 18; the application module 14 receives the YUV After the data in the format and the corresponding metadata, the data in the YUV format and the corresponding metadata are sent to the algorithm post-processing module 16. The algorithm post-processing module 16 takes pictures and post-processes the data in the YUV format according to the metadata and processes the processed data. The result data is sent back to the application module 14.
  • the data in the first image format in the hardware abstraction module 12 is directly transmitted by the image sensor 20.
  • the data in the first image format stored in the hardware abstraction module 12 is processed by the algorithm post-processing module 16 to achieve post-photographing processing and is returned by the application module 14 to be obtained.
  • the photographing device 100 of the embodiment of the present application includes the image processor 10 and the image sensor 20 of any of the above embodiments.
  • the image processor 10 and the image sensor 20 are connected.
  • the image sensor 20 includes an image acquisition unit (sensor) 22 and a RAW image data unit (Image Front-end, IFE) 24.
  • the image acquisition unit 22 is used to receive light to acquire image data (RAW images, that is, the aforementioned first image format). Data), the RAW image data unit 24 is used to transmit the image data collected by the image collection unit 22 to the image processor 10.
  • the RAW image data unit 24 can process the RAW image collected by the image collection unit 22 and output the processed image data. RAW image to the image processor 10.
  • the image processor 10 includes a hardware abstraction module 12, an application program module 14, an algorithm post-processing module 16 and a camera service module 18.
  • the hardware abstraction module 12 is used to receive RAW images, convert the RAW images into YUV images (that is, data in the aforementioned second image format), and transmit RAW images and/or YUV images.
  • the hardware abstraction module 12 may be connected with the image sensor 20.
  • the hardware abstraction module 12 may include a buffer queue 122 connected to the image sensor 20, a RAW-to-RGB processing unit (Bayer Process Segment, BPS) 124, and a noise reduction and YUV post-processing connected to the application module 14.
  • Unit (Image Process Engine, IPE) 126 is used to buffer the RAW image from the image sensor 20 and transmit it to the algorithm post-processing module 16 through the camera service module 18 and the application module 14.
  • the RAW-to-RGB processing unit 124 is used to convert the RAW image from the buffer unit 122 into an RGB image.
  • the noise reduction and YUV post-processing unit 126 is configured to process RGB images to obtain YUV images and transmit the YUV images to the algorithm post-processing module 16 through the camera service module 18 and the application module 14.
  • the hardware abstraction module 12 can also transmit metadata of image data.
  • the metadata includes 3a (automatic exposure control AE, automatic focus control AF, automatic white balance control AWB) information, picture information (such as image width, height), exposure parameters (aperture Size, shutter speed and sensitivity aperture value), etc., metadata can be used to assist in the post-processing of RAW images and/or YUV images (such as beauty processing, filter processing, rotation processing, watermark processing, and blur processing) , HDR processing, and at least one of multi-frame processing).
  • the metadata includes sensitivity (ISO) information, and the brightness of the RAW image and/or YUV image can be assisted to adjust the brightness of the RAW image and/or YUV image according to the sensitivity information, so as to achieve post-photographing processing related to adjusting the brightness.
  • the hardware abstraction module 12 Since the hardware abstraction module 12 only receives RAW images, converts RAW images into YUV images, and transmits RAW images and/or YUV images, and does not post-process the RAW images and/or YUV images, the image processing algorithms for post-photographic processing are not required To perform process truncation on the algorithm architecture of the hardware abstraction module 12 itself, only external compatibility is required, and the design difficulty is greatly reduced.
  • the application programming interface builds the hardware abstraction module into a pipeline. Since the creation of the pipeline requires a lot of time and memory, it is necessary to use the camera's corresponding working mode when the camera is started. All pipelines are created first, and in order to implement various image processing algorithms, it is generally necessary to create a large number of pipelines (for example, more than three pipelines), which will cause the camera startup to take a lot of time and take up a lot of memory.
  • the hardware abstraction module 12 of the embodiment of the present application only receives RAW images, converts RAW images to YUV images, and transmits RAW images and/or YUV images, and does not perform post-photographic processing on RAW images and/or YUV images. Therefore, hardware abstraction The module 12 only needs to establish a small number of (for example, one or two) pipelines, and does not need to establish a large number of pipelines, thereby saving memory and making the startup speed of the camera faster.
  • the application program module 14 and the hardware abstraction module 12 are connected through the camera service module 18.
  • the application module 14 can be used to generate control instructions according to user input and send the control instructions to the image sensor 20 through the camera service module 18 and the hardware abstraction module 12 to control the work of the image sensor 20 accordingly.
  • the application module 14 can run in 64 bits, and the static data connection library (lib) of the image processing algorithm for post-photographing processing can be configured in 64 bits to improve the calculation speed.
  • the application module 14 receives the RAW image and/or YUV image transmitted by the hardware abstraction module 12 through the camera service module 18, it can post-process the RAW and/or YUV image, or transmit the RAW and/or YUV image to the algorithm
  • the post-processing module 16 performs photographing post-processing.
  • the application module 14 can also perform some post-photographing processing (such as beauty processing, filter processing, rotation processing, watermark processing, blurring processing, etc.), and the algorithm post-processing module 16 can perform other post-photo processing (such as HDR). Processing, multi-frame processing, etc.).
  • the application module 14 transmits the RAW and/or YUV image to the algorithm post-processing module 16 for post-photographic processing.
  • the algorithm post-processing module 16 is connected to the hardware abstraction module 12 through the application module 14, the camera service module 18, and the algorithm post-processing module 16 stores at least one image processing algorithm (for example, including beauty processing algorithm, filter processing algorithm, rotation At least one of processing algorithm, watermark processing algorithm, blur processing algorithm, HDR processing algorithm, and multi-frame processing algorithm), the algorithm post-processing module 16 is used to process RAW images and/or YUV images by using image processing algorithms to realize taking pictures Post-processing. Since the post-processing of the RAW image and/or YUV image can be implemented by the algorithm post-processing module 16, there is no need to perform process truncation on the algorithm architecture of the hardware abstraction module 12 itself, and only need to be externally compatible, which reduces the design difficulty.
  • image processing algorithm for example, including beauty processing algorithm, filter processing algorithm, rotation At least one of processing algorithm, watermark processing algorithm, blur processing algorithm, HDR processing algorithm, and multi-frame processing algorithm
  • the function of the algorithm post-processing module 16 is simpler and more focused, which can achieve effects such as faster transplantation and simple extension of new image processing algorithms.
  • some post-photographic processing such as beauty processing, filter processing, rotation processing, watermark processing, blur processing, etc.
  • the algorithm post-processing module 16 will perform other post-photo processing (such as HDR processing).
  • the application module 14 may also store at least one image processing algorithm (for example, including beauty processing algorithm, filter processing algorithm, rotation processing algorithm, watermark processing algorithm, blur processing algorithm, HDR At least one of a processing algorithm and a multi-frame processing algorithm), the application module 14 is also used to process the RAW image and/or YUV image by using the image processing algorithm to implement post-photographing processing. Since the post-processing of RAW images and/or YUV images is implemented by the application module 14 and the algorithm post-processing module 16, there is no need to cut the process on the algorithm architecture of the hardware abstraction module 12 itself, and only need to be compatible and coupled externally. The design difficulty is also greatly reduced.
  • image processing algorithm for example, including beauty processing algorithm, filter processing algorithm, rotation processing algorithm, watermark processing algorithm, blur processing algorithm, HDR At least one of a processing algorithm and a multi-frame processing algorithm
  • the hardware abstraction module 12 may only transmit RAW images (there is no need to convert RAW images to YUV images); after the algorithm
  • the processing module 16 only processes YUV images (for example, when the image processing algorithm processes YUV images)
  • the hardware abstraction module 12 can only transmit YUV images
  • the algorithm post-processing module 16 processes RAW images and YUV images
  • the hardware abstraction module 12 can transmit RAW image and YUV image.
  • the image sensor 20 After the image sensor 20 performs a shooting (exposure imaging), it transmits the shooting data (RAW image) to the hardware abstraction module 12.
  • the algorithm post-processing module 16 receives the RAW image and/or YUV image corresponding to the shooting data, the image sensor 20 can perform the next shooting, or the image sensor 20 can be closed, or the application module 14 can be closed, or the application module 14 can exit the application interface. Since the post-photographic processing is implemented by the algorithm post-processing module 16, after the RAW image and/or YUV image corresponding to the captured data is transmitted to the algorithm post-processing module 16, only the algorithm post-processing module 16 is required to implement the post-photo processing. The time image sensor 20 and the application module 14 do not need to participate in post-photographing processing.
  • the image sensor 20 can be closed or perform the next shooting, and the application module 14 can close or exit the application interface.
  • the photographing device 100 can realize snapshots, and when the algorithm post-processing module 16 performs photographing post-processing, the application module 14 can be closed or the application program interface can be exited, so as to perform some other operations on the electronic device (for example, irrelevant to the photographing device 100). Operations such as browsing web pages, watching videos, making phone calls, etc.), so that the user does not need to spend a lot of time waiting for the completion of the post-photograph processing, which is convenient for the user to use the electronic device.
  • the algorithm post-processing module 16 may include an encoding unit 162 for converting the YUV image into a JPG image (or JPEG image, etc.). Specifically, when the algorithm post-processing module 14 processes a YUV image, the encoding unit 162 may directly encode the YUV image to form a JPG image, thereby improving the output speed of the image.
  • the post-algorithm processing module 14 can return the RAW images processed to achieve post-photographing processing to the hardware abstraction module 12 via the application module 14 and the camera service module 18, for example, to
  • the RAW-to-RGB processing unit 124 the RAW-to-RGB processing unit 124 can be used to convert the RAW image processed by the algorithm post-processing module 14 to achieve post-photographing processing and return via the application module 14 to RGB images
  • noise reduction and YUV post-processing unit 126 can convert the RGB image into a YUV image
  • the YUV image can be transmitted to the encoding unit 162 of the algorithm post-processing module 16 again to convert the YUV image into a JPG image.
  • the algorithm post-processing module 14 may also send the RAW image processed for post-photographing to the buffer unit 122 via the application module 14 and the camera service module 18, and the returned RAW image is processed by RAW conversion to RGB.
  • the unit 124 and the noise reduction and YUV post-processing unit 126 form a YUV image, which is then transmitted to the encoding unit 162 to form a JPG image.
  • the algorithm post-processing module 16 can be used to transfer the JPG image to the memory for storage.
  • the algorithm post-processing module 16 includes a logic processing call layer 164, an algorithm module interface layer 166 and an algorithm processing layer 168.
  • the logic processing call layer 164 is used to communicate with the application program module 14.
  • the algorithm module interface layer 166 is used to maintain the algorithm interface.
  • the algorithm processing layer 168 includes at least one image processing algorithm.
  • the algorithm module interface layer 166 is used to perform at least one operation of registration, deregistration, invocation, and callback on the image processing algorithm of the algorithm processing layer 168 through the algorithm interface.
  • the logic processing call layer 164 may include a thread queue. After the algorithm post-processing module 16 receives the post-photographing task of the RAW image and/or YUV image, it can buffer the post-photograph processing task in the thread queue for processing, where the thread queue can be cached Multiple post-photographing processing tasks, in this way, a snapshot (ie a snapshot mechanism) can be implemented through the logic processing call layer 164.
  • the logic processing call layer 164 may also receive instructions such as initialization (init) and process (process) sent by the application module 14, and save the corresponding instructions and data in the thread queue.
  • the logic processing call layer 164 performs specific logic calls (ie, specific logic call combinations) according to the tasks in the thread queue.
  • the logic processing calling layer 164 may also return the processed thumbnail to the application module 14 for display (ie, thumbnail echo).
  • the algorithm module interface layer 166 is used to call the algorithm interface, and the call command can also be saved in the thread queue.
  • the algorithm processing layer 168 receives the call command of the thread queue, it can parse the parameters of the call command to obtain the image processing algorithm to be called.
  • the algorithm module interface layer 166 registers an image processing algorithm, it can add an image processing algorithm to the algorithm processing layer 168; when the algorithm module interface layer 166 cancels the image processing algorithm, it can delete one of the algorithm processing layers 168 Image processing algorithm; when the algorithm module interface layer 166 calls the image processing algorithm, it can call an image processing algorithm in the algorithm processing layer 168; when the algorithm module interface layer 166 calls back the image processing algorithm, it can use the data processed by the algorithm And the status is sent back to the application module 14.
  • a unified interface can be used to implement operations such as registration, cancellation, invocation, and callback of image processing algorithms.
  • Each image processing algorithm in the algorithm processing layer 168 is independent, so that operations such as registration, deregistration, calling, and callback can be conveniently implemented for the image processing algorithm.
  • the camera service module 18 is connected to the hardware abstraction module 12 and the application module 14 respectively.
  • the camera service module 18 encapsulates the RAW image and/or YUV image and transmits the encapsulated RAW image and/or YUV image to the application module 14, and transmits the RAW image returned by the application module 14 to the hardware abstraction module 12. .
  • the efficiency of image transmission can be improved, and the security of image transmission can be improved.
  • the data transmitted between the hardware abstraction module 12 and the application module 14 needs to pass through the camera service module 18.
  • the hardware abstraction module 12 when the hardware abstraction module 12 transmits the RAW image and/or YUV image to the application module 14, the hardware abstraction module 12 first transmits the RAW image and/or YUV image to the camera service module 18, and the camera service module 18 processes the RAW image and/or YUV image. Or the YUV image is packaged and the packaged RAW image and/or YUV image are transmitted to the application module 14.
  • the hardware abstraction module 12 when the hardware abstraction module 12 transmits metadata to the application module 14, the hardware abstraction module 12 first transmits the metadata to the camera service module 18, and the camera service module 18 encapsulates the metadata and transmits the encapsulated metadata to Application module 14.
  • the hardware abstraction module 12 when the hardware abstraction module 12 transmits the frame number recommendation to the application module 14, the hardware abstraction module 12 first transmits the frame number recommendation to the camera service module 18. The camera service module 18 encapsulates the frame number recommendation and the packaged frame The number is recommended to be transferred to application module 14.
  • the hardware abstraction module 12 when the hardware abstraction module 12 transmits algorithm suggestions to the application module 14, the hardware abstraction module 12 first transmits the algorithm suggestions to the camera service module 18. The camera service module 18 encapsulates the algorithm suggestions and transmits the encapsulated algorithm suggestions to Application module 14.
  • the hardware abstraction module 12 can transmit the sensitivity information, the jitter condition of the gyroscope, the AR scene detection result, etc., to the camera service module 18, and the camera service module 18 is based on the sensitivity information and the jitter of the gyroscope.
  • the frame number recommendations and/or algorithm recommendations are obtained based on the situation, AR scene detection results, etc., and then the frame number recommendations and/or algorithm recommendations are transmitted to the application module 14.
  • the electronic device 1000 of the embodiment of the present application includes the photographing device 100 and the housing 200 of any of the above embodiments, and the photographing device 100 is combined with the housing 200.
  • the housing 200 can be used as a mounting carrier for the functional elements of the electronic device 1000.
  • the housing 200 may provide protections such as dustproof, anti-drop, and waterproof for the functional elements, and the functional elements may be a display screen, the camera 100, a receiver, and the like.
  • the housing 200 includes a main body 210 and a movable bracket 220.
  • the movable bracket 220 can move relative to the main body 210 under the driving of the driving device.
  • the movable bracket 220 can slide relative to the main body 210.
  • Part of the functional element can be installed on the main body 210, and another part of the functional element (for example, the camera 100) can be installed on the movable bracket 220, and the movement of the movable bracket 220 can drive the other part of the functional element to retract into or from the main body 210. Stretched out.
  • a collection window is opened on the housing 200, and the shooting device 100 is aligned and installed with the collection window so that the shooting device 100 can receive external light through the collection window to form an image, or the shooting device 100 is installed below the display screen , The camera 100 receives external light passing through the display screen to form an image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Studio Devices (AREA)

Abstract

一种图像处理方法、图像处理器(10)、拍摄装置(100)和电子设备(1000)。图像处理方法包括：(01)应用程序模块(14)向相机服务模块(18)请求原始数据；(02)相机服务模块(18)在接收到请求后，将请求发送给硬件抽象模块(12)；(03)硬件抽象模块(12)在接收到请求后，将原始数据经相机服务模块(18)发送至应用程序模块(14)；(04)应用程序模块(14)在接收到原始数据后，将原始数据发送至算法后处理模块(16)；和(05)算法后处理模块(16)对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块(14)。

Description

图像处理方法、图像处理器、拍摄装置和电子设备
优先权信息
本申请请求2019年4月10日向中国国家知识产权局提交的、专利申请号为201910285175.4的专利申请的优先权和权益，并且通过参照将其全文并入此处。
技术领域
本申请涉及图像处理技术领域，更具体而言，涉及一种图像处理方法、图像处理器、拍摄装置和电子设备。
背景技术
随着电子技术的发展，相机的功能越来越多，模块也越来越复杂。当用户按下相机的拍照键后，需要在拍照界面等待一段时间以完成图像处理，在这一过程中，用户不能执行其他操作，也不能退出相机。
发明内容
本申请实施方式提供一种图像处理方法、图像处理器、拍摄装置和电子设备。
本申请实施方式的图像处理方法包括：应用程序模块向相机服务模块请求原始数据；所述相机服务模块在接收到所述请求后，将所述请求发送给硬件抽象模块；所述硬件抽象模块在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块；所述应用程序模块在接收到所述原始数据后，将所述原始数据发送至算法后处理模块；和所述算法后处理模块对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
本申请实施方式的图像处理器包括应用程序模块、相机服务模块、硬件抽象模块和算法后处理模块。所述应用程序模块用于向所述相机服务模块请求原始数据。所述相机服务模块用于在接收到所述请求后，将所述请求发送给所述硬件抽象模块。所述硬件抽象模块用于在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块。所述应用程序模块还用于在接收到所述原始数据后，将所述原始数据发送至所述算法后处理模块。所述算法后处理模块用于对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
本申请实施方式的拍摄装置包括图像处理器和图像传感器，所述图像传感器与所述图像处理器连接。所述图像处理器包括应用程序模块、相机服务模块、硬件抽象模块和算法后处理模块。所述应用程序模块用于向所述相机服务模块请求原始数据。所述相机服务模块用于在接收到所述请求后，将所述请求发送给所述硬件抽象模块。所述硬件抽象模块用于在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块。所述应用程序模块还用于在接收到所述原始数据后，将所述原始数据发送至所述算法后处理模块。所述算法后处理模块用于对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
本申请实施方式的电子设备包括拍摄装置和壳体，所述拍摄装置与所述壳体结合。所述拍摄装置包括图像处理器和图像传感器，所述图像传感器与所述图像处理器连接。所述图像处理器包括应用程序模块、相机服务模块、硬件抽象模块和算法后处理模块。所述应用程序模块用于向所述相机服务模块请求原始数据。所述相机服务模块用于在接收到所述请求后，将所述请求发送给所述硬件抽象模块。所述硬件抽象模块用于在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块。所述应用程序模块还用于在接收到所述原始数据后，将所述原始数据发送至所述算法后处理模块。所述算法后处理模块用于对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
本申请的实施方式的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实施方式的实践了解到。
附图说明
本申请的上述和/或附加的方面和优点从结合下面附图对实施方式的描述中将变得明显和容易理解，其中：
图1是本申请某些实施方式的图像处理器的模块示意图；
图2是本申请某些实施方式的图像处理方法的流程示意图；
图3是本申请某些实施方式的图像处理方法的流程示意图；
图4是本申请某些实施方式的图像处理方法的流程示意图；
图5是本申请某些实施方式的图像处理方法的流程示意图；
图6是本申请某些实施方式的图像处理方法的流程示意图；
图7是本申请某些实施方式的图像处理方法的流程示意图；
图8是本申请某些实施方式的拍摄装置的模块示意图；
图9是本申请某些实施方式的算法后处理模块的示意图；
图10是本申请某些实施方式的电子设备的结构示意图；
图11是本申请某些实施方式的电子设备的结构示意图。
具体实施方式
下面详细描述本申请的实施方式，实施方式的示例在附图中示出，相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本申请，而不能理解为对本申请的限制。
请参阅图1和图2，本申请实施方式的图像处理方法包括：
01：应用程序模块14向相机服务模块18请求原始数据；
02：相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；
03：硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14；
04：应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16；和
05：算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1和图3，在某些实施方式中，应用程序模块14向相机服务模块18请求的原始数据为多帧。
硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14(即03)，包括：
032：硬件抽象模块12在接收到请求后，依次将多帧原始数据经相机服务模块18发送至应用程序模块14；
应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16(即04)，包括：
042：应用程序模块14在接收到多帧原始数据后，将多帧原始数据打包发送至算法后处理模块16。
请参阅图1，在某些实施方式中，在算法后处理模块16对原始数据进行拍照后处理的过程中，应用程序模块14能够关闭或者退出应用界面。
请参阅图1和图4，在某些实施方式中，图像处理方法还包括：
06：相机服务模块18向应用程序模块14发送关于原始数据的帧数建议；
应用程序模块14向相机服务模块18请求原始数据(即01)，包括：
012：应用程序模块14根据帧数建议向相机服务模块18请求原始数据。
请参阅图1和图5，在某些实施方式中，图像处理方法还包括：
07：相机服务模块18向应用程序模块14发送对原始数据进行拍照后处理的算法建议；
08：应用程序模块14根据算法建议设置预定算法并将预定算法发送至硬件抽象模块12；
算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14(即05)，包括：
052：算法后处理模块16根据预定算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14。
请参阅图1和图6，在某些实施方式中，原始数据包括待处理数据和元数据。应用程序模块14向相机服务模块18请求原始数据(即01)，包括：
014：应用程序模块14向相机服务模块18请求待处理数据；
硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14(即03)，包括：
034：硬件抽象模块12在接收到请求后，将待处理数据和与待处理数据对应的元数据经相机服务模块18发送至应用程序模块14；
应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16(即04)，包括：
044：应用程序模块14在接收到待处理数据和元数据后，将待处理数据和元数据发送至算法后处理模块16；
算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14(即05)，包括：
054：算法后处理模块16根据元数据对待处理数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1和图7，在某些实施方式中，硬件抽象模块12内的数据为第一图像格式。其中：应用程序模块14向相机服务模块18请求的原始数据也为第一图像格式的数据。或者，应用程序模块14向相机服务模块18请求的原始数据为第二图像格式的数据，图像处理方法还包括：
09：硬件抽象模块12将第一图像格式的数据转换为第二图像格式的数据。
请参阅图1，本申请实施方式的图像处理器10包括应用程序模块14、相机服务模块18、硬件抽象模块12和算法后处理模块16。应用程序模块14用于向相机服务模块18请求原始数据。相机服务模块18用于在接收到请求后，将请求发送给硬件抽象模块12。硬件抽象模块12用于在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14。应用程序模块14还用于在接收到原始数据后，将原始数据发送至算法后处理模块16。算法后处理模块16用于对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1，在某些实施方式中，应用程序模块14向相机服务模块18请求的原始数据为多帧。硬件抽象模块12可用于执行032中的方法，应用程序模块14可用于执行042中的方法。也即是说，硬件抽象模块12用于在接收到请求后，依次将多帧原始数据经相机服务模块18发送至应用程序模块14。应用程序模块14用于在接收到多帧原始数据后，将多帧原始数据打包发送至算法后处理模块16。
请参阅图1，在某些实施方式中，在算法后处理模块16对原始数据进行拍照后处理的过程中，应用程序模块14能够关闭或者退出应用界面。
请参阅图1，在某些实施方式中，相机服务模块18还用于向应用程序模块14发送关于原始数据的帧数建议。应用程序模块14还用于根据帧数建议向相机服务模块18请求原始数据。
请参阅图1，在某些实施方式中，相机服务模块18还用于向应用程序模块14发送对原始数据进行拍照后处理的算法建议。应用程序模块14还用于根据算法建议设置预定算法并将预定算法发送至硬件抽象模块12。算法后处理模块16还用于根据预定算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14。
请参阅图1，在某些实施方式中，原始数据包括待处理数据和元数据。应用程序模块14用于向相机服务模块18请求待处理数据。硬件抽象模块12用于在接收到请求后，将待处理数据和与待处理数据对应的元数据经相机服务模块18发送至应用程序模块14。应用程序模块14用于在接收到待处理数据和元数据后，将待处理数据和元数据发送至算法后处理模块16。算法后处理模块16用于根据元数据对待处理数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1，在某些实施方式中，硬件抽象模块12内的数据为第一图像格式。其中：应用程序模块14向相机服务模块18请求的原始数据也为第一图像格式的数据。或者，应用程序模块14 向相机服务模块18请求的原始数据为第二图像格式的数据。硬件抽象模块12还用于将第一图像格式的数据转换为第二图像格式的数据。
请参阅图8，本申请实施方式的拍摄装置100包括上述任一实施方式的图像处理器10和图像传感器20。图像处理器10和图像传感器20连接。
请参阅图10和图11，本申请实施方式的电子设备1000包括上述任一实施方式的拍摄装置100和壳体200，拍摄装置100与壳体200结合。
请参阅图1和图2，本申请实施方式的图像处理方法包括：
01：应用程序模块14向相机服务模块18请求原始数据；
02：相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；
03：硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14；
04：应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16；和
05：算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1，本申请实施方式的图像处理器10包括应用程序模块(APP)14、相机服务模块(Camera Service)18、硬件抽象模块(Hardware Abstraction Layer，HAL)12和算法后处理模块(Algo Process Service，APS)16。本申请实施方式的图像处理方法可由本申请实施方式的图像处理器10实现。例如，应用程序模块14可用于执行01和04中的方法，相机服务模块18可用于执行02中的方法，硬件抽象模块12可用于执行03中的方法，算法后处理模块16可用于执行05中的方法。
也即是说，应用程序模块14可以用于向相机服务模块18请求原始数据。相机服务模块18可以用于在接收到请求后，将请求发送给硬件抽象模块12。硬件抽象模块12可以用于在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14。应用程序模块14还可以用于在接收到原始数据后，将原始数据发送至算法后处理模块16。算法后处理模块16可以用于对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
可以理解，随着电子技术的发展，相机的功能越来越多，模块也越来越复杂。当用户按下相机的拍照键后，需要在拍照界面等待一段时间以完成图像处理，在这一过程中，用户不能执行其他操作，也不能退出相机，拍照体验不佳。
本申请实施方式的图像处理方法和图像处理器10通过算法后处理模块16实现数据分流，可以减少硬件抽象模块12的负载，使得系统运行流畅，不会出现卡顿现象，而算法后处理模块16的拍照后处理可以在后台运行，不需要用户长时间等待，极大地提升了拍照体验。
具体地，应用程序模块14向相机服务模块18请求原始数据，原始数据为图像数据，原始数据可以是RAW格式的数据或者是YUV格式的数据；相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14，即硬件抽象模块12先将原始数据发送至相机服务模块18，相机服务模块18再将原始数据发送至应用程序模块14；应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16，算法后处理模块16内存储有至少一种图像处理算法，图像处理算法包括多帧处理算法、美颜处理算法、虚化处理算法、高动态范围图像(High-Dynamic Range，HDR)处理算法、滤镜处理算法、旋转处理算法、水印处理算法等；算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14，即：算法后处理模块16采用上述图像处理算法处理RAW格式的数据或者是YUV格式的数据，并将处理后的结果数据传回给应用程序模块14。
请参阅图1和图3，在某些实施方式中，应用程序模块14向相机服务模块18请求的原始数据为多帧。
硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14(即03)，包括：
032：硬件抽象模块12在接收到请求后，依次将多帧原始数据经相机服务模块18发送至应 用程序模块14；
应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16(即04)，包括：
042：应用程序模块14在接收到多帧原始数据后，将多帧原始数据打包发送至算法后处理模块16。
请参阅图1，在某些实施方式中，应用程序模块14向相机服务模块18请求的原始数据为多帧。硬件抽象模块12可用于执行032中的方法，应用程序模块14可用于执行042中的方法。也即是说，硬件抽象模块12可以用于在接收到请求后，依次将多帧原始数据经相机服务模块18发送至应用程序模块14。应用程序模块14可以用于在接收到多帧原始数据后，将多帧原始数据打包发送至算法后处理模块16。
例如，应用程序模块14向相机服务模块18请求的原始数据为4帧，相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，首先，硬件抽象模块12将第1帧原始数据发送至相机服务模块18，相机服务模块18将第1帧原始数据发送至应用程序模块14；其次，硬件抽象模块12将第2帧原始数据发送至相机服务模块18，相机服务模块18将第2帧原始数据发送至应用程序模块14；然后，硬件抽象模块12将第3帧原始数据发送至相机服务模块18，相机服务模块18将第3帧原始数据发送至应用程序模块14；最后，硬件抽象模块12将第4帧原始数据发送至相机服务模块18，相机服务模块18将第4帧原始数据发送至应用程序模块14。在这一过程中，硬件抽象模块12将前一帧原始数据发送至相机服务模块18后，即可开始将下一帧原始数据发送至相机服务模块18，而无需等待相机服务模块18将前一帧原始数据发送至应用程序模块14，再开始将下一帧原始数据发送至相机服务模块18。这样可以节省将多帧原始数据经相机服务模块18发送至应用程序模块14所需的总时长，避免用户长时间等待，有利于提升拍照体验。
应用程序模块14在接收到第1帧原始数据、第2帧原始数据、第3帧原始数据和第4帧原始数据后，将第1帧原始数据、第2帧原始数据、第3帧原始数据和第4帧原始数据打包发送至算法后处理模块16。可以理解，当应用程序模块14向相机服务模块18请求的原始数据为多帧时，算法后处理模块16一般是需要对多帧原始数据进行拍照后处理，而若是应用程序模块14依次将多帧原始数据发送至算法后处理模块16，算法后处理模块16在仅接收到第1帧原始数据，或者仅接收到第1帧原始数据和第2帧原始数据，或者仅接收到第1帧原始数据、第2帧原始数据和第3帧原始数据时，算法后处理模块16并不能完成对多帧原始数据的拍照后处理。在本申请实施方式中，应用程序模块14将多帧原始数据打包发送至算法后处理模块16，可以高效地利用传输资源，在应用程序模块14将多帧原始数据打包发送至算法后处理模块16之后，算法后处理模块16即可对多帧原始数据进行拍照后处理，并将处理后的结果数据传回给所述应用程序模块14。
当然，在其他实施方式中，相机服务模块18也可以实现将多帧原始数据打包的功能。也即是说，硬件抽象模块12在接收到请求后，依次将多帧原始数据发送至相机服务模块18；相机服务模块18将多帧原始数据打包发送至应用程序模块14；应用程序模块14在接收到打包好的多帧原始数据后，将打包好的多帧原始数据发送至算法后处理模块16。
请参阅图1，在某些实施方式中，在算法后处理模块16对原始数据进行拍照后处理的过程中，应用程序模块14能够关闭或者退出应用界面。
具体地，在算法后处理模块16对原始数据进行拍照后处理的过程中，用户能够选择性的控制应用程序模块14关闭或者退出应用界面。也即是说，当用户仍需要使用应用程序模块14的应用界面时，用户也可以保持打开应用程序模块14的应用界面，而用户不需要使用应用程序模块14的应用界面时，由于算法后处理模块16对原始数据进行拍照后处理的过程不涉及应用程序模块14的工作，应用程序模块14能够根据用户的控制关闭或者退出应用界面。用户无需在应用程序模块14的应用界面长时间等待，导致无法执行其他操作。
请参阅图1和图4，在某些实施方式中，图像处理方法还包括：
06：相机服务模块18向应用程序模块14发送关于原始数据的帧数建议；
应用程序模块14向相机服务模块18请求原始数据(即01)，包括：
012：应用程序模块14根据帧数建议向相机服务模块18请求原始数据。
请参阅图1，在某些实施方式中，相机服务模块18可用于执行06中的方法，应用程序模块14可用于执行012中的方法。也即是说，相机服务模块18还可以用于向应用程序模块14发送关于原始数据的帧数建议。应用程序模块14还可以用于根据帧数建议向相机服务模块18请求原始数据。
具体地，帧数建议表示帧数的具体数值，例如1帧、2帧、3帧、4帧等。若相机服务模块18向应用程序模块14发送的关于原始数据的帧数建议为4帧，则应用程序模块14向相机服务模块18请求4帧原始数据；若相机服务模块18向应用程序模块14发送的关于原始数据的帧数建议为5帧，则应用程序模块14向相机服务模块18请求5帧原始数据。
硬件抽象模块12可以获取感光度(ISO值)信息、陀螺仪的抖动情况、AR场景检测结果(检测场景类型，例如人物、动物、风景等)等，硬件抽象模块12将这些信息发送至相机服务模块18，相机服务模块18根据这些信息向应用程序模块14发送关于原始数据的帧数建议。例如，当抖动较大时，相机服务模块18向应用程序模块14建议的帧数可以相对多一些，以更好地实现拍照后处理；当几乎不存在抖动时，相机服务模块18向应用程序模块14建议的帧数可以相对少一些，以减少数据传输量。也即是说，相机服务模块18向应用程序模块14建议的帧数与陀螺仪的抖动程度成正相关。当然，硬件抽象模块12也可以直接根据这些信息给出关于原始数据的帧数建议，相机服务模块18再向应用程序模块14转发关于原始数据的帧数建议。
请参阅图1和图5，在某些实施方式中，图像处理方法还包括：
07：相机服务模块18向应用程序模块14发送对原始数据进行拍照后处理的算法建议；
08：应用程序模块14根据算法建议设置预定算法并将预定算法发送至硬件抽象模块12；
算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14(即05)，包括：
052：算法后处理模块16根据预定算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14。
请参阅图1，在某些实施方式中，相机服务模块18可用于执行07中的方法，应用程序模块14可用于执行08中的方法，算法后处理模块16可用于执行052中的方法。也即是说，相机服务模块18还可以用于向应用程序模块14发送对原始数据进行拍照后处理的算法建议。应用程序模块14还可以用于根据算法建议设置预定算法并将预定算法发送至硬件抽象模块12。算法后处理模块16还可以用于根据预定算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14。
具体地，算法建议和预定算法均可以是上述多帧处理算法、美颜处理算法、虚化处理算法、HDR处理算法、滤镜处理算法、旋转处理算法、水印处理算法中一种或多种。例如，若相机服务模块18向应用程序模块14发送的对原始数据进行拍照后处理的算法建议为多帧处理算法，则应用程序模块14根据算法建议设置多帧处理算法，并将多帧处理算法发送至硬件抽象模块12，算法后处理模块16根据多帧处理算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14；若相机服务模块18向应用程序模块14发送的对原始数据进行拍照后处理的算法建议为美颜处理算法和HDR处理算法，则应用程序模块14根据算法建议设置美颜处理算法和HDR处理算法，并将美颜处理算法和HDR处理算法发送至硬件抽象模块12，算法后处理模块16根据美颜处理算法和HDR处理算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14。
硬件抽象模块12可以获取感光度(ISO值)信息、陀螺仪的抖动情况、AR场景检测结果(检测场景类型，例如人物、动物、风景等)等，硬件抽象模块12将这些信息发送至相机服务模块18，相机服务模块18根据这些信息向应用程序模块14发送对原始数据进行拍照后处理的算法建议。例如，当陀螺仪检测到的抖动较大时，相机服务模块18建议多帧处理算法，以根据多帧处理消除抖动；当AR场景检测结果检测的场景类型为人物时，相机服务模块18建议美颜处理算法，以对人物进行美颜；当AR场景检测结果检测的场景类型为风景时，相机服务模块18建议HDR处理算法，以形成高动态范围的风景图像。当然，硬件抽象模块12也可以直接根据这些信息给 出对原始数据进行拍照后处理的算法建议，相机服务模块18再向应用程序模块14转发对原始数据进行拍照后处理的算法建议。
在某些实施方式中，图像处理方法还包括：相机服务模块18一并向应用程序模块14发送关于原始数据的帧数建议和对原始数据进行拍照后处理的算法建议；应用程序模块14根据帧数建议向相机服务模块18请求原始数据；应用程序模块14根据算法建议设置预定算法并将预定算法发送至硬件抽象模块12；算法后处理模块16根据预定算法对原始数据进行拍照后处理，并将处理后的结果数据传回给应用程序模块14。
请参阅图1和图6，在某些实施方式中，原始数据包括待处理数据和元数据。应用程序模块14向相机服务模块18请求原始数据(即01)，包括：
014：应用程序模块14向相机服务模块18请求待处理数据；
硬件抽象模块12在接收到请求后，将原始数据经相机服务模块18发送至应用程序模块14(即03)，包括：
034：硬件抽象模块12在接收到请求后，将待处理数据和与待处理数据对应的元数据经相机服务模块18发送至应用程序模块14；
应用程序模块14在接收到原始数据后，将原始数据发送至算法后处理模块16(即04)，包括：
044：应用程序模块14在接收到待处理数据和元数据后，将待处理数据和元数据发送至算法后处理模块16；
算法后处理模块16对原始数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14(即05)，包括：
054：算法后处理模块16根据元数据对待处理数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1，在某些实施方式中，原始数据包括待处理数据和元数据。应用程序模块14可用于执行014中的方法，硬件抽象模块12可用于执行034中的方法，应用程序模块14可用于执行044中的方法，算法后处理模块16可用于执行054中的方法。
也即是说，应用程序模块14可以用于向相机服务模块18请求待处理数据。硬件抽象模块12可以用于在接收到请求后，将待处理数据和与待处理数据对应的元数据经相机服务模块18发送至应用程序模块14。应用程序模块14可以用于在接收到待处理数据和元数据后，将待处理数据和元数据发送至算法后处理模块16。算法后处理模块16可以用于根据元数据对待处理数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
具体地，元数据(metadata)与待处理数据对应。元数据包括待处理数据的相关信息，例如3a(自动曝光控制AE、自动聚焦控制AF、自动白平衡控制AWB)信息、图片信息(图片的宽度和高度)、曝光参数(光圈大小、快门速度和感光度光圈值)等。算法后处理模块16可以依据元数据实现对待处理数据的拍照后处理。例如，算法后处理模块16依据元数据中的ISO值调节待处理数据对应的图像亮度等。
当原始数据包括待处理数据和元数据时，应用程序模块14向相机服务模块18请求待处理数据；相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，将待处理数据和与待处理数据对应的元数据经相机服务模块18发送至应用程序模块14；应用程序模块14在接收到待处理数据和元数据后，将待处理数据和元数据发送至算法后处理模块16；算法后处理模块16根据元数据对待处理数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
进一步地，当应用程序模块14向相机服务模块18请求的原始数据为多帧时，相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，依次将多帧待处理数据和分别与多帧待处理数据对应的多帧元数据经相机服务模块18发送至应用程序模块14；应用程序模块14在接收到多帧待处理数据和多帧元数据后，将多帧待处理数据和多帧元数据打包发送至算法后处理模块16；算法后处理模块16根据多帧元数据对多帧待处理数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
例如，应用程序模块14向相机服务模块18请求4帧待处理数据，相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，首先，硬件抽象模块12将第1帧待处理数据(即图1中的待处理数据1)和第1帧元数据(即图1中的元数据1)发送至相机服务模块18，相机服务模块18将第1帧待处理数据和第1帧元数据发送至应用程序模块14；其次，硬件抽象模块12将第2帧待处理数据(即图1中的待处理数据2)和第2帧元数据(即图1中的元数据2)发送至相机服务模块18，相机服务模块18将第2帧待处理数据和第2帧元数据发送至应用程序模块14；然后，硬件抽象模块12将第3帧待处理数据(即图1中的待处理数据3)和第3帧元数据(即图1中的元数据3)发送至相机服务模块18，相机服务模块18将第3帧待处理数据和第3帧元数据发送至应用程序模块14；最后，硬件抽象模块12将第4帧待处理数据(即图1中的待处理数据4)和第4帧元数据(即图1中的元数据4)发送至相机服务模块18，相机服务模块18将第4帧待处理数据和第4帧元数据发送至应用程序模块14。在这一过程中，硬件抽象模块12将前一帧待处理数据和元数据发送至相机服务模块18后，即可开始将下一帧待处理数据和元数据发送至相机服务模块18，而无需等待相机服务模块18将前一帧待处理数据和元数据发送至应用程序模块14，再开始将下一帧待处理数据和元数据发送至相机服务模块18。这样可以节省将多帧待处理数据和元数据经相机服务模块18发送至应用程序模块14所需的总时长，避免用户长时间等待，有利于提升拍照体验。
应用程序模块14在接收到第1帧待处理数据和第1帧元数据、第2帧待处理数据和第2帧元数据、第3帧待处理数据和第3帧元数据、第4帧待处理数据和第4帧元数据后，将第1帧待处理数据和第1帧元数据、第2帧待处理数据和第2帧元数据、第3帧待处理数据和第3帧元数据、第4帧待处理数据和第4帧元数据打包发送至算法后处理模块16。可以理解，当应用程序模块14向相机服务模块18请求的待处理数据为多帧时，算法后处理模块16一般是需要对多帧待处理数据进行拍照后处理，而若是应用程序模块14依次将多帧待处理数据和对应的元数据发送至算法后处理模块16，算法后处理模块16在仅接收到第1帧待处理数据和第1帧元数据，或者仅接收到第1帧待处理数据和第1帧元数据、第2帧待处理数据和第2帧元数据，或者仅接收到第1帧待处理数据和第1帧元数据、第2帧待处理数据和第2帧元数据、第3帧待处理数据和第3帧元数据时，算法后处理模块16并不能完成对多帧待处理数据的拍照后处理，在本申请实施方式中，应用程序模块14将多帧待处理数据和对应的元数据打包发送至算法后处理模块16，可以高效地利用传输资源，在应用程序模块14将多帧待处理数据和对应的元数据打包发送至算法后处理模块16之后，算法后处理模块16即可根据多帧元数据对多帧待处理数据进行拍照后处理，并将处理后的结果数据传回给所述应用程序模块14。
请参阅图1和图7，在某些实施方式中，硬件抽象模块12内的数据为第一图像格式。其中：应用程序模块14向相机服务模块18请求的原始数据也为第一图像格式的数据。或者，应用程序模块14向相机服务模块18请求的原始数据为第二图像格式的数据，图像处理方法还包括：
09：硬件抽象模块12将第一图像格式的数据转换为第二图像格式的数据。
请参阅图1，在某些实施方式中，硬件抽象模块12内的数据为第一图像格式。其中：应用程序模块14向相机服务模块18请求的原始数据也为第一图像格式的数据。或者，应用程序模块14向相机服务模块18请求的原始数据为第二图像格式的数据。硬件抽象模块12可用于执行09中的方法。也即是说，硬件抽象模块12还可以用于将第一图像格式的数据转换为第二图像格式的数据。
具体地，第一图像格式可以是RAW格式，第二图像格式可以是YUV格式。硬件抽象模块12内的数据为第一图像格式，即RAW格式。
第一种情况，应用程序模块14向相机服务模块18请求的原始数据也为RAW格式。此时，硬件抽象模块12内的数据与应用程序模块14向相机服务模块18请求的原始数据格式相同。硬件抽象模块12无需对RAW格式的数据进行格式转换。
当应用程序模块14向相机服务模块18请求多帧RAW格式的原始数据时，相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，依次将多帧RAW格式的数据经相机服务模块18发送至应用程序模块14；应用程序模块14在接收到多帧 RAW格式的数据后，将多帧RAW格式的数据打包发送至算法后处理模块16；算法后处理模块16对多帧RAW格式的数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
当原始数据包括待处理数据和元数据时，应用程序模块14向相机服务模块18请求RAW格式的数据；相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，将RAW格式的数据和对应的元数据经相机服务模块18发送至应用程序模块14；应用程序模块14在接收到RAW格式的数据和对应的元数据后，将RAW格式的数据和对应的元数据发送至算法后处理模块16；算法后处理模块16根据元数据对RAW格式的数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
第二种情况，应用程序模块14向相机服务模块18请求的原始数据为YUV格式。此时，硬件抽象模块12内的数据与应用程序模块14向相机服务模块18请求的原始数据格式不同。硬件抽象模块12需要将RAW格式的数据转换为YUV格式的数据。
当应用程序模块14向相机服务模块18请求多帧YUV格式的原始数据时，相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，先将多帧RAW格式的数据转换为多帧YUV格式的数据，再依次将多帧YUV格式的数据经相机服务模块18发送至应用程序模块14；应用程序模块14在接收到多帧YUV格式的数据后，将多帧YUV格式的数据打包发送至算法后处理模块16；算法后处理模块16对多帧YUV格式的数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
当原始数据包括待处理数据和元数据时，应用程序模块14向相机服务模块18请求YUV格式的数据；相机服务模块18在接收到请求后，将请求发送给硬件抽象模块12；硬件抽象模块12在接收到请求后，先将RAW格式的数据转换为YUV格式的数据，再将YUV格式的数据和对应的元数据经相机服务模块18发送至应用程序模块14；应用程序模块14在接收到YUV格式的数据和对应的元数据后，将YUV格式的数据和对应的元数据发送至算法后处理模块16；算法后处理模块16根据元数据对YUV格式的数据进行拍照后处理并将处理后的结果数据传回给应用程序模块14。
请参阅图1和图8，在某些实施方式中，硬件抽象模块12内的第一图像格式的数据通过图像传感器20直接传输获得。或者，硬件抽象模块12内存储的第一图像格式的数据通过被算法后处理模块16处理实现拍照后处理并经应用程序模块14回传获得。
请参阅图8，本申请实施方式的拍摄装置100包括上述任一实施方式的图像处理器10和图像传感器20。图像处理器10和图像传感器20连接。图像传感器20包括图像采集单元(sensor)22和RAW图像数据单元(Image Front-end，IFE)24，图像采集单元22用于接收光线以采集获得图像数据(RAW图像，即前述第一图像格式的数据)，RAW图像数据单元24用于将图像采集单元22采集的图像数据传输至图像处理器10，其中，RAW图像数据单元24可以对图像采集单元22采集获得的RAW图像进行处理并输出处理后的RAW图像至图像处理器10。
图像处理器10包括硬件抽象模块12、应用程序模块14、算法后处理模块16和相机服务模块18。
硬件抽象模块12用于接收RAW图像、将RAW图像转换为YUV图像(即前述第二图像格式的数据)、及传输RAW图像和/或YUV图像。硬件抽象模块12可以与图像传感器20连接。具体地，硬件抽象模块12可以包括与图像传感器20连接的缓存单元(buffer queue)122、RAW转RGB处理单元(Bayer Process Segment，BPS)124和与应用程序模块14连接的降噪及YUV后处理单元(Image Process Engine，IPE)126。缓存单元122用于缓存来自图像传感器20的RAW图像并通过相机服务模块18、应用程序模块14传输给算法后处理模块16。RAW转RGB处理单元124用于将来自缓存单元122的RAW图像转换为RGB图像。降噪及YUV后处理单元126用于处理RGB图像得到YUV图像并将YUV图像通过相机服务模块18、应用程序模块14传输给算法后处理模块16。硬件抽象模块12还可传输图像数据的元数据，元数据包括3a(自动曝光控制AE、自动聚焦控制AF、自动白平衡控制AWB)信息、图片信息(例如图像宽度、高度)、曝光参数(光圈大小、快门速度和感光度光圈值)等，可以利用元数据辅助实现对RAW图像和/或YUV图像的拍照后处理(例如包括美颜处理、滤镜处理、旋转处理、水印处理、虚化处理、 HDR处理、及多帧处理中的至少一种)。在一个实施例中，元数据包括感光度(ISO)信息，根据感光度信息可以辅助调节RAW图像和/或YUV图像的亮度，从而实现与调节亮度相关的拍照后处理。
由于硬件抽象模块12只接收RAW图像、将RAW图像转换为YUV图像、及传输RAW图像和/或YUV图像，而不对RAW图像和/或YUV图像进行拍照后处理，拍照后处理的图像处理算法无需在硬件抽象模块12本身的算法架构上做流程截断，只需在外部做兼容，设计难度大大减小。
在相关技术中，应用程序接口(API)将硬件抽象模块建立成管道(pipeline)的方式，由于管道的创建需要大量的时间和内存，因此在相机启动时需要将相机对应的工作模式用到的所有管道均先创建好，而为了实现各种图像处理算法，一般需要创建大量管道(例如超过三条管道)，这会导致相机的启动需要耗费大量的时间，并且占用大量的内存。本申请实施方式的硬件抽象模块12只接收RAW图像、将RAW图像转换为YUV图像、及传输RAW图像和/或YUV图像，而不对RAW图像和/或YUV图像进行拍照后处理，因此，硬件抽象模块12只需建立少量(例如一条或两条)管道即可，无需建立大量的管道，从而能够节约内存，并且可以使得相机的启动速度变快。
应用程序模块14与硬件抽象模块12通过相机服务模块18连接。应用程序模块14可以用于根据用户的输入产生控制指令并将该控制指令通过相机服务模块18、硬件抽象模块12发送给图像传感器20以对图像传感器20的工作进行相应的控制。其中，应用程序模块14可以以64比特位(bit)运行，并且拍照后处理的图像处理算法的静态数据连接库(lib)可以配置为64比特位，以提高运算速度。应用程序模块14接收硬件抽象模块12通过相机服务模块18传输的RAW图像和/或YUV图像后，可以对RAW和/或YUV图像进行拍照后处理，也可以将RAW和/或YUV图像传输至算法后处理模块16进行拍照后处理。当然，还可以是应用程序模块14进行一些拍照后处理(例如美颜处理、滤镜处理、旋转处理、水印处理、虚化处理等)，算法后处理模块16进行另外一些拍照后处理(例如HDR处理、多帧处理等)。在本申请实施方式中，应用程序模块14将RAW和/或YUV图像传输至算法后处理模块16进行拍照后处理。
算法后处理模块16通过应用程序模块14、相机服务模块18与硬件抽象模块12连接，算法后处理模块16内存储有至少一种图像处理算法(例如包括美颜处理算法、滤镜处理算法、旋转处理算法、水印处理算法、虚化处理算法、HDR处理算法、及多帧处理算法中的至少一种)，算法后处理模块16用于采用图像处理算法处理RAW图像和/或YUV图像以实现拍照后处理。由于对RAW图像和/或YUV图像进行拍照后处理可由算法后处理模块16实现，从而无需在硬件抽象模块12本身的算法架构上做流程截断，只需在外部做兼容，设计难度减小。并且由于拍照后处理由算法后处理模块16实现，因此算法后处理模块16的功能更单一、更聚焦，从而可以达到移植快，扩展新的图像处理算法简单等效果。当然，若是在应用程序模块14进行一些拍照后处理(例如美颜处理、滤镜处理、旋转处理、水印处理、虚化处理等)，算法后处理模块16进行另外一些拍照后处理(例如HDR处理、多帧处理等)时，应用程序模块14内也可存储有至少一种图像处理算法(例如包括美颜处理算法、滤镜处理算法、旋转处理算法、水印处理算法、虚化处理算法、HDR处理算法、及多帧处理算法中的至少一种)，应用程序模块14还用于采用图像处理算法处理RAW图像和/或YUV图像以实现拍照后处理。由于对RAW图像和/或YUV图像进行拍照后处理由应用程序模块14与算法后处理模块16实现，从而无需在硬件抽象模块12本身的算法架构上做流程截断，只需在外部做兼容，耦合设计难度同样也是大大减小。
在算法后处理模块16只处理RAW图像(例如图像处理算法针对RAW图像进行处理)时，硬件抽象模块12可以只传输RAW图像(此时可以不需要将RAW图像转换为YUV图像)；在算法后处理模块16只处理YUV图像(例如图像处理算法针对YUV图像进行处理)时，硬件抽象模块12可以只传输YUV图像；在算法后处理模块16处理RAW图像和YUV图像时，硬件抽象模块12可以传输RAW图像和YUV图像。
图像传感器20进行一次拍摄(曝光成像)后，将拍摄数据(RAW图像)传输给硬件抽象模块12，在算法后处理模块16接收到与拍摄数据对应的RAW图像和/或YUV图像后，图像传感 器20能够进行下一次拍摄、或图像传感器20能够关闭、或应用程序模块14能够关闭、或应用程序模块14能够退出应用界面。由于拍照后处理由算法后处理模块16实现，因此在将拍摄数据对应的RAW图像和/或YUV图像传输给算法后处理模块16后，只需要算法后处理模块16就可实现拍照后处理，此时图像传感器20和应用程序模块14无需参与拍照后处理，因此，图像传感器20能够关闭或执行下一次拍摄，应用程序模块14能够关闭或退出应用界面。如此，拍摄装置100能够实现快拍，并且在算法后处理模块16进行拍照后处理时可以关闭应用程序模块14或退出应用程序界面，从而在电子设备上进行一些其他操作(例如与拍摄装置100无关的操作，比如浏览网页、看视频、打电话等)，从而用户无需花费大量的时间等待拍照后处理的完成，便于用户使用电子设备。
算法后处理模块16可以包括编码单元162，编码单元162用于将YUV图像转换为JPG图像(或者JPEG图像等)。具体地，在算法后处理模块14处理的是YUV图像时，编码单元162可以直接对YUV图像进行编码以形成JPG图像，从而提高图像的输出速度。在算法后处理模块14处理的是RAW图像时，算法后处理模块14可以将处理实现拍照后处理的RAW图像经应用程序模块14、相机服务模块18回传至硬件抽象模块12，例如回传至RAW转RGB处理单元124，RAW转RGB处理单元124可以用于将算法后处理模块14处理实现拍照后处理并经应用程序模块14回传的RAW图像转换为RGB图像，降噪及YUV后处理单元126可以将RGB图像转换为YUV图像，该YUV图像可以再次传输至算法后处理模块16的编码单元162中以将该YUV图像转换为JPG图像。在某些实施方式中，算法后处理模块14也可以将处理实现拍照后处理的RAW图像经应用程序模块14、相机服务模块18回传至缓存单元122，回传的RAW图像经过RAW转RGB处理单元124和降噪及YUV后处理单元126形成YUV图像，再传输至编码单元162以形成JPG图像。在形成JPG图像后，算法后处理模块16可以用于将JPG图像传输至存储器中保存。
请参阅图9，算法后处理模块16包括逻辑处理调用层164、算法模块接口层166和算法处理层168。逻辑处理调用层164用于与应用程序模块14通信。算法模块接口层166用于维护算法接口。算法处理层168包括至少一种图像处理算法。算法模块接口层166用于通过算法接口对算法处理层168的图像处理算法进行注册、注销、调用和回调中的至少一种操作。
逻辑处理调用层164可以包括线程队列，算法后处理模块16接收到RAW图像和/或YUV图像的拍照后处理任务后，可以将拍照后处理任务缓存在线程队列中处理，其中，线程队列可以缓存多个拍照后处理任务，如此，可以通过逻辑处理调用层164实现快拍(即快拍机制)。另外，逻辑处理调用层164也可以接收应用程序模块14发送的初始化(init)、进程(process)等指令，并将对应的指令和数据保存到线程队列中。逻辑处理调用层164根据线程队列中的任务进行具体逻辑的调用(即具体逻辑调用组合)。逻辑处理调用层164还可以将处理获得的缩略图(thumbnail)回传给应用程序模块14进行显示(即缩略图回显)。
算法模块接口层166用于调用算法接口，调用命令也可以保存到线程队列中，算法处理层168在接收到线程队列的调用命令时，即可解析调用命令的参数获得需要调用的图像处理算法。算法模块接口层166对图像处理算法进行注册时，可以在算法处理层168中新增一种图像处理算法；算法模块接口层166对图像处理算法进行注销时，可以删除算法处理层168中一种图像处理算法；算法模块接口层166对图像处理算法进行调用时，可以调用算法处理层168中一种图像处理算法；算法模块接口层166对图像处理算法进行回调时，可以将算法处理后的数据和状态回传给应用程序模块14。其中，可以采用统一接口实现图像处理算法的注册、注销、调用、回调等操作。算法处理层168中的每一种图像处理算法都是独立的，如此可以方便对图像处理算法实现注册、注销、调用、回调等操作。
请参阅图8，相机服务模块18分别与硬件抽象模块12和应用程序模块14连接。相机服务模块18对RAW图像和/或YUV图像进行封装并将封装后的RAW图像和/或YUV图像传输至应用程序模块14、及将应用程序模块14回传的RAW图像传输至硬件抽象模块12。如此，通过相机服务模块18对图像进行封装，可以提高图像传输的效率，并且能够提高图像传输的安全性。硬件抽象模块12和应用程序模块14之间传输的数据需要通过相机服务模块18。例如，硬件抽象模块12向应用程序模块14传输RAW图像和/或YUV图像时，硬件抽象模块12先将RAW图像和 /或YUV图像传输至相机服务模块18，相机服务模块18对RAW图像和/或YUV图像进行封装并将封装后的RAW图像和/或YUV图像传输至应用程序模块14。又例如，硬件抽象模块12向应用程序模块14传输元数据时，硬件抽象模块12先将元数据传输至相机服务模块18，相机服务模块18对元数据进行封装并将封装后的元数据传输至应用程序模块14。又例如，硬件抽象模块12向应用程序模块14传输帧数建议时，硬件抽象模块12先将帧数建议传输至相机服务模块18，相机服务模块18对帧数建议进行封装并将封装后的帧数建议传输至应用程序模块14。又例如，硬件抽象模块12向应用程序模块14传输算法建议时，硬件抽象模块12先将算法建议传输至相机服务模块18，相机服务模块18对算法建议进行封装并将封装后的算法建议传输至应用程序模块14。当然，在某些实施方式中，硬件抽象模块12可将感光度信息、陀螺仪的抖动情况、AR场景检测结果等传输至相机服务模块18，相机服务模块18根据感光度信息、陀螺仪的抖动情况、AR场景检测结果等获得帧数建议和/或算法建议，再将帧数建议和/或算法建议传输至应用程序模块14。
请参阅图10和图11，本申请实施方式的电子设备1000包括上述任一实施方式的拍摄装置100和壳体200，拍摄装置100与壳体200结合。壳体200可以作为电子设备1000的功能元件的安装载体。壳体200可以为功能元件提供防尘、防摔、防水等保护，功能元件可以是显示屏、拍摄装置100、受话器等。其中，在一个实施方式中，壳体200包括主体210和可动支架220，可动支架220在驱动装置的驱动下可以相对于主体210运动，例如可动支架220可以相对于主210体滑动，以滑入主体210(例如图10的状态)或从主体210滑出(例如图11的状态)。部分功能元件可以安装在主体210上，另一部分功能元件(例如拍摄装置100)可以安装在可动支架220上，可动支架220运动可带动该另一部分功能元件缩回主体210内或从主体210中伸出。在另一个实施方式中，壳体200上开设有采集窗口，拍摄装置100与采集窗口对准安装以使拍摄装置100能够通过采集窗口接收外界光线以形成图像，或者拍摄装置100设置在显示屏下方，拍摄装置100接收穿过显示屏的外界光线以形成图像。
尽管上面已经示出和描述了本申请的实施方式，可以理解的是，上述实施方式是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型。

Claims (22)

  1. 一种图像处理方法，其特征在于，包括：
    应用程序模块向相机服务模块请求原始数据；
    所述相机服务模块在接收到所述请求后，将所述请求发送给硬件抽象模块；
    所述硬件抽象模块在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块在接收到所述原始数据后，将所述原始数据发送至算法后处理模块；和
    所述算法后处理模块对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述应用程序模块向所述相机服务模块请求的所述原始数据为多帧；
    所述硬件抽象模块在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块，包括：
    所述硬件抽象模块在接收到所述请求后，依次将多帧所述原始数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块在接收到所述原始数据后，将所述原始数据发送至算法后处理模块，包括：
    所述应用程序模块在接收到多帧所述原始数据后，将多帧所述原始数据打包发送至所述算法后处理模块。
  3. 根据权利要求1所述的图像处理方法，其特征在于，在所述算法后处理模块对所述原始数据进行拍照后处理的过程中，所述应用程序模块能够关闭或者退出应用界面。
  4. 根据权利要求1所述的图像处理方法，其特征在于，所述图像处理方法还包括：
    所述相机服务模块向所述应用程序模块发送关于所述原始数据的帧数建议；
    所述应用程序模块向相机服务模块请求原始数据，包括：
    所述应用程序模块根据所述帧数建议向所述相机服务模块请求所述原始数据。
  5. 根据权利要求1所述的图像处理方法，其特征在于，所述图像处理方法还包括：
    所述相机服务模块向所述应用程序模块发送对所述原始数据进行拍照后处理的算法建议；
    所述应用程序模块根据所述算法建议设置预定算法并将所述预定算法发送至所述硬件抽象模块；
    所述算法后处理模块对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块，包括：
    所述算法后处理模块根据所述预定算法对所述原始数据进行拍照后处理，并将处理后的所述结果数据传回给所述应用程序模块。
  6. 根据权利要求1所述的图像处理方法，其特征在于，所述原始数据包括待处理数据和元数据；所述应用程序模块向相机服务模块请求原始数据，包括：
    所述应用程序模块向所述相机服务模块请求所述待处理数据；
    所述硬件抽象模块在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块，包括：
    所述硬件抽象模块在接收到所述请求后，将所述待处理数据和与所述待处理数据对应的所述元数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块在接收到所述原始数据后，将所述原始数据发送至算法后处理模块，包括：
    所述应用程序模块在接收到所述待处理数据和所述元数据后，将所述待处理数据和所述元数据发送至算法后处理模块；
    所述算法后处理模块对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块，包括：
    所述算法后处理模块根据所述元数据对所述待处理数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
  7. 根据权利要求1所述的图像处理方法，其特征在于，所述硬件抽象模块内的数据为第一图像格式；其中：
    所述应用程序模块向所述相机服务模块请求的所述原始数据也为所述第一图像格式的数据；或者
    所述应用程序模块向所述相机服务模块请求的所述原始数据为第二图像格式的数据，所述图像处理方法还包括：
    所述硬件抽象模块将所述第一图像格式的数据转换为所述第二图像格式的所述数据。
  8. 一种图像处理器，其特征在于，包括应用程序模块、相机服务模块、硬件抽象模块和算法后处理模块；
    所述应用程序模块用于向所述相机服务模块请求原始数据；
    所述相机服务模块用于在接收到所述请求后，将所述请求发送给所述硬件抽象模块；
    所述硬件抽象模块用于在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块还用于在接收到所述原始数据后，将所述原始数据发送至所述算法后处理模块；和
    所述算法后处理模块用于对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
  9. 根据权利要求8所述的图像处理器，其特征在于，所述应用程序模块向所述相机服务模块请求的所述原始数据为多帧；
    所述硬件抽象模块用于在接收到所述请求后，依次将多帧所述原始数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块用于在接收到多帧所述原始数据后，将多帧所述原始数据打包发送至所述算法后处理模块。
  10. 根据权利要求8所述的图像处理器，其特征在于，在所述算法后处理模块对所述原始数据进行拍照后处理的过程中，所述应用程序模块能够关闭或者退出应用界面。
  11. 根据权利要求8所述的图像处理器，其特征在于，
    所述相机服务模块还用于向所述应用程序模块发送关于所述原始数据的帧数建议；
    所述应用程序模块还用于根据所述帧数建议向所述相机服务模块请求所述原始数据。
  12. 根据权利要求8所述的图像处理器，其特征在于，
    所述相机服务模块还用于向所述应用程序模块发送对所述原始数据进行拍照后处理的算法建议；
    所述应用程序模块还用于根据所述算法建议设置预定算法并将所述预定算法发送至所述硬件抽象模块；
    所述算法后处理模块还用于根据所述预定算法对所述原始数据进行拍照后处理，并将处理后的所述结果数据传回给所述应用程序模块。
  13. 根据权利要求8所述的图像处理器，其特征在于，所述原始数据包括待处理数据和元数据；
    所述应用程序模块用于向所述相机服务模块请求所述待处理数据；
    所述硬件抽象模块用于在接收到所述请求后，将所述待处理数据和与所述待处理数据对应的所述元数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块用于在接收到所述待处理数据和所述元数据后，将所述待处理数据和所述元数据发送至算法后处理模块；
    所述算法后处理模块用于根据所述元数据对所述待处理数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
  14. 根据权利要求8所述的图像处理器，其特征在于，所述硬件抽象模块内的数据为第一图像格式；其中：
    所述应用程序模块向所述相机服务模块请求的所述原始数据也为所述第一图像格式的数据； 或者
    所述应用程序模块向所述相机服务模块请求的所述原始数据为第二图像格式的数据，所述硬件抽象模块还用于将所述第一图像格式的数据转换为所述第二图像格式的所述数据。
  15. 一种拍摄装置，其特征在于，包括图像处理器和图像传感器，所述图像传感器与所述图像处理器连接；所述图像处理器包括应用程序模块、相机服务模块、硬件抽象模块和算法后处理模块；
    所述应用程序模块用于向所述相机服务模块请求原始数据；
    所述相机服务模块用于在接收到所述请求后，将所述请求发送给所述硬件抽象模块；
    所述硬件抽象模块用于在接收到所述请求后，将所述原始数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块还用于在接收到所述原始数据后，将所述原始数据发送至所述算法后处理模块；
    所述算法后处理模块用于对所述原始数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
  16. 根据权利要求15所述的拍摄装置，其特征在于，所述应用程序模块向所述相机服务模块请求的所述原始数据为多帧；
    所述硬件抽象模块用于在接收到所述请求后，依次将多帧所述原始数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块用于在接收到多帧所述原始数据后，将多帧所述原始数据打包发送至所述算法后处理模块。
  17. 根据权利要求15所述的拍摄装置，其特征在于，在所述算法后处理模块对所述原始数据进行拍照后处理的过程中，所述应用程序模块能够关闭或者退出应用界面。
  18. 根据权利要求15所述的拍摄装置，其特征在于，
    所述相机服务模块还用于向所述应用程序模块发送关于所述原始数据的帧数建议；
    所述应用程序模块还用于根据所述帧数建议向所述相机服务模块请求所述原始数据。
  19. 根据权利要求15所述的拍摄装置，其特征在于，所述相机服务模块还用于向所述应用程序模块发送对所述原始数据进行拍照后处理的算法建议；
    所述应用程序模块还用于根据所述算法建议设置预定算法并将所述预定算法发送至所述硬件抽象模块；
    所述算法后处理模块还用于根据所述预定算法对所述原始数据进行拍照后处理，并将处理后的所述结果数据传回给所述应用程序模块。
  20. 根据权利要求15所述的拍摄装置，其特征在于，所述原始数据包括待处理数据和元数据；
    所述应用程序模块用于向所述相机服务模块请求所述待处理数据；
    所述硬件抽象模块用于在接收到所述请求后，将所述待处理数据和与所述待处理数据对应的所述元数据经所述相机服务模块发送至所述应用程序模块；
    所述应用程序模块用于在接收到所述待处理数据和所述元数据后，将所述待处理数据和所述元数据发送至算法后处理模块；
    所述算法后处理模块用于根据所述元数据对所述待处理数据进行拍照后处理并将处理后的结果数据传回给所述应用程序模块。
  21. 根据权利要求15所述的拍摄装置，其特征在于，所述硬件抽象模块内的数据为第一图像格式；其中：所述应用程序模块向所述相机服务模块请求的所述原始数据也为所述第一图像格式的数据；或者
    所述应用程序模块向所述相机服务模块请求的所述原始数据为第二图像格式的数据，所述硬件抽象模块还用于将所述第一图像格式的数据转换为所述第二图像格式的所述数据。
  22. 一种电子设备，其特征在于，包括权利要求15至21任意一项所述的拍摄装置和壳体，所述拍摄装置与所述壳体结合。
PCT/CN2020/079534 2019-04-10 2020-03-16 图像处理方法、图像处理器、拍摄装置和电子设备 WO2020207200A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910285175.4 2019-04-10
CN201910285175.4A CN110086967B (zh) 2019-04-10 2019-04-10 图像处理方法、图像处理器、拍摄装置和电子设备

Publications (1)

Publication Number Publication Date
WO2020207200A1 true WO2020207200A1 (zh) 2020-10-15

Family

ID=67414749

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/079534 WO2020207200A1 (zh) 2019-04-10 2020-03-16 图像处理方法、图像处理器、拍摄装置和电子设备

Country Status (2)

Country Link
CN (1) CN110086967B (zh)
WO (1) WO2020207200A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114125277A (zh) * 2021-11-12 2022-03-01 北京达佳互联信息技术有限公司 控制应用的视频拍摄特效的方法及设备

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110062161B (zh) * 2019-04-10 2021-06-25 Oppo广东移动通信有限公司 图像处理器、图像处理方法、拍摄装置和电子设备
CN110086967B (zh) * 2019-04-10 2021-02-05 Oppo广东移动通信有限公司 图像处理方法、图像处理器、拍摄装置和电子设备
CN110401800B (zh) * 2019-08-26 2021-06-04 Oppo广东移动通信有限公司 图像处理方法、图像处理器、拍摄装置和电子设备
CN110942047B (zh) * 2019-12-09 2023-07-07 Oppo广东移动通信有限公司 应用优化方法及相关产品
CN110941821A (zh) * 2019-12-09 2020-03-31 Oppo广东移动通信有限公司 数据处理方法、装置及存储介质
CN110971830B (zh) * 2019-12-09 2021-11-02 Oppo广东移动通信有限公司 视频拍摄的防抖方法及相关装置
CN110995994B (zh) * 2019-12-09 2021-09-14 上海瑾盛通信科技有限公司 图像拍摄方法及相关装置
CN110958390B (zh) * 2019-12-09 2021-07-20 Oppo广东移动通信有限公司 图像处理方法及相关装置
CN110941344B (zh) * 2019-12-09 2022-03-15 Oppo广东移动通信有限公司 获取注视点数据的方法及相关装置
CN110933314B (zh) * 2019-12-09 2021-07-09 Oppo广东移动通信有限公司 追焦拍摄方法及相关产品
CN110996089B (zh) * 2019-12-09 2021-11-30 上海瑾盛通信科技有限公司 体积计算方法及相关装置
CN110996012B (zh) * 2019-12-23 2022-03-22 Oppo广东移动通信有限公司 连拍处理方法、图像处理器、拍摄装置和电子设备
CN111225153B (zh) * 2020-01-21 2021-08-06 Oppo广东移动通信有限公司 一种图像数据处理方法、图像数据处理装置及移动终端
CN111314606B (zh) * 2020-02-21 2021-06-18 Oppo广东移动通信有限公司 拍照方法、装置、电子设备及存储介质
CN113554718B (zh) * 2020-04-24 2023-04-28 Oppo（重庆）智能科技有限公司 图像编码方法、存储介质和电子设备
CN112291475B (zh) * 2020-10-30 2022-04-12 维沃移动通信（杭州）有限公司 拍照方法、装置及电子设备
CN113645409B (zh) * 2021-08-16 2022-08-19 展讯通信（上海）有限公司 拍照处理方法及装置、拍照方法、装置及系统、终端设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030048361A1 (en) * 1998-05-29 2003-03-13 Safai Mohammad A. Digital camera
JP2005284411A (ja) * 2004-03-26 2005-10-13 Fuji Photo Film Co Ltd 画像処理方法、及び画像処理装置、及び画像処理システム、及び画像処理プログラム
CN105827971A (zh) * 2016-03-31 2016-08-03 维沃移动通信有限公司 一种图像处理方法及移动终端
CN105827940A (zh) * 2015-09-24 2016-08-03 维沃移动通信有限公司 一种拍照方法及电子设备
CN108833803A (zh) * 2018-09-20 2018-11-16 Oppo广东移动通信有限公司 成像方法、装置和电子设备
CN110086967A (zh) * 2019-04-10 2019-08-02 Oppo广东移动通信有限公司 图像处理方法、图像处理器、拍摄装置和电子设备

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4847161B2 (ja) * 2006-03-02 2011-12-28 キヤノン株式会社 画像送信装置および撮像装置
CN103294493B (zh) * 2012-02-28 2017-03-01 联想(北京)有限公司 一种外围设备的初始化方法、初始化装置及照相机
CN102799469A (zh) * 2012-05-17 2012-11-28 江苏中科梦兰电子科技有限公司 一种安卓应用软件调用拍照模块的方法
US9030571B2 (en) * 2012-07-11 2015-05-12 Google Inc. Abstract camera pipeline for uniform cross-device control of image capture and processing
US10491812B2 (en) * 2015-03-23 2019-11-26 Intel Corporation Workload scheduler for computing devices with camera
CN105827957A (zh) * 2016-03-16 2016-08-03 上海斐讯数据通信技术有限公司 一种图像处理的系统和方法
CN109101352B (zh) * 2018-08-30 2021-08-06 Oppo广东移动通信有限公司 图像处理算法架构、算法调用方法、装置、存储介质及移动终端

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030048361A1 (en) * 1998-05-29 2003-03-13 Safai Mohammad A. Digital camera
JP2005284411A (ja) * 2004-03-26 2005-10-13 Fuji Photo Film Co Ltd 画像処理方法、及び画像処理装置、及び画像処理システム、及び画像処理プログラム
CN105827940A (zh) * 2015-09-24 2016-08-03 维沃移动通信有限公司 一种拍照方法及电子设备
CN105827971A (zh) * 2016-03-31 2016-08-03 维沃移动通信有限公司 一种图像处理方法及移动终端
CN108833803A (zh) * 2018-09-20 2018-11-16 Oppo广东移动通信有限公司 成像方法、装置和电子设备
CN110086967A (zh) * 2019-04-10 2019-08-02 Oppo广东移动通信有限公司 图像处理方法、图像处理器、拍摄装置和电子设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114125277A (zh) * 2021-11-12 2022-03-01 北京达佳互联信息技术有限公司 控制应用的视频拍摄特效的方法及设备
CN114125277B (zh) * 2021-11-12 2024-02-09 北京达佳互联信息技术有限公司 控制应用的视频拍摄特效的方法及设备

Also Published As

Publication number Publication date
CN110086967B (zh) 2021-02-05
CN110086967A (zh) 2019-08-02

Similar Documents

Publication Publication Date Title
WO2020207200A1 (zh) 图像处理方法、图像处理器、拍摄装置和电子设备
CN109963083B (zh) 图像处理器、图像处理方法、拍摄装置和电子设备
WO2020259250A1 (zh) 图像处理方法、图像处理器、拍摄装置和电子设备
WO2020207192A1 (zh) 图像处理器、图像处理方法、拍摄装置和电子设备
CN110290288B (zh) 图像处理器、图像处理方法、拍摄装置和电子设备
WO2021147482A1 (zh) 一种长焦拍摄的方法及电子设备
CN110177215A (zh) 图像处理方法、图像处理器、拍摄装置和电子设备
CN110266951A (zh) 图像处理器、图像处理方法、拍摄装置和电子设备
CN110996012B (zh) 连拍处理方法、图像处理器、拍摄装置和电子设备
CN110278373A (zh) 图像处理器、图像处理方法、拍摄装置和电子设备
CN110300240B (zh) 图像处理器、图像处理方法、拍照装置及电子设备
CN110177214B (zh) 图像处理器、图像处理方法、拍照装置及电子设备
CN110121022A (zh) 拍摄装置的控制方法、拍摄装置和电子设备
CN111193866B (zh) 图像处理方法、图像处理器、拍摄装置和电子设备
CN110418061B (zh) 图像处理方法、图像处理器、拍照装置及电子设备
WO2024045670A1 (zh) 生成高动态范围视频的方法和电子设备
WO2023226612A1 (zh) 一种曝光参数确定方法和装置
WO2023060921A1 (zh) 图像处理方法与电子设备
CN111193867A (zh) 图像处理方法、图像处理器、拍摄装置和电子设备
CN113992854B (zh) 图像预览方法、装置、电子设备和计算机可读存储介质
WO2024183345A1 (zh) 多摄像头画面预览方法、装置、电子设备及存储介质
CN110278375A (zh) 图像处理方法、装置、存储介质及电子设备
CN111491101B (zh) 图像处理器、图像处理方法、拍摄装置和电子设备
WO2024109207A1 (zh) 显示缩略图像的方法和电子设备
CN111510629A (zh) 数据显示方法、图像处理器、拍摄装置和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20786995

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20786995

Country of ref document: EP

Kind code of ref document: A1