CN116419049A

CN116419049A - Image processing method, image processing system, device and electronic equipment

Info

Publication number: CN116419049A
Application number: CN202111630133.3A
Authority: CN
Inventors: 李超锋
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2021-12-28
Filing date: 2021-12-28
Publication date: 2023-07-11

Abstract

The application discloses an image processing method, an image processing system, an image processing device and electronic equipment, and belongs to the technical field of electronic products. The image processing method comprises the following steps: receiving initial image data acquired by an image sensor based on a main control chip; processing the initial image data based on an image processing module in the main control chip to obtain processed first image data; detecting a shooting scene of the first image data based on the main control chip, and transmitting the first image data to an image enhancement chip based on the main control chip; and under the condition that the shooting scene is a preset scene, processing the first image data based on the image enhancement chip to obtain processed second image data.

Description

Image processing method, image processing system, device and electronic equipment

Technical Field

The present disclosure relates to the field of electronic products, and in particular, to an image processing method, an image processing system, an image processing device, and an electronic device.

Background

With the development of communication technology, the functions in electronic devices are becoming more and more powerful, and for example, users can take images using electronic devices. However, in the process of photographing with the existing electronic device, the quality of the photographed image depends to some extent on the external photographing environment, and when there are many interference factors that affect photographing from the outside, the photographed image is often not ideal. Therefore, the existing electronic equipment has the problem of poor shooting effect.

Disclosure of Invention

According to the image processing method, the image processing system, the image processing device and the electronic equipment, the quality of the image output by the camera in the electronic equipment can be improved.

In a first aspect, an embodiment of the present application provides an image processing method, including:

receiving initial image data acquired by an image sensor based on a main control chip;

processing the initial image data based on an image processing module in the main control chip to obtain processed first image data;

detecting a shooting scene of the first image data based on the main control chip, and transmitting the first image data to an image enhancement chip based on the main control chip;

and under the condition that the shooting scene is a preset scene, processing the first image data based on the image enhancement chip to obtain processed second image data.

In a second aspect, embodiments of the present application provide an image processing system, including: the image sensor, the main control chip and the image enhancement chip are respectively and electrically connected with the main control chip;

The image processing module is used for receiving initial image data acquired by the image sensor and processing the initial image data to obtain processed first image data;

the main control chip is used for transmitting the first image data to the image enhancement chip;

the image enhancement chip is used for processing the first image data based on the image enhancement chip under the condition that the shooting scene is a preset scene to obtain processed second image data.

In a third aspect, an embodiment of the present application provides an image processing apparatus, including the image processing system described in the second aspect.

In a fourth aspect, an embodiment of the present application provides an image processing apparatus, including:

the acquisition module is used for receiving initial image data acquired by the image sensor based on the main control chip;

the first processing module is used for processing the initial image data based on the image processing module in the main control chip to obtain processed first image data;

the second processing module is used for detecting shooting scenes of the first image data based on the main control chip and transmitting the first image data to the image enhancement chip based on the main control chip;

And the third processing module is used for processing the first image data based on the image enhancement chip under the condition that the shooting scene is a preset scene to obtain processed second image data.

In a fifth aspect, embodiments of the present application provide an electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the method as described in the first aspect.

In a sixth aspect, an embodiment of the present application provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, where the program or the instruction implements the steps described in the first aspect when executed by a processor.

In a seventh aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the steps of the image processing method according to the first aspect.

In the embodiment of the application, after the primary image data is processed by the main control chip to obtain the first image data, the shooting scene of the first image data is further detected based on the main control chip, and under the condition that the shooting scene of the first image data is detected to be a preset scene, the first image data is further enhanced based on the image enhancement chip to obtain the second image data, so that the quality of the image output by the electronic equipment can be improved.

Drawings

Fig. 1 is a schematic flow chart of an image processing method according to an embodiment of the present application;

fig. 2 is a schematic diagram of a hardware structure inside an electronic device according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an image processing system according to an embodiment of the present disclosure;

FIG. 4 is a second schematic diagram of an image processing system according to an embodiment of the present disclosure;

FIG. 5 is a second flowchart of an image processing method according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 8 is a second schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The image processing method, the image processing device and the electronic equipment provided by the embodiment of the application are described in detail through specific embodiments and application scenes thereof with reference to the accompanying drawings.

Referring to fig. 1, a flowchart of an image processing method according to an embodiment of the present application is provided, where the method includes the following steps:

step S101, receiving initial image data acquired by an image sensor based on a main control chip;

step S102, processing the initial image data based on an image processing module in the main control chip to obtain processed first image data;

step S103, detecting a shooting scene of the first image data based on the main control chip, and transmitting the first image data to an image enhancement chip based on the main control chip;

step S104, processing the first image data based on the image enhancement chip to obtain processed second image data when the shooting scene is a preset scene.

The image processing method can be applied to various electronic devices with shooting functions, can also be applied to various monitoring systems, and can be applied to an access control monitoring system for example.

The main control Chip may be a System On Chip (SOC), and the image Sensor may be a Camera Sensor (Camera Sensor) of a Camera in the electronic device. The image processing module may be an image processor (Image Signal Processor, ISP) integrated in the main control chip. The image enhancement chip may be an add-on image enhancement chip independent of the main control chip, wherein the image enhancement chip is electrically connected to the main control chip, for example, the image enhancement chip and the main control chip may be connected by using a mobile industry processor interface (Mobile Industry Processor Interface, MIPI) or a high-speed serial computer expansion bus interface (Peripheral Component Interconnect Express, PCIE).

Referring to fig. 2, in an embodiment of the present application, a schematic hardware structure of an electronic device includes: dynamic random access memory (Dynamic Random Access Memory, DRAM), a main control chip, an image enhancement chip and a display screen. The DRAM is electrically connected with the main control chip through a memory controller (DDR controller), a transmitting end (MIPI Transmit, MIPI TX) of the main control chip is electrically connected with a receiving end (MIPI Receive, MIPI RX) of the image enhancement chip, a first transmitting end (MIPI TX 0) of the image enhancement chip is electrically connected with the display screen, and a second transmitting end (MIPI TX 1) of the image enhancement chip is electrically connected with the receiving end (MIPI RX 1) of the main control chip. The main control chip is internally provided with processing units such as a display controller (Display Controller), a central processing unit (Central Processing Unit, CPU), a graphic processor (Graphics Processing Unit, GPU) and the like. And a central processing unit (Central Processing Unit, CPU) and processing units such as digital signal processing (Digital Signal Process, DSP) are arranged in the image enhancement chip.

Specifically, referring to fig. 2, the receiving end MIPI RX0 and the first transmitting end MIPI TX0 of the image enhancement chip are connected through a first path and a second path connected in parallel, where the first path is a BYPASS path, and when image data is transmitted from the receiving end MIPI RX0 to the first transmitting end MIPI TX0 through the first path, the image enhancement chip does not perform enhancement processing on the image data. The second path includes a Static Random-Access Memory (SRAM), and the image enhancement chip performs enhancement processing on the image data when the image data is transferred from the receiving end MIPI RX0 to the first transmitting end MIPI TX0 through the second path. Therefore, when the shooting scene of the first image data is a preset scene, the first image data can be transmitted through the second path, and when the shooting scene of the first image data is a scene other than the preset scene, the first image data can be transmitted through the first path.

The above-mentioned preset scene may be a scene with a poor shooting environment, for example, a scene with a dark brightness, or a scene with a high contrast in the scene, because of the poor shooting environment, there may be a large number of interference factors affecting shooting in the process of shooting the initial image data, so the quality of the initial image data obtained by shooting by the image sensor is generally poor, and after the image processing module of the main control chip processes the initial image data, the image quality of the obtained first image data may still be poor. At this time, the first image data may be further processed based on the image enhancement chip to improve the image quality of the camera output.

Specifically, the initial image data is an image acquired by the image sensor for sensitization acquisition of an external shooting scene. The image processing module may process the initial image data based on conventional image processing means, for example, may perform sharpness adjustment, white balance, etc. on the initial image data to obtain the first image data. After the processed first image data is obtained, the shooting scene of the first image data is determined by further detecting the shooting scene of the first image data based on the main control chip.

Specifically, the contrast and brightness of the first image data may be detected based on a Camera hardware abstraction module (Camera Hardware Abstraction Layer, camera HAL) of the main control chip, and a judgment logic for judging whether a shooting scene of the first image data is a preset scene may be set in advance in the main control chip, for example, a relevant threshold judgment condition of the contrast and brightness may be set in advance, after the Camera HAL detects the contrast and brightness of the first image data, a detected value of the contrast and brightness is compared with the threshold judgment condition, and if the threshold judgment condition is met, it is determined that the first image data is in the preset scene; accordingly, under the condition that the threshold judgment condition is not met, determining that the first image data is in other scenes outside the preset scene.

When the first image data is in the preset scene, that is, when the shooting environment is poor, the processing such as brightness enhancement, color enhancement, contrast enhancement and the like can be performed on the first image data based on the image enhancement chip, so that the quality of an image output by a camera of the electronic device is improved.

In addition, in the case where the first image data is located in another scene other than the preset scene, that is, in the case where the photographing environment is good, at this time, since the photographing environment is good, the quality of the initial image data obtained by the image sensor is generally high, and after the initial image data is processed by the main control chip, the quality of the obtained image is further improved, at this time, since the image quality of the first image data is already high, even if the first image data is further processed based on the image processing module, it is difficult to improve the output image quality, and therefore, in this case, the first image data can be directly output, so that the time of image processing can be reduced, the real-time of image output can be improved, and at the same time, the power consumption of the electronic device can be reduced.

In this embodiment, after the primary image data is processed by the main control chip to obtain the first image data, the shooting scene of the first image data is further detected based on the main control chip, and in the case that the shooting scene of the first image data is detected to be the preset scene, the first image data is further enhanced based on the image enhancement chip to obtain the second image data, so that the quality of the image output by the electronic device can be improved.

Optionally, the transmitting the first image data to an image enhancement chip based on the main control chip includes:

transmitting the first image data to a camera hardware abstraction module of the main control chip based on the image processing module;

transmitting the first image data to a camera application module of the main control chip based on the camera hardware abstraction module;

drawing a user interface corresponding to the first image data based on the camera application module, and transmitting the first image data and the user interface to the image enhancement chip based on the camera application module;

the processing the first image data based on the image enhancement chip to obtain processed second image data includes:

And processing the first image data and the user interface based on the image enhancement chip to obtain the second image data.

Specifically, after the image processing module processes the first image data, the first image data may be stored in a DRAM, and the camera hardware abstraction module acquires the first image data from the DRAM, so as to implement the transfer of the first image data to the camera hardware abstraction module of the main control chip.

The Camera hardware abstraction module may sequentially transmit the first image data to the Camera application module (Camera Application, APP) through a Camera Service (Camera Service) and a Camera frame (Camera frame), and since a user may generally adjust a shot preview image based on a user interface during a shot preview process, the Camera application module may draw a user interface corresponding to the first image data, so as to facilitate subsequent synthesis of the first image data and the user interface and then display the first image data and the user interface.

Optionally, the processing the first image data and the user interface based on the image enhancement chip to obtain the second image data includes:

Processing the first image data based on the image enhancement chip to obtain target preview image data;

and synthesizing the target preview image data and the user interface based on the image enhancement chip to obtain the second image data.

Specifically, the processing such as brightness enhancement, color enhancement, contrast enhancement and the like can be performed on the first image data based on the image enhancement chip, so as to improve the quality of the preview image, and after the target preview image data is obtained, the image enhancement chip can directly synthesize the user interface with the target preview image data, so as to obtain the preview image finally displayed to the user.

Optionally, after the synthesizing the target preview image data and the user interface based on the image enhancement chip to obtain the second image data, the method further includes:

transmitting the second image data to a display screen based on the image enhancement chip;

and displaying the second image data based on the display screen.

In this embodiment, after the second image data is synthesized in the image enhancement chip, the second image data may be directly transmitted to the display screen through the image enhancement chip, and the display screen displays the second image data. Therefore, the encapsulator (Direct Rendering Manager Wrapper, DRM Wrapper) of the manager is not required to be directly rendered through the display system, so that the real-time performance of image display is improved, and the method is applicable to image processing processes of scenes with high requirements on the real-time performance of images, such as visual access control.

Optionally, after the processing the initial preview image data based on the image enhancement chip to obtain the target preview image data, the method further includes:

transmitting the target preview image data to the camera hardware abstraction module based on the image enhancement chip;

processing the target preview image data based on the camera hardware abstraction module to obtain an initial video, and transmitting the initial video to a video encoder based on the camera hardware abstraction module;

and encoding the initial video based on the video encoder to obtain a target video.

In a scene that the image processing method is applied to a monitoring system, the image enhancement chip synthesizes the target preview image data with the user interface after processing the target preview image data so as to output a real-time monitoring picture. The target preview image data can be transmitted to a Video encoder (Video Codec) for encoding and storage, so that the history monitoring record can be conveniently checked by the following related personnel.

Referring to fig. 3, a schematic structural diagram of an image processing system according to an embodiment of the present application is provided, and in order to further explain an image processing method according to an embodiment of the present application, a specific implementation process of the embodiment of the present application is further explained with reference to fig. 3, where the image processing method includes the following steps:

The image sensor collects initial image data, the initial image data is transmitted to an image processing module (ISP) based on a port CSI0 (Camera Serial Interface) of the image processing module ISP, the ISP processes the initial image data to obtain first image data, ISP hardware is controlled by a Driver (ISP Driver), the first image data is stored in a DRAM, a Camera hardware abstract module (Camera HAL) acquires the first image data from the DRAM, processes such as adding a filter and a template pattern to the first image data to obtain initial preview image data, then the Camera HAL sequentially transmits the initial preview image data to a Camera application module (Camera APP) through a Camera Service module (Camera Service) and a Camera frame module (Camera frame work), the Camera application module draws a User Interface (UI) corresponding to the initial preview image data, and transmits the initial preview image data and the UI to a display frame module (Surface filter), and the Camera filter can separate the initial preview image data from the initial preview image data to facilitate the subsequent preview image data to be processed independently. After the initial preview image data and the UI are separated by the Surface Flinger, the initial preview image data and the UI are sequentially transferred to a display processor (Display Processing Unit, DPU) through a display hardware synthesizer (Hardware Composer) and a display direct rendering manager (Direct Rendering Manager, DRM), the DPU transfers the UI to a receiving end RX0 of the image enhancement chip through an interface DSI0, and simultaneously, the DPU transfers the initial preview image data to a receiving end RX1 of the image enhancement chip through a DSI1 port, the image enhancement chip acquires the initial preview image data received by the receiving end RX1, and acquires the UI received by the receiving end RX0, then processes the initial preview image data to obtain target preview image data, synthesizes the target preview image data with the UI to obtain the second image data, and transmits the second image data to a display screen to display. Meanwhile, the image enhancement chip sends target preview image data to a CSI 1 port of the ISP through a sending end TX1, the ISP stores the target preview image data to a DRAM through a first driving module (ISP Driver), a Camera hardware abstraction module (Camera HAL) acquires the target preview image data from the DRAM, the Camera HAL sequentially transmits the target preview image data to a Camera application module (Camera APP) through a Camera Service (Camera Service) and a Camera frame (Camera frame), and the Camera APP sequentially transmits the target preview image data to a Video encoder (Video Codec) through a Media Codec (Media Codec), an open multimedia acceleration module (Open Media Acceleration, OMX) and a second driving module (Video Codec Driver) and encodes the target preview image data to obtain a target Video. Thereby completing the image processing process. In addition, the Video encoder (Video Codec) described above may also be used to encode the image to obtain the target image.

transmitting the first image data to an image enhancement chip based on a camera hardware abstraction module of the main control chip;

the method further comprises the steps of after the first image data are processed based on the image enhancement chip to obtain processed second image data:

transmitting the second image data to the camera hardware abstraction module based on the image enhancement chip;

processing the second image data based on the camera hardware abstraction module to obtain stored image data and initial preview image data;

transmitting the stored image data and the initial preview image data to a camera application module of the main control chip based on the camera hardware abstraction module;

transmitting the stored image data to a video encoder based on the camera application module, drawing a user interface corresponding to the initial preview image data based on the camera application module, and transmitting the initial preview image data and the user interface to a display frame based on the camera application module;

Synthesizing the initial preview image data and the user interface based on the display frame to obtain a third image, and transmitting the third image to a display screen based on the display frame;

and displaying the third image based on the display screen.

Specifically, referring to fig. 4, after processing the initial image data to obtain processed first image data, the image processing module may transfer the first image data to a Camera hardware abstraction module (Camera HAL) in the main control chip. The camera hardware abstraction module may be connected to the display system direct rendering manager (Direct Rendering Manager, DRM) through an encapsulator (Direct Rendering Manager Wrapper, DRM Wrapper) of the display system direct rendering manager such that the camera hardware abstraction module may pass the first image data to the DRM through the DRM Wrapper. After the DRM receives the first image data, the DRM transmits the first image data to a display processor (Display Processing Unit, DPU), the DPU transmits the first image data to a receiving end RX1 of an image enhancement chip through a DSI1 port, and the image enhancement chip performs brightness enhancement, color enhancement, contrast enhancement and the like on the first image data and determines the processed image as second image data. Because the second image data does not include a user interface, the second image data cannot be directly transmitted to the display screen for display, the image enhancement chip can transmit the second image data to a CSI 1 port of the ISP through the TXI port, the ISP can transmit the second image data to a Camera HAL through the first driving module, the Camera HAL can copy the second image data, so as to obtain two second image data, the two second image data are respectively determined to be the stored image data and the initial preview image data, and the Camera HAL sequentially transmits the initial video and the initial preview image data to a Camera application module (Camera APP) through a Camera Service (Camera Service) and a Camera frame (Camera frame). And the Camera APP transmits the stored image data to a Video Codec, and the Video Codec encodes the initial Video to obtain a target Video. Meanwhile, a Camera APP draws a User Interface (UI) corresponding to the initial preview image data, and the initial preview image data and the UI are sent to a display framework (surface eFlinger) based on the Camera APP; and synthesizing the initial preview image data and the UI based on the SurfaceFlinger to obtain a third image, transmitting the third image to a display screen based on the display frame, and displaying the third image based on the display screen so as to finish an image processing process.

Optionally, after the detecting, based on the main control chip, the shooting scene of the first image data, the method further includes:

transmitting the first image data to a display screen based on the image enhancement chip under the condition that the shooting scene is a scene other than the preset scene;

and displaying the first image data based on the display screen.

Wherein, based on the main control chip, the first image data is transmitted to an image enhancement chip, including:

processing the first image data based on the main control chip to obtain stored image data and initial preview image data under the condition that the shooting scene is a scene other than the preset scene,

And synthesizing the initial preview image data and the user interface based on the display frame to obtain a fourth image, and transmitting the fourth image to an image enhancement chip based on the display frame.

Accordingly, the transmitting the first image data to the display screen based on the image enhancement chip includes:

the fourth image is transmitted to a display screen based on the image enhancement chip;

the displaying the first image data based on the display screen includes:

and displaying the fourth image based on the display screen.

Specifically, the scene of the first image data may be detected and judged based on a Camera HAL of the main control chip, and when the Camera HAL determines that the first image data is a scene other than the preset scene, the first image data may be copied, so as to obtain two first image data, and the two first image data are respectively determined as storage image data and initial preview image data, and the storage image data and the initial preview image data are sequentially transferred to a Camera application module (Camera APP) through a Camera Service (Camera Service) and a Camera frame (Camera frame). And the Camera APP transmits the initial Video to a Video Codec, and the Video Codec encodes the stored image data to obtain a target image. Meanwhile, a Camera APP draws a User Interface (UI) corresponding to the initial preview image data, and the initial preview image data and the UI are sent to a display framework (surface eFlinger) based on the Camera APP; and synthesizing the initial preview image data and the UI based on the Surface Flinger to obtain a fourth image, transmitting the fourth image to a display screen based on the display frame, and displaying the fourth image based on the display screen so as to finish an image processing process.

Please refer to fig. 3, which is an image processing system provided in an embodiment of the present application, including an image sensor, a main control chip and an image enhancement chip, wherein the main control chip includes an image processing module (ISP), the image sensor and the image enhancement chip are respectively electrically connected to the main control chip, and the image sensor is electrically connected to a receiving end CSI 0 of the image processing module.

Optionally, the main control chip further includes a Camera hardware abstraction module (Camera HAL), a first Driver module (ISP Driver) and a Camera application module (Camera APP), where the image processing module, the first Driver module, the Camera hardware abstraction module and the Camera application module are sequentially connected, and the Camera hardware abstraction module is sequentially connected with the Camera application module through a Camera Service module (Camera Service) and a Camera frame module (Camera frame).

The image processing module is used for transmitting the first image data to the camera hardware abstraction module based on the first driving module;

the camera hardware abstraction module is used for transmitting the first image data to the camera application module;

the camera application module is used for drawing a user interface corresponding to the first image data and transmitting the first image data and the user interface to the image enhancement chip, wherein the camera application module is connected with the image enhancement chip through a display frame module (Surface Flinger), a display hardware synthesizer (Hardware Composer), a display direct rendering manager (Direct Rendering Manager, DRM) and a display processor (Display Processing Unit, DPU) in sequence, wherein a DSI0 port of the display processor is electrically connected with a receiving end RX0 of the image enhancement chip, and a DSI1 port of the display processor is electrically connected with a receiving end RX1 of the image enhancement chip. The transmitting end TX1 of the image enhancement chip is electrically connected with the CSI 1 port of the ISP;

The image enhancement chip is used for processing the first image data and the user interface to obtain the second image data.

Optionally, the image enhancement chip is specifically configured to process the first image data based on the image enhancement chip to obtain target preview image data, and is configured to synthesize the target preview image data and the user interface to obtain the second image data.

Optionally, the system further comprises a display screen, and the image enhancement chip is electrically connected with the display screen;

the image enhancement chip is further used for transmitting the second image data to a display screen;

the display screen is used for displaying the second image data based on the display screen.

Optionally, the system further comprises a Media Codec (Media Codec), an open multimedia acceleration module (OMX), a second driving module (Video Codec Driver) and an image encoder (Video Codec), and the camera application module is connected with the image encoder through the Media Codec, the open multimedia acceleration module and the second driving module in sequence;

the image enhancement chip is further used for transmitting the target preview image data to the camera hardware abstraction module through the image processing module and the first driving module in sequence;

The camera hardware abstraction module is further configured to transmit the target preview image data to the image encoder sequentially through the camera application module, the media codec, the open multimedia acceleration module, and the second driving module;

the image encoder is used for encoding the target preview image data to obtain target image data.

The image processing system provided in this embodiment can implement each process in the foregoing method embodiment, and can implement the same beneficial effects, and in order to avoid repetition, a description is omitted here.

Referring to fig. 4, another image processing system provided in an embodiment of the present application includes: the image sensor, main control chip and image enhancement chip, main control chip includes image processing module (ISP), image sensor with image enhancement chip respectively with main control chip electricity is connected, wherein, image sensor with image processing module's receiving terminal CSI 0 electricity is connected.

Optionally, the main control chip further comprises a Camera hardware abstraction module (Camera HAL) and a first Driver module (ISP Driver), and the image processing module is connected with the Camera hardware abstraction module through the first Driver module;

the image enhancement chip is further configured to transmit the second image data to the camera hardware abstraction module sequentially through the image processing module and the first driving module.

Optionally, the main control chip further includes a Camera application module (Camera APP), a Media Codec (Media Codec), an open multimedia acceleration module (OMX), a second driving module (Video Codec Driver), an image encoder (Video Codec), a display frame module (Surface player), a display hardware synthesizer (Hardware Composer), a display direct rendering manager (Direct Rendering Manager, DRM), a display processor (Display Processing Unit, DPU), and a display screen, where the Camera hardware abstraction module, the Camera application module, the Media Codec, the open multimedia acceleration module, the second driving module, and the image encoder are sequentially connected, and the Camera application module, the display frame module, the display hardware synthesizer, the display direct rendering manager, the display processor, the image enhancement chip, and the display screen are sequentially electrically connected;

The Camera hardware abstract module is connected with the Camera application module through a Camera Service module (Camera Service) and a Camera frame module (Camera frame) in sequence. The DSI0 port of the display processor is electrically connected with the receiving end RX0 of the image enhancement chip, and the DSI1 port of the display processor is electrically connected with the receiving end RX1 of the image enhancement chip. The transmitting end TX1 of the image enhancement chip is electrically connected with the CSI 1 port of the ISP.

The camera hardware abstraction module is used for processing the second image data to obtain stored image data and initial preview image data, and transmitting the stored image data and the initial preview image data to the camera application module;

the camera application module is used for transmitting the stored image data to the image encoder through the media codec, the open multimedia acceleration module and the second driving module in sequence, drawing a user interface corresponding to the initial preview image data and transmitting the initial preview image data and the user interface to the display frame module;

the display framework module is used for synthesizing the initial preview image data and the user interface to obtain a third image, and transmitting the third image to the display screen through the display hardware synthesizer, the display direct rendering manager, the display processor and the image enhancement chip in sequence;

And the display screen is used for displaying the third image.

Optionally, the system further comprises a encapsulator (Direct Rendering Manager Wrapper, DRM Wrapper) of a display system direct rendering manager, the camera hardware abstraction module being connected to the display direct rendering manager through the encapsulator of the display system direct rendering manager;

the camera hardware abstraction module is used for driving the display direct rendering manager based on a wrapper of the display system direct rendering manager and transmitting the first image data to the display processor based on the display direct rendering manager;

the display processor is used for transmitting the first image data to the image enhancement chip.

Referring to fig. 4, after processing the initial image data to obtain processed first image data, the image processing module may transmit the first image data to a Camera hardware abstraction module (Camera HAL) in the main control chip. The camera hardware abstraction module may be connected to the display direct rendering manager (Direct Rendering Manager, DRM) through a Wrapper (Direct Rendering Manager Wrapper, DRM Wrapper) of the display system direct rendering manager, such that the camera hardware abstraction module may drive the DRM through the DRM Wrapper and then pass the first image data to the DRM. After receiving the first image data, the DRM transmits the first image data to a display processor (Display Processing Unit, DPU), and the DPU transmits the first image data to a receiving terminal RX1 of the image enhancement chip through a DSI1 port.

Optionally, the image enhancement chip is configured to transmit the first image data to the display screen when the shooting scene is a scene other than the preset scene;

the display screen is used for displaying the first image data.

An embodiment of the present application provides an image processing apparatus, including the image processing system described in the foregoing embodiment.

The image processing device provided in this embodiment, because of including the image processing system, can implement each process of the image processing system, and has the same beneficial effects, and in order to avoid repetition, will not be described herein again.

Referring to fig. 5, a flowchart of an image processing method according to another embodiment of the present application is provided, where the method is specifically applied to an image preview or photographing process, and the method includes the following steps: acquiring initial image data; processing the initial image data based on the main control chip to obtain first image data; detecting a shooting scene of the first image data; judging whether the shooting scene is a preset scene, starting an image enhancement chip under the condition that the shooting scene is the preset scene, performing image preview based on the image enhancement chip after the successful starting state of the image enhancement chip is normal, and executing a normal preview flow under the condition that the shooting scene is other scenes except the preset scene or before the successful starting of the image enhancement chip or when the state is wrong.

The specific process based on the image enhancement chip preview comprises the following steps: and inputting the first image data into an image enhancement chip for enhancement processing, then combining the enhanced first image data with the UI to obtain preview image data, and transmitting the preview image data to a display screen for preview display. Accordingly, the normal preview process includes: and synthesizing the first image data with the UI to obtain preview image data, and transmitting the preview image data to a display screen for preview display.

According to the image processing method provided by the embodiment of the application, the execution subject can be an image processing device. In the embodiments of the present application, a method for performing image processing by an image processing apparatus is taken as an example, and an apparatus of the image processing method provided in the embodiments of the present application is described.

Referring to fig. 6, a schematic structural diagram of an image processing apparatus 600 according to an embodiment of the present application is provided, where the image processing apparatus 600 includes:

The acquisition module 601 is configured to receive initial image data acquired by the image sensor based on the main control chip;

the first processing module 602 is configured to process the initial image data based on an image processing module in the main control chip, to obtain processed first image data;

a second processing module 603, configured to detect a shooting scene of the first image data based on the main control chip, and transmit the first image data to an image enhancement chip based on the main control chip;

and a third processing module 604, configured to process the first image data based on the image enhancement chip to obtain processed second image data when the shooting scene is a preset scene.

Optionally, the second processing module 603 includes:

the first transmission sub-module is used for transmitting the first image data to the camera hardware abstraction module of the main control chip based on the image processing module;

the first processing sub-module is used for transmitting the first image data to a camera application module of the main control chip based on the camera hardware abstraction module;

a second processing sub-module, configured to draw a user interface corresponding to the first image data based on the camera application module, and transmit the first image data and the user interface to the image enhancement chip based on the camera application module;

The third processing module 604 is specifically configured to process the first image data and the user interface based on the image enhancement chip, so as to obtain the second image data.

Optionally, the third processing module 604 includes:

the third processing sub-module is used for processing the initial preview image data based on the image enhancement chip to obtain target preview image data;

and the synthesis submodule is used for synthesizing the target preview image data and the user interface based on the image enhancement chip to obtain the second image data.

Optionally, the apparatus further comprises:

the first transmission module is used for transmitting the second image data to a display screen based on the image enhancement chip;

and the display module is used for displaying the second image data based on the display screen.

Optionally, the apparatus further comprises:

the second transmission module is used for transmitting the target preview image data to the camera hardware abstraction module based on the image enhancement chip;

a fourth processing module for transmitting the target preview image data to an image encoder based on the camera hardware abstraction module;

And the encoding module is used for encoding the target preview image data based on the image encoder to obtain target image data.

Optionally, the second processing module 603 is specifically configured to transmit the first image data to an image enhancement chip based on a camera hardware abstraction module of the main control chip;

the apparatus further comprises:

a third transmission module for transmitting the second image data to the camera hardware abstraction module based on the image enhancement chip;

a fifth processing module, configured to process the second image data based on the camera hardware abstraction module, to obtain stored image data and initial preview image data;

the fourth transmission module is used for respectively transmitting the stored image data and the initial preview image data to the camera application module of the main control chip based on the camera hardware abstraction module;

a sixth processing module for transmitting the stored image data to an image encoder based on the camera application module, and drawing a user interface corresponding to the initial preview image data based on the camera application module and transmitting the initial preview image data and the user interface to a display frame based on the camera application module;

The synthesizing module is used for synthesizing the initial preview image data and the user interface based on the display frame to obtain a third image, and transmitting the third image to a display screen based on the display frame;

and the display module is used for displaying the third image based on the display screen.

Optionally, the second processing module 603 is configured to drive the display direct rendering manager by the camera hardware abstraction module based on a wrapper of the display system direct rendering manager, and transmit the first image data to a display processor based on the display direct rendering manager;

the second processing module 603 is configured to transmit the first image data to the image enhancement chip based on the display processor.

Optionally, the apparatus further comprises:

a fifth transmission module, configured to transmit the first image data to a display screen based on the image enhancement chip when the shooting scene is a scene other than the preset scene;

and the display module is used for displaying the first image data based on the display screen.

In this embodiment, after the primary image data is processed by the main control chip to obtain the first image data, the shooting scene of the first image data is further detected based on the main control chip, and in the case that the shooting scene of the first image data is detected to be the preset scene, the first image data is further enhanced based on the image enhancement chip to obtain the second image data, so that the quality of the image output by the camera in the electronic device can be improved.

The image processing apparatus 600 in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The image processing apparatus 600 in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The image processing apparatus 600 provided in this embodiment of the present application can implement each process implemented by the method embodiments of fig. 1 to 5, and in order to avoid repetition, a description is omitted here.

Optionally, as shown in fig. 7, the embodiment of the present application further provides an electronic device 700, including a processor 701, a memory 702, and a program or an instruction stored in the memory 702 and capable of running on the processor 701, where the program or the instruction implements each process of the embodiment of the image processing method when executed by the processor 701, and the process can achieve the same technical effect, and for avoiding repetition, a description is omitted herein.

It should be noted that, the interface display device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 8 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 800 includes, but is not limited to: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

The input unit 804 is configured to receive initial image data acquired by the image sensor based on the main control chip;

The processor 810 is configured to process the initial image data based on an image processing module in the main control chip, to obtain processed first image data;

the processor 810 is configured to detect a shooting scene of the first image data based on the main control chip, and transmit the first image data to an image enhancement chip based on the main control chip;

the processor 810 is configured to process the first image data based on the image enhancement chip to obtain processed second image data when the shooting scene is a preset scene.

Optionally, the processor 810 is configured to transfer the first image data to a camera hardware abstraction module of the main control chip based on the image processing module;

the processor 810 is configured to transfer the first image data to a camera application module of the main control chip;

the processor 810 is configured to draw a user interface corresponding to the first image data based on the camera application module, and transmit the first image data and the user interface to the image enhancement chip based on the camera application module;

The processor 810 is configured to process the first image data and the user interface based on the image enhancement chip to obtain the second image data.

Optionally, the processor 810 is configured to process the initial preview image data based on the image enhancement chip to obtain target preview image data;

the processor 810 is configured to synthesize the target preview image data and the user interface based on the image enhancement chip to obtain the second image data.

Optionally, the processor 810 is configured to transmit the second image data to a display screen based on the image enhancement chip;

the display unit 806 is configured to display the second image data based on the display screen.

Optionally, the processor 810 is configured to transmit the target preview image data to the camera hardware abstraction module based on the image-enhancement chip;

the processor 810 is configured to send the target preview image data to an image encoder based on the camera hardware abstraction module;

the processor 810 is configured to encode the target preview image data to obtain target image data based on the image encoder.

Optionally, the processor 810 is configured to drive the display direct rendering manager based on an encapsulator of the display system direct rendering manager by the camera hardware abstraction module, and transmit the first image data to a display processor based on the display direct rendering manager;

the processor 810 is configured to transmit the first image data to the image enhancement chip based on the display processor.

Optionally, the processor 810 is configured to transmit the first image data to an image enhancement chip based on a camera hardware abstraction module of the main control chip;

the processor 810 is configured to transmit the second image data to the camera hardware abstraction module based on the image-enhancement chip;

the processor 810 is configured to process the second image data based on the camera hardware abstraction module to obtain stored image data and initial preview image data;

the processor 810 is configured to transmit the stored image data and the initial preview image data to a camera application module of the main control chip based on the camera hardware abstraction module, respectively;

the processor 810 is configured to transmit the stored image data to an image encoder based on the camera application module, and draw a user interface corresponding to the initial preview image data based on the camera application module and transmit the initial preview image data and the user interface to a display frame based on the camera application module;

The processor 810 is configured to synthesize the initial preview image data and the user interface based on the display frame to obtain a third image, and transmit the third image to a display screen based on the display frame;

the display unit 806 is configured to display the third image based on the display screen.

Optionally, the processor 810 is configured to transmit the first image data to a display screen based on the image enhancement chip when the shooting scene is a scene other than the preset scene;

the display unit 806 is configured to display the first image data based on the display screen.

Those skilled in the art will appreciate that the electronic device 800 may also include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 810 by a power management system to perform functions such as managing charge, discharge, and power consumption by the power management system. The electronic device structure shown in fig. 8 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.

It should be appreciated that in embodiments of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, with the graphics processor 8041 processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. Touch panel 8071, also referred to as a touch screen. The touch panel 8071 may include two parts, a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.

The memory 809 can be used to store software programs as well as various data. The memory 809 may mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 809 may include volatile memory or nonvolatile memory, or the memory 809 may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (ddr SDRAM), enhanced SDRAM (Enhanced SDRAM), synchronous DRAM (SLDRAM), and Direct RAM (DRRAM). Memory 809 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

The processor 810 may include one or more processing units; optionally, the processor 810 integrates an application processor that primarily processes operations involving an operating system, user interface, application programs, etc., and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 810.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the embodiment of the image processing method, and the same technical effects can be achieved, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is used for running a program or an instruction, so as to implement each process of the embodiment of the image processing method, and achieve the same technical effect, so that repetition is avoided, and no redundant description is provided here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

The embodiments of the present application provide a computer program product stored in a storage medium, where the program product is executed by at least one processor to implement the respective processes of the embodiments of the image processing method described above, and achieve the same technical effects, and are not repeated herein.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Claims

1. An image processing method, comprising:

transmitting the first image data to an image enhancement chip based on the main control chip;

2. The method of claim 1, wherein the transmitting the first image data to an image enhancement chip based on the master chip comprises:

3. The method of claim 2, wherein the processing the first image data and the user interface based on the image enhancement chip to obtain the second image data comprises:

4. The method of claim 3, wherein the synthesizing the target preview image data and the user interface based on the image enhancement chip results in the second image data, the method further comprising:

and displaying the second image data based on the display screen.

5. The method of claim 3, wherein after processing the first image data based on the image enhancement chip to obtain target preview image data, the method further comprises:

transmitting the target preview image data to an image encoder based on the camera hardware abstraction module;

and encoding the target preview image data based on the image encoder to obtain target image data.

6. The method of claim 1, wherein the transmitting the first image data to an image enhancement chip based on the master chip comprises:

and transmitting the second image data to the camera hardware abstraction module based on the image enhancement chip.

7. The method of claim 6, wherein after the transmitting the second image data to the camera hardware abstraction module based on the image-enhancement chip, the method further comprises:

transmitting the stored image data to an image encoder based on the camera application module, drawing a user interface corresponding to the initial preview image data based on the camera application module, and transmitting the initial preview image data and the user interface to a display frame based on the camera application module;

and displaying the third image based on the display screen.

8. The method of claim 6, wherein the camera hardware abstraction module based on the master chip transmits the first image data to an image-enhancement chip, comprising:

the camera hardware abstraction module drives the display direct rendering manager based on a wrapper of the display system direct rendering manager, and transmits the first image data to a display processor based on the display direct rendering manager;

The first image data is transmitted to the image enhancement chip based on the display processor.

9. The method according to claim 1, wherein after the detecting of the shooting scene of the first image data based on the main control chip, the method further comprises:

and displaying the first image data based on the display screen.

10. An image processing system, comprising: the image sensor, the main control chip and the image enhancement chip are respectively and electrically connected with the main control chip;

11. The system of claim 10, wherein the master control chip further comprises a camera hardware abstraction module, a first driver module, and a camera application module, the image processing module, the first driver module, the camera hardware abstraction module, and the camera application module being connected in sequence;

the camera application module is used for drawing a user interface corresponding to the first image data and transmitting the first image data and the user interface to the image enhancement chip;

12. The system of claim 11, wherein the image enhancement chip is specifically configured to process the first image data based on the image enhancement chip to obtain target preview image data, and to synthesize the target preview image data with the user interface to obtain the second image data.

13. The system of claim 12, further comprising a display screen, wherein the image intensifier chip is electrically connected to the display screen;

14. The system of claim 12, further comprising a media codec, an open multimedia acceleration module, a second drive module, and an image encoder, the camera application module being connected to the image encoder sequentially through the media codec, the open multimedia acceleration module, the second drive module;

15. The system of claim 10, wherein the master control chip further comprises a camera hardware abstraction module and a first driver module, the image processing module being coupled to the camera hardware abstraction module through the first driver module;

16. The system of claim 15, wherein the master control chip further comprises a camera application module, a media codec, an open multimedia acceleration module, a second driver module, an image encoder, a display framework module, a display hardware synthesizer, a display direct rendering manager, a display processor, a display screen, the camera hardware abstraction module, the camera application module, the media codec, the open multimedia acceleration module, the second driver module, and the image encoder are connected in sequence, the camera application module, the display framework module, the display hardware synthesizer, the display direct rendering manager, the display processor, the image enhancement chip, and the display screen are connected in sequence;

and the display screen is used for displaying the third image.

17. The system of claim 16, further comprising an encapsulator of a display system direct rendering manager, the camera hardware abstraction module being coupled to the display direct rendering manager through the encapsulator of the display system direct rendering manager;

18. The system according to claim 16, wherein the image enhancement chip is configured to transmit the first image data to the display screen in a case where the shooting scene is a scene other than the preset scene;

the display screen is used for displaying the first image data.

19. An image processing apparatus comprising the image processing system of any one of claims 10 to 18.

20. An image processing apparatus, comprising:

21. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the image processing method of any one of claims 1 to 9.

22. A readable storage medium, characterized in that the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps of the image processing method according to any of claims 1 to 9.

23. A chip comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, the processor being for running a program or instructions to implement the steps of the image processing method of any of claims 1 to 9.