CN112449085A - Image processing method and device, electronic equipment and readable storage medium - Google Patents
Image processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112449085A CN112449085A CN201910817264.9A CN201910817264A CN112449085A CN 112449085 A CN112449085 A CN 112449085A CN 201910817264 A CN201910817264 A CN 201910817264A CN 112449085 A CN112449085 A CN 112449085A
- Authority
- CN
- China
- Prior art keywords
- image
- image quality
- model
- quality compensation
- display screen
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
- H04N7/144—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
Abstract
The disclosure relates to an image processing method and device, an electronic device and a readable storage medium. An image processing method is suitable for an electronic device provided with a camera module under a display screen, and comprises the following steps: acquiring original image data output by a sensor in the camera module; responding to the processed original image data to reach a designated processing stage, and performing image quality compensation on the image data of the designated processing stage by adopting a preset image quality compensation model; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen; and continuously processing the image data after the picture quality compensation until the target image data is obtained. According to the embodiment, the image quality compensation is carried out on the image data at the appointed processing stage, for example, the contrast and/or the definition are/is carried out, and the influence of the display screen on the image shot by the camera module is eliminated or slowed down.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a readable storage medium.
Background
At present, more and more electronic equipment adopts comprehensive display screen, and the module setting of making a video recording is in the below of comprehensive display screen. Because there is the pixel circuit of metal material in the display screen, not only can reduce the transmissivity, still can scatter and diffract the light of part light zone that passes through, can reduce the contrast and the definition of the module of making a video recording image like this.
In order to solve the above problems, the related art is implemented by a camera module lifting mode, such as a module lifting mechanism, a sliding mechanism, a lateral rotation lifting mechanism, and the like, and is implemented by digging a hole on a display screen. However, the module lifting manner increases the complexity of the internal structure of the electronic device, and damages its integrity, resulting in higher cost; the hole digging method cannot achieve 100% of the full display screen.
Disclosure of Invention
The present disclosure provides an image processing method and apparatus, an electronic device, and a readable storage medium to solve the deficiencies of the related art.
According to a first aspect of the embodiments of the present disclosure, there is provided an image processing method suitable for an electronic device in which a camera module is disposed below a display screen, the camera module including an image sensor for receiving light passing through the display screen and generating raw image data, the method including:
acquiring raw image data output by the image sensor;
responding to the processed original image data to reach a designated processing stage, and performing image quality compensation on the image data of the designated processing stage by adopting a preset image quality compensation model; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen;
and continuously processing the image data after the picture quality compensation until the target image data is obtained.
Alternatively,
the designated processing stages include at least one of the following image processing stages:
RAW domain processing before an ISP stage of image signal processing;
an ISP stage of image signal processing;
YUV domain processing after the image signal processing ISP stage.
Optionally, the image quality compensation model includes at least one of: diffraction models, deblurring models, and scattering models.
Optionally, the image quality compensation model is obtained by:
inputting an actual image into the image quality compensation model, and performing image quality compensation on the actual image by using the image quality compensation model; the actual image is obtained in a scene that the display screen is arranged on the camera module;
comparing the actual image after image quality compensation with a preset alignment image; the target image is obtained in a scene that the display screen is not arranged on the camera module;
and if the difference value of the image quality damage amount of the actual image after image quality compensation and the targeted image exceeds a set difference threshold value, adjusting the parameters of the image quality compensation model and then performing image quality compensation on the actual image again until the difference value of the image quality damage amount of the actual image after image quality compensation and the targeted image is less than the set difference threshold value.
Optionally, the image quality compensation model is obtained by:
acquiring a first optical model and a second optical model which are arranged in front of and behind the display screen on the camera module;
determining an optical offset model representing the influence of the display screen on light according to the first optical model and the second optical model;
and training the image quality compensation model by using the optical offset model so as to enable the difference value of the image quality damage quantity of the images acquired before and after the setting of the display screen to be smaller than a set difference threshold value.
Optionally, the image quality compensation model is obtained by:
acquiring a second optical model on the camera module after the display screen is arranged;
acquiring actual values of all parameters in the second optical model and corresponding calibration ranges of the actual values;
and comparing the actual value with the calibration range, adjusting the actual value of the parameter exceeding the calibration range to the calibration range, wherein the second optical model for adjusting the actual value of the parameter is the image quality compensation model.
According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus suitable for an electronic device in which a camera module is disposed below a display screen, the camera module including an image sensor for receiving light passing through the display screen and generating raw image data, the image processing apparatus including:
the original image acquisition module is used for acquiring original image data output by the image sensor;
the image quality compensation module is used for responding to the situation that the processed original image data reaches a specified processing stage, and performing image quality compensation on the image data of the specified processing stage by adopting a preset image quality compensation model; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen;
and the target image acquisition module is used for continuously processing the image data after the picture quality compensation until the target image data is obtained.
Alternatively,
the designated processing stages include at least one of the following image processing stages:
RAW domain processing before an ISP stage of image signal processing;
an ISP stage of image signal processing;
YUV domain processing after the image signal processing ISP stage.
Optionally, the image quality compensation model includes at least one of: diffraction models, deblurring models, and scattering models.
Optionally, the apparatus further includes a compensation model obtaining module, where the compensation model obtaining module includes:
an image quality compensation unit for inputting an actual image into the image quality compensation model and performing image quality compensation on the actual image by the image quality compensation model; the actual image is obtained in a scene that the display screen is arranged on the camera module;
the image comparison unit is used for comparing the actual image after the image quality compensation with a preset alignment image; the target image is obtained in a scene that the display screen is not arranged on the camera module;
and a parameter adjusting unit, configured to adjust parameters of the image quality compensation model and then perform image quality compensation on the actual image again until a difference between the image quality damage amounts of the actual image after image quality compensation and the target image is smaller than a set difference threshold value, when the difference between the image quality damage amounts of the actual image after image quality compensation and the target image exceeds the set difference threshold value.
Optionally, the apparatus further includes a compensation model obtaining module, where the compensation model obtaining module includes:
the optical model acquisition unit is used for acquiring a first optical model and a second optical model which are arranged on the camera module and in front of and behind the display screen;
the offset model acquisition unit is used for determining an optical offset model representing the influence of the display screen on light according to the first optical model and the second optical model;
and the compensation model training unit is used for training the image quality compensation model by using the optical offset model so as to enable the difference value of the image quality damage quantity of the images acquired before and after the setting of the display screen to be smaller than a set difference threshold value.
Optionally, the apparatus further includes a compensation model obtaining module, where the compensation model obtaining module includes:
the optical model acquisition unit is used for acquiring a second optical model after the display screen is arranged on the camera module;
the parameter value acquisition unit is used for acquiring actual values of all parameters in the second optical model and corresponding calibration ranges of the actual values;
and the parameter value adjusting unit is used for comparing the actual value with the calibration range, adjusting the actual value of the parameter beyond the calibration range to the calibration range, and adjusting the second optical model of the actual value of the parameter, namely the image quality compensation model.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
a display screen;
the camera module is arranged below the display screen;
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to execute executable instructions in the memory to implement the steps of the method of any of the first aspects.
According to a fourth aspect of embodiments of the present disclosure, there is provided a readable storage medium having stored thereon executable instructions that, when executed by a processor, implement the steps of the method of any one of the first aspect.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
as can be seen from the foregoing embodiments, in the embodiment of the present disclosure, at a designated processing stage for processing original image data, image quality compensation is performed on image data at the designated processing stage by using a preset image quality compensation model, so that an amount of image quality damage of an image captured by the image capturing module by the display screen can be compensated, and target image data with satisfactory image quality can be obtained by continuously processing image data after image quality compensation. In this way, the present embodiment eliminates or reduces the influence of the display screen on the image captured by the camera module by performing image quality compensation, such as contrast and/or sharpness, on the image data at the designated processing stage.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1(a) is a schematic illustration of a display screen shown in accordance with an exemplary embodiment;
3 FIG. 31 3 ( 3 b 3) 3 is 3 a 3 cross 3- 3 sectional 3 view 3 in 3 the 3 direction 3 A 3- 3 A 3' 3 of 3 FIG. 31 3, 3 shown 3 in 3 accordance 3 with 3 an 3 exemplary 3 embodiment 3; 3
FIG. 2 is a flow diagram illustrating a method of image processing according to an exemplary embodiment;
FIG. 3 is a schematic diagram illustrating specified processing stages in accordance with an exemplary embodiment;
fig. 4 is a flow diagram illustrating an acquisition of an image quality compensation model according to an example embodiment;
fig. 5 is a flow diagram illustrating another method for obtaining an image quality compensation model according to an example embodiment;
fig. 6 is a flow diagram illustrating yet another method for obtaining an image quality compensation model according to an example embodiment;
fig. 7(a) is an effect diagram of a target image before image quality compensation;
FIG. 7(b) is an effect diagram of the target image after image quality compensation;
FIGS. 8 to 11 are block diagrams illustrating an image processing apparatus according to an exemplary embodiment;
FIG. 12 is a block diagram illustrating an electronic device in accordance with an example embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure as recited in the claims below.
At present, more and more electronic equipment adopts the full screen, and the module setting of making a video recording is in the below of full display screen. As shown in fig. 1(a), the display screen 10 includes a main display area 11 and a sub display area 12; the camera module 13 is disposed below the sub-display area 12, and the effect is as shown in fig. 1 (b). Referring to fig. 1(b), light passes through the display screen 10 and the lens 131 in the camera module 13 to reach the image sensor 132, and the image sensor 132 is configured to receive the light and generate raw image data.
Because there is the pixel circuit of metal material in the display screen, not only can reduce the transmissivity, still can scatter and diffract the light of part light zone that passes through, can reduce the contrast and the definition of the module of making a video recording image like this.
In order to solve the above problem, an embodiment of the present disclosure provides an image processing method, which is suitable for an electronic device in which a camera module is disposed below a display screen of the electronic device, where the camera module includes an image sensor, the image sensor is configured to receive light passing through the display screen and generate raw image data, the raw image data may be input to an image processing platform in the electronic device, and the image processing platform may be implemented by hardware and/or software, such as an AI image processing chip, which is not limited herein. FIG. 2 is a flow diagram illustrating an image processing method according to an exemplary embodiment. Referring to fig. 2, an image processing method includes steps 201 to 203, in which:
in step 201, raw image data output by an image sensor is acquired.
In this embodiment, the electronic device can control the camera module to shoot original image data, and output the original image data through the image sensor. The RAW image data refers to RAW image data, i.e., image data in RAW format. The electronic device may capture RAW image data in response to a user's capture operation, store to a local cache or memory, at which time the electronic device may read RAW image data from the local cache or memory. Of course, the electronic apparatus may directly acquire the original image data in response to a photographing operation by the user.
In step 202, in response to the processing of the original image data reaching a designated processing stage, performing image quality compensation on the image data of the designated processing stage by using a preset image quality compensation model; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen.
In this embodiment, the image processing platform may process the raw image data after acquiring the raw image data to obtain target image data in a set format, for example, the set format may be YUV format, JPEG, or the like. Taking the conversion into the YUV format as an example, referring to fig. 3, the image data in raw format output by the image sensor in the camera module is processed by the image processing platform 20 and then output to the image encoder for encoding. The image processing platform 20 includes an image signal processing ISP, so that the processing stage of the image data in RAW format in the image processing platform may include RAW domain processing before the image signal processing ISP stage, and YUV domain processing after the image signal processing ISP stage.
In combination with the characteristics of the image data output from each stage, in this embodiment, one or more stages of the multiple image processing stages in the image processing platform 20 may be used as the designated processing stage, including: RAW domain processing A before the ISP phase of image signal processing, image signal processing ISP phase C and YUV domain processing B after the ISP phase of image signal processing. That is, in this embodiment, the image quality compensation model may be preset in one or more of the RAW domain processing a before the image signal processing ISP stage, the image signal processing ISP stage C, and the YUV domain processing B after the image signal processing ISP stage, so as to achieve the effect of compensating the image quality of the captured image.
It should be noted that, in this embodiment, the pixel values in the image data may be compensated to achieve the effect of compensating the image quality deterioration amount. In an embodiment, the image quality compensation model may also compensate different parameters of the whole frame of image, such as brightness, noise filtering, white balance, color correction, saturation, hue, and the like of the image, and the effect of compensating the image quality of the image can be achieved.
In this embodiment, an image quality compensation model may be preset in the electronic device, which substantially eliminates or alleviates the influence of the display screen on the image quality damage amount in the captured image. The number of the image quality compensation models can be set according to specific scenes:
when the number is one, the image quality compensation model may be set in the phase a, the phase B, or the phase C.
When the number is plural, the image quality compensation models may be provided in two or three of the stages a, B, and C. In consideration of the possibility that the parameters of each processing stage may be different, the image quality compensation models provided in different stages may compensate the image quality deterioration amount of the map from different angles, for example, a partial image quality compensation model may compensate the variation amount of the pixel value, and another partial image quality compensation model may compensate the variation amount of the parameters of the image.
In this embodiment, the image quality compensation model may include at least one of the following: diffraction models, deblurring models, and scattering models. Taking the deblurring model as an example, a Gaussian filter (Gaussian filter), a Gaussian mixture filter or a wiener filter (Lucy-Richardson algorithm) may be included. Taking scattering models as examples, the scattering models can include a Lambert (Lambert) model, a bipartite graph perfect matching KM algorithm, a Rayleigh scattering Rayleigh algorithm, and the like. It should be noted that, a skilled person may select an appropriate image quality compensation model according to the influence of the display screen on the light entering the lens, such as scattering, diffraction, or blurring, and if the image quality compensation function can be realized, the corresponding solution falls within the scope of the present application.
In this embodiment, the image quality compensation model may be obtained by:
first, referring to fig. 4, the electronic device may obtain one or more frames of actual images, where the actual images are obtained in a scene where the camera module is disposed on the display screen, input the actual images into the image quality compensation model, and perform image quality compensation on the actual images by using the image quality compensation model, so as to obtain an image quality compensated actual image (corresponding to step 401 in fig. 4). Then, the electronic device may acquire a preset alignment image, which is acquired in a scene where the display screen is not disposed on the camera module, and compare the actual image after the image quality compensation with the alignment image (corresponding to step 402 in fig. 4). For example, the electronic device may extract the image quality deterioration amounts of the actual image and the target image, respectively, and then calculate a difference value between the image quality deterioration amounts of the actual image and the target image.
Then, the electronic device may obtain a preset difference threshold, determine a relationship between the difference between the quality-damaged amount and the set difference threshold, and if the difference between the quality-damaged amount of the compensated actual image and the quality-damaged amount of the target image exceeds the set difference threshold, adjust parameters of the image quality compensation model and then perform image quality compensation on the actual image again, that is, adjust parameters of the image quality compensation model and then perform steps 401 to 403 again until the difference between the quality-damaged amount of the compensated actual image and the quality-damaged amount of the target image is smaller than the set difference threshold (corresponding to step 403 in fig. 4).
In a second way, referring to fig. 5, the electronic device may obtain the first optical model and the second optical model before and after the display screen is set on the camera module (corresponding to step 501 in fig. 5). For example, a light source can be disposed above the camera module, and the direction of the light emitted by the light source is determined, so that an optical model of the lens plane in the camera module can be determined. The electronic device can then determine an optical shift model characterizing the effect of the display screen on the light based on the first optical model and the second optical model (corresponding to step 502 in FIG. 5). Then, the electronic device may train the image quality compensation model using the optical offset model so that a difference between image quality damages of the acquired images before and after the setting of the display screen is smaller than a set difference threshold (corresponding to step 503 in fig. 5).
In a third way, referring to fig. 6, the electronic device may obtain a second optical model after the display screen is disposed on the camera module (corresponding to step 601 in fig. 6). The electronic device may then obtain the actual values of the parameters in the second optical model and their corresponding calibration ranges (corresponding to step 602 in fig. 6). Then, the electronic device may compare the actual value with the calibration range, and adjust the actual value of the parameter exceeding the calibration range to the calibration range, where the second optical model for adjusting the actual value of the parameter is the image quality compensation model (corresponding to step 603 in fig. 6).
In the process of training the image quality compensation model by using the optical offset model, the structures of the display screen, such as the transparent area and the opaque area, may be simultaneously input into the image quality compensation model, so as to improve the compensation accuracy of the image quality compensation model.
In this embodiment, the image quality compensation model is trained to reversely simulate the influence of the display screen on the image quality of the captured image, that is, the image quality compensation model can compensate the image quality damage amount of the display screen on the image data, which is beneficial to improving the image quality of the captured image. In other words, the display screen affects the captured image, and the image quality compensation model compensates the captured image, thereby eliminating the effect of the display screen on the image quality damage amount of the image.
It should be noted that the image quality compensation model may also be obtained by other methods, such as optical design simulation, or calibrated by a calibration environment such as laser beam scanner, precision calibration system, or calibrated by a shooting iso12233 graphic card and a pin grid pattern, and in the case that the image quality compensation model can simulate the influence of a display screen on a shot image, the corresponding scheme falls within the scope of the present application.
In this embodiment, the electronic device may acquire the designated processing stage, then detect whether the designated processing stage is reached in the process of processing the original image data, and if it is detected that the designated processing stage is reached, perform image quality compensation on the image data of the designated processing stage by using a preset image quality compensation model in response to that the processed original image data reaches the designated processing stage, so as to obtain image data after image quality compensation.
In step 203, the image data after the image quality compensation is continuously processed until the target image data is obtained.
In this embodiment, the image processing platform may continue to process the image data after the image quality compensation until the target image data is obtained, and the compensation effect is as shown in fig. 7, where fig. 7(a) is an image before the compensation and fig. 7(b) is an image after the compensation.
Therefore, in the embodiment of the present disclosure, at a designated processing stage for processing original image data, a preset image quality compensation model is used to perform image quality compensation on the image data at the designated processing stage, so that an amount of image quality damage of a display screen to an image captured by a camera module can be compensated, and target image data with an image quality meeting requirements is obtained. In this way, the present embodiment eliminates or slows down the influence of the display screen on the image captured by the camera module by performing image quality compensation on the image data at the designated processing stage.
On the basis of the image processing method provided by the embodiment of the present disclosure, an embodiment of the present disclosure further provides an image processing apparatus, where a camera module is disposed in an electronic device below a display screen, the camera module includes an image sensor, the image sensor is configured to receive light passing through the display screen and generate raw image data, and fig. 8 is a block diagram of an image processing apparatus according to an exemplary embodiment. Referring to fig. 8, an image processing apparatus 800 includes:
an original image obtaining module 801, configured to obtain original image data output by a sensor in the camera module;
an image quality compensation module 802, configured to perform quality compensation on image data in a specified processing stage by using a preset quality compensation model in response to that the processed original image data reaches the specified processing stage; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen;
and the target image acquisition module 803 is configured to continue to process the image data after the image quality compensation until the target image data is obtained.
In one embodiment of the present invention, the substrate is,
the designated processing stages include at least one of the following image processing stages:
RAW domain processing before an ISP stage of image signal processing;
an ISP stage of image signal processing;
YUV domain processing after the image signal processing ISP stage.
In an embodiment, the image quality compensation model includes at least one of the following: diffraction models, deblurring models, and scattering models.
Fig. 9 is a block diagram of an image processing apparatus according to an exemplary embodiment, and based on the image processing apparatus shown in fig. 8, referring to fig. 9, the apparatus 800 further includes a compensation model obtaining module 804, and the compensation model obtaining module 804 includes:
an image quality compensation unit 901 configured to input an actual image into the image quality compensation model, and perform image quality compensation on the actual image by using the image quality compensation model; the actual image is obtained in a scene that the display screen is arranged on the camera module;
an image comparison unit 902, configured to compare the actual image after image quality compensation with a preset alignment image; the target image is obtained in a scene that the display screen is not arranged on the camera module;
a parameter adjusting unit 903, configured to adjust parameters of the image quality compensation model and then perform image quality compensation on the actual image again until a difference between the image quality damage amounts of the image quality compensated actual image and the target image is smaller than a set difference threshold when a difference between the image quality damage amounts of the image quality compensated actual image and the target image exceeds the set difference threshold.
Fig. 10 is a block diagram of an image processing apparatus according to an exemplary embodiment, and based on the image processing apparatus shown in fig. 8, referring to fig. 10, the apparatus 800 further includes a compensation model obtaining module 804, and the compensation model obtaining module 804 includes:
an optical model obtaining unit 1001 configured to obtain a first optical model and a second optical model before and after the display screen is set on the camera module;
an offset model obtaining unit 1002, configured to determine, according to the first optical model and the second optical model, an optical offset model representing an influence of the display screen on light;
a compensation model training unit 1003, configured to train the image quality compensation model by using the optical offset model, so that a difference between image quality damage amounts of the acquired images before and after the setting of the display screen is smaller than a set difference threshold.
Fig. 11 is a block diagram of an image processing apparatus according to an exemplary embodiment, and based on the image processing apparatus shown in fig. 8, referring to fig. 11, the apparatus 800 further includes a compensation model obtaining module 804, and the compensation model obtaining module 804 includes:
an optical model obtaining unit 1101, configured to obtain a second optical model after the display screen is disposed on the camera module;
a parameter value obtaining unit 1102, configured to obtain actual values of each parameter in the second optical model and a calibration range corresponding to the actual values;
a parameter value adjusting unit 1103, configured to compare the actual value with the calibration range, and adjust the actual value of the parameter exceeding the calibration range to the calibration range, where the second optical model that adjusts the actual value of the parameter is the image quality compensation model.
It can be understood that the apparatus provided in the embodiment of the present invention corresponds to the content of the above method embodiments, and specific content may refer to the content of each method embodiment, which is not described herein again.
Therefore, in the embodiment of the present disclosure, at a designated processing stage for processing original image data, a preset image quality compensation model is used to perform image quality compensation on the image data at the designated processing stage, so that an amount of image quality damage of a display screen to an image captured by a camera module can be compensated, and target image data with an image quality meeting requirements is obtained. In this way, the present embodiment eliminates or reduces the influence of the display screen on the image captured by the camera module by performing image quality compensation, such as contrast and/or sharpness, on the image data at the designated processing stage.
FIG. 12 is a block diagram illustrating an electronic device in accordance with an example embodiment. For example, the electronic device 1200 may be a smart phone, a computer, a digital broadcast terminal, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like, which includes a transmitting end and/or a receiving end of a wireless charging device.
Referring to fig. 12, electronic device 1200 may include one or more of the following components: a processing component 1202, a memory 1204, a power component 1206, a multimedia component 1208, an audio component 1210, an input/output (I/O) interface 1212, a sensor component 1214, a communications component 1216, and an image capture component 1218.
The processing component 1202 generally provides for overall operation of the electronic device 1200, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 1202 may include one or more processors 1220 to execute instructions. Further, the processing component 1202 can include one or more modules that facilitate interaction between the processing component 1202 and other components. For example, the processing component 1202 can include a multimedia module to facilitate interaction between the multimedia component 1208 and the processing component 1202. In interacting, processor 1220 may read executable instructions from memory 1204 to implement the steps of the methods of fig. 2-7.
The memory 1204 is configured to store various types of data to support operation at the electronic device 1200. Examples of such data include instructions for any application or method operating on the electronic device 1200, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 1204 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component 1206 provides power to the various components of the electronic device 1200. The power components 1206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 1200.
The multimedia component 1208 includes a screen providing an output interface between the electronic device 1200 and the target object. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a target object. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
The audio component 1212 is configured to output and/or input audio signals. For example, the audio assembly 1210 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 1200 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 1204 or transmitted via the communication component 1216. In some embodiments, audio assembly 1210 further includes a speaker for outputting audio signals.
The I/O interface 1212 provides an interface between the processing component 1202 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc.
The sensor assembly 1214 includes one or more sensors for providing various aspects of state assessment for the electronic device 1200. For example, the sensor assembly 1214 may detect an open/closed state of the electronic device 1200, the relative positioning of components, such as a display screen and keypad of the electronic device 1200, the sensor assembly 1214 may also detect a change in the position of the electronic device 1200 or one of the components, the presence or absence of a target object in contact with the electronic device 1200, an orientation or acceleration/deceleration of the electronic device 1200, and a change in the temperature of the electronic device 1200.
The communications component 1216 is configured to facilitate communications between the electronic device 1200 and other devices in a wired or wireless manner. The electronic device 1200 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1216 receives the broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communications component 1216 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 1200 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components.
In an exemplary embodiment, a non-transitory readable storage medium including executable instructions, such as memory 1204 including instructions, that are executable by processor 1220 of electronic device 1200 to implement the steps of the method shown in fig. 2-7, is also provided. The readable storage medium may be, among others, ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (14)
1. An image processing method, adapted to an electronic device having a camera module disposed below a display screen, the camera module including an image sensor for receiving light passing through the display screen and generating raw image data, comprising:
acquiring raw image data output by the image sensor;
responding to the processed original image data to reach a designated processing stage, and performing image quality compensation on the image data of the designated processing stage by adopting a preset image quality compensation model; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen;
and continuously processing the image data after the picture quality compensation until the target image data is obtained.
2. The image processing method according to claim 1, wherein the specified processing stages comprise at least one of the following image processing stages:
RAW domain processing before an ISP stage of image signal processing;
an ISP stage of image signal processing;
YUV domain processing after the image signal processing ISP stage.
3. The image processing method of claim 1, wherein the image quality compensation model comprises at least one of: diffraction models, deblurring models, and scattering models.
4. The image processing method according to claim 1, wherein the image quality compensation model is obtained by:
inputting an actual image into the image quality compensation model, and performing image quality compensation on the actual image by using the image quality compensation model; the actual image is obtained in a scene that the display screen is arranged on the camera module;
comparing the actual image after image quality compensation with a preset alignment image; the target image is obtained in a scene that the display screen is not arranged on the camera module;
and if the difference value of the image quality damage amount of the actual image after image quality compensation and the targeted image exceeds a set difference threshold value, adjusting the parameters of the image quality compensation model and then performing image quality compensation on the actual image again until the difference value of the image quality damage amount of the actual image after image quality compensation and the targeted image is less than the set difference threshold value.
5. The image processing method according to claim 1, wherein the image quality compensation model is obtained by:
acquiring a first optical model and a second optical model which are arranged in front of and behind the display screen on the camera module;
determining an optical offset model representing the influence of the display screen on light according to the first optical model and the second optical model;
and training the image quality compensation model by using the optical offset model so as to enable the difference value of the image quality damage quantity of the images acquired before and after the setting of the display screen to be smaller than a set difference threshold value.
6. The image processing method according to claim 1, wherein the image quality compensation model is obtained by:
acquiring a second optical model on the camera module after the display screen is arranged;
acquiring actual values of all parameters in the second optical model and corresponding calibration ranges of the actual values;
and comparing the actual value with the calibration range, adjusting the actual value of the parameter exceeding the calibration range to the calibration range, wherein the second optical model for adjusting the actual value of the parameter is the image quality compensation model.
7. An image processing apparatus adapted to an electronic device having a camera module disposed below a display screen, the camera module including an image sensor for receiving light passing through the display screen and generating raw image data, comprising:
the original image acquisition module is used for acquiring original image data output by the image sensor;
the image quality compensation module is used for responding to the situation that the processed original image data reaches a specified processing stage, and performing image quality compensation on the image data of the specified processing stage by adopting a preset image quality compensation model; the image quality compensation model is used for compensating the image quality damage amount of the image shot by the camera module group by the display screen;
and the target image acquisition module is used for continuously processing the image data after the picture quality compensation until the target image data is obtained.
8. The image processing apparatus according to claim 7, wherein the specified processing stages include at least one of the following image processing stages:
RAW domain processing before an ISP stage of image signal processing;
an ISP stage of image signal processing;
YUV domain processing after the image signal processing ISP stage.
9. The image processing apparatus according to claim 7, wherein the image quality compensation model comprises at least one of: diffraction models, deblurring models, and scattering models.
10. The image processing apparatus according to claim 7, further comprising a compensation model acquisition module, the compensation model acquisition module comprising:
an image quality compensation unit for inputting an actual image into the image quality compensation model and performing image quality compensation on the actual image by the image quality compensation model; the actual image is obtained in a scene that the display screen is arranged on the camera module;
the image comparison unit is used for comparing the actual image after the image quality compensation with a preset alignment image; the target image is obtained in a scene that the display screen is not arranged on the camera module;
and a parameter adjusting unit, configured to adjust parameters of the image quality compensation model and then perform image quality compensation on the actual image again until a difference between the image quality damage amounts of the actual image after image quality compensation and the target image is smaller than a set difference threshold value, when the difference between the image quality damage amounts of the actual image after image quality compensation and the target image exceeds the set difference threshold value.
11. The image processing apparatus according to claim 7, further comprising a compensation model acquisition module, the compensation model acquisition module comprising:
the optical model acquisition unit is used for acquiring a first optical model and a second optical model which are arranged on the camera module and in front of and behind the display screen;
the offset model acquisition unit is used for determining an optical offset model representing the influence of the display screen on light according to the first optical model and the second optical model;
and the compensation model training unit is used for training the image quality compensation model by using the optical offset model so as to enable the difference value of the image quality damage quantity of the images acquired before and after the setting of the display screen to be smaller than a set difference threshold value.
12. The image processing apparatus according to claim 7, further comprising a compensation model acquisition module, the compensation model acquisition module comprising:
the optical model acquisition unit is used for acquiring a second optical model after the display screen is arranged on the camera module;
the parameter value acquisition unit is used for acquiring actual values of all parameters in the second optical model and corresponding calibration ranges of the actual values;
and the parameter value adjusting unit is used for comparing the actual value with the calibration range, adjusting the actual value of the parameter beyond the calibration range to the calibration range, and adjusting the second optical model of the actual value of the parameter, namely the image quality compensation model.
13. An electronic device, characterized in that the electronic device comprises:
a display screen;
the camera module is arranged below the display screen;
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to execute executable instructions in the memory to implement the steps of the method of any of claims 1 to 6.
14. A readable storage medium having stored thereon executable instructions, which when executed by a processor, perform the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910817264.9A CN112449085A (en) | 2019-08-30 | 2019-08-30 | Image processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910817264.9A CN112449085A (en) | 2019-08-30 | 2019-08-30 | Image processing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112449085A true CN112449085A (en) | 2021-03-05 |
Family
ID=74735013
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910817264.9A Pending CN112449085A (en) | 2019-08-30 | 2019-08-30 | Image processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112449085A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113132626A (en) * | 2021-03-26 | 2021-07-16 | 联想(北京)有限公司 | Image processing method and electronic equipment |
| CN114302051A (en) * | 2021-03-29 | 2022-04-08 | 海信视像科技股份有限公司 | Camera shooting method and display device |
| CN114612437A (en) * | 2022-03-15 | 2022-06-10 | 广州康盛网络科技有限公司 | AMOLED-based display image quality improving method |
| CN115776532A (en) * | 2021-09-07 | 2023-03-10 | 荣耀终端有限公司 | Method for capturing image in video and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1708092A (en) * | 2004-06-10 | 2005-12-14 | 奥林巴斯株式会社 | Image recording apparatus, image recording method, image processing apparatus, image processing method and image processing system |
| CN101257577A (en) * | 2007-03-02 | 2008-09-03 | 索尼株式会社 | Image-capturing apparatus and image processing method |
| JP2017153156A (en) * | 2017-06-07 | 2017-08-31 | 株式会社ニコン | Electronic camera |
| CN107222681A (en) * | 2017-06-30 | 2017-09-29 | 维沃移动通信有限公司 | The processing method and mobile terminal of a kind of view data |
| CN109143607A (en) * | 2018-09-17 | 2019-01-04 | 深圳奥比中光科技有限公司 | It compensates display screen, shield lower optical system and electronic equipment |
| CN109285860A (en) * | 2017-07-21 | 2019-01-29 | 京东方科技集团股份有限公司 | A kind of electroluminescence display panel, display device and its obtain image display method |
| CN109979382A (en) * | 2019-04-23 | 2019-07-05 | 清华大学 | Imaging system color correcting method and system under screen based on screen transmitted spectrum |
-
2019
- 2019-08-30 CN CN201910817264.9A patent/CN112449085A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1708092A (en) * | 2004-06-10 | 2005-12-14 | 奥林巴斯株式会社 | Image recording apparatus, image recording method, image processing apparatus, image processing method and image processing system |
| CN101257577A (en) * | 2007-03-02 | 2008-09-03 | 索尼株式会社 | Image-capturing apparatus and image processing method |
| JP2017153156A (en) * | 2017-06-07 | 2017-08-31 | 株式会社ニコン | Electronic camera |
| CN107222681A (en) * | 2017-06-30 | 2017-09-29 | 维沃移动通信有限公司 | The processing method and mobile terminal of a kind of view data |
| CN109285860A (en) * | 2017-07-21 | 2019-01-29 | 京东方科技集团股份有限公司 | A kind of electroluminescence display panel, display device and its obtain image display method |
| CN109143607A (en) * | 2018-09-17 | 2019-01-04 | 深圳奥比中光科技有限公司 | It compensates display screen, shield lower optical system and electronic equipment |
| CN109979382A (en) * | 2019-04-23 | 2019-07-05 | 清华大学 | Imaging system color correcting method and system under screen based on screen transmitted spectrum |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113132626A (en) * | 2021-03-26 | 2021-07-16 | 联想(北京)有限公司 | Image processing method and electronic equipment |
| CN114302051A (en) * | 2021-03-29 | 2022-04-08 | 海信视像科技股份有限公司 | Camera shooting method and display device |
| CN115776532A (en) * | 2021-09-07 | 2023-03-10 | 荣耀终端有限公司 | Method for capturing image in video and electronic equipment |
| CN115776532B (en) * | 2021-09-07 | 2023-10-20 | 荣耀终端有限公司 | Method for capturing images in video and electronic equipment |
| CN114612437A (en) * | 2022-03-15 | 2022-06-10 | 广州康盛网络科技有限公司 | AMOLED-based display image quality improving method |
| CN114612437B (en) * | 2022-03-15 | 2024-04-09 | 深圳市科泰兴业科技有限公司 | AMOLED-based display image quality improvement method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109345485B (en) | Image enhancement method and device, electronic equipment and storage medium | |
| CN112449085A (en) | Image processing method and device, electronic equipment and readable storage medium | |
| KR101434796B1 (en) | Method, apparatus and computer program product for capturing video content | |
| US20140347521A1 (en) | Simulating High Dynamic Range Imaging with Virtual Long-Exposure Images | |
| CN108154465B (en) | Image processing method and device | |
| US9247152B2 (en) | Determining image alignment failure | |
| CN106131441B (en) | Photographing method and device and electronic equipment | |
| CN108986053B (en) | Screen display method and device | |
| CN110569822A (en) | image processing method and device, electronic equipment and storage medium | |
| CN111709891B (en) | Training method of image denoising model, image denoising method, device and medium | |
| CN114096994A (en) | Image alignment method and device, electronic equipment and storage medium | |
| CN106982327B (en) | Image processing method and device | |
| CN109784327B (en) | Boundary box determining method and device, electronic equipment and storage medium | |
| CN112033527B (en) | Ambient brightness detection method, device, equipment and storage medium | |
| CN105472228B (en) | Image processing method and device and terminal | |
| CN112750081A (en) | Image processing method, device and storage medium | |
| CN108156381B (en) | Photographing method and device | |
| CN111277754B (en) | Mobile terminal shooting method and device | |
| CN112950503A (en) | Training sample generation method and device and truth value image generation method and device | |
| CN115641269A (en) | Image repairing method and device and readable storage medium | |
| CN114331852A (en) | Method and device for processing high dynamic range image and storage medium | |
| CN112288657A (en) | Image processing method, image processing apparatus, and storage medium | |
| CN116347248B (en) | Image acquisition method and device, electronic equipment, medium and chip | |
| CN116416505A (en) | Training method of image processing model, image processing method, device and storage medium | |
| CN115249208A (en) | Image processing method, video processing method and device, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210305 |