WO2024174711A1

WO2024174711A1 - Image processing method and terminal device

Info

Publication number: WO2024174711A1
Application number: PCT/CN2023/140999
Authority: WO
Inventors: 李先明; 林志杰; 张文浩
Original assignee: 华为技术有限公司
Priority date: 2023-02-23
Filing date: 2023-12-22
Publication date: 2024-08-29
Also published as: CN118540449A

Abstract

The present application relates to the field of terminals. Provided are an image processing method and a terminal device. The method comprises: acquiring effect parameters of at least one first camera module according to a real-time image captured by the at least one first camera module; inputting the effect parameters of the at least one first camera module into a trained mathematical model, so as to obtain effect parameters of a second camera module, wherein the at least one first camera module and the second camera module have a co-visibility region; and controlling the second camera module to capture an image by using the effect parameters of the second camera module, and/or by using the effect parameters of the second camera module, performing image processing on a real-time image captured by the second camera module. In the method, effect parameters are calculated according to captured images, without the need to configure corresponding hardware resources or software resources for a second camera module, such that the image effect and quality of a plurality of cameras can be improved under the constraints of power consumption, loads, and hardware costs.

Description

Image processing method and terminal device

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 23, 2023, with application number 202310203636.5 and application name “Image Processing Method and Terminal Device”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of image processing, and in particular to an image processing method and a terminal device.

Background Art

Extended reality (XR) is a general term for multiple technologies such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). In XR scenarios, users can use multiple cameras to perceive, locate, or interact with the real world. In order to have a better XR experience, different cameras need to be able to stably capture high-quality images in complex user environments. The image effect debugging used by existing XR headsets is divided into two categories: hardware debugging and software image algorithm debugging. If multiple cameras use hardware debugging, the power consumption and cost will be high; if multiple cameras use software debugging, the system load will be high.

Therefore, how to improve the image quality of multiple cameras under the constraints of power consumption, load and hardware cost is an urgent problem to be solved.

Summary of the invention

The present application provides an image processing method and a terminal device, which can improve the image effect quality of multiple cameras under the constraints of power consumption, load and hardware cost.

In a first aspect, an image processing method is provided, which is applied to a terminal device. The terminal device includes at least one first camera module and a second camera module, and there is a common viewing area between the at least one first camera module and the second camera module. The method includes: obtaining effect parameters of the at least one first camera module according to the real-time image captured by the at least one first camera module; inputting the effect parameters of the at least one first camera module into a trained mathematical model to obtain the effect parameters of the second camera module, the trained mathematical model is obtained by training the mathematical model, the input of the mathematical model is a sample image captured by the at least one first camera module, and the supervision data of the mathematical model is the effect parameters of the second camera model obtained according to the sample image captured by the second camera module; controlling the second camera module to use the effect parameters of the second camera module to capture the image, and/or using the effect parameters of the second camera module to perform image processing on the real-time image captured by the second camera module.

According to the image processing method provided by the present application, for multiple camera modules with a common viewing area, only the effect parameters of some (i.e., one or more) camera modules are calculated, and the effect parameters of other camera modules are derived using a pre-trained mathematical model, and then other camera modules use the derived effect parameters to shoot images or use the derived effect parameters to perform corresponding image processing on images shot by other camera modules. This method does not require the configuration of corresponding hardware resources or software resources for other camera modules to calculate effect parameters based on the captured images, and can improve the image effect quality of multiple cameras under the constraints of power consumption, load and hardware cost.

In one possible implementation, the mathematical model may be a deep learning model, such as a neural network model.

In a possible implementation, the effect parameter includes one or more of the following: exposure time, exposure gain, white balance coefficient, color correction matrix, or sharpening coefficient.

In a possible implementation, the terminal device is a mixed reality MR device.

In a possible implementation, the at least one first camera module is connected to at least one image processing unit, and the at least one image processing unit is a hardware resource. The method of obtaining the effect parameters of the at least one first camera module according to the real-time image captured by the at least one first camera module includes: controlling the at least one image processing unit to perform image processing on the real-time image captured by the at least one first camera module to obtain the effect parameters of the at least one first camera module. The solution can calculate the effect parameters of the at least one first camera module through the image processing unit.

Exemplarily, the image processing unit may be an image signal processing (ISP), a graphics processing (GPU), or a digital signal processing (DSP), or other hardware resources capable of performing image processing.

In a possible implementation, the real-time image captured by the second camera module is processed using the effect parameters of the second camera module, including: controlling the image algorithm module to use part or all of the effect parameters to perform image processing on the real-time image captured by the second camera module, and the image algorithm module is a software resource.

In a second aspect, a terminal device is provided, comprising a functional module/unit for executing the method of the first aspect and any possible implementation manner of the first aspect.

According to a third aspect, a computer-readable medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes a method for executing the first aspect or any one of the implementations of the first aspect.

According to a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method according to the first aspect or any one of the implementations of the first aspect.

It should be noted that the above-mentioned computer program code can be stored in whole or in part on the first storage medium, wherein the first storage medium can be packaged together with the processor or separately packaged with the processor, and the embodiments of the present application do not specifically limit this.

In a fifth aspect, a chip system is provided, which is applied to a terminal device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method in the first aspect or any one of the implementations of the first aspect.

Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the first aspect or any one of the implementation methods of the first aspect.

In a sixth aspect, a terminal device is provided, comprising one or more processors, a memory, and multiple camera modules; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect or any one of the implementations of the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an XR head display provided in an embodiment of the present application;

FIG2 is a schematic diagram of a process of performing image processing by hardware provided in an embodiment of the present application;

FIG3 is a schematic diagram of a flow chart of image processing by software provided in an embodiment of the present application;

FIG4 is a schematic diagram of a camera system of a terminal device provided in an embodiment of the present application;

FIG5 is a schematic diagram of calculating effect parameters of a camera module provided by an embodiment of the present application;

FIG6 is a schematic diagram of a machine learning model training provided in an embodiment of the present application;

FIG7 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

FIG8 is another schematic flow chart of another image processing method provided in an embodiment of the present application;

FIG9 is a schematic block diagram of a terminal device provided in an embodiment of the present application;

FIG. 10 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and/or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character "/" generally indicates that the associated objects before and after are in a kind of "or" relationship.

References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some other embodiments", "in other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment that allows human-computer interaction. XR is also a general term for multiple technologies such as AR, VR, and MR. In XR scenarios, users can use multiple camera modules in XR terminal devices (such as XR headsets) to perceive, locate, or interact with the real world.

Exemplarily, FIG1 shows an XR head display intention. Referring to FIG1 , a plurality of camera modules are provided on the XR head display 100, such as the camera modules 101 to 104 shown in the figure. At least two of the camera modules 101 to 104 have a common viewing area, that is, at least two of the camera modules 101 to 104 partially overlap in perspective. For example, the perspectives of camera module 101 and camera module 102 partially overlap, and the perspectives of camera module 103 and camera module 104 partially overlap. For example, referring to FIG1 , the images captured by camera module 101 and camera module 102 have some of the same pictures (i.e., pictures of people lifting bicycles). In addition, camera module 102 and camera module 103 may also have partial perspective overlap. In some embodiments, the functions or types of the two camera modules with partial perspective overlap are different. For example, the viewing angles of the camera module 101 and the camera module 102 partially overlap, the camera module 101 can realize the video perspective function, and the camera module 102 can realize the vector space positioning function.

In order to provide a better XR experience, different camera modules need to be able to stably capture high-quality images in complex user environments. However, due to the physical defects of the lens and image sensor in the camera module, the original image output by the camera module is generally of poor quality. In order to obtain a usable image from the original image output by the camera module, it is usually necessary to further process the original image. For example, the original image can be processed by auto exposure (AE), auto white balance (AWB), color correction, sharpening, etc. The following is a brief description of these processing methods.

Automatic exposure: Automatically adjust the exposure time and exposure gain according to the intensity of light, so as to adjust the exposure amount, ensure that the photos taken in different lighting conditions and scenes are accurately exposed and have appropriate brightness, and prevent overexposure or underexposure.

Automatic white balance: related to color temperature, used to measure the color authenticity and accuracy of an image. Specifically, the color temperature of the light source can be determined based on the input image, and then the R/G/B coefficients (or gains) can be calculated based on the color temperature. Finally, the R/G/B coefficients can be adjusted to correct the input image, thereby accurately restoring the original color of the object in various complex scenes.

Color correction: mainly to correct the color error caused by color penetration between color blocks at the filter plate. The general color correction process is to first compare the image captured by the image sensor with the standard image to calculate a correction matrix. This matrix is the color correction matrix of the image sensor. During the application of the image sensor, the matrix can be used to correct all images captured by the image sensor to obtain an image that is closest to the true color of the object.

Sharpening: It is an image processing method that makes the image edge clearer. Its main principle is to extract the high-frequency components of the original image, and then superimpose them with the original image according to certain rules (involving the sharpening coefficient). The final image is the sharpened image.

Each of the above processing methods corresponds to one or more parameters, which can be called effect parameters of the camera module. In the same environment or scene, the camera module or the corresponding processing unit can use the effect parameters to capture images or perform image processing on the captured images. For example, after automatic exposure, a suitable exposure time and exposure gain can be obtained, and the exposure time and exposure gain can be considered as two effect parameters of the camera module, and the camera module can use the exposure time and exposure gain to capture images. For another example, after automatic white balance, a suitable white balance coefficient can be obtained, and the white balance coefficient can be considered as an effect parameter of the camera module, and the camera module can use the white balance coefficient to capture images. Similarly, after color correction and sharpening processing, the corresponding color correction matrix and sharpening coefficient can be obtained, and the color correction matrix and the sharpening coefficient are both effect parameters of the camera module, and the color correction matrix and the sharpening coefficient can be used to process the image captured by the camera module.

It should be understood that only some image processing methods are introduced above. In fact, more or fewer processing methods can be used to process the original image. For example, the original image can also be processed by noise removal, bad pixel removal, interpolation, etc.; accordingly, the effect parameters can include the parameters used for the corresponding processing.

The image processing described above can be implemented in hardware or through software algorithms. These two implementation methods are briefly described.

FIG2 is a flow chart of image processing by hardware. Referring to FIG2 , the camera module outputs an image signal, and the image processing unit can perform post-processing on the image signal, such as automatic exposure, automatic white balance, color correction, sharpening, etc., so as to better restore the details of the scene under different optical conditions. After processing the image signal, the image processing unit outputs the processed image signal to the processor, which further processes it, such as presenting an image corresponding to the processed image signal on a display screen.

Exemplarily, the image processing unit in the present application is a hardware resource capable of performing image processing. For example, the image processing unit in the present application may be an image signal processing (ISP), a graphics processing (GPU), or a digital signal processing (DSP) or other hardware resources capable of performing image processing.

Exemplarily, the processor in the present application can run a variety of image processing algorithms and control peripheral devices. For example, the processor can be a central processing unit (CPU), GPU or other types of processors.

Depending on the processing capabilities of the image processing unit, an image processing unit may only be able to process the image signal output by one camera module at a time, or it may be able to process the image signals input by multiple (usually 2) camera modules at the same time. If multiple camera modules of the XR headset use hardware for image processing, then multiple image processing units are required, which will result in higher costs and power consumption.

FIG3 is a flow chart of image processing by software. Referring to FIG3 , the camera module outputs an image signal to the processor, and the image algorithm module in the processor performs post-processing on the image signal, such as automatic exposure, automatic white balance, color correction, sharpening, etc., so as to better restore the details of the scene under different optical conditions. The image signal processed by the processor can be further processed by other modules in the processor, such as presenting an image corresponding to the processed image signal on a display screen.

Exemplarily, the image algorithm module in the present application refers to a software algorithm for image signal processing.

The image algorithm module will occupy a certain load in the process of processing image signals. If multiple camera modules of the XR headset adopt this software debugging method, it will bring a higher load.

However, to ensure the normal operation of XR headsets, there are usually certain constraints on power consumption, load, and hardware cost. How to improve the image quality of multi-camera modules under the constraints of power consumption, load, and hardware cost is an issue that needs to be considered.

In view of this, the present application provides an image processing method, which can calculate the effect parameters of only some camera modules in a scenario where multiple camera modules with a common viewing area are used together, and use a pre-trained mathematical model to derive the effect parameters of other camera modules, and then other camera modules use the derived effect parameters to shoot images or use the derived effect parameters to perform corresponding image processing on images shot by other camera modules. This method does not need to configure corresponding hardware resources or software resources for other camera modules to calculate effect parameters based on the captured images, and can improve the image effect quality of multiple cameras under the constraints of power consumption, load and hardware cost.

The method provided in the present application can be applied to a terminal device including multiple camera modules having a common viewing area. For example, the terminal device can be a VR device or an MR device. The present application does not limit the type of the terminal device.

It should be understood that a camera module is sometimes also referred to as a camera or a camera.

Exemplarily, FIG4 shows a schematic diagram of a camera system of a terminal device provided by the present application. Referring to FIG4, the camera system may include multiple camera modules, for example, including camera module 1 to camera module N, N ≥ 2. Among them, the image processing units corresponding to camera module 1 to camera module M can respectively perform image processing on the image signals output by camera module 1 to camera module M. For example, the image processing unit can perform automatic exposure, automatic white balance, color correction, sharpening and other processing on the image. The processor can calculate the effect parameters of the remaining part or all of the camera modules, such as camera module (M+1) to camera module N, based on the effect parameters of the image processing units corresponding to camera module 1 to camera module M, using a pre-trained mathematical model. For example, if camera module 1, camera module 3 and camera module 4 all have a common viewing area, the effect parameters of camera module 3 and camera module 4 can be obtained according to the effect parameters of camera module 1, and if camera module 2 and camera module 5 have a common viewing area, the effect parameters of camera module 5 can be obtained according to the effect parameters of camera module 2. After obtaining the effect parameters of the remaining part or all of the camera modules, the effect parameters of the corresponding camera module may be used to capture images and/or perform image processing on the image signals of the camera modules.

In some embodiments, referring to (a) in FIG. 4 , the processor may send the calculated effect parameters of camera module (M+1) to camera module N to camera module (M+1) to camera module N respectively, and camera module (M+1) to camera module N may use their corresponding effect parameters to perform image processing on the image signal, and then may output the processed image signal to the processor. 4, see (b), the processor can send the calculated effect parameters of camera module (M+1) to camera module N to the image algorithm modules corresponding to camera module (M+1) to camera module N respectively, and the image algorithm module can use the effect parameters of the camera module to perform image processing on the image signal output by the camera module, and output the processed image signal. In some embodiments, the camera module can use part of the effect parameters to capture the image, and the image algorithm module can use the remaining part of the effect parameters to perform image processing on the image signal output by the camera module, and output the processed image signal. It should be understood that the image algorithm module can be a software algorithm module, which can be executed by other hardware resources in the camera system, such as a processor, or other hardware resources with processing functions.

It should be understood that in the camera system shown in Figure 4, one image processing unit processes the image signal of one camera module. However, in practice, if one image processing unit can process the image signals of multiple camera modules, then multiple camera modules only need to be equipped with one image processing unit.

In addition, in the camera system shown in FIG. 4 , the image processing unit is disposed outside the processor, but it should be understood that, in practice, the image processing unit may be disposed inside the processor, that is, the processor may include one or more image processing units.

It should be understood that only several units in the camera system related to the invention of the present application are shown in Fig. 4. In an actual camera system, other units or modules involved in image signal processing may also be included, which, although not shown in the present application, do not mean that they are excluded from the protection scope of the present application.

It should be understood that the terminal shown in FIG. 4 may also include other camera modules in addition to camera module 1 to module N.

It should be understood that the camera system shown in FIG. 4 can be configured in the XR head display shown in FIG. 1, that is, the camera modules in FIG. 4 are part or all of the camera modules in FIG. 1. For example, camera module 1 to module module M in FIG. 4 are camera modules 101 and 103 in FIG. 1, and camera module (M+1) to module module N in FIG. 4 are camera modules 102 and 104 in FIG. 1. In one example, the effect parameters of camera module 3 (equivalent to camera module 102 in FIG. 1) can be obtained according to the effect parameters of camera module 1 (equivalent to camera module 101 in FIG. 1), and the effect parameters of camera module 4 (equivalent to camera module 104 in FIG. 1) can be obtained according to the effect parameters of camera module 2 (equivalent to camera module 103 in FIG. 1). In another example, the effect parameters of camera module 1 (equivalent to camera module 101 in Figure 1) can be obtained according to the effect parameters of camera module 3 (equivalent to camera module 102 in Figure 1), and the effect parameters of camera module 2 (equivalent to camera module 103 in Figure 1) can be obtained according to the effect parameters of camera module 4 (equivalent to camera module 104 in Figure 1).

The solution provided in this application is described in detail below.

In some embodiments, the method provided in the present application is implemented on the basis of a trained mathematical model (the model is referred to as a trained mathematical model in the present application). Based on the trained mathematical model, the effect parameters of one or more camera modules can be predicted according to the effect parameters of one or more camera modules.

In one example, the mathematical model may be a machine learning model. The following is a brief description of how to perform machine learning model training.

The multiple camera modules with a common viewing area in the terminal device work in the same environment, and the color temperature, overall brightness, and environmental content (bedroom scene, office scene, etc.) of the environment are the same or have a small difference, so there is a certain mapping relationship between the effect parameters of the multiple cameras. In this application, multiple camera modules can be used to collect images in the same environment first, and then the effect parameters of the multiple camera modules can be determined based on the images collected by the multiple camera modules in the same environment. Finally, the effect parameters of the multiple camera modules are used for machine model learning to obtain a trained machine learning model.

Optionally, images (ie, sample images) captured by multiple camera modules under the same environment may be input into an image processing unit, and the image processing unit may obtain effect parameters of the multiple camera modules according to the sample images.

For example, FIG5 shows a schematic diagram of calculating the effect parameters of a camera module. Referring to FIG5 , both camera module 1 and camera module 2 have a common viewing area with camera module 3. The images collected by camera module 1 to camera module 3 in the same environment (for example, an office scene or an outdoor scene, etc.) are input into the ISPs to which they are connected. After the ISP processes the input images, the effect parameters of the corresponding camera modules in the environment can be obtained. For example, the ISP performs automatic exposure, automatic white balance, color correction, sharpening and other processing on the images input by each camera module, and then the exposure time, exposure gain, white balance coefficient, color correction matrix, sharpening coefficient and other effect parameters corresponding to each camera module can be obtained.

It should be understood that the sample images may also be calculated using software algorithms to obtain the effect parameters of the multiple camera modules.

For example, FIG6 shows a schematic diagram of a machine learning model training. Referring to FIG6 , the input parameter vector of the machine learning model is the effect parameters of camera module 1 and camera module 2 calculated in FIG5 . Among them, [t _i , g _i , _rwi , _gwi , _bwi , _Xi , k _i ] is the effect parameter of camera module i, including exposure time t _i , exposure gain g _i , white balance coefficient ( _rwi , _gwi , _bwi ), color correction Matrix _Xi , sharpening coefficient k _i . The output parameter vector of the machine learning model is the predicted effect parameter of the camera module 3. The output effect parameter of the camera module 3 is compared with the effect parameter of the camera module 3 calculated in FIG5 (i.e., the supervision data or the true value), and the difference is calculated; the obtained difference is used to reversely update the relevant parameters in the machine learning model, and the above steps are repeated until the difference between the effect parameter of the camera module 3 output by the machine learning model and the effect parameter of the supervision data is less than a certain threshold, thereby completing the training of the machine learning model.

It should be understood that the machine learning model in the present application can be any machine learning model, for example, it can be the deep learning model shown in the figure, such as a neural network model, that is, other machine learning models.

In another example, the mathematical model may be a linear model.

After obtaining the trained mathematical model, the image processing method provided by the present application can be executed. The image processing method provided by the present application is described in detail below in conjunction with the flowchart shown in FIG7 .

FIG7 is a schematic flow chart of an image processing method provided by the present application. The method can be executed by a terminal device, which includes at least one first camera module and a second camera module, and the at least one first camera module and the second camera module have a common viewing area. For example, the terminal device may be the XR glasses shown in FIG1, the at least one first camera module may be camera module 101, the second camera module may be camera module 102, and camera module 101 and camera module 102 have a common viewing area. The method may include S710 to S730, and each step is described below.

S710: Acquire effect parameters of the at least one first camera module according to the real-time image captured by the at least one first camera module.

Specifically, the effect parameters of any first camera module may be obtained by performing image processing on a real-time image captured by any first camera module.

It should be understood that the real-time image refers to the image currently being captured.

In one example, the effect parameters of the at least one first camera module can be obtained through an image processing unit connected to the at least one first camera module. Specifically, the image taken by any first camera module can be input to the image processing unit connected thereto, and the image processing unit can obtain the effect parameters of the first camera module by processing the input image. For example, the image processing unit can obtain the effect parameters such as exposure time, exposure gain, white balance coefficient, color correction matrix, sharpening coefficient, etc. by performing automatic exposure, automatic white balance, color correction, sharpening and other processing on the input image.

S720: Input the effect parameters of the at least one first camera module into the trained mathematical model to obtain the effect parameters of the second camera module output by the trained mathematical model.

The trained mathematical model is obtained by training the mathematical model. When the mathematical model is trained, the input of the mathematical model is the sample image taken by the at least one first camera module, and the supervision data of the mathematical model is the effect parameter of the second camera model obtained according to the sample image taken by the second camera module. It can be understood that the trained mathematical model can characterize the relationship between the effect parameter of the at least one first camera module and the effect parameter of the second camera module.

Regarding how to train the mathematical model and obtain the trained mathematical model, please refer to the previous description, which will not be repeated here. It can be understood that the at least one first camera module here can be the camera module 1 and the camera module 2 described in Figures 5 and 6, and the second camera module can be the camera module 3.

S730, controlling the second camera module to use the effect parameters of the second camera module to capture an image, and/or using the effect parameters of the second camera module to perform image processing on the real-time image captured by the second camera module.

The effect parameters of the second camera module may include only parameters for image capture, may include only parameters for image processing, or may include both parameters for image capture and parameters for image processing. If the effect parameters of the second camera module include parameters for image capture, the second camera module may use the parameters for image capture to capture images. If the effect parameters of the second camera module include parameters for image processing, the parameters for image processing may be used to process the real-time image captured by the second camera module.

For example, when the effect parameters of the second camera module include one or more of exposure time, exposure gain, or white balance coefficient, the second camera module can use the exposure time, the exposure gain, or one or more of the white balance coefficient to capture the image. When the effect parameters of the second camera module include a color correction matrix and/or a sharpening coefficient, the color correction matrix and/or the sharpening coefficient can be used to color correct and/or sharpen the real-time image captured by the second camera module.

In summary, according to the image processing method provided by the present application, for multiple camera modules having a common viewing area, only the effect parameters of some (i.e., one or more) camera modules are calculated, and the effect parameters of other camera modules are derived using a pre-trained mathematical model. Then, the other camera modules use the derived effect parameters to take images or use the derived effect parameters to process other camera modules. The method does not need to configure corresponding hardware resources or software resources for other camera modules to calculate effect parameters according to the captured images, and can improve the image effect quality of multiple cameras under the constraints of power consumption, load and hardware cost.

FIG8 is another schematic flow chart of the image processing method provided by the present application. Referring to FIG8 , camera modules 801 to 804 are camera modules in the same terminal device, and camera module 801 and camera module 802, camera module 803 and camera module 804 all have a common viewing area, and the viewing angles of camera module 802, camera module 803 and camera module 804 are not completely the same. The image signal output by camera module 801 is input into ISP 811; ISP 811 can obtain the effect parameters of camera module 801 by processing the input image signal. Then, the effect parameters of camera module 801 are respectively input into the trained mathematical model A, the trained mathematical model B and the trained mathematical model C, and the effect parameters of camera module 802, camera module 803 and camera module 804 can be obtained. Then, camera module 802, camera module 803 and camera module 804 can respectively use their respective effect parameters to perform image processing on the images they have captured, so as to obtain usable images. Alternatively, the image algorithm module may respectively use the effect parameters of the camera module 802 , the camera module 803 , and the camera module 804 to process the image signals output by the corresponding camera modules to obtain a usable image.

In this method, only the effect parameters of camera module 801 can be calculated through ISP, and the effect parameters of camera module 802, camera module 803 and camera module 804 can be derived using the trained mathematical model and sent to camera module 802, camera module 803 and camera module 804 or transmitted to the image algorithm module 821 of the image for use, thereby achieving the migration and reuse of effect parameters, thereby saving ISP resources and ensuring the effects of multiple camera modules.

It should be understood that the trained mathematical model A can be obtained by training the mathematical model according to the effect parameters of the camera module 801 and the camera module 802 in the same environment; the trained mathematical model B can be obtained by training the mathematical model according to the effect parameters of the camera module 801 and the camera module 803 in the same environment; the trained mathematical model C can be obtained by training the mathematical model according to the effect parameters of the camera module 801 and the camera module 804 in the same environment. The trained mathematical model A, the trained mathematical model B, and the trained mathematical model C can be of the same or different types, and this application does not limit this.

It should be understood that the camera module 801 shown in FIG. 8 may be at least one first camera module in the method 700 , and the camera module 802 , the camera module 803 or the camera module 804 may be a second camera module in the method 700 .

The above describes in detail the image processing method provided by the embodiment of the present application. The following will describe in detail the device embodiment of the present application in conjunction with Figures 9 and 10. It should be understood that the terminal device in the embodiment of the present application can execute the image processing method in the aforementioned embodiment of the present application. The specific working process of the terminal device can refer to the corresponding process in the aforementioned method embodiment.

FIG9 is a schematic block diagram of a terminal device provided in an embodiment of the present application. It should be understood that the terminal device 900 can execute the image processing method shown in FIG7. The terminal device 900 includes: a processing unit 910. The terminal device 900 also includes at least one first camera module and a second camera module, and the at least one first camera module and the second camera module have a common viewing area.

The processing unit 910 is used to: obtain effect parameters of the at least one first camera module based on the real-time image captured by the at least one first camera module; input the effect parameters of the at least one first camera module into a trained mathematical model to obtain effect parameters of the second camera module, wherein the trained mathematical model is obtained by training the mathematical model, the input of the mathematical model is the sample image captured by the at least one first camera module, and the supervision data of the mathematical model is the effect parameters of the second camera model obtained based on the sample image captured by the second camera module; control the second camera module to capture images using the effect parameters of the second camera module, and/or perform image processing on the real-time image captured by the second camera module using the effect parameters of the second camera module.

Optionally, the effect parameter includes one or more of the following: exposure time, exposure gain, white balance coefficient, color correction matrix, or sharpening coefficient.

Optionally, the terminal device is a mixed reality MR device.

Optionally, the processing unit includes at least one image processing unit, and the at least one first camera module is connected to the at least one image processing unit, and the at least one image processing unit is used to: perform image processing on the real-time image taken by the at least one first camera module to obtain effect parameters of the at least one first camera module.

Optionally, the processing unit further includes an image algorithm module, configured to perform image processing on the real-time image captured by the second camera module using part or all of the effect parameters of the second camera module.

It should be noted that the terminal device 900 is implemented in the form of functional units. The term "unit" here can be implemented in the form of software and/or hardware, and is not specifically limited to this.

For example, a "unit" may be a software program, a hardware circuit, or a combination of the two to implement the above functions. The hardware circuit may include an application An application specific integrated circuit (ASIC), electronic circuit, processor (e.g., shared processor, dedicated processor or group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuits and/or other suitable components supporting the described functions.

Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

Fig. 10 shows a schematic diagram of the structure of a terminal device provided by the present application. The terminal device 1000 can be used to implement the method described in the above method embodiment.

The terminal device 1000 includes a plurality of camera modules 1006 and one or more processors 1001. The plurality of camera modules may include at least one first camera module and a second camera module as described above. The one or more processors 1001 may support the image processing method in the method embodiment implemented by the terminal device 1000. The processor 1001 may include a general-purpose processor and/or a dedicated processor. For example, the processor 1001 may include one or more of the following: a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, such as discrete gates, transistor logic devices or discrete hardware components.

The processor 1001 can be used to control the terminal device 1000, execute software programs, and process data of the software programs.

The terminal device 1000 may further include a communication unit 1005 for implementing input (reception) and output (transmission) of signals.

For example, the terminal device 1000 may be a chip, the communication unit 1005 may be an input and/or output circuit of the chip, or the communication unit 1005 may be a communication interface of the chip, and the chip may be a component of a terminal device or other terminal devices.

For another example, the terminal device 1000 may be a terminal device, the communication unit 1005 may be a transceiver of the terminal device, or the communication unit 1005 may be a transceiver circuit of the terminal device.

The terminal device 1000 may include one or more memories 1002 on which a program 1004 is stored. The program 1004 can be executed by the processor 1001 to generate instructions 1003, so that the processor 1001 executes the image processing method described in the above method embodiment according to the instructions 1003.

Optionally, data may also be stored in the memory 1002 .

Optionally, the processor 1001 may also read data stored in the memory 1002 . The data may be stored at the same storage address as the program 1004 , or may be stored at a different storage address from the program 1004 .

The processor 1001 and the memory 1002 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of a terminal device.

Exemplarily, the memory 1002 may be used to store a program 1004 related to the image processing method provided in an embodiment of the present application, and the processor 1001 may be used to execute the program 1004 related to the image processing method stored in the memory 1002 .

Optionally, the processor 1001 may be configured to execute various steps/functions of the embodiment shown in FIG. 7 .

The present application also provides a computer program product, which, when executed by the processor 1001, implements the image processing method of any method embodiment in the present application.

The computer program product may be stored in the memory 1002 , for example, a program 1004 , which is converted into an executable target file that can be executed by the processor 1001 after preprocessing, compiling, assembling, and linking.

The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the image processing method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

Optionally, the computer-readable storage medium is, for example, memory 1002. Memory 1002 may be a volatile memory or a non-volatile memory, or memory 1002 may include both volatile memory and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), Enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

It should be noted that although the terminal device 1000 shown in FIG. 10 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the terminal device 1000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the terminal device 1000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the terminal device 1000 may also include only the devices necessary for implementing the embodiments of the present application, and does not necessarily include all the devices shown in FIG. 10.

The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

It should be understood that the term "and/or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character "/" in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and/or" relationship. Please refer to the context for specific understanding.

In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple.

It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

An image processing method, characterized in that it is applied to a terminal device, the terminal device includes at least one first camera module and a second camera module, the at least one first camera module and the second camera module have a common viewing area, and the method includes:

Acquiring effect parameters of the at least one first camera module according to the real-time image captured by the at least one first camera module;

Inputting the effect parameters of the at least one first camera module into a trained mathematical model to obtain the effect parameters of the second camera module, wherein the trained mathematical model is obtained by performing model training on the mathematical model, the input of the mathematical model is a sample image taken by the at least one first camera module, and the supervision data of the mathematical model is the effect parameters of the second camera model obtained according to the sample image taken by the second camera module;

Control the second camera module to use the effect parameters of the second camera module to capture images, and/or use the effect parameters of the second camera module to perform image processing on the real-time image captured by the second camera module.
The method according to claim 1, characterized in that the effect parameters include one or more of the following: exposure time, exposure gain, white balance coefficient, color correction matrix, or sharpening coefficient.
The method according to claim 1 or 2, characterized in that the terminal device is a mixed reality MR device.
The method according to any one of claims 1 to 3, characterized in that the at least one first camera module is connected to at least one image processing unit, the at least one image processing unit is a hardware resource, and the obtaining of the effect parameters of the at least one first camera module according to the real-time image captured by the at least one first camera module comprises:

The at least one image processing unit is controlled to perform image processing on the real-time image captured by the at least one first camera module to obtain effect parameters of the at least one first camera module.
The method according to any one of claims 1 to 4, characterized in that the using the effect parameter of the second camera module to perform image processing on the real-time image captured by the second camera module comprises:

The image algorithm module is controlled to use part or all of the effect parameters to perform image processing on the real-time image taken by the second camera module, and the image algorithm module is a software resource.
A terminal device, characterized in that it comprises: at least one first camera module, a second camera module and a processing unit, wherein the at least one first camera module and the second camera module have a common viewing area;

The processing unit is used for:

Acquiring effect parameters of the at least one first camera module according to the real-time image captured by the at least one first camera module;

Inputting the effect parameters of the at least one first camera module into a trained mathematical model to obtain the effect parameters of the second camera module, wherein the trained mathematical model is obtained by performing model training on the mathematical model, the input of the mathematical model is a sample image taken by the at least one first camera module, and the supervision data of the mathematical model is the effect parameters of the second camera model obtained according to the sample image taken by the second camera module;

Control the second camera module to use the effect parameters of the second camera module to capture images, and/or use the effect parameters of the second camera module to perform image processing on the real-time image captured by the second camera module.
The terminal device as described in claim 6 is characterized in that the effect parameters include one or more of the following: exposure time, exposure gain, white balance coefficient, color correction matrix, or sharpening coefficient.
The terminal device as described in claim 6 or 7 is characterized in that the terminal device is a mixed reality MR device.
The terminal device according to any one of claims 6 to 8, characterized in that the processing unit includes at least one image processing unit, the at least one first camera module is connected to the at least one image processing unit, and the at least one image processing unit is used to:

Perform image processing on the real-time image captured by the at least one first camera module to obtain effect parameters of the at least one first camera module.
The terminal device according to any one of claims 6 to 9, characterized in that the processing unit further comprises an image algorithm module, which is used to:

The real image taken by the second camera module is processed by using part or all of the effect parameters of the second camera module. The image is processed.
A terminal device, characterized by comprising:

One or more processors, memories, at least one first camera module and a second camera module, wherein the at least one first camera module and the second camera module have a common viewing area;

The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the terminal device to execute the method according to any one of claims 1 to 5.
A chip system, characterized in that the chip system is applied to a terminal device, the chip system comprises one or more processors, and the processor is used to call computer instructions so that the terminal device executes the method as described in any one of claims 1 to 5.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 5.
A computer program product, characterized in that the computer program product comprises computer program code, and when the computer program code is executed by a processor, the processor executes the method according to any one of claims 1 to 5.