CN115334239A

CN115334239A - Method for fusing photographing of front camera and photographing of rear camera, terminal equipment and storage medium

Info

Publication number: CN115334239A
Application number: CN202210955202.6A
Authority: CN
Inventors: 张培龙; 周春萌; 朱众微
Original assignee: Hisense Mobile Communications Technology Co Ltd
Current assignee: Hisense Mobile Communications Technology Co Ltd
Priority date: 2022-08-10
Filing date: 2022-08-10
Publication date: 2022-11-11
Anticipated expiration: 2042-08-10
Also published as: CN115334239B

Abstract

The application discloses a method for fusing photographing of front and rear cameras, terminal equipment and a storage medium, which are used for solving the problem that the front and rear cameras in a three-dimensional space cannot realize fused photographing in the related art. Acquiring a figure image through a front camera, acquiring a background image and a background depth image through a rear camera, determining relevant image fusion parameters for image fusion according to three-dimensional space coordinates of the figure, a figure region coefficient matrix and the figure image, and finally fusing the figure image and the background image by using a third figure region coefficient matrix to obtain a fused image. Compared with the prior art, the method and the device can only determine the related image fusion parameters according to the portrait coordinates acquired by the front-facing camera in the set two-dimensional space, realize the image fusion in the three-dimensional space, are more real in the shielding relation and the size proportion of the portrait to the surrounding environment objects, have better fusion effect, can artificially adjust the relative distance between the portrait and the background, and improve the use experience of users.

Description

Method for fusing front camera and rear camera shooting, terminal equipment and storage medium

Technical Field

The application belongs to the technical field of image processing, and particularly relates to a method for fusing photographing of front and rear cameras, terminal equipment and a storage medium.

Background

At present, the photographing function of the mobile terminal is becoming more and more abundant, and the single camera is developed to double-shot, three-shot, four-shot and the like. At present, a front shooting and a rear double shooting become basic configurations of a mobile terminal, the front shooting is generally used for shooting a portrait, and the rear shooting can be used for shooting a scene and also shooting a person. The traditional cameras generally use front and back shooting independently, but recently manufacturers start to fuse front and back shooting images, and the front shooting image and the back shooting image are spliced or fused into a pair of images to achieve the function of double-scene shooting.

In the related method for shooting by combining the front shot and the back shot, only two-dimensional space combination can be carried out, the portrait area extracted from the front shot can only adjust the two-dimensional coordinate position of the portrait area in the background image of the back shot, but cannot adjust the depth direction. Meanwhile, the illumination directions of the light source of the front shot image and the light source of the back shot image are possibly inconsistent, so that the illumination of the portrait of the fused image is inconsistent with the illumination of the whole background.

Therefore, how to realize fusion shooting by front and back cameras in a three-dimensional space is a concern in the industry.

Disclosure of Invention

The application aims to provide a method for fusing front and back camera photographing, terminal equipment and a storage medium, and is used for solving the problem that the front and back cameras in a three-dimensional space cannot be fused for photographing in the related technology.

In a first aspect, the present application provides a method for fusion of front and back camera photographing, where the method includes:

determining a figure image according to data acquired by the front camera, and determining a background image and a background depth image according to the acquisition of background data by the rear camera;

carrying out person region segmentation on the person image to obtain a person region coefficient matrix;

determining three-dimensional space coordinates of a person in the person image in the background image based on the background depth image and triggering a fusion instruction, wherein the three-dimensional space coordinates comprise a depth coordinate, a transverse coordinate and a longitudinal coordinate;

in response to the fusion instruction, scaling the character area coefficient matrix and the character image according to the scaling ratio, and converting the character area coefficient matrix into a second character area coefficient matrix and a second character image corresponding to the focal length and the visual field corresponding to the rear camera;

determining a third character image and a third character area coefficient matrix corresponding to the three-dimensional space coordinate of the character to be adjusted to the corresponding three-dimensional space coordinate according to the three-dimensional space coordinate of the character, the second character area coefficient matrix and the second character image;

and fusing the third person image and the background image by using a third person region coefficient matrix to obtain a fused image.

In one possible embodiment, the determining the background image and the background depth image includes:

acquiring two frames of background images by using a dual-purpose rear camera, and determining a background depth image according to the two frames of background images based on a triangulation principle; or alternatively

And determining a background depth image by adopting a TOF depth rear camera, and determining a background image by adopting an RGB color rear camera.

In one possible embodiment, the scaling is determined as follows:

wherein S represents the scaling, f ₁ Denotes the front focal length of the front camera, f ₂ Indicating the back focal length of the rear camera lens, z ₁ A depth coordinate, z, in the background image representing the set person ₂ Representing front camera acquisitionAnd the physical distance between the person and the front camera when the person images.

In one possible embodiment, determining a third image corresponding to the person adjusted to the corresponding three-dimensional space coordinates based on the second person region coefficient matrix and the second person image includes:

creating an initial third person image, and setting the value of a pixel point at any position in the initial third person image to be 0, wherein the row and column number of the initial third person image is the same as the row and column number of the background image;

if the abscissa of the pixel point at any position in the initial third character image is not less than x1 and not more than the smaller value of W2 and x1+ W1, and the ordinate of the pixel point at any position in the initial third character image is not less than y1 and not more than the smaller value of H2 and y1+ H1, assigning the value of the pixel point at the same position of the second character image to the pixel point at the same position in the initial third character image to obtain a third character image;

wherein x1 represents the horizontal coordinate of the set person in the background image, W1 represents the number of columns of the matrix corresponding to the second personal image, W2 represents the number of columns of the matrix corresponding to the initial third personal image, y1 represents the vertical coordinate of the set person in the background image, H1 represents the number of rows of the matrix corresponding to the second personal image, and H2 represents the number of rows of the matrix corresponding to the initial third personal image.

In one possible embodiment, determining a third person region coefficient matrix corresponding to the person adjusted to the corresponding three-dimensional space coordinates based on the three-dimensional space coordinates of the person, the second person region coefficient matrix, and the second person image includes:

creating an initial third human figure area coefficient matrix, and setting the value of the initial third human figure area coefficient matrix to be 0, wherein the row and column number of the initial third human figure area coefficient matrix is the same as the row and column number of the matrix corresponding to the background image;

if the abscissa of the element at any position in the initial third person region coefficient matrix is not less than x1 and not more than the smaller value of W2 and x1+ W1, the ordinate of the element at any position in the initial third person region coefficient matrix is not less than y1 and not more than the smaller value of H2 and y1+ H1, and the depth coordinate of the set person in the background image is not more than D1 (i, j), assigning the value of the element at the same position of the second person region coefficient matrix to the element at the same position of the initial third person region coefficient matrix to obtain a third person region coefficient matrix;

wherein x1 represents the horizontal coordinate of the set person in the background image, W1 represents the column number of the matrix corresponding to the second person image, W2 represents the column number of the initial third person region coefficient matrix, y1 represents the vertical coordinate of the set person in the background image, H1 represents the row number of the matrix corresponding to the second person image, H2 represents the row number of the initial third person region coefficient matrix, and D1 (i, j) represents the value of the pixel point at the (i, j) position in the background depth image.

In a possible implementation manner, the fusing the third person image and the background image by using a third person region coefficient matrix to obtain a fused image includes:

determining a fused image by adopting the following formula:

P ₅ ＝β×P ₄ +(1-β)×P ₂

wherein, P ₅ Representing the fused image, beta represents the fusion coefficient, P ₄ Representing a fused image of a person, P ₂ Representing a background image.

In one possible embodiment, the method further comprises:

displaying the fusion image on a display interface, and determining a spatial depth sliding bar corresponding to the adjustment range of the depth coordinate of the person in the background image according to the background depth image;

in response to the sliding instruction, determining the distance between the person in the preset depth range adjusting fusion image and a rear camera of the mobile terminal to obtain the updated depth coordinate of the person;

responding to the dragging instruction, determining the position of a person area in the preset area range adjusting fusion image, and determining the updated transverse coordinate and the updated longitudinal coordinate of the person;

taking the three-dimensional space coordinate after the character is updated as the three-dimensional space coordinate of the character, and triggering the fusion instruction again; the updated three-dimensional space coordinates of the character comprise updated depth coordinates, abscissa and ordinate.

In one possible implementation, determining the person image according to the data acquired by the front camera comprises:

acquiring an original figure image by adopting a front camera;

processing the background image by adopting a deep learning relighting model to determine a lighting position;

and generating a person image according to the illumination position and the original person image.

In a second aspect, the present application further provides a device for photographing and fusing a front camera and a rear camera, the device including:

the image determining module is configured to determine a person image according to data acquired by the front camera, and determine a background image and a background depth image according to the acquisition of background data by the rear camera;

the person region coefficient matrix determining module is configured to perform person region segmentation on the person image to obtain a person region coefficient matrix;

a three-dimensional space coordinate determination module configured to determine three-dimensional space coordinates of a person in the person image in the background image based on the background depth image and trigger a fusion instruction, wherein the three-dimensional space coordinates include a depth coordinate, a horizontal coordinate and a vertical coordinate;

the data conversion module is configured to respond to a fusion instruction, zoom the person region coefficient matrix and the person image according to a zoom ratio, and convert the person region coefficient matrix and the person image into a second person region coefficient matrix and a second person image which correspond to a focal length and a visual field which correspond to the rear camera;

the fusion parameter determining module is configured to determine a third person image and a third person area coefficient matrix corresponding to the three-dimensional space coordinate of the person to be adjusted according to the three-dimensional space coordinate of the person, the second person area coefficient matrix and the second person image;

and the image fusion module is configured to fuse the third person image and the background image by using a third person region coefficient matrix to obtain a fused image.

In one possible implementation, the determining the background image and the background depth image is performed, and the image determination module is configured to:

acquiring two frames of background images by using a dual-purpose rear camera, and determining a background depth image according to the two frames of background images based on a triangulation principle; or

In one possible embodiment, the scaling is determined as follows:

wherein S represents the scaling, f ₁ Indicating the front focal length of the front camera, f ₂ Indicating the back focal length of the rear camera lens, z ₁ Indicating the depth coordinate, z, of the set person in the background image ₂ And the physical distance between the person and the front camera when the front camera captures the person image is represented.

In one possible embodiment, determining a third image of the person corresponding to the adjustment of the person to the corresponding three-dimensional space coordinates based on the second person region coefficient matrix and the second person image is performed, and the fusion parameter determination module is configured to:

In one possible embodiment, determining a third human figure region coefficient matrix corresponding to the person adjusted to the corresponding three-dimensional space coordinate is performed based on the three-dimensional space coordinate of the person, the second human figure region coefficient matrix, and the second human figure image, and the fusion parameter determination module is configured to:

In one possible embodiment, the fusing the third person image with the background image using a third person region coefficient matrix is performed to obtain a fused image, and the fusing module is configured to:

determining a fused image using the following formula:

P ₅ ＝β×P ₄ +(1-β)×P ₂

wherein, P ₅ Denotes a fusion image, beta denotes a fusion coefficient, P ₄ Representing a fused image of a person, P ₂ Representing a background image.

In a possible embodiment, the apparatus further comprises:

the display module is configured to display the fusion image on a display interface and determine a spatial depth sliding bar corresponding to the adjustment range of the depth coordinate of the person in the background image according to the background depth image;

the first coordinate determination module is configured to respond to a sliding instruction, determine the distance between a person in the adjusted fusion image and a rear camera of the mobile terminal within a preset depth range, and obtain an updated depth coordinate of the person;

the second coordinate determination module is configured to respond to the dragging instruction, determine the position of a person area in the preset area range adjusting fusion image, and determine the updated transverse coordinate and longitudinal coordinate of the person;

the fusion instruction triggering module is configured to take the updated three-dimensional space coordinate of the character as the three-dimensional space coordinate of the character and re-trigger the fusion instruction; the updated three-dimensional space coordinates of the character comprise updated depth coordinates, abscissa and ordinate.

In a possible embodiment, the determining of the image of the person from the data acquired by the front camera is performed, the image determination module being configured to:

acquiring an original figure image by adopting a front camera;

In a third aspect, an embodiment of the present application provides a terminal device, including:

a display for displaying the acquired image;

a memory for storing executable instructions of the processor;

a processor, configured to execute the executable instructions to implement any one of the front and back camera photographing fusion methods provided in the first aspect of the present application.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where instructions in the computer-readable storage medium, when executed by a processor of a terminal device, enable the terminal device to perform any one of the method for front and back camera shooting fusion provided in the first aspect of the present application.

In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method for front-back camera photographing fusion as described in any one of the aspects provided in the first aspect of the present application is implemented.

The technical scheme provided by the embodiment of the application at least has the following beneficial effects:

according to the method and the device, the figure image is obtained through the front camera, the background image and the background depth image are obtained through the rear camera, relevant image fusion parameters for image fusion are determined according to the three-dimensional space coordinate of the figure, the figure region coefficient matrix and the figure image, and finally the figure image and the background image are fused through the third figure region coefficient matrix to obtain a fused image. Compared with the prior art that the related image fusion parameters can be determined only according to the portrait coordinates acquired by the front-facing camera in the set two-dimensional space, the image fusion in the three-dimensional space is realized, the shielding relation and the size proportion of the portrait to the surrounding environment are more real, the fusion effect is better, meanwhile, when a user uses the scheme provided by the application, the relative distance between the portrait and the background can be adjusted manually, and the use experience of the user is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application. On the basis of the common knowledge in the field, the above preferred conditions can be combined randomly to obtain the preferred embodiments of the application.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a terminal according to an embodiment of the present disclosure;

fig. 2 is a schematic diagram of a software architecture of a terminal according to an embodiment of the present application;

fig. 3 is a schematic view of an application scenario of a method for fusion of front and rear camera photographing provided in an embodiment of the present application;

fig. 4 is a schematic view of an application interface for a user to start a camera for taking a picture according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a method for fusing photographing of front and rear cameras provided in the embodiment of the present application;

fig. 6 is a schematic diagram of a type of a rear camera of a terminal device according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a person image provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of a background image provided in an embodiment of the present application;

fig. 9 is a flowchart illustrating the step 505 of determining a third person image according to the embodiment of the present application;

fig. 10 is a schematic diagram illustrating that a pixel point at any position in an initial third human image is located in a human region of a background image according to an embodiment of the present application;

fig. 11 is a schematic diagram illustrating that a pixel point at any position in an initial third human image is located in a human region of a background image according to an embodiment of the present application;

fig. 12 is a schematic flowchart illustrating a process of determining the coefficient matrix of the third person region in step 505 according to the embodiment of the present application;

FIG. 13 is a schematic diagram of a person image in front of a background image according to an embodiment of the present disclosure;

FIG. 14 is a schematic diagram of a person image behind a background image according to an embodiment of the present application;

FIG. 15 is a schematic flowchart of relative distances between a person and a background according to an embodiment of the present disclosure;

FIG. 16 is a schematic diagram of a display interface provided by an embodiment of the present application;

fig. 17 is a schematic diagram illustrating an effect of the human image in front of the background image according to the embodiment of the present application;

fig. 18 is a schematic diagram illustrating an effect of a human image located behind a background image according to an embodiment of the present application;

fig. 19 is a schematic flowchart of determining a person image according to data acquired by a front camera in step 501 according to an embodiment of the present application;

fig. 20 is a schematic structural diagram of a device for photographing and fusing front and rear cameras provided in the embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. The embodiments described are some, but not all embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

Also, in the description of the embodiments of the present application, "/" indicates an inclusive meaning unless otherwise specified, for example, a/B may indicate a or B; "and/or" in the text is only an association relationship describing an associated object, and means that three relationships may exist, for example, a and/or B may mean: three cases of a alone, a and B both, and B alone exist, and in addition, "a plurality" means two or more than two in the description of the embodiments of the present application.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as implying or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first", "second", may explicitly or implicitly include one or more of that feature.

First, fig. 1 shows a schematic structural diagram of a terminal 100.

The following describes an embodiment specifically by taking the terminal 100 as an example. It should be understood that the terminal 100 shown in fig. 1 is merely an example, and that the terminal 100 may have more or fewer components than shown in fig. 1, may combine two or more components, or may have a different configuration of components. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

A block diagram of a hardware configuration of the terminal 100 according to an exemplary embodiment is exemplarily shown in fig. 1. As shown in fig. 1, the terminal 100 includes: a Radio Frequency (RF) circuit 110, a memory 120, a display unit 130, a camera 140, a sensor 150, an audio circuit 160, a Wireless Fidelity (Wi-Fi) module 170, a processor 180, a bluetooth module 181, and a power supply 190.

The RF circuit 110 may be used for receiving and transmitting signals during information transmission and reception or during a call, and may receive downlink data of a base station and then send the downlink data to the processor 180 for processing; the uplink data may be transmitted to the base station. In general, RF circuitry includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like.

The memory 120 may be used to store software programs and data. The processor 180 performs various functions of the terminal 100 and data processing by executing software programs or data stored in the memory 120. The memory 120 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. The memory 120 stores an operating system that enables the terminal 100 to operate. The memory 120 may store an operating system and various application programs, and may also store program codes for performing the methods described in the embodiments of the present application.

The display unit 130 may be used to receive input numeric or character information and generate signal input related to user settings and function control of the terminal 100, and particularly, the display unit 130 may include a touch screen 131 disposed on the front surface of the terminal 100 and may collect touch operations of a user thereon or nearby, such as starting a camera, closing the camera, clicking a button, dragging a scroll box, and the like.

The display unit 130 may also be used to display a Graphical User Interface (GUI) of information input by or provided to the user and various menus of the terminal 100. Specifically, the display unit 130 may include a display screen 132 disposed on the front surface of the terminal 100. The display screen 132 may be configured in the form of a liquid crystal display, a light emitting diode, or the like. The display unit 130 may be configured to display an interface for a user to start a camera for photographing as described in this application.

The touch screen 131 may cover the display screen 132, or the touch screen 131 and the display screen 132 may be integrated to implement the input and output functions of the terminal 100, and after the integration, the touch screen may be referred to as a touch display screen for short. In the present application, the display unit 130 may display the application programs and the corresponding operation steps.

The camera 140 may be used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing elements convert the light signals into electrical signals which are then passed to the processor 180 for conversion into digital image signals.

The terminal 100 may further comprise at least one sensor 150, such as an acceleration sensor 151, a distance sensor 152, a fingerprint sensor 153, a temperature sensor 154. The terminal 100 may also be configured with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, motion sensor, etc.

Audio circuitry 160, speaker 161, and microphone 162 may provide an audio interface between a user and terminal 100. The audio circuit 160 may transmit the electrical signal converted from the received audio data to the speaker 161, and convert the electrical signal into a sound signal for output by the speaker 161. The terminal 100 may also be provided with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, which is then output to the RF circuit 110 for transmission to, for example, another terminal or to the memory 120 for further processing. In this application, the microphone 162 may capture the voice of the user.

Wi-Fi belongs to a short-distance wireless transmission technology, and the terminal 100 can help a user to receive and send e-mails, browse web pages, access streaming media and the like through the Wi-Fi module 170, and provides wireless broadband internet access for the user.

The processor 180 is a control center of the terminal 100, connects various parts of the entire terminal using various interfaces and lines, and performs various functions of the terminal 100 and processes data by running or executing software programs stored in the memory 120 and calling data stored in the memory 120. In some embodiments, processor 180 may include one or more processing units; the processor 180 may also integrate an application processor, which mainly handles operating systems, user interfaces, applications, etc., and a baseband processor, which mainly handles wireless communications. It will be appreciated that the baseband processor described above may not be integrated into the processor 180. In the present application, the processor 180 may run an operating system, an application program, a user interface display, and a touch response, as well as the methods described in the embodiments of the present application. Further, the processor 180 is coupled with the display unit 130.

And the bluetooth module 181 is configured to perform information interaction with other bluetooth devices having a bluetooth module through a bluetooth protocol. For example, the terminal 100 may establish a bluetooth connection with a wearable terminal device (e.g., a smart watch) having a bluetooth module via the bluetooth module 181, so as to perform data interaction.

The terminal 100 also includes a power supply 190 (e.g., a battery) to power the various components. The power supply may be logically coupled to the processor 180 through a power management system to manage charging, discharging, and power consumption functions through the power management system. The terminal 100 may also be configured with power buttons for powering the terminal on and off, and for locking the screen.

Fig. 2 is a block diagram of a software configuration of the terminal 100 according to the embodiment of the present application.

The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system may be divided into four layers, an application layer, an application framework layer, an Android runtime (Android runtime) and system library, and a kernel layer, from top to bottom, respectively.

The application layer may include a series of application packages.

As shown in fig. 2, the application package may include applications such as camera, gallery, calendar, phone call, map, navigation, WLAN, bluetooth, music, video, short message, etc.

The application framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions.

As shown in fig. 2, the application framework layer can be divided into a java side and a native side, wherein the java side includes a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, an application manager, and the like.

As shown in FIG. 2, the application framework layers may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and the like.

The window manager is used for managing window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, short messages, etc.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the interface including the front camera of the control list and the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide a communication function of the terminal 100. Such as management of call status (including on, off, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

The notification manager allows the application to display notification information (e.g., message digest of short message, message content) in the status bar, can be used to convey notification-type messages, and can automatically disappear after a short dwell without user interaction. Such as a notification manager used to inform download completion, message alerts, etc. The notification manager may also be a notification that appears in the form of a chart or scroll bar text at the top status bar of the system, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. For example, text information is prompted in the status bar, a prompt tone is given, the terminal vibrates, and an indicator light flashes.

And the service of the native side is positioned at the native side of the application program framework layer and is adjacent to the system library.

The Android Runtime comprises a core library and a virtual machine. The Android runtime is responsible for scheduling and managing an Android system.

The core library comprises two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes java files of the application layer and the application framework layer as binary files. The virtual machine is used for performing the functions of object life cycle management, stack management, thread management, safety and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface managers (surface managers), media Libraries (Media Libraries), three-dimensional graphics processing Libraries (e.g., openGL ES), 2D graphics engines (e.g., SGL), and camera services, among others.

The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc.

The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

A 2D (one way of animation) graphics engine is a drawing engine for 2D drawing.

The camera service is used for common logical camera objects, and corresponding parameter information and the like are configured for the camera objects.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

The following describes exemplary workflow of the terminal 100 software and hardware in connection with capturing a photo scene.

When the touch screen 131 receives a touch operation, a corresponding hardware interrupt is issued to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, a time stamp of the touch operation, and other information). The raw input events are stored at the kernel layer. And the application program framework layer acquires the original input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and taking a control corresponding to the click operation as a control of a camera application icon as an example, the camera application calls an interface of an application framework layer, starts the camera application, further starts a camera drive by calling a kernel layer, and captures a still image or a video through the camera 140.

The terminal 100 in this embodiment may be a mobile phone, a tablet computer, a wearable device, a notebook computer, a television, and other terminal devices having a front camera and a rear camera.

The method for fusing photographing of front and back cameras provided by the present application will be described below with reference to embodiments.

The inventive concept of the present application can be summarized as follows: firstly, determining a character image according to data acquired by a front camera, determining a background image and a background depth image according to the acquisition of background data by a rear camera, carrying out character region segmentation on the character image to obtain a character region coefficient matrix, determining a three-dimensional space coordinate of a character in the character image in the background image and triggering a fusion instruction based on the background depth image, wherein the three-dimensional space coordinate comprises a depth coordinate, a horizontal coordinate and a vertical coordinate, responding to the fusion instruction, scaling the character region coefficient matrix and the character image according to a scaling ratio, converting the scaled character region coefficient matrix into a second character region coefficient matrix and a second character image corresponding to a focal length and a visual field corresponding to the rear camera, determining a third character image and a third character region coefficient matrix corresponding to the adjusted character to the corresponding three-dimensional space coordinate according to the character three-dimensional space coordinate, the second character region coefficient matrix and the second character image, and fusing the third character image and the background image by using the third character region coefficient matrix to obtain a fused image.

To sum up, the depth map is obtained through the rear camera in the embodiment of the application, and according to the three-dimensional space coordinate of the portrait obtained by the set front camera, the second person area coefficient matrix and the second person image, the relevant image fusion parameters for image fusion are determined, compared with the prior art that the relevant image fusion parameters can only be determined according to the portrait coordinate obtained by the set front camera in the two-dimensional space, the image fusion in the three-dimensional space is realized, the shielding relation and the size proportion of the portrait to the surrounding environment are more real, the fusion effect is better, and meanwhile, when a user uses the scheme provided by the application, the relative distance between the portrait and the background can be adjusted manually, and the use experience of the user is improved.

After the main inventive concepts of the embodiments of the present application are introduced, some simple descriptions are provided below for application scenarios to which the technical solutions of the embodiments of the present application can be applied, and it should be noted that the application scenarios described below are only used for describing the embodiments of the present application and are not limited. In specific implementation, the technical scheme provided by the embodiment of the application can be flexibly applied according to actual needs.

Fig. 3 is a schematic view of a scene in which a user uses a terminal device to take a picture according to an embodiment of the present disclosure. The figure includes: a user, a terminal device and a subject of photographing. In a scene that a user uses a terminal device to take a picture, the method provided by the embodiment of the application can acquire the relevant data required by generating the image by using the terminal device, and realize the image fusion of the portrait and the background according to the relevant data.

Only a single terminal device is detailed in the description in the present application, but it should be understood by those skilled in the art that the illustrated user, terminal device and object of photographing are intended to represent the operations of the user, terminal device and object of photographing to which the technical aspects of the present application relate. The detailed description of a single terminal device is at least for convenience of description and does not imply a limitation on the number, type, or location of terminal devices. It should be noted that the underlying concepts of the example embodiments of the present application may not be altered if additional modules are added or removed from the illustrated environments.

Certainly, the use scenario provided in the embodiment of the present application is not limited to the application scenario shown in fig. 3, and may also be used in other possible application scenarios, and the embodiment of the present application is not limited.

Referring to fig. 4, a schematic view of an interface for a user to start a camera for taking a picture is provided in the embodiment of the present application. The interface is an interface with a camera shooting function, and comprises functions of shooting, portrait and the like, the terminal equipment at least comprises a front camera and a rear camera, the front camera and the rear camera are adopted for carrying out image acquisition on related objects, and the image fusion of the portrait and the background is realized through the method provided by the embodiment of the application.

Based on the above description, an embodiment of the present application provides a method for fusion of front and back camera shooting, a flowchart of the method is shown in fig. 5, and the method may include the following steps:

in step 501, a person image is determined according to data collected by the front camera, and a background image and a background depth image are determined according to the collection of background data by the rear camera.

In one possible implementation, the types of the front camera and the rear camera of the terminal device are various, and as shown in fig. 6, the following two cases are mainly included:

the first condition is as follows: the rear camera is a binocular rear camera.

And in the second case, the rear camera is a TOF depth rear camera and an RGB color rear camera.

In view of the above first situation, the determining the background image and the background depth image in step 501 may be implemented as:

two frames of background images are collected by using a dual-purpose rear camera, and a background depth image is determined according to the two frames of background images based on a triangulation principle. For example, a person image acquired by the front camera is an image P1, two background images acquired by the rear camera are P2 and P3, respectively, and a background depth image D1 is determined according to the two background images P2 and P3 based on a triangulation principle.

For the second case, the determining the background image and the background depth image in step 501 may be implemented as:

and determining a background depth image by adopting a TOF depth rear camera, and determining a background image by adopting an RGB color rear camera. For example, the image of a person obtained by the front camera is an image P1, the background depth image collected by the TOF depth rear camera is D1, and the background image obtained by the RGB color rear camera is P2.

Thus, in step 501, the embodiment of the present application acquires the person image P1, the background image P2, and the background depth image D1, and performs a step of fusing the subsequent person image and the background image.

In step 502, a person region of the person image is segmented to obtain a person region coefficient matrix.

In a possible implementation manner, the person region segmentation may use a Deep learning semantic segmentation model, such as deplab v3, unet, PSPnet, FCN, bisegnet, etc., or a Matting model Deep Image matching, indexent matching, adamanting, etc., and after using the above models to perform the person region segmentation, the embodiment of the present application obtains a person region coefficient matrix, such as named α, where α has the same row and column number as the person Image P1, and according to the person region coefficient matrix α and the person Image P1, the embodiment of the present application can obtain a person region Image F, where F = α × P1, and α ∈ [0,1].

In step 503, based on the background depth image, three-dimensional space coordinates of the person in the person image in the background image are determined and a fusion instruction is triggered, wherein the three-dimensional space coordinates include a depth coordinate, a horizontal coordinate and a vertical coordinate.

In one possible implementation, the person image P1 and the background image P2 are fused based on the background depth image D1, for example, the person image P1 is as shown in fig. 7, the background image P2 is as shown in fig. 8, wherein the black circles are the central point of the person image P1 and the central point of the background image P2, respectively, the central point of the person image P1 and the central point of the background image P2 are overlapped, and based on the background depth image D1, the embodiment of the present application is capable of determining three-dimensional space coordinates of the person in the person image in the background image, wherein the three-dimensional space coordinates include depth coordinates (i.e., the distance between the person in the person image and the background in the background image), horizontal coordinates (i.e., horizontal coordinates of the person in the background image), and vertical coordinates (i.e., vertical coordinates of the person in the background image), and triggering a fusion instruction.

It should be noted that the three-dimensional space coordinates are obtained when the human image P1 and the background image P2 are overlapped and fused, and if the fusion effect of the fused image is not good, the three-dimensional space coordinates may be adjusted according to the requirement in the embodiment of the present application.

In step 504, in response to the fusion instruction, the person region coefficient matrix and the person image are scaled according to the scaling ratio and converted into a second person region coefficient matrix and a second person image corresponding to the focal length and the view field corresponding to the rear camera.

In one possible embodiment, the scaling is determined as follows:

wherein S represents a scaling, f ₁ Denotes the front focal length of the front camera, f ₂ Indicating the back focal length of the rear camera lens, z ₁ Indicating the depth coordinate of the set person in the background image, z ₂ And the physical distance between the person and the front camera when the front camera captures the person image is represented.

It is added that the above-mentioned front focal length f ₁ And a back focal length f ₂ The two parameters are fixed parameters of the camera and can be obtained according to the camera specification. In the embodiment of the application, after the human figure area coefficient matrix and the human figure image are scaled according to the scaling, a new human figure area coefficient matrix α 'and a new human figure image P1' are obtained, that is, a second human figure area coefficient matrix α 'and a second human figure image P1' are obtained, and the number of rows and columns of the second human figure area coefficient matrix α 'and the second human figure image P1' are respectively H1 and W1.

In step 505, a third person image and a third person region coefficient matrix corresponding to the three-dimensional space coordinate of the person to be adjusted to the corresponding three-dimensional space coordinate are determined based on the three-dimensional space coordinate of the person, the second person region coefficient matrix, and the second person image.

In one possible embodiment, in step 505, a third image corresponding to the person adjusted to the corresponding three-dimensional space coordinate is determined according to the second person region coefficient matrix and the second person image, and the flowchart is shown in fig. 9, and includes the following contents:

in step 901, an initial third human figure image is created, the value of a pixel point at any position in the initial third human figure image is set to 0, and the number of rows and columns of the initial third human figure image is the same as the number of rows and columns of the background image.

In step 902, if the abscissa of the pixel at any position in the initial third human image is not less than x1 and not greater than the smaller value of W2 and x1+ W1, the ordinate of the pixel at any position in the initial third human image is not less than y1 and not greater than the smaller value of H2 and y1+ H1, assigning the value of the pixel at the same position in the second human image to the pixel at the same position in the initial third human image, and obtaining a third human image.

Where x1 denotes the horizontal coordinates of the set person in the background image, W1 denotes the number of columns of the matrix corresponding to the second personal image, W2 denotes the number of columns of the matrix corresponding to the initial third personal image, y1 denotes the vertical coordinates of the set person in the background image, H1 denotes the number of rows of the matrix corresponding to the second personal image, and H2 denotes the number of rows of the matrix corresponding to the initial third personal image.

For example, a new person image, that is, an initial third person image is created, the value of the pixel at any position in the initial third person image is set to 0, and the number of rows and columns of the initial third person image is the same as the number of rows and columns of the background image P2, which are H2 and W2, respectively. Assigning values to pixel points at any position in the initial third figure image respectively, wherein the assignment formula is as follows:

p4= P1' (if any position pixel (i, j), min (W2, x1+ W1) ≧ i ≧ x1, min (H2, y1+ H1) ≧ j ≧ y 1), otherwise, P4=0.

In the step 902 provided in this embodiment of the application, if the abscissa of the pixel point at any position in the initial third personal image is not less than x1 and not greater than the smaller value of W2 and x1+ W1, and the ordinate of the pixel point at any position in the initial third personal image is not less than y1 and not greater than the smaller value of H2 and y1+ H1, the value of the pixel point at the same position in the second personal image is assigned to the pixel point at the same position in the initial third personal image, so as to obtain the third personal image P4. That is, if any position pixel point in the initial third person image is located within the person region range of the background image, as shown in fig. 10, the value of the pixel point at the same position in the second person image is assigned to the pixel point at the same position in the third person image, and if any position pixel point in the initial third person image is located outside the person region range of the background image, as shown in fig. 11, the value of the pixel point at the same position in the third person image is assigned to 0.

In another possible embodiment, in step 505, a third person region coefficient matrix corresponding to the three-dimensional space coordinate of the person to be adjusted to the corresponding three-dimensional space coordinate is determined according to the three-dimensional space coordinate of the person, the second person region coefficient matrix and the second person image, and the flowchart is as shown in fig. 12, and includes the following steps:

in step 1201, an initial third human figure region coefficient matrix is created, and the value of the initial third human figure region coefficient matrix is set to 0, and the number of rows and columns of the initial third human figure region coefficient matrix is the same as the number of rows and columns of the matrix corresponding to the background image.

In step 1202, if the abscissa of the element at any position in the initial third person region coefficient matrix is not less than x1 and is not greater than the smaller of W2 and x1+ W1, the ordinate of the element at any position in the initial third person region coefficient matrix is not less than y1 and is not greater than the smaller of H2 and y1+ H1, and the set depth coordinate of the person in the background image is not greater than D1 (i, j), the values of the elements at the same position of the second person region coefficient matrix are assigned to the elements at the same position of the initial third person region coefficient matrix, so as to obtain a third person region coefficient matrix.

Where x1 denotes the horizontal coordinate of the set person in the background image, W1 denotes the number of columns of the matrix corresponding to the second person image, W2 denotes the number of columns of the initial third person region coefficient matrix, y1 denotes the vertical coordinate of the set person in the background image, H1 denotes the number of rows of the matrix corresponding to the second person image, H2 denotes the number of rows of the initial third person region coefficient matrix, and D1 (i, j) denotes the value of the pixel point at the (i, j) position in the background depth image.

For example, a new human figure region coefficient matrix, i.e., an initial third human figure region coefficient matrix is created, and the value of the initial third human figure region coefficient matrix is set to 0, and the number of rows and columns of the initial third human figure image is the same as the number of rows and columns of the background image P2, which are H2 and W2, respectively. And respectively assigning values to elements at any position in the initial third person region coefficient matrix, wherein the assignment formula is as follows:

β = α' (if min (W2, x1+ W1) ≧ i ≧ x1, min (H2, y1+ H1) ≧ j ≧ y1, and z1 ≦ D1 (i, j)) at the element position (i, j) at any position, otherwise β =0.

The above assignment formula can be understood as step 1102 provided in this embodiment of the present application: if the abscissa of the element at any position in the initial third human figure region coefficient matrix is not less than x1 and not greater than the smaller value of W2 and x1+ W1, the ordinate of the element at any position in the initial third human figure region coefficient matrix is not less than y1 and not greater than the smaller value of H2 and y1+ H1, and the set depth coordinate of the human in the background image is not greater than D1 (i, j), the value of the element at the same position of the second human figure region coefficient matrix is assigned to the element at the same position of the initial third human figure region coefficient matrix, so as to obtain a third human figure region coefficient matrix. That is, if the person image is located in front of the background image, as shown in fig. 13, the value of the element at the same position of the second person region coefficient matrix is assigned to the element at the same position of the initial third person region coefficient matrix, that is, the person in the region at the position shades the background; if the image of the person is behind the background image, as shown in fig. 14, the value of the element at the same position in the initial third person region coefficient matrix is assigned to 0, that is, the background of the position region shades the person.

In step 506, the third person image and the background image are fused by using the third person region coefficient matrix to obtain a fused image.

In one possible implementation manner, in step 506, a third person image is fused with the background image by using a third person region coefficient matrix, so as to obtain a fused image, which includes the following contents:

determining a fused image by adopting the following formula:

P ₅ ＝β×P ₄ +(1-β)×P ₂

wherein, P ₅ Representing the fused image, beta represents the fusion coefficient matrix, P ₄ Representing a fused image of a person, P ₂ Representing a background image. Compared with the prior art, the method can only determine the related image fusion parameters according to the portrait coordinates acquired by the front-facing camera in the set two-dimensional space, realizes the image fusion in the three-dimensional space, is more real in the shielding relation and the size ratio of the portrait to the surrounding environment, and has better fusion effect.

In order to realize a better fusion effect, the embodiment of the application can artificially adjust the relative distance between the portrait and the background, and improve the use experience of a user.

In one possible embodiment, the relative distance between the character and the background is artificially adjusted, and the flow chart is shown in fig. 15, and includes the following contents:

in step 1501, the fusion image is displayed on a display interface, and a spatial depth sliding bar corresponding to the adjustment range of the depth coordinate of the person in the background image is determined according to the background depth image.

In step 1502, in response to the sliding instruction, the distance between the person in the adjusted fusion image and the rear camera of the mobile terminal in the preset depth range is determined, and the updated depth coordinate of the person is obtained.

In step 1503, in response to the dragging instruction, the position of the person area in the preset area range adjustment fusion image is determined, and the updated horizontal coordinate and the updated vertical coordinate of the person are determined.

In step 1504, the updated three-dimensional space coordinates of the person are used as the three-dimensional space coordinates of the person, and the fusion instruction is triggered again; the updated three-dimensional space coordinates of the character comprise an updated depth coordinate, an abscissa and an ordinate.

As shown in fig. 16, the fused image is displayed on the display interface, a spatial depth slider is displayed, the distance between the person in the fused image and the rear camera of the mobile terminal is adjusted within a preset depth range by sliding the spatial depth slider in the image, wherein the preset depth range is 0-10 m, and after the sliding instruction is completed, the updated depth coordinate of the person is obtained. For example, as shown in fig. 17, the character image is located at the front of the background image, and as shown in fig. 18, the character image is located at the back of the background image, and thus, the method provided by the embodiment of the application fully considers the occlusion relationship between the character and the background object and the size ratio between the character and the surrounding object, and the fusion effect is better.

In a possible implementation manner, considering that the illumination directions of the light source of the image taken by the front camera and the light source of the image taken by the rear camera may be inconsistent, which causes the illumination of the portrait of the fused image to be inconsistent with the illumination of the whole background, the embodiment of the application determines the illumination position first and then performs image fusion. In step 501, a person image is determined according to the data collected by the front camera, and the flow chart is shown in fig. 19 and includes the following contents:

in step 1901, an original person image is captured using a front-facing camera.

In step 1902, the background image is processed using the deep learning relighting model to determine the illumination location.

In step 1903, a person image is generated from the illumination position and the original person image.

After the figure image containing the illumination position is obtained, the image fusion method provided by the application can be executed, and by adopting the method, the illumination of the figure image can be consistent with that of the background image, so that the fusion effect is more real.

Based on the same inventive concept, an embodiment of the present application further provides a device for photographing and fusing front and back cameras, as shown in fig. 20, where the device 2000 includes:

an image determining module 2001 configured to determine a person image according to data acquired by the front camera, and determine a background image and a background depth image according to acquisition of background data by the rear camera;

a person region coefficient matrix determining module 2002 configured to perform person region segmentation on the person image to obtain a person region coefficient matrix;

a three-dimensional space coordinate determination module 2003 configured to determine three-dimensional space coordinates of a person in the person image in the background image based on the background depth image, the three-dimensional space coordinates including a depth coordinate, a horizontal coordinate, and a vertical coordinate, and trigger a fusion instruction;

the data conversion module 2004 is configured to respond to the fusion instruction, scale the person region coefficient matrix and the person image according to a scaling ratio, and convert the person region coefficient matrix and the person image into a second person region coefficient matrix and a second person image corresponding to the focal length and the visual field corresponding to the rear camera;

a fusion parameter determination module 2005 configured to determine, according to the three-dimensional space coordinates of the person, the second person region coefficient matrix, and the second person image, a third person image and a third person region coefficient matrix corresponding to the three-dimensional space coordinates of the person to be adjusted;

an image fusion module 2006 configured to fuse the third person image and the background image by using a third person region coefficient matrix to obtain a fused image.

In one possible embodiment, the scaling is determined as follows:

In one possible embodiment, determining a third image corresponding to the adjustment of the person to the corresponding three-dimensional spatial coordinates is performed based on the second person region coefficient matrix and the second person image, and the fusion parameter determination module is configured to:

wherein x1 represents the horizontal coordinate of the set person in the background image, W1 represents the number of columns of the matrix corresponding to the second person image, W2 represents the number of columns of the matrix corresponding to the initial third person image, y1 represents the vertical coordinate of the set person in the background image, H1 represents the number of rows of the matrix corresponding to the second person image, and H2 represents the number of rows of the matrix corresponding to the initial third person image.

creating an initial third person area coefficient matrix, and setting the value of the initial third person area coefficient matrix to be 0, wherein the row and column number of the initial third person area coefficient matrix is the same as the row and column number of the matrix corresponding to the background image;

if the abscissa of the element at any position in the initial third character area coefficient matrix is not less than x1 and not more than the smaller value of W2 and x1+ W1, the ordinate of the element at any position in the initial third character area coefficient matrix is not less than y1 and not more than the smaller value of H2 and y1+ H1, and the depth coordinate of the set character in the background image is not more than D1 (i, j), assigning the value of the element at the same position of the second character matting coefficient matrix to the element at the same position of the initial third character area coefficient matrix to obtain a third character area coefficient matrix;

determining a fused image by adopting the following formula:

P ₅ ＝β×P ₄ +(1-β)×P ₂

In one possible embodiment, the apparatus further comprises:

the second coordinate determination module is configured to respond to the dragging instruction, determine the position of a person area in the preset area range adjusting fusion image, and determine the updated transverse coordinate and the updated longitudinal coordinate of the person;

the fusion instruction triggering module is configured to take the three-dimensional space coordinate after the character is updated as the three-dimensional space coordinate of the character and re-trigger the fusion instruction; the updated three-dimensional space coordinates of the character comprise an updated depth coordinate, an abscissa and an ordinate.

acquiring an original figure image by adopting a front camera;

In an exemplary embodiment, the present application further provides a computer-readable storage medium, such as the memory 120, including instructions, which are executable by the processor 180 of the terminal device 100 to perform the method for front-back camera photographing fusion. Alternatively, the computer readable storage medium may be a non-transitory computer readable storage medium, for example, which may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product is also provided, comprising a computer program, which when executed by the processor 180, implements the method of front and back camera shot fusion as provided herein.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method for fusing front camera shooting and back camera shooting is characterized by comprising the following steps:

in response to the fusion instruction, scaling the human area coefficient matrix and the human image according to the scaling ratio, and converting the human area coefficient matrix into a second human area coefficient matrix and a second human image corresponding to the focal length and the visual field corresponding to the rear camera;

determining a third person image and a third person area coefficient matrix corresponding to the three-dimensional space coordinate of the person to be adjusted according to the three-dimensional space coordinate of the person, the second person area coefficient matrix and the second person image;

2. The method of claim 1, wherein determining the background image and the background depth image comprises:

And determining a background depth image by adopting a TOF depth rear camera and determining a background image by adopting an RGB color rear camera.

3. The method of claim 1, wherein the scaling is determined as follows:

wherein S represents the scaling, f ₁ Denotes the front focal length of the front camera, f ₂ Indicating the back focal length of the rear camera lens, z ₁ Indicating the depth coordinate, z, of the set person in the background image ₂ And the physical distance between the person and the front camera when the front camera captures the person image is represented.

4. The method of claim 1, wherein determining a third image of the person corresponding to the adjusted person to the corresponding three dimensional space coordinates based on the second matrix of person region coefficients and the second image of the person comprises:

5. The method of claim 1, wherein determining a third human figure coefficient matrix corresponding to the person adjusted to the corresponding three-dimensional space coordinates based on the three-dimensional space coordinates of the person, the second human figure coefficient matrix, and the second human figure image comprises:

6. The method of claim 1, wherein fusing the third person image with the background image using a third person region coefficient matrix to obtain a fused image comprises:

determining a fused image using the following formula:

P ₅ ＝β×P ₄ +(1-β)×P ₂

7. The method of claim 1, further comprising:

taking the updated three-dimensional space coordinate of the character as the three-dimensional space coordinate of the character, and re-triggering the fusion instruction; the updated three-dimensional space coordinates of the character comprise an updated depth coordinate, an abscissa and an ordinate.

8. The method of claim 1, wherein determining the image of the person based on the data collected by the front facing camera comprises:

acquiring an original figure image by adopting a front camera;

9. A terminal device, comprising:

a display for displaying the acquired image;

a memory for storing executable instructions of the processor;

a processor for executing the executable instructions to implement the method for front-back camera photographical fusion as claimed in any one of claims 1-8.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of a terminal device, enable the terminal device to perform the method for front-back camera shot fusion according to any one of claims 1-8.