WO2022262313A1 - 基于画中画的图像处理方法、设备、存储介质和程序产品 - Google Patents

基于画中画的图像处理方法、设备、存储介质和程序产品 Download PDF

Info

Publication number
WO2022262313A1
WO2022262313A1 PCT/CN2022/078618 CN2022078618W WO2022262313A1 WO 2022262313 A1 WO2022262313 A1 WO 2022262313A1 CN 2022078618 W CN2022078618 W CN 2022078618W WO 2022262313 A1 WO2022262313 A1 WO 2022262313A1
Authority
WO
WIPO (PCT)
Prior art keywords
image frame
video
picture
image
frame
Prior art date
Application number
PCT/CN2022/078618
Other languages
English (en)
French (fr)
Inventor
李玮
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Publication of WO2022262313A1 publication Critical patent/WO2022262313A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/62—Control of parameters via user interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
    • H04N5/445—Receiver circuitry for the reception of television signals according to analogue transmission standards for displaying additional information
    • H04N5/45—Picture in picture, e.g. displaying simultaneously another television channel in a region of the screen

Definitions

  • the present application relates to the field of computer technology, and in particular to a picture-in-picture-based image processing method, device, storage medium and program product.
  • the picture-in-picture technology is a technology that uses two playback windows to superimpose to present two video signals at the same time.
  • the playback interface displayed by the terminal device includes a main window and an auxiliary window, and the terminal device displays two video signals in the main window and the auxiliary window respectively.
  • the size of the main window is larger than the window size of the auxiliary window, and the main window is usually the entire playback interface, and the auxiliary window is usually superimposed and displayed on the main window.
  • electronic devices such as mobile phones and tablet computers are usually configured with multiple cameras, for example, a front camera and a rear camera are respectively configured on the electronic device. Users can choose the corresponding shooting mode according to their own needs, for example, front camera mode, rear camera mode, front and rear dual camera mode, etc.
  • Users can use at least two cameras to shoot in picture-in-picture mode.
  • the user may need to adjust the size or position of the auxiliary window during the video shooting process.
  • the size and position of the image frame of the auxiliary window are fixed, and the size or position of the image frame of the auxiliary window cannot be adjusted, which cannot meet the needs of users, resulting in user experience poor.
  • the embodiment of the present application provides a picture-in-picture-based image processing method, device, storage medium, and program product, so as to solve the problem that the existing technology cannot adjust the size or position of the auxiliary window, resulting in poor user experience. bad question.
  • the embodiment of the present application provides an image processing method based on picture-in-picture, which is applied to electronic equipment, including:
  • the video image includes a first image frame and at least one second image frame, the first image frame at least partially surrounds the at least one second image frame, and the first image frame at least partially surrounds the at least one second image frame.
  • An image frame and the at least one second image frame are video images synthesized from frame images of at least two channels of video data;
  • the synthesis strategy is information for synthesizing the display position and display size of the first image frame and at least one second image frame;
  • the frame images of the at least two channels of video data are synthesized into a video image to be displayed.
  • the method also includes:
  • the first adjustment operation includes at least one of a zoom operation or a drag operation.
  • adjusting the synthesis strategy of the video image according to the first adjustment operation includes:
  • the first adjustment operation determining the superimposed position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame, wherein the auxiliary image frame is the at least one second image frame;
  • composition strategy is adjusted according to the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame.
  • the adjustment of the synthesis strategy of the video image according to the first adjustment operation includes:
  • the composition strategy is adjusted.
  • determining the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame includes:
  • the initial touch point of the first adjustment operation is a single touch point and falls within the adjustment range of the auxiliary image frame, determine that the first adjustment operation is a zoom operation
  • the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame are determined.
  • determining the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame includes:
  • the initial touch points of the first adjustment operation are at least two touch points
  • the touch tracks of the at least two touch points determine the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame.
  • determining the superposition position of the auxiliary image frame on the first image frame includes:
  • the superimposition position of the auxiliary image frame on the first image frame is determined.
  • the adjusting the synthesis strategy according to the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame includes:
  • the preset size threshold is the size of the spliced display of the first image frame and the auxiliary image frame in the spliced display mode.
  • the method also includes:
  • the composition strategy is updated to the composition strategy of the spliced display mode.
  • said recording video images according to said video shooting operation includes:
  • the frame images of the at least two channels of video data are synthesized into video images according to the synthesis strategy of the picture-in-picture display mode.
  • the frame images of the at least two channels of video data are synthesized according to the synthesis strategy of the picture-in-picture display mode, and the synthesis of video images includes:
  • an embodiment of the present application provides an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, trigger The electronic device executes the method described in any one of the first aspect.
  • the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned The method of any one of the first aspects.
  • an embodiment of the present application provides a computer program product, the computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer executes any one of the above-mentioned first aspects. described method.
  • the first adjustment operation is received, and the composition strategy is adjusted according to the received first adjustment operation, so that the second image of the picture-in-picture
  • the purpose of adjusting the size and position of the screen is to better meet user needs and improve user experience.
  • FIG. 1 is a schematic diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of a scene in a picture-in-picture shooting mode provided by an embodiment of the present application
  • FIG. 3 is a schematic flow chart of a picture-in-picture-based image processing method provided by an embodiment of the present application
  • FIG. 4a is a schematic diagram of a shooting scene in a front-back dual-camera mode provided by an embodiment of the present application
  • FIG. 4b is a schematic diagram of a shooting scene in a front-to-back picture-in-picture mode provided by an embodiment of the present application;
  • FIG. 4c is a schematic diagram of a post-picture-in-picture mode shooting scene provided by the embodiment of the present application.
  • FIG. 5a is a schematic diagram of a scene adjustment of an auxiliary image provided by an embodiment of the present application.
  • Fig. 5b is a schematic diagram of another auxiliary image screen adjustment scene provided by the embodiment of the present application.
  • Fig. 5c is a schematic diagram of another auxiliary image adjustment scene provided by the embodiment of the present application.
  • Fig. 5d is a schematic diagram of another auxiliary image adjustment scene provided by the embodiment of the present application.
  • FIG. 6a is a schematic diagram of a scene of a spliced display mode provided by an embodiment of the present application.
  • FIG. 6b is a schematic diagram of another spliced display mode provided by the embodiment of the present application.
  • Fig. 6c is a schematic diagram of another spliced display mode provided by the embodiment of the present application.
  • FIG. 7 is a schematic flowchart of another picture-in-picture-based image processing method provided by the embodiment of the present application.
  • FIG. 8 is a software structural block diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a rendering scene provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of another rendering scene provided by the embodiment of the present application.
  • Fig. 11a is a schematic diagram of a composite video image scene provided by an embodiment of the present application.
  • Fig. 11b is a schematic diagram of another synthetic video image scene provided by the embodiment of the present application.
  • FIG. 12 is a schematic flowchart of another picture-in-picture-based image processing method provided by the embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 1 it is a schematic diagram of an electronic device provided by an embodiment of the present application.
  • the mobile phone 100 is taken as an example to illustrate the electronic device.
  • FIG. Four rear cameras 121 , 122 , 123 , 124 are configured on the rear side of the mobile phone 100 .
  • users can be provided with various shooting modes, such as front camera mode, rear camera mode, front and rear dual camera mode, etc. The user can select a corresponding shooting mode to shoot according to the shooting scene, so as to improve the user experience.
  • FIG. 1 is only an exemplary illustration, and should not be taken as a limitation of the protection scope of the present application.
  • the electronic devices involved in the embodiments of the present application can also be tablet computers, personal computers (personal computers, PCs), personal digital assistants (personal digital assistants, PDAs), smart watches, netbooks, wearable electronic devices, Augmented reality (augmented reality, AR) equipment, virtual reality (virtual reality, VR) equipment, vehicle equipment, smart car, smart audio, robot, smart glasses, smart TV, etc.
  • an electronic device may also be called a terminal device, a user equipment (User Equipment, UE), etc., which is not limited in this embodiment of the present application.
  • UE User Equipment
  • electronic devices now include at least two cameras, and at least two cameras can be turned on at the same time for video recording, so that the video image of the video file obtained by the video recording includes the shooting pictures of multiple cameras (based on the camera mode selected by the user) , according to the picture combination strategy corresponding to the camera mode, the shooting pictures of multiple cameras are combined into one image picture).
  • This mode of simultaneously using at least two cameras to shoot the same video file can be called a multi-channel video recording mode.
  • the user can adjust the shooting mode without interrupting the video recording during the recording process, such as adjusting the camera used to record the video and/or the combination mode of the multi-channel camera images in the video image, so as to Generate a video file containing shooting mode switching.
  • the display screen of the electronic device will display the synchronous display screen of the recorded video file. The displayed picture is consistent with the picture in the recorded video file.
  • FIG. 2 it is an example diagram of a scene captured in a picture-in-picture mode according to an embodiment of the present application.
  • the captured video images can be displayed in real time on the display interface.
  • the display interface also includes a shooting mode selection window, and the user can select a corresponding shooting mode in the shooting mode selection window for video shooting. For example, front single camera mode, rear single camera mode, front and rear dual camera mode, front and rear picture-in-picture mode, etc.
  • the user selects the front-to-back picture-in-picture shooting mode.
  • the foreground picture captured by the front camera and the background picture captured by the rear camera are combined into one picture to achieve the picture-in-picture effect. That is to say, in the front and rear dual-camera mode, the front camera and the rear camera are respectively used to collect the foreground picture and the background picture, and display the composite picture of the foreground picture and the background picture respectively in the display interface, and , encode and save the composite picture of the foreground picture and the background picture as a video file in the background.
  • the displayed display screen when the electronic device shoots a video in a picture-in-picture mode, the displayed display screen includes two parts in terms of visual effect, a first window and a second window.
  • the size of the first window is larger than that of the second window, and the first window is usually the entire playback interface, while the second window is usually superimposed and displayed on the first window.
  • the background of the electronic device acquires two channels of video data of the foreground image and the background image, and processes the foreground image as the first window image and the background image as the second window according to the image synthesis strategy corresponding to the picture-in-picture display mode. image, combining the first window image and the second window image into one video image, and then sending them to the display interface for display and encoding and saving respectively.
  • the important image content in the foreground picture in the recorded video picture may be blocked by the background picture, or the user is not sure about the foreground picture in the currently recorded video picture.
  • the layout of the screen and the background screen is not satisfactory; at this time, the user may wish to adjust the size or position of the background screen (auxiliary window).
  • auxiliary window is only a visual display effect presented on the video screen, it is not an independent display window, therefore, it cannot be adjusted based on the adjustment method for window controls, so it cannot meet user needs. resulting in poor user experience.
  • the user can input the first adjustment operation, adjust the synthesis method of the foreground picture and the background picture according to the first adjustment operation, and generate a video image corresponding to the user's first adjustment operation , sending the video image to the display interface for display and encoding storage, so as to achieve the purpose of adjusting the size and/or position of the second window image in the recorded video image during the recording process of the picture-in-picture mode, and better meet the User needs, improve user experience.
  • the captured video needs to be encoded into a video file (for example, a video file in MP4 format) and stored in the electronic device.
  • a video file for example, a video file in MP4 format
  • FIG. 3 it is a schematic flowchart of a picture-in-picture-based image processing method provided by an embodiment of the present application.
  • the method can be applied to the electronic device shown in FIG. 1 , and as shown in FIG. 3 , it mainly includes the following steps.
  • Step S301 receiving a video shooting operation in a picture-in-picture shooting mode.
  • the shooting modes involved in the electronic device may include a single-shot mode and a multi-shot mode.
  • single camera mode may include front single camera mode, rear single camera mode, etc.
  • multi-camera mode may include front dual camera mode, rear dual camera mode, front and rear dual camera mode, front picture in picture mode, rear camera mode, etc.
  • one camera is used for video shooting; in the multi-camera mode, two or more cameras are used for video shooting.
  • a front camera is used for video shooting; in the rear single camera mode, a rear camera is used for video shooting; in the front dual camera mode, two front cameras are used Camera for video shooting; in rear dual camera mode, two rear cameras are used for video shooting; in front and rear dual camera mode, one front camera and one rear camera are used for video shooting; front picture-in-picture
  • two front cameras are used for video shooting, and the picture taken by one front camera is placed in the picture taken by the other front camera
  • in the rear picture-in-picture mode two rear cameras are used For video shooting, and the picture taken by one rear camera is placed in the picture taken by another rear camera; in the picture-in-picture mode, one front camera and one rear camera are used for video shooting, and the The picture taken by the front camera or the rear camera is placed in the picture taken by the rear camera or the front camera.
  • FIG. 4 a it is a schematic diagram of a shooting scene in a front-back dual-camera mode provided by an embodiment of the present application.
  • a front camera is used to collect the foreground image
  • a rear camera is used to collect the background image
  • the foreground image and the background image are simultaneously displayed in the display interface.
  • FIG. 4 b it is a schematic diagram of a shooting scene in a front-to-back picture-in-picture mode provided by an embodiment of the present application.
  • a front camera is used to collect the foreground image
  • a rear camera is used to collect the background image
  • the foreground image is placed in the background image.
  • FIG. 4c it is a schematic diagram of a post-picture-in-picture mode shooting scene provided by an embodiment of the present application.
  • one rear camera is used to collect distant images
  • the other rear camera is used to collect close-up images
  • the near-view images are placed in the distant view images.
  • the shooting mode involved in the embodiment of the present application may be any one of the front picture-in-picture mode, the rear picture-in-picture mode, and the front-back picture-in-picture mode.
  • the user if the user needs to use the picture-in-picture shooting mode to shoot video, he can input the video shooting operation in the picture-in-picture shooting mode.
  • the user may input the video shooting operation through a touch screen, physical buttons, gesture control, voice control, etc., which is not specifically limited in this embodiment of the present application.
  • Step S302 taking video images according to the video shooting operation.
  • the video image includes a first image frame and at least one second image frame, and the first image frame at least partially surrounds the at least one second image frame.
  • the first image frame and the at least one second image frame are respectively video images synthesized from frame images of at least two channels of video data.
  • the first image picture is a picture of a frame image in the first road video data
  • at least one second image picture is respectively a picture of a frame image in at least one second road video data; the first road video data and at least one second road video data
  • the video data collected by different camera devices.
  • first image frame in the video image refers to the above-mentioned first window image frame
  • second image frame refers to the above-mentioned second window image frame
  • At least two channels of video data can be collected through the camera in the picture-in-picture shooting mode, and the synthesis strategy can be determined as picture-in-picture shooting mode.
  • the electronic device can synthesize the frame images of at least two channels of video data collected into one video image according to the synthesis strategy of the picture-in-picture display mode.
  • the preset display position and display size of the first image frame and the display position and display size of each second image frame can be set according to the synthesis strategy of the picture-in-picture display mode , setting the video data corresponding to the first image frame at its corresponding display position, and setting the video data corresponding to each second image frame at its corresponding display position, and setting the video data corresponding to the first image frame
  • the size of the frame image is adjusted to its corresponding display size
  • the size of the frame image of the video data corresponding to each second image screen is adjusted to its respective corresponding display size
  • the frame images of the video data of each channel are synthesized to obtain a video image.
  • the position and size of each second image frame in the video image is the preset display position and display size of the second image frame in the synthesis strategy of the picture-in-picture display mode. That is to say, in the embodiment of the present application, the captured frame images of at least two channels of video data are synthesized into one video image, and then subsequent display and encoding are performed.
  • the first image frame can be displayed according to the size of the display area of the display screen.
  • the display position of the first image frame can be set as a point in the display area of the display screen, and the display size can be set as a point of the display area of the display screen. The size of the display area.
  • recording video images according to the video shooting operation includes:
  • the electronic device when it receives a shooting operation, it may turn on corresponding at least two cameras according to the shooting operation, so as to collect at least two channels of video data through the at least two cameras.
  • the size of at least two channels of video data collected by the electronic device through at least two cameras can be different.
  • the size of the video data collected by the front camera is 540*480
  • the size of the video data collected by the rear camera is 1080*960.
  • the sizes of at least two channels of video data collected by at least two cameras may also be the same, which is not limited in this application.
  • the electronic device After the electronic device acquires at least two channels of video data, since it receives the shooting operation of the picture-in-picture shooting mode, the electronic device can set the synthesis strategy as the synthesis strategy of the picture-in-picture display mode, and combine the video data of each channel
  • the frame images are rendered and merged according to the display size and display position corresponding to the frame images of each channel of video data preset in the synthesis strategy to obtain a video image.
  • the video image captured by the front camera is scaled according to the display size corresponding to the video image captured by the front camera in the synthesis strategy of the preset picture-in-picture display mode, and the video image captured by the front camera in the synthesis strategy is scaled.
  • the display position corresponding to the video image embed the video image captured by the front camera into the corresponding position inside the image captured by the rear camera, and realize the merging of the frame images of at least two channels of video data collected by the front camera and the rear camera , the combined image is in a picture-in-picture display mode to form a video image.
  • the frame images of at least two channels of video data are synthesized according to the synthesis strategy of the picture-in-picture display mode, and the synthesis of video images includes:
  • the display position and display size of the frame image of each channel of video data are determined.
  • the size of the frame image of each channel of video data is adjusted according to the display size corresponding to the frame image of each channel of video data.
  • the texture information of the frame image of each channel of video data and the position information corresponding to the texture information are determined.
  • the display position of the frame image of each channel of video data, the texture information of the frame image of each channel of video data, and the position information corresponding to the texture information the frame images of at least two channels of video data are correspondingly synthesized into one video image.
  • the electronic device when setting the compositing strategy of the picture-in-picture display mode, it is to set the display position and display size of each image frame in the picture-in-picture display mode. Therefore, when the user chooses to use the picture-in-picture shooting mode for recording, the electronic device can obtain the preset synthesis strategy of the picture-in-picture display mode from the storage medium, and obtain the display position and display position of each image frame in the synthesis strategy. The relevant information of the size can then determine the display position and display size of the frame image of each channel of video data.
  • the size of the frame image of each channel of video data is adjusted separately, for example, zooming is performed, or image extraction processing of a corresponding size is performed to obtain a frame of a corresponding size of each channel of video data image.
  • the texture information of the frame image of each channel of video data and the position information corresponding to the texture information are extracted from the frame image of each channel of video data after resizing.
  • the display position corresponding to the frame image of each channel of video data, the texture information of the frame image of each channel of video data, and the position information corresponding to the texture information the frame images of at least two channels of video data are correspondingly synthesized into a video image.
  • Step S303 receiving a first adjustment operation.
  • the first adjustment operation is used to indicate to adjust the position and/or size of the second image frame.
  • the user may input a first adjustment operation to the electronic device, so as to adjust the corresponding auxiliary display area.
  • the user may input the first adjustment operation through a touch screen, physical buttons, gesture control, voice control, etc., which is not specifically limited in this embodiment of the present application.
  • the first adjustment operation includes at least one of a zoom operation and a drag operation. That is, the user can adjust the size and display position of the second image frame through a zoom operation. Only the display position of the second image frame can be adjusted through a drag operation.
  • the second image frames to be adjusted by the first adjustment operation are collectively referred to as auxiliary image frames.
  • the existing electronic equipment generally has a touch screen, so the user can input the first adjustment operation to the electronic equipment by touching the touch screen.
  • the user when the user needs to adjust the size and/or position of the second image screen, the user can input a touch operation to the electronic device through the touch screen, and the electronic device can The touch operation determines the first adjustment operation, so as to adjust the second image frame.
  • the first adjustment operation is a zoom operation.
  • an adjustment area that can adjust the display size of the second image frame is preset in each second image frame, for example, it may be a display boundary of the second image frame.
  • the electronic device determines the adjustment of the auxiliary image screen according to the user's touch track. That is, when the user inputs a touch operation through touch, if the initial touch point of the touch operation is a single touch point, it means that the user touches the touch screen with a single finger, and the initial touch point falls within the adjustment range of the auxiliary image screen.
  • the electronic device determines the first adjustment operation as a scaling operation.
  • the first adjustment operation is a zoom operation.
  • the touch points input by the user may be single or at least two.
  • the touch point is a single control zoom, you can refer to the above content, which will not be repeated here.
  • the first adjustment operation is a zoom operation. That is, the user adjusts the size and display position of the auxiliary image frame by using at least two fingers.
  • the electronic device when the initial touch points of the first adjustment operation are at least two touch points, it may be further detected whether the at least two touch points are within the auxiliary image frame, and if the touch points are outside the auxiliary image frame, it means yes The manipulation of other image screens is not the manipulation of auxiliary image screens. Therefore, after the electronic device detects at least two initial touch points of the first adjustment operation, it may first detect whether the coordinates of the at least two initial touch points are in the auxiliary image frame, for example, detect at least two initial touch points Whether the coordinate position of is within the coordinate range corresponding to the secondary image screen. If it is in the auxiliary image frame, it indicates that it is an operation on the auxiliary image frame, and at this time, the electronic device may determine the first adjustment operation as a zoom operation. The first adjustment operation is performed by the user to adjust the size of the auxiliary image frame by using at least two fingers. In this way, the user adjusts the size of the secondary image screen by pinching with two fingers.
  • the initial touch point of the first adjustment operation is a single touch point and does not fall within the adjustment area of the auxiliary image frame, it may be determined that the first adjustment operation is a drag operation.
  • the touch point is used to adjust the position of the auxiliary image frame. That is, after the electronic device detects the initial touch point of the first adjustment operation, it can detect whether the coordinate position of the initial touch point falls within the adjustment area of the auxiliary image screen, if not, it means that the electronic device can determine
  • the first adjustment operation is a drag operation.
  • determining whether the first adjustment operation is a drag operation may also be implemented in other ways in the prior art, which is not limited in the present application.
  • Step S304 according to the first adjustment operation, adjust the composite strategy of the video image in real time.
  • the compositing strategy is information for compositing the display position and display size of the first image frame and at least one second image frame.
  • the electronic device after the electronic device determines the first adjustment operation, it can determine the second image frame to be adjusted according to the first adjustment operation, that is, the superimposition position and the auxiliary image frame on the first image frame
  • the size of the image frame, and then the adjusted merge strategy can be determined, that is, to determine the display position and display size of the image frame corresponding to each video data in the video image to be synthesized, update the existing merge strategy in real time, so that the follow-up
  • the frame images of at least two channels of video data are correspondingly synthesized into one video image.
  • the adjusted position and size are displayed.
  • the auxiliary image frame is the second image frame to be adjusted by the first adjustment operation among at least one second image frame.
  • the first adjustment operation may be a zoom operation or a drag operation.
  • the adjustment of the auxiliary image frame is also different, as follows.
  • adjusting the synthesis strategy of the video image includes:
  • the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame are determined.
  • the composition strategy is adjusted.
  • the size of the auxiliary image frame adjusted by the user can be determined according to the first adjustment operation, and then the superimposition position of the auxiliary image frame on the first image frame before resizing can be determined and the size of the adjusted auxiliary image frame, determine the superimposed position of the adjusted auxiliary image frame on the first image frame, and then adjust the superimposed position of the adjusted auxiliary image frame on the first image frame and the auxiliary image
  • the zoom operation can be input by the user through a single-point touch screen, or through at least two-point touch screen input, according to the number of input initial touch points, it is determined that the adjusted secondary image screen is in the first image
  • the superimposition position on the screen and the size of the auxiliary image screen are also different, as follows.
  • determining the superposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame includes:
  • the initial touch point of the first adjustment operation is a single touch point and falls within the adjustment range of the auxiliary image frame
  • determine that the first adjustment operation is a zoom operation
  • determine the auxiliary image frame according to the touch track of the first adjustment operation The superposition position on the first image frame and the size of the secondary image frame.
  • the electronic device may determine the superimposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame according to the zoom operation.
  • the initial touch point input by the user is a single touch point and falls within the adjustment area of the auxiliary image screen, it means that the first adjustment operation is a zoom operation , at this time, the initial touch point can be regarded as the adjustment point of the auxiliary window screen.
  • the electronics can determine the adjustment position of the adjustment point of the auxiliary image frame according to the touch track of the first adjustment operation.
  • the electronic device can calculate the size of the auxiliary image frame according to the position before adjustment and the adjusted position of the adjustment point of the auxiliary image frame, and adjust the superposition position of the former auxiliary image frame on the first image frame according to the size of the auxiliary image frame, After the auxiliary image frame is resized, the superimposition position of the auxiliary image frame on the first image frame is determined. In this way, after the superimposition position and size of the auxiliary image frame on the first image frame are determined, the display position and display size of the auxiliary image frame recorded in the synthesis strategy can be adjusted. That is, adjust the display position and display size of the frame image of one channel of video data corresponding to the auxiliary image in the video image in the video image to be synthesized, and update the synthesis strategy, as shown in Figure 5a.
  • determining the superposition position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame includes:
  • the initial touch points of the first adjustment operation are at least two touch points
  • the first adjustment operation can be determined.
  • the adjustment operation is a scaling operation, and the electronic device can determine the superimposed position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame according to the touch traces of at least two touch points. In this way, after determining the superimposed position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame, the composite image can be adjusted accordingly according to the superimposed position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame.
  • the display position and display size of the auxiliary image screen recorded in the strategy is to adjust the synthesis strategy, as shown in FIG. 5b.
  • the first adjustment operation is a zoom operation
  • the zoom operation is performed on the size of the auxiliary image frame
  • the display position of the auxiliary image frame will also change. Therefore, when adjusting the compositing strategy, it is necessary to simultaneously determine the size of the auxiliary image frame and the superimposition position of the auxiliary image frame on the first image frame.
  • adjusting the composite strategy of the video image includes:
  • the superposition position of the auxiliary image frame on the first image frame is determined. According to the superimposition position of the auxiliary image frame on the first image frame, the composition strategy is adjusted.
  • a drag operation can be input, that is, the first adjustment operation is dragging operate.
  • the electronic device After the electronic device receives the drag operation, it can determine the superposition position of the adjusted auxiliary image frame on the first image frame according to the drag operation, and then adjust the composition according to the superimposed position of the auxiliary image frame on the first image frame Strategy.
  • determining the superposition position of the auxiliary image frame on the first image frame includes:
  • the superimposition position of the auxiliary image frame on the first image frame is determined.
  • the electronic device may determine the superimposition position of the auxiliary image frame on the first image frame according to the touch track of the first adjustment operation. In this way, after the superimposition position of the auxiliary image frame on the first image frame is determined, the display position of the auxiliary image frame recorded in the synthesis strategy can be adjusted according to the superimposition position of the auxiliary image frame on the first image frame, that is, adjusting The synthesis strategy is shown in Figure 5c.
  • determining the size of the auxiliary image frame according to the first adjustment operation may be determining a scaling ratio of the auxiliary image frame.
  • the scaling ratio is recorded in the compositing strategy, so that the electronic device can perform scaling processing on the frame image of the video data corresponding to the auxiliary image frame according to the scaling ratio to obtain an image of a corresponding size.
  • the size of the auxiliary image frame is recorded in the synthesis strategy, so that the electronic device can directly create a video image corresponding to the auxiliary image frame according to the size of the auxiliary image frame.
  • the image of the corresponding size is extracted from the frame image of the data.
  • adjusting the synthesis strategy includes:
  • the composition strategy is adjusted according to the superimposed position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame.
  • the preset size threshold is the size of the spliced display of the first image frame and the auxiliary image frame in the spliced display mode.
  • the sizes of the first image frame and the auxiliary image frame can be equal to each other, as shown in FIG. 6a, or the first image frame and the auxiliary image frame can be spliced and displayed at equal widths, as shown in FIG. 6b. It is also possible that the first image frame and the auxiliary image frame are spliced and displayed with equal lengths, as shown in FIG. 6 c .
  • other spliced display modes are also possible, which is not limited in this application.
  • equal width means equal width along the short side direction of the electronic device
  • equal length means equal length along the long side direction of the electronic device.
  • the electronic device forms a video image in a picture-in-picture display mode from frame images of at least two channels of video data captured by at least two cameras, and then displays it on a display interface. Since in the embodiment of the present application, the user can adjust the size of the second image frame according to actual needs, in order to prevent the second image frame from being too large and affecting the display of the first image frame, the display size threshold of the second image frame can be limited. Based on this, the display size threshold of the second image frame can be preset, and the electronic device first determines the auxiliary image frame that needs to be adjusted before adjusting the composition strategy according to the superimposed position of the auxiliary image frame on the first image frame and the size of the auxiliary image frame.
  • the adjusted size of the auxiliary image frame is smaller than the display size threshold, it means that the size of the auxiliary image frame is within the normal range.
  • the superposition position and the size of the auxiliary image frame are adjusted to the synthesis strategy.
  • the preset size threshold can be set as the size when the auxiliary image frame and the first image frame are spliced and displayed in the spliced display mode.
  • the preset size threshold can also be set to other size values. This is not limited.
  • the above method further includes: when the size of the auxiliary image frame is not smaller than the preset size threshold, updating the compositing strategy to the compositing strategy in the spliced display mode.
  • the electronic device detects that the size of the adjusted auxiliary image frame is equal to or greater than the preset size threshold, it means that the auxiliary image frame will affect the display of other image frames.
  • the picture display mode is updated to the synthesis strategy of the splicing display mode, so that subsequent frame images of multiple channels of video data can be displayed in the splicing display mode instead of picture-in-picture display, as shown in Figure 5d.
  • Step S305 correspondingly synthesizing the frame images of at least two channels of video data into a video image to be displayed according to the adjusted synthesis strategy.
  • the electronic device adjusts the synthesis strategy
  • the adjusted synthesis strategy is used so that the auxiliary image in the subsequent synthesized video images
  • the display position and display size of the screen are displayed according to the display position and display size adjusted by the user.
  • the display position and display size of the image picture corresponding to the frame image of each channel of video data is recorded in the synthesis strategy.
  • the electronic device can know the image picture corresponding to the frame image of each channel of video data by analyzing the synthesis strategy.
  • the frame images of each video data are synthesized into a video image according to their respective display positions and display sizes.
  • the electronic device learns the adjusted display position of the auxiliary image frame in the synthesis strategy, it can place the frame image of one channel of video data corresponding to the auxiliary image frame at the corresponding position in the video image to be synthesized, and according to the auxiliary image frame
  • the size of the image frame Extract the video frame of the corresponding size from the frame image of the video data corresponding to the auxiliary image frame or zoom the frame image to obtain an image of the corresponding size, and set the texture information of the image at the corresponding display position place.
  • FIG. 7 it is a schematic flowchart of another picture-in-picture-based image processing method provided by the embodiment of the present application.
  • the method can be applied to the electronic device shown in FIG. 1 , which further includes generating a video file on the basis of the embodiment shown in FIG. 3 , and mainly includes the following steps.
  • Step S601 receiving a video shooting operation in a picture-in-picture shooting mode.
  • step S301 For details, refer to step S301, which will not be repeated here.
  • Step S602 taking video images according to the video shooting operation.
  • the video image includes a first image frame and at least one second image frame, the first image frame at least partially surrounds at least one second image frame, and the first image frame and the at least one second image frame are respectively at least two video channels
  • the frame image of the data is synthesized into a video image.
  • step S302 For details, refer to step S302, which will not be repeated here.
  • Step S603 receiving a first adjustment operation.
  • the first adjustment operation is used to adjust the position and/or size of the second image frame.
  • step S303 For details, refer to step S303, which will not be repeated here.
  • Step S604. According to the first adjustment operation, adjust the synthesis strategy of the video image in real time.
  • the compositing strategy is information for compositing the display position and display size of the first image frame and at least one second image frame.
  • step S304 which will not be repeated here.
  • Step S605 synthesize the frame images of at least two channels of video data into a video image to be displayed.
  • Step S606 encode the video image adjusted in real time during the shooting process, generate a video file, and store the video file.
  • the video image adjusted in real time during the shooting process is sent to the encoding module, so as to realize encoding while recording, and realize the effect of what you see is what you get. That is, in this application, after the video image is synthesized, the video image can be sent to the corresponding encoding module for encoding, a video file is generated, and the generated video file is stored, so that when the video file is played again, the display interface The video image displayed in is also displayed in the picture-in-picture mode, and the adjustment to the second image can be reflected in the played video image.
  • FIG. 8 it is a software structural block diagram of an electronic device provided by an embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
  • the Android (Android) system is divided into four layers, which are respectively an application layer, a framework layer, a hardware abstraction layer and a hardware layer from top to bottom.
  • the application layer may include a series of application packages.
  • the application package may include a camera application.
  • the application layer can be further divided into display interface (UI) and application logic.
  • the application interface of the camera application includes a single-view mode, a multi-view mode, a picture-in-picture mode, and the like. Among them, only one shooting picture is displayed in the single-view mode; at least two shooting pictures are displayed side by side in the multi-view mode; at least two shooting pictures are displayed in the picture-in-picture mode, and at least one shooting picture is located between the other shooting pictures middle.
  • the application logic of the camera application includes the multi-camera framework and camera management.
  • the multi-camera framework includes a picture-in-picture event management module, a picture-in-picture coordinate management module, a picture-in-picture drawing management module, and a multi-camera encoding module.
  • the picture-in-picture event management module is used to control the adjustment of the synthesis strategy;
  • the picture-in-picture coordinate management module is used to determine the display position and display size of the image picture corresponding to the frame image of each video data;
  • the picture-in-picture drawing management module is used to pass Call the Open GL renderer to render the frame images in at least two channels of video data.
  • the multi-cam encoding module is used to encode the video images displayed on the display interface to generate video files.
  • Camera management includes device management module, Surface management module, session management module, etc.
  • Surface corresponds to a screen buffer, which is used to save the pixel data of the current window.
  • the framework layer provides application programming interface (application programming interface, API) and programming framework for the application program of the application layer, including some predefined functions.
  • the framework layer includes the camera access interface (Camera2 API).
  • Camera2 API is a set of interfaces introduced by Android to access the camera device. It adopts a pipeline design to make the data stream flow from the camera to the Surface.
  • Camera2 API includes camera management (CameraManager) and camera device (CameraDevice).
  • CameraManager is the management class of the Camera device, through which the camera device information of the device can be queried to obtain the CameraDevice object.
  • CameraDevice provides a series of fixed parameters related to Camera devices, such as basic settings and output formats.
  • the hardware abstraction layer is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform, provides a virtual hardware platform for the operating system, makes it hardware-independent, and can be transplanted on various platforms.
  • the HAL includes the camera hardware abstraction layer (Camera HAL), and the Camera HAL includes device (Device) 1, device (Device) 2, device (Device) 3, etc. It can be understood that the Device1, Device2 and Device3 are abstract devices.
  • the hardware layer (HardWare, HW) is the hardware at the bottom of the operating system.
  • the HW includes a camera device (CameraDevice) 1, a camera device (CameraDevice) 2, a camera device (CameraDevice) 3, and the like.
  • CameraDevice1, CameraDevice2 and CameraDevice3 may correspond to multiple cameras on the electronic device.
  • the picture-in-picture drawing management module may call an open graphics library (Open Graphics Library, OpenGL) renderer to implement rendering processing on video data.
  • the picture-in-picture drawing management module includes an Open GL display rendering engine, an Open GL coding rendering engine and a display buffer area. Take a frame of video image in the video data as an example to describe the rendering process, as follows:
  • the Open GL display rendering engine monitors the video images gathered by the two cameras of the electronic device through two monitoring modules. That is, the Open GL display rendering engine monitors the video images captured by the first camera and the second camera respectively through the first monitoring module and the second monitoring module.
  • the Open GL display rendering engine transmits the two-way video images and synthesis strategies monitored by the first monitoring module and the second monitoring module to the Open GL renderer.
  • the Open GL renderer synthesizes two video images into one video image according to the synthesis strategy, and transfers the video image to the display buffer for caching.
  • the video images buffered in the display buffer area are respectively transmitted to the display interface (SurfaceView) and the encoding buffer area.
  • the video image is displayed in the display interface.
  • the Open GL encoding and rendering engine obtains the video image in the encoding buffer area, and renders the root of the video image, such as performing beauty treatment on the video image, or adding a watermark to the video image, and sending the rendered video image to to the encoding module, so that the encoding module can perform corresponding encoding processing to generate a video file, as shown in Fig. 9 and Fig. 10 .
  • the video file generated by the encoding module is in MP4 format
  • other video images are in RGB format. That is to say, the video image monitored by the Open GL display rendering engine is an image in RGB format, and the video image output by the Open GL renderer after rendering and compositing is also in RGB format. That is, the video images buffered in the display buffer area are in RGB format, and the video images sent to the display interface and encoding buffer area are also in RGB format.
  • the Open GL encoding and rendering engine obtains the video image in RGB format, and renders the video image according to the image rendering operation input by the user, and the rendered video image is in RGB format.
  • the video image received by the encoding module is in RGB format, and the video image in RGB format is encoded to generate a video file in MP4 format.
  • Open GL renderer synthesizes two video images into one video image according to the synthesis strategy, specifically:
  • the Open GL renderer analyzes the compositing strategy to know the display position of the image frame corresponding to each video image. For example, the vertex coordinates of the image frame corresponding to each video image can be marked in the compositing strategy, such as the vertex coordinates of the upper left corner.
  • the Open GL renderer can determine the display position of the image frame corresponding to each video image according to the vertex coordinates of the image frame corresponding to each video image.
  • the texture information of each video image and the position information corresponding to the texture information are acquired according to each video image.
  • the Open GL renderer synthesizes two video images into one video image according to the display position of the image frame corresponding to each video image, the texture information of each video image, and the position information corresponding to the texture information, as shown in Figure 11a and Figure 11b Show.
  • the Open GL display rendering engine can monitor the video images through the first monitoring module and the second monitoring module respectively, and respectively monitor the two monitoring modules.
  • One video image is used for display rendering and the other is used for encoding rendering. It is also possible to use only one monitoring module to monitor video images, display and render the monitored video images, and encode and render the displayed and rendered video images.
  • the Open GL display rendering engine can monitor video images collected by the first camera of the electronic device through two monitoring modules. That is, the Open GL display rendering engine respectively monitors the video images collected by the first camera through the first monitoring module and the second monitoring module.
  • the Open GL display rendering engine transmits the video images monitored by the first monitoring module and the video images monitored by the second monitoring module to the Open GL renderer, and the Open GL renderer will obtain the first monitoring module of the Open GL display rendering engine
  • the transmitted video image is transmitted to the display buffer area for caching, and the video image of the road is used for displaying on the display interface, and the video image transmitted by the second monitoring module of the obtained Open GL display rendering engine is transmitted to the encoding buffer area, and the video image of the road is transmitted to the encoding buffer area.
  • Images are used for encoding processing in the multi-cam encoding module.
  • the video images buffered in the display buffer area are transmitted to the display interface.
  • the video image is displayed in the display interface.
  • the Open GL encoding and rendering engine obtains the video image in the encoding buffer area, and renders the root of the video image, such as performing beauty treatment on the video image, or adding a watermark to the video image, and sending the rendered video image to Up to the multi-camera encoding module, so that the multi-camera encoding module performs corresponding encoding processing to generate video files, as shown in FIG. 9 .
  • the video image monitored by the first monitor module and the second monitor module of the Open GL display rendering engine can also be Instead of passing through the Open GL renderer, the video images monitored by the first monitoring module of the Open GL display rendering engine are directly transmitted to the display buffer area, and the video images monitored by the second monitoring module are transmitted to the encoding buffer area. No limit.
  • the Open GL display rendering engine can monitor the video image collected by the first camera of the electronic device through a monitoring module. For example, the Open GL display rendering engine monitors the video images collected by the first camera through the first monitoring module. The Open GL display rendering engine transmits the video image monitored by the first monitoring module to the Open GL renderer, and the Open GL renderer transmits the acquired video image to the display buffer area for caching. The video images buffered in the display buffer area are transmitted to the display interface. The video image is displayed in the display interface, and the video image is transmitted to the coding buffer area.
  • the Open GL encoding and rendering engine obtains the video image in the encoding buffer area, and renders the root of the video image, such as performing beauty treatment on the video image, or adding a watermark to the video image, and sending the rendered video image to to the encoding module, so that the encoding module performs corresponding encoding processing to generate a video file, as shown in FIG. 10 .
  • the video image monitored by the first monitoring module of the Open GL display rendering engine may also not be rendered by Open GL. Instead, the video image monitored by the first monitoring module of the Open GL display rendering engine is directly transmitted to the display buffer area, which is not limited by the present application.
  • the shared context can be used to implement the encoding buffer and the display interface to share the video images buffered in the display buffer, so that the display thread and the encoding thread share texture information through the SharedContext to realize video recording.
  • its corresponding display interface may be in a multi-view mode or in a picture-in-picture mode.
  • its corresponding display interface is a single-view mode.
  • Open GL display rendering engine, the Open GL renderer and the display buffer area in the single-view mode in the accompanying drawings 9 and 10 are the same as the Open GL display rendering engine, the Open GL renderer and the display buffer in the dual-view mode.
  • the display buffer area is the same Open GL display rendering engine, Open GL renderer and display buffer area respectively.
  • Open GL display rendering engine, Open GL rendering device and display buffer area are all drawn under single-view mode and dual-view mode.
  • FIG. 12 it is a schematic flowchart of another picture-in-picture-based image processing method provided by the embodiment of the present application. This method can be applied to the software structure shown in FIG. 8, and as shown in FIG. 12, it mainly includes the following steps.
  • S901. Trigger a video shooting operation in a picture-in-picture shooting mode.
  • the video shooting operation in the picture-in-picture shooting mode is triggered.
  • the video image includes a first image frame and at least one second image frame, the first image frame at least partially surrounds the at least one second image frame, the first image frame and the at least one second image frame are respectively A video image synthesized from frame images of at least two channels of video data.
  • the video capture operation of the picture-in-picture event management module is transmitted to the picture-in-picture drawing management module, and the picture-in-picture drawing management module renders the frame images of at least two channels of video data captured to obtain a video image, which can be transmitted to The pre-display surface is displayed and transmitted to the multi-cam encoding module for encoding processing.
  • the picture-in-picture drawing management module performs rendering processing on at least two frames of captured video data to obtain a video image, which can be transmitted to the multi-cam encoding module for encoding processing to generate a video file.
  • the user needs to adjust the position and/or size of the second image frame displayed on the display interface.
  • the first adjustment operation is triggered, and the first adjustment operation is used to instruct to adjust the position and/or size of the second image frame. or size.
  • the picture-in-picture event management module starts the adjustment of the secondary image picture.
  • the picture-in-picture event management module After receiving the first adjustment operation, the picture-in-picture event management module starts the adjustment of the secondary image screen.
  • the auxiliary image frame is the second image frame to be adjusted by the first adjustment operation among the at least two second image frames.
  • the picture-in-picture event management module sends the first adjustment operation to the picture-in-picture coordinate management module.
  • the picture-in-picture coordinate management module may determine the superimposition position of the auxiliary image frame in the first image frame and the size of the auxiliary image frame according to the first adjustment operation. For example, the coordinate position of the vertex of the auxiliary image frame and the scaling ratio of the auxiliary image frame can be determined.
  • the picture-in-picture drawing management module acquires the superimposed position and zoom ratio in the first image picture from the picture-in-picture coordinate management module, and adjusts the frame image of the video data corresponding to the auxiliary image picture in the synthesis strategy in the video image. Display position and display size. And call the Open GL renderer to render the frame images in at least two channels of video data.
  • the compositing strategy is information for compositing the display position and display size of the first image frame and at least one second image frame.
  • the picture-in-picture drawing management module acquires the superposition position and scaling ratio in the first image frame in the picture-in-picture coordinate management module, it can adjust the compositing strategy according to the superimposing position and scaling ratio in the first image frame.
  • the Open GL renderer renders the frame images in at least two channels of video data according to the adjusted synthesis strategy to obtain the video image to be displayed. Display the video image to be displayed through the display interface, and encode it through the multi-cam encoding module.
  • the picture-in-picture drawing management module renders the frame images in at least two channels of video data by calling the Open GL renderer to obtain the video image to be displayed, which is the adjusted video image.
  • the picture-in-picture drawing management module sends the video image to be displayed to the multi-cam encoding module, and the multi-cam encoding module encodes the received video image to generate a video file.
  • the video file generated in step S903 and the video file generated in this step are one video file, and the video file is stored in the electronic device.
  • the present application also provides an electronic device, the electronic device is used for storing computer program instructions in a memory and for executing the program instructions, wherein, when the computer program instructions are processed When the device is executed, the electronic device is triggered to execute some or all of the steps in the above method embodiments.
  • the electronic device 1400 may include a processor 1410, an external memory interface 1420, an internal memory 1421, a universal serial bus (universal serial bus, USB) interface 1430, a charging management module 1440, a power management module 1441, a battery 1442, antenna 1, antenna 2, mobile communication module 1450, wireless communication module 1460, audio module 1470, speaker 1470A, receiver 1470B, microphone 1470C, earphone jack 1470D, sensor module 1480, button 1490, motor 1491, indicator 1492, camera 1493, a display screen 1494, and a subscriber identification module (subscriber identification module, SIM) card interface 1495, etc.
  • SIM subscriber identification module
  • the sensor module 1480 may include a pressure sensor 1480A, a gyroscope sensor 1480B, an air pressure sensor 1480C, a magnetic sensor 1480D, an acceleration sensor 1480E, a distance sensor 1480F, a proximity light sensor 1480G, a fingerprint sensor 1480H, a temperature sensor 1480J, a touch sensor 1480K, and ambient light Sensor 1480L, bone conduction sensor 1480M, etc.
  • the structure shown in the embodiment of the present invention does not constitute a specific limitation on the electronic device 1400 .
  • the electronic device 1400 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 1410 may include one or more processing units, for example: the processor 1410 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor (ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • graphics processing unit graphics processing unit
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 1410 for storing instructions and data.
  • the memory in processor 1410 is a cache memory.
  • the memory may hold instructions or data that the processor 1410 has just used or recycled. If the processor 1410 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 1410 is reduced, thus improving the efficiency of the system.
  • processor 1410 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL).
  • processor 1410 may include multiple sets of I2C buses.
  • the processor 1410 may be respectively coupled to the touch sensor 1480K, the charger, the flashlight, the camera 1493 and the like through different I2C bus interfaces.
  • the processor 1410 may be coupled to the touch sensor 1480K through the I2C interface, so that the processor 1410 and the touch sensor 1480K communicate through the I2C bus interface to realize the touch function of the electronic device 1400 .
  • the I2S interface can be used for audio communication.
  • processor 1410 may include multiple sets of I2S buses.
  • the processor 1410 may be coupled to the audio module 1470 through an I2S bus to implement communication between the processor 1410 and the audio module 1470 .
  • the audio module 1470 can transmit audio signals to the wireless communication module 1460 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.
  • the PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal.
  • the audio module 1470 and the wireless communication module 1460 can be coupled through a PCM bus interface.
  • the audio module 1470 can also transmit audio signals to the wireless communication module 1460 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • a UART interface is generally used to connect the processor 1410 and the wireless communication module 1460 .
  • the processor 1410 communicates with the Bluetooth module in the wireless communication module 1460 through the UART interface to realize the Bluetooth function.
  • the audio module 1470 can transmit audio signals to the wireless communication module 1460 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.
  • the MIPI interface can be used to connect the processor 1410 with the display screen 1494, the camera 1493 and other peripheral devices.
  • MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc.
  • the processor 1410 communicates with the camera 1493 through the CSI interface to realize the shooting function of the electronic device 1400 .
  • the processor 1410 communicates with the display screen 1494 through the DSI interface to realize the display function of the electronic device 1400 .
  • the GPIO interface can be configured by software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 1410 with the camera 1493 , the display screen 1494 , the wireless communication module 1460 , the audio module 1470 , the sensor module 1480 and so on.
  • the GPIO interface can also be configured as I14C interface, I14S interface, UART interface, MIPI interface, etc.
  • the USB interface 1430 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like.
  • the USB interface 1430 can be used to connect a charger to charge the electronic device 1400, and can also be used to transmit data between the electronic device 1400 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.
  • the interface connection relationship among the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 1400 .
  • the electronic device 1400 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
  • the charging management module 1440 is used for receiving charging input from the charger.
  • the charger may be a wireless charger or a wired charger.
  • the charging management module 1440 can receive charging input from a wired charger through the USB interface 1430 .
  • the charging management module 1440 may receive wireless charging input through a wireless charging coil of the electronic device 1400 . While the charging management module 1440 is charging the battery 1442 , it can also provide power for electronic devices through the power management module 1441 .
  • the power management module 1441 is used for connecting the battery 1442 , the charging management module 1440 and the processor 1410 .
  • the power management module 1441 receives the input of the battery 1442 and/or the charging management module 1440, and supplies power for the processor 1410, the internal memory 1421, the display screen 1494, the camera 1493, and the wireless communication module 1460, etc.
  • the power management module 1441 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance).
  • the power management module 1441 can also be set in the processor 1410 .
  • the power management module 1441 and the charging management module 1440 may also be set in the same device.
  • the wireless communication function of the electronic device 1400 can be realized by the antenna 1, the antenna 2, the mobile communication module 1450, the wireless communication module 1460, the modem processor and the baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in electronic device 1400 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna may be used in conjunction with a tuning switch.
  • the mobile communication module 1450 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 1400 .
  • the mobile communication module 1450 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.
  • the mobile communication module 1450 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module 1450 can also amplify the signal modulated by the modem processor, convert it into electromagnetic wave and radiate it through the antenna 1 .
  • at least part of the functional modules of the mobile communication module 1450 may be set in the processor 1410 .
  • at least part of the functional modules of the mobile communication module 1450 and at least part of the modules of the processor 1410 may be set in the same device.
  • a modem processor may include a modulator and a demodulator.
  • the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the low-frequency baseband signal is passed to the application processor after being processed by the baseband processor.
  • the application processor outputs sound signals through audio equipment (not limited to speaker 1470A, receiver 1470B, etc.), or displays images or videos through display screen 1494 .
  • the modem processor may be a stand-alone device.
  • the modem processor may be independent from the processor 1410, and be set in the same device as the mobile communication module 1450 or other functional modules.
  • the wireless communication module 1460 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • the wireless communication module 1460 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 1460 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 1410 .
  • the wireless communication module 1460 can also receive the signal to be sent from the processor 1410 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
  • the antenna 1 of the electronic device 1400 is coupled to the mobile communication module 1450, and the antenna 2 is coupled to the wireless communication module 1460, so that the electronic device 1400 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc.
  • GSM global system for mobile communications
  • GPRS general packet radio service
  • code division multiple access code division multiple access
  • CDMA broadband Code division multiple access
  • WCDMA wideband code division multiple access
  • time division code division multiple access time-division code division multiple access
  • TD-SCDMA time-division code division multiple
  • the GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • Beidou navigation satellite system beidou navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the electronic device 1400 implements a display function through a GPU, a display screen 1494, and an application processor.
  • the GPU is a microprocessor for image processing, connected to the display screen 1494 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 1410 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 1494 is used to display images, videos and the like.
  • Display 1494 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device 1400 may include 1 or N display screens 1494, where N is a positive integer greater than 1.
  • the electronic device 1400 can realize the shooting function through the ISP, the camera 1493 , the video codec, the GPU, the display screen 1494 and the application processor.
  • the ISP is used to process the data fed back by the camera 1493 .
  • the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
  • ISP can also perform algorithm optimization on image noise, brightness, and skin color.
  • ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be located in the camera 1493.
  • Camera 1493 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other image signals.
  • the electronic device 1400 may include 1 or N cameras 1493, where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 1400 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 1400 may support one or more video codecs.
  • the electronic device 1400 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
  • the NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • Applications such as intelligent cognition of the electronic device 1400 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.
  • the external memory interface 1420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1400.
  • the external memory card communicates with the processor 1410 through the external memory interface 1420 to implement a data storage function. Such as saving music, video and other files in the external memory card.
  • the internal memory 1421 may be used to store computer-executable program codes including instructions.
  • the internal memory 1421 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data created during the use of the electronic device 1400 (such as audio data, phonebook, etc.) and the like.
  • the internal memory 1421 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the processor 1410 executes various functional applications and data processing of the electronic device 1400 by executing instructions stored in the internal memory 1421 and/or instructions stored in a memory provided in the processor.
  • the electronic device 1400 may implement an audio function through an audio module 1470 , a speaker 1470A, a receiver 1470B, a microphone 1470C, an earphone interface 1470D, and an application processor. Such as music playback, recording, etc.
  • the audio module 1470 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.
  • the audio module 1470 may also be used to encode and decode audio signals.
  • the audio module 1470 may be set in the processor 1410 , or some functional modules of the audio module 1470 may be set in the processor 1410 .
  • Loudspeaker 1470A also called “horn" is used to convert audio electrical signals into sound signals.
  • Electronic device 1400 can listen to music through speaker 1470A, or listen to hands-free calls.
  • Receiver 1470B also called “earpiece” is used to convert audio electrical signals into audio signals.
  • the receiver 1470B can be placed close to the human ear to receive the voice.
  • Microphone 1470C also known as “microphone” or “microphone” is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 1470C with the human mouth, and input the sound signal to the microphone 1470C.
  • the electronic device 1400 may be provided with at least one microphone 1470C. In some other embodiments, the electronic device 1400 may be provided with two microphones 1470C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the electronic device 1400 can also be provided with three, four or more microphones 1470C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.
  • the earphone interface 1470D is used to connect wired earphones.
  • the earphone interface 1470D may be a USB interface 1430, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the pressure sensor 1480A is used to sense the pressure signal and convert the pressure signal into an electrical signal.
  • pressure sensor 1480A may be located on display screen 1494 .
  • pressure sensors 1480A such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors.
  • a capacitive pressure sensor may be comprised of at least two parallel plates with conductive material.
  • the electronic device 1400 may also calculate the touched position according to the detection signal of the pressure sensor 1480A.
  • touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.
  • the gyro sensor 1480B can be used to determine the motion posture of the electronic device 1400 .
  • the angular velocity of the electronic device 1400 around three axes may be determined by the gyro sensor 1480B.
  • the gyro sensor 1480B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 1480B detects the shaking angle of the electronic device 1400, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 1400 through reverse motion to achieve anti-shake.
  • the gyro sensor 1480B can also be used for navigation and somatosensory game scenes.
  • the air pressure sensor 1480C is used to measure air pressure.
  • the electronic device 1400 calculates the altitude based on the air pressure value measured by the air pressure sensor 1480C to assist positioning and navigation.
  • the magnetic sensor 1480D includes a Hall sensor.
  • the electronic device 1400 may detect opening and closing of the flip holster using the magnetic sensor 1480D.
  • the electronic device 1400 can detect the opening and closing of the flip according to the magnetic sensor 1480D.
  • features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 1480E can detect the acceleration of the electronic device 1400 in various directions (generally three axes).
  • the magnitude and direction of gravity can be detected when the electronic device 1400 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
  • the electronic device 1400 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 1400 can use the distance sensor 1480F for distance measurement to achieve fast focusing.
  • Proximity light sensor 1480G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes.
  • the light emitting diodes may be infrared light emitting diodes.
  • the electronic device 1400 emits infrared light through the light emitting diode.
  • Electronic device 1400 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 1400 . When insufficient reflected light is detected, the electronic device 1400 may determine that there is no object near the electronic device 1400 .
  • the electronic device 1400 can use the proximity light sensor 1480G to detect that the user holds the electronic device 1400 close to the ear to make a call, so as to automatically turn off the screen to save power.
  • the proximity light sensor 1480G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.
  • the ambient light sensor 1480L is used for sensing ambient light brightness.
  • the electronic device 1400 can adaptively adjust the brightness of the display screen 1494 according to the perceived ambient light brightness.
  • the ambient light sensor 1480L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 1480L can also cooperate with the proximity light sensor 1480G to detect whether the electronic device 1400 is in the pocket to prevent accidental touch.
  • the fingerprint sensor 1480H is used to collect fingerprints.
  • the electronic device 1400 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.
  • the temperature sensor 1480J is used to detect temperature.
  • the electronic device 1400 uses the temperature detected by the temperature sensor 1480J to implement a temperature processing strategy. For example, when the temperature reported by the temperature sensor 1480J exceeds the threshold, the electronic device 1400 may reduce the performance of a processor located near the temperature sensor 1480J, so as to reduce power consumption and implement thermal protection.
  • the electronic device 1400 when the temperature is lower than another threshold, the electronic device 1400 heats the battery 1442 to prevent the electronic device 1400 from being shut down abnormally due to the low temperature.
  • the electronic device 1400 boosts the output voltage of the battery 1442 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 1480K also known as "touch device”.
  • the touch sensor 1480K can be arranged on the display screen 1494, and the touch sensor 1480K and the display screen 1494 form a touch screen, also called “touch screen”.
  • the touch sensor 1480K is used to detect a touch operation acting on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • Visual output related to touch operations can be provided through the display screen 1494 .
  • the touch sensor 1480K may also be disposed on the surface of the electronic device 1400 , which is different from the position of the display screen 1494 .
  • the bone conduction sensor 1480M can acquire vibration signals. In some embodiments, the bone conduction sensor 1480M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 1480M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 1480M can also be disposed in the earphone, combined into a bone conduction earphone.
  • the audio module 1470 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the vocal part acquired by the bone conduction sensor 1480M, so as to realize the voice function.
  • the application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 1480M, so as to realize the heart rate detection function.
  • the keys 1490 include a power key, a volume key, and the like. Key 1490 may be a mechanical key. It can also be a touch button.
  • the electronic device 1400 may receive key input and generate key signal input related to user settings and function control of the electronic device 1400 .
  • the motor 1491 can generate a vibrating reminder.
  • the motor 1491 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.
  • touch operations applied to different applications may correspond to different vibration feedback effects.
  • the motor 1491 can also correspond to different vibration feedback effects for touch operations on different areas of the display screen 1494 .
  • Different application scenarios for example: time reminder, receiving information, alarm clock, games, etc.
  • the touch vibration feedback effect can also support customization.
  • the indicator 1492 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • SIM card interface 1495 is used for connecting SIM card.
  • the SIM card can be connected and separated from the electronic device 1400 by inserting it into the SIM card interface 1495 or pulling it out from the SIM card interface 1495 .
  • the electronic device 1400 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
  • SIM card interface 1495 can support Nano SIM card, Micro SIM card, SIM card, etc.
  • the same SIM card interface 1495 can insert multiple cards at the same time. The types of the multiple cards may be the same or different.
  • the SIM card interface 1495 is also compatible with different types of SIM cards.
  • the SIM card interface 1495 is also compatible with external memory cards.
  • the electronic device 1400 interacts with the network through the SIM card to implement functions such as calling and data communication.
  • the electronic device 1400 adopts an eSIM, that is, an embedded SIM card.
  • the eSIM card can be embedded in the electronic device 1400 and cannot be separated from the electronic device 1400 .
  • the present application also provides a computer storage medium, wherein the computer storage medium can store a program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the parts in the above-mentioned embodiments or all steps.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (English: read-only memory, abbreviated: ROM) or a random access memory (English: random access memory, abbreviated: RAM), etc.
  • an embodiment of the present application also provides a computer program product, the computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer executes part or part of the above method embodiments. All steps.
  • "at least one” means one or more, and “multiple” means two or more.
  • “And/or” describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate that A exists alone, A and B exist simultaneously, or B exists alone. Among them, A and B can be singular or plural.
  • the character “/” generally indicates that the contextual objects are an “or” relationship.
  • “At least one of the following” and similar expressions refer to any combination of these items, including any combination of single items or plural items.
  • At least one of a, b, and c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c may be single or multiple.
  • any function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, referred to as ROM), random access memory (random access memory, referred to as RAM), magnetic disk or optical disc, etc., which can store program codes. medium.

Abstract

本申请实施例提供了一种基于画中画的图像处理方法、设备、存储介质和程序产品，所述方法包括接收画中画拍摄模式下的视频拍摄操作；根据视频拍摄操作拍摄视频图像，视频图像包含第一图像画面及至少一个第二图像画面，第一图像画面至少部分包围至少一个第二图像画面；接收第一调整操作，第一调整操作用以调整第二图像画面的位置和/或大小；根据第一调整操作，实时调整视频图像的合成策略，合成策略是用以合成第一图像画面及至少一个第二图像画面的显示位置及显示大小的信息；根据调整后的合成策略，将至少两路视频数据的帧图像合成待显示的视频图像。用以对画中画的第二图像画面的大小及位置进行调整的目的，更好的满足用户需求，提高用户体验。

Description

基于画中画的图像处理方法、设备、存储介质和程序产品
本申请要求于2021年6月16日提交中国专利局、申请号为202110668551.5、申请名称为“基于画中画的图像处理方法、设备、存储介质和程序产品”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，具体地涉及一种基于画中画的图像处理方法、设备、存储介质和程序产品。
背景技术
画中画技术是使用两个播放窗口叠加，进而同时呈现两路视频信号的技术。
在现有的基于画中画的视频播放方案中，终端设备显示的播放界面中包括主窗口和辅窗口，终端设备在主窗口和辅窗口中分别显示两路视频信号。其中，主窗口大小大于辅助窗口的窗口大小，且主窗口通常为整个播放界面，而辅窗口通常叠加显示在主窗口之上。
为了提高用户体验，手机、平板电脑等电子设备通常配置多个摄像头，例如在电子设备上分别配置一个前置摄像头和一个后置摄像头。用户可以根据自己的需求选择相应的拍摄模式，例如，前摄模式、后摄模式、前后双摄模式等。
用户可以使用至少两个摄像头，采用画中画模式进行拍摄。而在视频拍摄的场景中，用户可能在视频拍摄过程中需要对辅窗口的大小或位置进行调整。但是，现有技术中画中画模式的拍摄场景中，辅窗口图像画面的大小及位置是固定不变的，无法对辅窗口图像画面的大小或位置进行调整，无法满足用户需求，导致用户体验较差。
发明内容
有鉴于此，本申请实施例提供一种基于画中画的图像处理方法、设备、存储介质和程序产品，以利于解决现有技术无法对辅窗口的大小或位置进行调整时，导致用户体验较差的问题。
第一方面，本申请实施例提供了一种基于画中画的图像处理方法，应用于电子设备，包括：
接收画中画拍摄模式下的视频拍摄操作；
根据所述视频拍摄操作拍摄视频图像，其中，所述视频图像包含第一图像画面及至少一个第二图像画面，所述第一图像画面至少部分包围所述至少一个第二图像画面，所述第一图像画面和所述至少一个第二图像画面分别是至少两路视频数据的帧图像合 成的视频图像；
接收第一调整操作，所述第一调整操作用以调整所述第二图像画面的位置和/或大小；
根据所述第一调整操作，实时调整所述视频图像的合成策略，所述合成策略是用以合成所述第一图像画面及至少一个第二图像画面的显示位置及显示大小的信息；
根据调整后的所述合成策略，将所述至少两路视频数据的帧图像合成待显示的视频图像。
优选地，所述方法还包括：
将拍摄过程中，实时调整的所述视频图像进行编码生成视频文件；存储所述视频文件。
优选地，所述第一调整操作包括缩放操作或拖动操作中的至少一种。
优选地，在所述第一调整操作为缩放操作时，所述根据所述第一调整操作，调整所述视频图像的合成策略包括：
根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，其中，所述辅图像画面是所述至少一个第二图像画面；
根据所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略。
优选地，在所述第一调整操作为拖动操作时，所述根据所述第一调整操作，调整视频图像的合成策略包括：
根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置；
根据所述辅图像画面在第一图像画面上的叠加位置，调整合成策略。
优选地，所述根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小包括：
在所述第一调整操作的初始触控点为单个触控点，且落入所述辅图像画面的调整范围时，确定所述第一调整操作为缩放操作；
根据第一调整操作的触控轨迹，确定所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
优选地，所述根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小包括：
在所述第一调整操作的初始触控点为至少两个触控点时，根据所述至少两个触控点的触控轨迹，确定所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
优选地，所述根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置包括：
根据所述第一调整操作的触控轨迹，确定所述辅图像画面在第一图像画面上的叠加位置。
优选地，所述根据所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略包括：
根据所述辅图像画面的大小，在所述辅图像画面的尺寸小于预设尺寸阈值时，根 据所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略，其中，所述预设尺寸阈值为拼接显示模式下，所述第一图像画面与所述辅图像画面拼接显示的尺寸。
优选地，所述方法还包括：
在所述辅图像画面的尺寸不小于预设尺寸阈值时，将所述合成策略更新为拼接显示模式的合成策略。
优选地，所述根据所述视频拍摄操作录制视频图像包括：
根据所述视频拍摄操作获取至少两路视频数据；
将所述至少两路视频数据的帧图像按照画中画显示模式的合成策略，合成视频图像。
优选地，所述将所述至少两路视频数据的帧图像按照画中画显示模式的合成策略，合成视频图像包括：
根据所述画中画显示模式的合成策略，确定每路视频数据的帧图像对应的显示位置及显示大小；
根据每路视频数据的帧图像对应的显示大小调整每路视频数据的帧图像的大小；
根据调整后的每路视频数据的帧图像确定每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息；
根据所述每路视频数据的帧图像对应的显示位置、每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息，将所述至少两路视频数据的帧图像合成为视频图像。
第二方面，本申请实施例提供了一种电子设备，包括用于存储计算机程序指令的存储器和用于执行程序指令的处理器，其中，当该计算机程序指令被所述处理器执行时，触发所述电子设备执行第一方面任一项所述的方法。
第三方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质包括存储的程序，其中，在所述程序运行时控制所述计算机可读存储介质所在设备执行上述第一方面中任意一项所述的方法。
第四方面，本申请实施例提供了一种计算机程序产品，所述计算机程序产品包含可执行指令，当所述可执行指令在计算机上执行时，使得计算机执行上述第一方面中任意一项所述的方法。
采用本申请实施例提供的技术方案，在根据视频拍摄操作录制视频图像时，接收到第一调整操作，通过根据接收的第一调整操作对合成策略进行调整，以对画中画的第二图像画面的大小及位置进行调整的目的，更好的满足用户需求，提高用户体验。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其它的附图。
图1为本申请实施例提供的一种电子设备示意图；
图2为本申请实施例提供的一种画中画拍摄模式的场景示意图；
图3为本申请实施例提供的一种基于画中画的图像处理方法流程示意图；
图4a为本申请实施例提供的一种前后双摄模式拍摄场景示意图；
图4b为本申请实施例提供的一种前后画中画模式拍摄场景示意图；
图4c为本申请实施例提供的一种后置画中画模式拍摄场景示意图；
图5a为本申请实施例提供的一种辅图像画面调整的场景示意图；
图5b为本申请实施例提供的另一种辅图像画面调整的场景示意图；
图5c为本申请实施例提供的另一种辅图像画面调整的场景示意图；
图5d为本申请实施例提供的另一种辅图像画面调整的场景示意图；
图6a为本申请实施例提供的一种拼接显示模式的场景示意图；
图6b为本申请实施例提供的另一种拼接显示模式的场景示意图；
图6c为本申请实施例提供的另一种拼接显示模式的场景示意图；
图7为本申请实施例提供的另一种基于画中画的图像处理方法流程示意图；
图8为本申请实施例提供的一种电子设备的软件结构框图；
图9为本申请实施例提供的一种渲染场景示意图；
图10为本申请实施例提供的另一种渲染场景示意图；
图11a为本申请实施例提供的一种合成视频图像的场景示意图；
图11b为本申请实施例提供的另一种合成视频图像的场景示意图；
图12为本申请实施例提供的另一种基于画中画的图像处理方法流程示意图；
图13为本申请实施例提供的一种电子设备的结构示意图。
具体实施方式
为了更好的理解本申请的技术方案，下面结合附图对本申请实施例进行详细描述。
应当明确，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其它实施例，都属于本申请保护的范围。
在本申请实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。
应当理解，本文中使用的术语“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，甲和/或乙，可以表示：单独存在甲，同时存在甲和乙，单独存在乙这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
参见图1，为本申请实施例提供的一种电子设备示意图。在图1中以手机100为例对电子设备进行示例性说明，图1中示出了手机100的前视图和后视图，在手机100的前侧配置有两个前置摄像头111、112，在手机100的后侧面配置有四个后置摄像头121、122、123、124。通过配置的多个摄像头，可以为用户提供多种拍摄模式，例如，前摄模式、后摄模式、前后双摄模式等。用户可以根据拍摄场景，选择相应的拍摄模式进行拍摄，以提高用户体验。
可理解，图1所示仅为一种示例性说明，并不应当将其作为本申请保护范围的限制。例如，不同的手机，其摄像头的配置数量和配置位置可能不同。另外，本申请实施例涉及的电子设备除了手机以外，还可以为平板电脑、个人计算机(personal computer，PC)、个人数字助理(personal digital assistant，PDA)、智能手表、上网本、可穿戴电子设备、增强现实技术(augmented reality，AR)设备、虚拟现实(virtual reality，VR)设备、车载设备、智能汽车、智能音响、机器人、智能眼镜、智能电视等。
需要指出的是，在一些可能的实现方式中，电子设备也可能称为终端设备、用户设备(User Equipment，UE)等，本申请实施例对此不作限制。
在实际应用场景中，现在电子设备包含有至少两个摄像头，可以同时开启至少两个摄像头进行录像，使得录像获取的视频文件的视频图像包含多路摄像头的拍摄画面(基于用户选定的摄像头模式，按照摄像头模式所对应的画面组合策略，多路摄像头的拍摄画面组合为一个图像画面)。
这种同时使用至少两个摄像头进行同一视频文件拍摄的模式可以称为多路录像模式。在多路录像模式下，在录像过程中，在不中断视频录制的状态下，用户可以调整拍摄模式，例如调整录制视频所使用的摄像头和/或视频图像中多路摄像头画面的组合模式，以生成一个包含拍摄模式切换视频文件。同步的，在视频录制的过程中，电子设备的显示屏上会显示录制的视频文件的同步显示画面。显示画面与录制的视频文件中的画面保持一致。
参见图2，为本申请实施例提供的一种画中画模式拍摄的场景示例图。如图2所示，用户在通过电子设备进行视频拍摄的过程中，显示界面内可以实时显示拍摄的视频画面。另外，在显示界面内还包括拍摄模式选择窗口，用户可以在拍摄模式选择窗口内选择相应的拍摄模式进行视频拍摄。例如，前置单摄模式、后置单摄模式、前后双摄模式、前后画中画模式等。
在图2所示的应用场景中，用户选择前后画中画拍摄模式，此时前置摄像头拍摄前景画面和后置摄像头拍摄的后景画面被组合成一张画面，以实现画中画的效果。也就是说，在前后双摄模式下，分别通过前置摄像头和后置摄像头进行前景画面的采集和后景画面的采集，并在显示界面内分别显示前景画面以及后景画面的合成画面，并且，在后台将前景画面以及后景画面的合成画面编码保存为视频文件。
如图2所示，在电子设备以画中画模式拍摄视频时，从视觉效果上，显示的显示画面包括两部分，第一窗口以及第二窗口。第一窗口大小大于第二窗口的窗口大小，且第一窗口通常为整个播放界面，而第二窗口通常叠加显示在第一窗口之上。
但是，从图像数据的角度看，显示画面内显示的是一个视频图像，并不是第一窗口和第二窗口的组合显示，并不是独立的第一窗口图像与独立的第二窗口图像的组合显示。而是，电子设备后台获取前景画面以及后景画面的两路视频数据，按照画中画显示模式对应的图像合成策略，将前景画面处理为第一窗口图像，将后景画面处理为第二窗口图像，将第一窗口图像以及第二窗口图像合并为一张视频图像，然后分别发送到显示界面显示以及编码保存。
进一步的，如图2所示的视频录像场景中，在视频拍摄过程中，录制的视频画面 中前景画面中重要的图像内容可能被后景画面遮挡，或者，用户对当前录制的视频画面中前景画面与后景画面的布局不满意；此时，用户可能希望对后景画面(辅窗口)的大小或位置进行调整。但是，由于辅窗口仅仅是一种在视频画面上呈现出的视觉显示效果，其并不是一个独立的显示窗口，因此，无法基于针对窗口控件的调整方法对其进行调整，因而无法满足用户需求，导致用户体验较差。
针对该问题，在图2所示的视频录像场景中，通过用户可以输入第一调整操作，根据第一调整操作调整前景画面以及后景画面的合成方式，生成对应用户第一调整操作的视频图像，将该视频图像发送到显示界面显示以及编码保存，从而实现在画中画模式录制过程中，对录制的视频图像中第二窗口图像的大小和/或位置进行调整的目的，更好的满足用户需求，提高用户体验。
可理解，在视频录制过程中，除了需要在显示界面内对拍摄的视频进行显示以外，还需要将拍摄的视频编码为视频文件(例如，MP4格式的视频文件)，并存储在电子设备中。
参见图3，为本申请实施例提供的一种基于画中画的图像处理方法流程示意图。该方法可应用于图1所示的电子设备，如图3所示，其主要包括以下步骤。
步骤S301、接收画中画拍摄模式下的视频拍摄操作。
在本申请实施例中，电子设备涉及的拍摄模式可能包括单摄模式和多摄模式。其中，单摄模式可能包括前置单摄模式、后置单摄模式等；多摄模式可能包括前置双摄模式、后置双摄模式、前后双摄模式、前置画中画模式、后置画中画模式、前后画中画模式等。
其中，在单摄模式下，采用一个摄像头进行视频拍摄；在多摄模式下采用两个或两个以上摄像头进行视频拍摄。
具体地，在前置单摄模式下，采用一个前置摄像头进行视频拍摄；在后置单摄模式下，采用一个后置摄像头进行视频拍摄；在前置双摄模式下，采用两个前置摄像头进行视频拍摄；在后置双摄模式下，采用两个后置摄像头进行视频拍摄；在前后双摄模式下，采用一个前置摄像头和一个后置摄像头进行视频拍摄；在前置画中画模式下，采用两个前置摄像头进行视频拍摄，且将一个前置摄像头拍摄的画面置于另一个前置摄像头拍摄的画面之中；在后置画中画模式下，采用两个后置摄像头进行视频拍摄，且将一个后置摄像头拍摄的画面置于另一个后置摄像头拍摄的画面之中；在前后画中画模式下，采用一个前置摄像头和一个后置摄像头进行视频拍摄，且将前置摄像头或后置摄像头拍摄的画面置于后置摄像头或前置摄像头拍摄的画面之中。
参见图4a，为本申请实施例提供的一种前后双摄模式拍摄场景示意图。在前后双摄模式下，采用一个前置摄像头采集前景画面，采用一个后置摄像头采集后景画面，前景画面和后景画面在显示界面内同时显示。
参见图4b，为本申请实施例提供的一种前后画中画模式拍摄场景示意图。在前后画中画模式下，采用一个前置摄像头采集前景画面，采用一个后置摄像头采集后景画面，将前景画面置于后景画面之中。
参见图4c，为本申请实施例提供的一种后置画中画模式拍摄场景示意图。在后置画中画模式下，采用一个后置摄像头采集远景画面，采用另一个后置摄像头采集近 景画面，将近景画面置于远景画面之中。
本申请实施例涉及的拍摄模式可以为前置画中画模式、后置画中画模式、前后画中画模式中的任意一种。
需要指出的是，上述拍摄模式仅是本申请实施例列举的一些可能的实现方式，本领域技术人员可以根据实际需要配置其它形式的画中画拍摄模式，本申请实施例对此不作具体限制。
用户在电子设备中，若需要采用画中画拍摄模式进行视频的拍摄时，可以输入画中画拍摄模式下的视频拍摄操作。其中，用户可以通过触摸屏、物理按键、手势控制、语音控制等方式输入视频拍摄操作，本申请实施例对此不作具体限制。
步骤S302、根据视频拍摄操作拍摄视频图像。
其中，视频图像包含第一图像画面及至少一个第二图像画面，第一图像画面至少部分包围至少一个第二图像画面。第一图像画面和至少一个第二图像画面分别是至少两路视频数据的帧图像合成的视频图像。且第一图像画面是第一路视频数据中的帧图像的画面，至少一个第二图像画面分别是至少一个第二路视频数据中的帧图像的画面；第一路视频数据及至少一个第二路视频数据通过不同摄像装置采集的视频数据。
需要说明的是，在视频图像中的第一图像画面是指上文所述的第一窗口图像的画面，第二图像画面是指上文所述的第二窗口图像的画面。
在本申请实施例中，电子设备接收到画中画拍摄模式下的视频拍摄操作后，可以在画中画拍摄模式下通过摄像头采集到至少两路视频数据，并将其合成策略可以确定为画中画显示模式的合成策略。此时，电子设备根据画中画显示模式的合成策略可以将采集的至少两路视频数据的帧图像合成一个视频图像。在进行至少两路视频数据的合成时，可以按照画中画显示模式的合成策略中，预先设定的第一图像画面的显示位置、显示大小，每个第二图像画面的显示位置及显示大小，将第一图像画面对应的视频数据设置在其对应的显示位置处，且将每个第二图像画面对应的视频数据设置在其对应的显示位置处，将第一图像画面对应的视频数据的帧图像的大小调整为其对应的显示大小，将每个第二图像画面对应的视频数据的帧图像的大小调整为其各自对应的显示大小，然后将各路视频数据的帧图像进行合成，得到一个视频图像。其中，该视频图像中各第二图像画面的位置及大小即为画中画显示模式的合成策略中，预先设定的第二图像画面的显示位置及显示大小。也就是说，在本申请实施例中，是将拍摄的至少两路视频数据的帧图像合成为一个视频图像后，在进行后续的显示及编码。
需要说明的是，第一图像画面可以是按照显示屏的显示区域大小进行显示，此时可以将第一图像画面的显示位置设置为显示屏的显示区域的一点，显示大小设定为显示屏的显示区域的尺寸。
进一步地，根据视频拍摄操作录制视频图像包括：
根据视频拍摄操作获取至少两路视频数据；将至少两路视频数据的帧图像按照画中画显示模式的合成策略，合成录制的视频图像。
具体的，电子设备在接收到拍摄操作时，可以根据拍摄操作开启相应的至少两个摄像头，以便通过至少两个摄像头采集至少两路视频数据。其中，电子设备通过至少两个摄像头采集的至少两路视频数据的大小可以不同，例如，前置摄像头采集的视频 数据的大小为540*480，后置摄像头采集的视频数据的大小为1080*960。当然，至少两个摄像头采集的至少两路视频数据的大小也可以相同，本申请对此不作限制。
电子设备在获取了至少两路视频数据后，由于接收的是画中画拍摄模式的拍摄操作，因此，电子设备可以将合成策略设定为画中画显示模式的合成策略，将各路视频数据的帧图像按照合成策略中预先设置的各路视频数据的帧图像对应的显示大小及显示位置，进行渲染合并处理，得到一个视频图像。例如，将前置摄像头采集的视频图像按照预设画中画显示模式的合成策略中前置摄像头采集的视频图像对应的显示大小，将其进行缩放处理，并按照合成策略中前置摄像头采集的视频图像对应的显示位置，将前置摄像头采集的视频图像嵌入至后置摄像头采集的图像内部对应的位置，实现采集的前置摄像头与后置摄像头采集的至少两路视频数据的帧图像的合并，该合并后的图像为画中画显示模式，形成一个视频图像。
进一步地，将至少两路视频数据的帧图像按照画中画显示模式的合成策略，合成视频图像包括：
根据预设画中画显示模式的合成策略，确定每路视频数据的帧图像的显示位置及显示大小。根据每路视频数据的帧图像对应的显示大小调整每路视频数据的帧图像的大小。根据每路视频数据的帧图像，确定每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息。根据每路视频数据的帧图像的显示位置、每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息，将至少两路视频数据的帧图像对应合成为一个视频图像。
需要说明的是，在设置画中画显示模式的合成策略时，即为设置画中画显示模式中，各个图像画面的显示位置及显示大小。因此，在用户选择采用画中画拍摄模式进行录制时，电子设备可以在存储介质中获取预先设置的画中画显示模式的合成策略，并在该合成策略中获取各个图像画面的显示位置及显示大小的相关信息，进而可以确定出每路视频数据的帧图像的显示位置及显示大小。根据每路视频数据的帧图像的显示大小，分别对每路视频数据的帧图像的大小进行调整，例如进行缩放处理，或者进行相应大小的图像提取处理，得到每路视频数据的对应大小的帧图像。在调整大小之后的每路视频数据中的帧图像中提取出每路视频数据的帧图像的纹理信息及该纹理信息对应的位置信息。根据每路视频数据的帧图像对应的显示位置、每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息，将至少两路视频数据的帧图像对应合成为一个视频图像。
步骤S303、接收第一调整操作。
其中，第一调整操作用以指示调整第二图像画面的位置和/或大小。
在实际应用中，用户可能在视频拍摄过程中需要调整第二图像画面的位置和/或大小时，可以向电子设备输入第一调整操作，以便对相应的辅显示区域进行调整。其中，用户可以通过触摸屏、物理按键、手势控制、语音控制等方式输入第一调整操作，本申请实施例对此不作具体限制。
进一步地，第一调整操作包括缩放操作及拖动操作中的至少一种。即为，用户可以通过缩放操作对第二图像画面的大小及显示位置进行调整。可以通过拖动操作仅对第二图像画面的显示位置进行调整。
需要说明的是，在本申请实施例中，在至少一个第二图像画面中，将第一调整操作需要调整的第二图像画面统称为辅图像画面。
需要说明的是，现有的电子设备普遍具有触摸屏，因此用户可以通过触控触摸屏向电子设备输入第一调整操作。
作为一种可能的实现方式，在本申请实施例中，用户在需要对第二图像画面的大小和/或位置进行调整时，可以通过触摸屏向电子设备输入触控操作，电子设备可以根据用户输入的触控操作确定第一调整操作，以便对第二图像画面进行调整。
具体的，在第一调整操作的初始触控点为单个触控点，且落入辅图像画面的调整范围时，确定第一调整操作为缩放操作。
需要说明的是，在每个第二图像画面中预先设置了可以对第二图像画面的显示大小进行调整的调整区域，例如可以是第二图像画面的显示边界。
在本申请实施例中，在用户输入触控操作时，若初始触控点落在辅图像画面的调整范围内，则用户可以通过调整初始触控点的位置实现对辅图像画面的缩放。用户可以直接触控触摸屏，电子设备根据用户的触摸轨迹确定对辅图像画面的调整。即为，用户通过触摸输入触控操作时，如果触控操作的初始触控点为单个触控点，说明用户通过单指触控触摸屏，并且初始触控点落入辅图像画面的调整范围内时，说明用户想要调整辅图像画面，实现辅图像画面的显示尺寸及显示位置的调整。此时，电子设备确定第一调整操作作为缩放操作。
或者，在第一调整操作的初始触控点为至少两个触控点，确定第一调整操作为缩放操作。
也就是说，用户输入的触控点可以是单个，也可以是至少两个。在触控点是单个控制缩放时，可以参考上述内容，在此不再赘述。用户输入的触控点为至少两个时，可以直接确定第一调整操作为缩放操作。即为，用户是通过至少双指调整辅图像画面的大小及显示位置。
进一步地，在第一调整操作的初始触控点为至少两个触控点时，可以进一步检测至少两个触控点是否在辅图像画面内，如果触控点在辅图像画面外，说明是对其他图像画面的操控，并不是对辅图像画面的操控。因此，在电子设备检测到第一调整操作的至少两个初始触控点后，可以先检测至少两个初始触控点的坐标是否在辅图像画面内，例如，检测至少两个初始触控点的坐标位置是否在辅图像画面对应的坐标范围内。如果在辅图像画面内，则说明是对辅图像画面的操作，此时电子设备可以将第一调整操作确定为缩放操作。该种第一调整操作是用户通过至少两指对辅图像画面进行尺寸调整的。此种方式用户通过双指捏合的方式调整辅图像画面的大小。
或者，在第一调整操作的初始触控点为单个触控点，且未落入辅图像画面的调整区域内时，可以确定第一调整操作为拖动操作。
也就是说，用户输入的触控点为单个时，若单个触控点并没有落入辅图像画面的调整区域内时，则说明该触控点是用于对辅图像画面的位置的调整。即为，电子设备在检测到第一调整操作的初始触控点后，可以检测初始触控点的坐标位置是否落入辅图像画面的调整区域内，若未落入，则说明电子设备可以确定第一调整操作为拖动操作。
需要说明的是，确定第一调整操作是否为拖动操作还可以通过现有技术中的其他方式实现，本申请对此不做限制。
步骤S304、根据第一调整操作，实时调整视频图像的合成策略。
其中，合成策略是用以合成第一图像画面及至少一个第二图像画面的显示位置及显示大小的信息。
在本申请实施例中，电子设备确定了第一调整操作后，可以根据第一调整操作，确定出需调整的第二图像画面即为辅图像画面的在第一图像画面上的叠加位置及辅图像画面的大小，进而可以确定出调整后的合并策略，即为确定出各个视频数据对应的图像画面在待合成的视频图像中的显示位置及显示大小，实时更新已有的合并策略，以便后续进行至少两路视频数据的帧图像的合成时，按照更新后的合并策略，将至少两路视频数据的帧图像对应合成为一个视频图像，此时，该视频图像中的辅图像画面是按照用户调整的位置及大小显示出来的。
其中，辅图像画面是至少一个第二图像画面中，第一调整操作需调整的第二图像画面。
如上文所述，第一调整操作可以是缩放操作也可以是拖动操作。根据第一调整操作的不同，对辅图像画面的调整也不同，具体如下。
在第一调整操作为缩放操作时，根据第一调整操作，调整视频图像的合成策略包括：
根据第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
根据辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略。
具体的，在第一调整操作为缩放操作时，可以根据第一调整操作确定出用户调整后的辅图像画面的大小，进而可以根据调整大小之前，辅图像画面在第一图像画面上的叠加位置及调整后的辅图像画面的大小，确定出调整后的辅图像画面在第一图像画面上的叠加位置，进而根据缩放操作调整后的辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略，调整合成策略。
由于缩放操作可以是用户通过单指向触摸屏输入的，也可以是用过通过至少两指向触摸屏输入的，根据输入的初始触控点的个数不同，其确定调整后的辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小的方式也不同，具体如下。
根据第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小包括：
在第一调整操作的初始触控点为单个触控点，且落入辅图像画面的调整范围时，确定第一调整操作为缩放操作；根据第一调整操作的触控轨迹，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
具体的，电子设备在接收的第一调整操作是缩放操作时，可以根据此缩放操作，确定出辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。如上文所述，在用户通过触摸屏输入第一调整操作时，若用户输入的初始触控点为单个触控点，且落入辅图像画面的调整区域内时，说明第一调整操作为缩放操作，此时可以将初始触 控点视为辅窗口画面的调整点。电子可以根据第一调整操作的触控轨迹，确定出辅图像画面的调整点的调整位置。电子设备可以根据辅图像画面的调整点调整前的位置及调整后的位置计算出辅图像画面的大小，并根据辅图像画面的大小，调整前辅图像画面在第一图像画面上的叠加位置，确定出调整辅图像画面的大小后，辅图像画面在第一图像画面上的叠加位置。这样，在确定出辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小后，可以调整合成策略中记录的辅图像画面的显示位置及显示大小。即为，调整视频图像中辅图像画面对应的一路视频数据的帧图像在待合成的视频图像中的显示位置及显示大小，更新合成策略，如图5a所示。
或者，根据第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小包括：
在第一调整操作的初始触控点为至少两个触控点时，根据至少两个触控点的触控轨迹，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
若用户通过触摸屏输入第一调整操作时，在用户输入的初始触控点为至少两个，则说明用户是通过至少双指的捏合操作实现对辅图像画面的大小调整，此时可以确定第一调整操作为缩放操作，电子设备可以根据至少两个触控点的触控轨迹确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。这样，在确定出辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小后，可以根据辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，相应的调整合成策略中记录的辅图像画面的显示位置及显示大小，即为调整合成策略，如图5b所示。
需要说明的是，在第一调整操作为缩放操作时，对辅图像画面的大小进行缩放操作时，会导致辅图像画面的显示位置也发生变化。因此，调整合成策略时，需要同时确定辅图像画面的大小及辅图像画面在第一图像画面的叠加位置。
在第一调整操作为拖动操作时，根据一调整操作，调整视频图像的合成策略包括：
根据第一调整操作，确定辅图像画面在第一图像画面上的叠加位置。根据辅图像画面在第一图像画面上的叠加位置，调整合成策略。
在本申请实施例中，若用户仅需对辅图像画面的显示位置进行调整，并不需要对辅图像画面的大小进行调整，此时可以输入拖动操作，即为第一调整操作为拖动操作。在电子设备接收到拖动操作后，可以根据拖动操作确定出调整后的辅图像画面在第一图像画面上的叠加位置，进而根据辅图像画面在第一图像画面上的叠加位置，调整合成策略。
进一步地，在用户通过触控触摸屏的方式输入拖动操作时，根据第一调整操作，确定辅图像画面在第一图像画面上的叠加位置包括：
根据第一调整操作的触控轨迹，确定辅图像画面在第一图像画面上的叠加位置。
具体的，用户通过触摸屏输入的第一调整操作为拖动操作时，电子设备可以根据第一调整操作的触控轨迹确定出辅图像画面在第一图像画面上的叠加位置。这样，在确定出辅图像画面在第一图像画面上的叠加位置后，可以根据辅图像画面在第一图像画面上的叠加位置，调整合成策略中记录的辅图像画面的显示位置，即为调整合成策略，如图5c所示。
需要说明的是，在本申请实施例中，当第一调整操作为缩放操作时，根据第一调整操作确定辅图像画面的大小可以是确定辅图像画面的缩放比例。此时，合成策略中记录该缩放比例，以便电子设备可以根据该缩放比例，对辅图像画面对应的视频数据的帧图像进行缩放处理，得到相应大小的图像。当然，也可以是直接确定出调整后的辅图像画面的大小，此时，合成策略中记录辅图像画面的大小，以便可以电子设备可以直接根据辅图像画面的大小，在辅图像画面对应的视频数据的帧图像中提取出相应大小的图像。
进一步地，根据辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略包括：
根据辅图像画面的大小，在辅图像画面的尺寸小于预设尺寸阈值时，根据辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略。
其中，预设尺寸阈值为拼接显示模式下，第一图像画面与辅图像画面拼接显示的尺寸。在拼接显示模式中，第一图像画面与辅图像画面拼接显示的尺寸可以相等，如图6a所示，也可以是第一图像画面与辅图像画面为等宽拼接显示，如图6b所示，还可以是第一图像画面与辅图像画面为等长拼接显示，如图6c所示，当然，还可以是其他的拼接显示模式，本申请对此不作限制。
需要说明的是，等宽是指沿电子设备的短边方向宽度相等，等长是指沿电子设备的长边方向长度相等。
在本申请实施例中，电子设备通过至少两个摄像头拍摄的至少两路视频数据的帧图像以画中画的显示模式形成视频图像后，在显示界面中显示出。由于在本申请实施例中，用户可以根据实际需求调整第二图像画面的大小，为了防止第二图像画面过大，影响第一图像画面的显示，可以限定第二图像画面的显示尺寸阈值。基于此，可以预先设置第二图像画面的显示尺寸阈值，电子设备在根据辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小对合成策略进行调整之前，先确定需要调整的辅图像画面的尺寸是否小于显示尺寸阈值，若调整后的辅图像画面的大小小于显示尺寸阈值，则说明辅图像画面的大小在正常范围内，此时可以根据辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小对合成策略进行调整。
需要说明的是，可以将预设尺寸阈值设置为拼接显示模式下，辅图像画面与第一图像画面拼接显示时的尺寸，当然，还可以将预设尺寸阈值设置为其他尺寸值，本申请对此不作限制。
进一步地，上述方法还包括：在辅图像画面的大小不小于预设尺寸阈值时，将合成策略更新为拼接显示模式的合成策略。
在本申请实施例中，电子设备检测到调整后的辅图像画面的大小等于或大于预设尺寸阈值，则说明辅图像画面会影响其他图像画面的显示，此时，可以将合成策略由画中画显示模式更新为拼接显示模式的合成策略，以便后续的多路视频数据的帧图像可以通过拼接显示模式显示，不再进行画中画显示，如图5d所示。
步骤S305、根据调整后的合成策略，将至少两路视频数据的帧图像对应合成待显示的视频图像。
在本申请实施例中，电子设备在调整了合成策略后，在后续将至少两路视频数据 的帧图像进行视频图像的合成时，使用调整后的合成策略，以便后续合成的视频图像中辅图像画面的显示位置及显示大小是按照用户调整后的显示位置及显示大小显示的。
具体的，在合成策略中记录了各路视频数据的帧图像对应的图像画面的显示位置及显示大小，此时，电子设备可以通过解析合成策略，获知各路视频数据的帧图像对应的图像画面的显示位置及显示大小，将各路视频数据的帧图像按照各自的显示位置及显示大小，合成为一个视频图像。例如，电子设备在合成策略中获知调整后的辅图像画面的显示位置，则可以将辅图像画面对应的一路视频数据的帧图像放置在待合成的视频图像中的相应的位置处，且根据辅图像画面的大小对辅图像画面对应的一路视频数据的帧图像进行相应大小的视频画面的提取或对帧图像进行缩放处理，得到相应大小的图像，将该图像的纹理信息设置在相应的显示位置处。
参见图7，为本申请实施例提供的另一种基于画中画的图像处理方法流程示意图。该方法可应用于图1所示的电子设备，其在图3所示实施例的基础上还包括生成视频文件，主要包括以下步骤。
步骤S601、接收画中画拍摄模式下的视频拍摄操作。
具体可参考步骤S301在此不再赘述。
步骤S602、根据视频拍摄操作拍摄视频图像。
其中，视频图像包含第一图像画面及至少一个第二图像画面，第一图像画面至少部分包围至少一个第二图像画面，第一图像画面和所述至少一个第二图像画面分别是至少两路视频数据的帧图像合成的视频图像。
具体可参考步骤S302在此不再赘述。
步骤S603、接收第一调整操作。
其中，第一调整操作用以调整所述第二图像画面的位置和/或大小。
具体可参考步骤S303在此不再赘述。
步骤S604、根据第一调整操作，实时调整所述视频图像的合成策略。
其中，合成策略是用以合成第一图像画面及至少一个第二图像画面的显示位置及显示大小的信息。
具体可参考步骤S304在此不再赘述。
步骤S605、根据调整后的合成策略，将至少两路视频数据的帧图像合成待显示的视频图像。
步骤S606、将拍摄过程中，实时调整的视频图像进行编码，生成视频文件，存储视频文件。
本申请实施例在视频拍摄过程中，需要将拍摄的视频图像编码为视频文件。此时，将拍摄过程中，实时调整的视频图像发送至编码模块，以便实现边录制边编码，实现所看即所得的效果。即为，在本申请中，在合成视频图像后，可以将该视频图像发送至相应的编码模块进行编码，生成视频文件，并存储生成的视频文件，从而可以保证再次播放视频文件时，显示界面中显示的视频图像也是通过画中画的模式进行显示，并且可以在播放的视频图像中体现出对第二图像画面的调整。
参见图8，为本申请实施例提供的一种电子设备的软件结构框图。分层架构将软 件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将安卓(Android)系统分为四层，从上至下分别为应用层、框架层、硬件抽象层和硬件层。
应用层(Application，App)可以包括一系列应用程序包。例如，该应用程序包可以包括相机应用。应用层又可以分为显示界面(UI)和应用逻辑。
相机应用的应用界面包括单景模式、多景模式、画中画模式等。其中，在单景模式下仅显示一个拍摄画面；在多景模式下并列显示至少两个拍摄画面；在画中画模式下显示至少两个拍摄画面，其中至少一个拍摄画面位于另一个拍摄画面之中。
相机应用的应用逻辑包括多摄框架和相机管理。其中，多摄框架包括画中画事件管理模块、画中画坐标管理模块、画中画绘制管理模块及多摄编码模块等。画中画事件管理模块用于控制合成策略的调整；画中画坐标管理模块用于确定各路视频数据的帧图像对应的图像画面的显示位置及显示大小；画中画绘制管理模块用于通过调用Open GL渲染器对至少两路视频数据中的帧图像进行渲染处理。多摄编码模块，用于对显示界面显示的视频图像进行编码，生成视频文件。相机管理包括设备管理模块、Surface管理模块、会话管理模块等。在Android系统中，Surface对应一块屏幕缓冲区，用于保存当前窗口的像素数据。
框架层(Framework，FWK)为应用层的应用程序提供应用编程接口(application programming interface，API)和编程框架，包括一些预先定义的函数。在图8中，框架层包括相机访问接口(Camera2 API)，Camera2 API是Android推出的一套访问摄像头设备的接口，其采用管道式的设计，使数据流从摄像头流向Surface。Camera2 API包括相机管理(CameraManager)和相机设备(CameraDevice)。CameraManager为Camera设备的管理类，通过该类对象可以查询设备的Camera设备信息，得到CameraDevice对象。CameraDevice提供了Camera设备相关的一系列固定参数，例如基础的设置和输出格式等。
硬件抽象层(HAL)是位于操作系统内核与硬件电路之间的接口层，其目的在于将硬件抽象化。它隐藏了特定平台的硬件接口细节,为操作系统提供虚拟硬件平台,使其具有硬件无关性,可在多种平台上进行移植。在图8中，HAL包括相机硬件抽象层(Camera HAL)，Camera HAL包括设备(Device)1、设备(Device)2、设备(Device)3等。可理解，该Device1、Device2和Device3为抽象的设备。
硬件层(HardWare，HW)是位于操作系统最底层的硬件。在图8中，HW包括相机设备(CameraDevice)1、相机设备(CameraDevice)2、相机设备(CameraDevice)3等。其中，CameraDevice1、CameraDevice2和CameraDevice3可对应于电子设备上的多个摄像头。
在一种可能的实现方式中，画中画绘制管理模块可以调用开放图形库(Open Graphics Library，OpenGL)渲染器实现对视频数据的渲染处理。画中画绘制管理模块包含Open GL显示渲染引擎、Open GL编码渲染引擎及显示缓存区。以视频数据中的一帧视频图像为例进行渲染处理说明，具体如下所述：
在电子设备通过多摄像头进行视频的拍摄时，以电子设备通过第一摄像头及第二摄像头进行视频的拍摄为例进行说明。Open GL显示渲染引擎通过两个监听模块监 听电子设备的两个摄像头采集的视频图像。即为，Open GL显示渲染引擎通过第一监听模块及第二监听模块分别监听第一摄像头及第二摄像头采集的视频图像。Open GL显示渲染引擎将第一监听模块及第二监听模块监听到的两路视频图像及合成策略传输至Open GL渲染器。Open GL渲染器根据合成策略，将两路视频图像合成为一个视频图像，并将该视频图像传输至显示缓存区进行缓存。将显示缓存区中缓存的视频图像分别传输至显示界面(SurfaceView)及编码缓存区。在显示界面内显示该视频图像。Open GL编码渲染引擎获取编码缓存区内的视频图像，并对该视频图像根进行相关渲染，例如对该视频图像进行美颜处理，或者在该视频图像中添加水印，将渲染后的视频图像发送至编码模块，以便编码模块进行相应的编码处理，生成视频文件，如图9及图10所示。
需说明的是，在上述过程中，除了编码模块生成的视频文件为MP4格式外，其他视频图像均是RGB格式的。也就是说，Open GL显示渲染引擎监听的视频图像为RGB格式的图像，Open GL渲染器渲染合成后输出的视频图像也是RGB格式。即为，显示缓存区缓存的视频图像为RGB格式，发送至显示界面及编码缓存区的视频图像也是RGB格式。Open GL编码渲染引擎获取到RGB格式的视频图像，并对该视频图像根据用户输入的图像渲染操作进行相关渲染，得到的渲染后的视频图像是RGB格式。编码模块接收的视频图像为RGB格式，对RGB格式的视频图像进行编码处理，生成MP4格式的视频文件。
进一步地，Open GL渲染器根据合成策略，将两路视频图像合成为一个视频图像具体为：
Open GL渲染器解析合成策略，获知每路视频图像对应的图像画面的显示位置，例如，可以合成策略中可以标记出每路视频图像对应的图像画面的顶点坐标，例如左上角的顶点坐标。Open GL渲染器根据每路视频图像对应的图像画面的顶点坐标，可以确定每路视频图像对应的图像画面的显示位置。根据每路视频图像获取每路视频图像的纹理信息及纹理信息对应的位置信息。Open GL渲染器根据每路视频图像对应的图像画面的显示位置、每路视频图像的纹理信息及纹理信息对应的位置信息，将两路视频图像合成为一个视频图像，如图11a及图11b所示。
需要说明的是，将单个摄像头拍摄的视频图像进行显示及编码的渲染过程中，Open GL显示渲染引擎可以分别通过第一监听模块及第二监听模块监听视频图像，分别将两个监听模块监听的视频图像一个用作显示渲染，一个用作编码渲染。也可以仅使用一个监听模块监听视频图像，将监听的视频图像进行显示渲染，并将显示渲染后的视频图像进行编码渲染。具体说明如下：
在电子设备通过单摄像头进行视频的拍摄时，以电子设备通过第一摄像头进行视频的拍摄为例进行说明。Open GL显示渲染引擎可以通过两个监听模块监听电子设备的第一摄像头采集的视频图像。即为，Open GL显示渲染引擎通过第一监听模块及第二监听模块分别监听第一摄像头采集的视频图像。Open GL显示渲染引擎将第一监听模块监听到的视频图像及第二监听模块监听到的视频图像均传输至Open GL渲染器，Open GL渲染器将获取的Open GL显示渲染引擎的第一监听模块传输的视频图像传输至显示缓存区进行缓存，该路视频图像用于在显示界面进行显示，将获取 的Open GL显示渲染引擎的第二监听模块传输的视频图像传输至编码缓存区，该路视频图像用于在多摄编码模块中进行编码处理。将显示缓存区中缓存的视频图像传输至显示界面。在显示界面内显示该视频图像。Open GL编码渲染引擎获取编码缓存区内的视频图像，并对该视频图像根进行相关渲染，例如对该视频图像进行美颜处理，或者在该视频图像中添加水印，将渲染后的视频图像发送至多摄编码模块，以便多摄编码模块进行相应的编码处理，生成视频文件，参考图9所示。
需要说明的是，在上述电子设备通过单摄像头进行视频拍摄时，由于无需对视频图像进行特殊的渲染处理，因此Open GL显示渲染引擎的第一监听模块及第二监听模块监听的视频图像也可以不经过Open GL渲染器，而是直接将Open GL显示渲染引擎的第一监听模块监听的视频图像传输至显示缓存区，将第二监听模块监听的视频图像传输至编码缓存区，本申请对此不作限制。
或者，在电子设备通过单摄像头进行视频的拍摄时，Open GL显示渲染引擎可以通过一个监听模块监听电子设备的第一摄像头采集的视频图像。例如，Open GL显示渲染引擎通过第一监听模块监听第一摄像头采集的视频图像。Open GL显示渲染引擎将第一监听模块监听到的视频图像传输至Open GL渲染器，Open GL渲染器将获取的视频图像传输至显示缓存区进行缓存。将显示缓存区中缓存的视频图像传输至显示界面。在显示界面内显示该视频图像，并将该视频图像传输至编码缓存区中。Open GL编码渲染引擎获取编码缓存区内的视频图像，并对该视频图像根进行相关渲染，例如对该视频图像进行美颜处理，或者在该视频图像中添加水印，将渲染后的视频图像发送至编码模块，以便编码模块进行相应的编码处理，生成视频文件，参考图10所示。
需要说明的是，在上述电子设备通过单摄像头进行视频拍摄时，由于无需对视频图像进行特殊的渲染处理，因此Open GL显示渲染引擎的第一监听模块监听的视频图像也可以不经过Open GL渲染器，而是直接将Open GL显示渲染引擎的第一监听模块监听的视频图像传输至显示缓存区，本申请对此不作限制。
在本申请实施例中，可以利用共享上下文(SharedContext)实现编码缓存区及显示界面共享显示缓存区中缓存的视频图像，使得显示线程与编码线程间通过SharedContext共享纹理信息实现视频录制。
需要说明的是，电子设备通过多摄像头进行视频的拍摄时，其对应的显示界面可以是多景模式，也可以是画中画模式。电子设备通过单个摄像头进行视频的拍摄时，其对应的显示界面是单景模式。
需要说明的是，在附图9及附图10中单景模式下的Open GL显示渲染引擎、Open GL渲染器及显示缓存区与双景模式下的Open GL显示渲染引擎、Open GL渲染器及显示缓存区分别是同一个Open GL显示渲染引擎、Open GL渲染器及显示缓存区。为了便于说明，在附图9及附图10中，在单景模式及双景模式下均绘制了Open GL显示渲染引擎、Open GL渲染器及显示缓存区。
参见图12，为本申请实施例提供的另一种基于画中画的图像处理方法流程示意图。该方法可应用于图8所示的软件结构，如图12所示，其主要包括以下步骤。
S901、触发画中画拍摄模式下的视频拍摄操作。
用户若需要电子设备进行画中画拍摄下的视频拍摄，则触发画中画拍摄模式下的视频拍摄操作。
S902、根据视频拍摄操作拍摄视频图像。
其中，视频图像包含第一图像画面及至少一个第二图像画面，所述第一图像画面至少部分包围所述至少一个第二图像画面，第一图像画面和所述至少一个第二图像画面分别是至少两路视频数据的帧图像合成的视频图像。
画中画事件管理模块视频拍摄操作传输至画中画绘制管理模块，画中画绘制管理模块将拍摄的至少两路视频数据的帧图像进行渲染处理，得到视频图像，可以将该视频图像传输至预显示面进行显示，并传输至多摄编码模块进行编码处理。
S903、将拍摄过程中，实时合成的视频图像传输至多摄编码模块进行编码处理。
画中画绘制管理模块将摄的至少两路视频数据的帧图像进行渲染处理，得到视频图像，可以将该视频图像传输至多摄编码模块进行编码处理，生成视频文件。
S904、触发第一调整操作。
用户在拍摄过程中，需要对显示界面中显示的第二图像画面的位置和/或大小进行调整，此时触发第一调整操作，第一调整操作用于指示调整第二图像画面的位置和/或大小。
S905、画中画事件管理模块启动辅图像画面的调整。
画中画事件管理模块接收到第一调整操作后，启动辅图像画面的调整。辅图像画面是至少两个第二图像画面中，第一调整操作需调整的第二图像画面。
S906、画中画事件管理模块向画中画坐标管理模块发送第一调整操作。
画中画坐标管理模块可以根据第一调整操作，确定辅图像画面的在第一图像画面中的叠加位置及辅图像画面的大小。例如，可以确定辅图像画面的顶点的坐标位置及辅图像画面的缩放比例。
S907、画中画绘制管理模块从画中画坐标管理模块中获取在第一图像画面中的叠加位置及缩放比例，调整合成策略中辅图像画面对应的一路视频数据的帧图像在视频图像中的显示位置及显示大小。并调用Open GL渲染器对至少两路视频数据中的帧图像进行渲染处理。
其中，合成策略是用以合成第一图像画面及至少一个第二图像画面的显示位置及显示大小的信息。
具体的，画中画绘制管理模块在画中画坐标管理模块中获取了第一图像画面中的叠加位置及缩放比例后，可以根据第一图像画面中的叠加位置及缩放比例，调整合成策略。调用Open GL渲染器，Open GL渲染器根据调整后的合成策略对至少两路视频数据中的帧图像进行渲染处理，得到待显示视频图像。将待显示的视频图像通过显示界面显示，并通过多摄编码模块进行编码。
S908、将拍摄过程中，实时调整的视频图像进行编码生成视频文件；存储视频文件。
画中画绘制管理模块通过调用Open GL渲染器对至少两路视频数据中的帧图像进行渲染处理，得到待显示的视频图像，即为调整的视频图像。画中画绘制管理模块 将该待显示视频图像发送至多摄编码模块，多摄编码模块对接收的视频图像进行编码，生成视频文件。其中，步骤S903生成的视频文件与该步骤生成的视频文件为一个视频文件，将此视频文件存储在电子设备中。
与上述方法实施例相对应，本申请还提供了一种电子设备，该电子设备用于存储计算机程序指令的存储器和用于执行程序指令的处理器，其中，当该计算机程序指令被所述处理器执行时，触发所述电子设备执行上述方法实施例中的部分或全部步骤。
参见图13，为本申请实施例提供的一种电子设备的结构示意图。如图13所示，该电子设备1400可以包括处理器1410，外部存储器接口1420，内部存储器1421，通用串行总线(universal serial bus，USB)接口1430，充电管理模块1440，电源管理模块1441，电池1442，天线1，天线2，移动通信模块1450，无线通信模块1460，音频模块1470，扬声器1470A，受话器1470B，麦克风1470C，耳机接口1470D，传感器模块1480，按键1490，马达1491，指示器1492，摄像头1493，显示屏1494，以及用户标识模块(subscriber identification module，SIM)卡接口1495等。其中传感器模块1480可以包括压力传感器1480A，陀螺仪传感器1480B，气压传感器1480C，磁传感器1480D，加速度传感器1480E，距离传感器1480F，接近光传感器1480G，指纹传感器1480H，温度传感器1480J，触摸传感器1480K，环境光传感器1480L，骨传导传感器1480M等。
可以理解的是，本发明实施例示意的结构并不构成对电子设备1400的具体限定。在本申请另一些实施例中，电子设备1400可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器1410可以包括一个或多个处理单元，例如：处理器1410可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphicsprocessing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器1410中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器1410中的存储器为高速缓冲存储器。该存储器可以保存处理器1410刚用过或循环使用的指令或数据。如果处理器1410需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器1410的等待时间，因而提高了系统的效率。
在一些实施例中，处理器1410可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose  input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器1410可以包含多组I2C总线。处理器1410可以通过不同的I2C总线接口分别耦合触摸传感器1480K，充电器，闪光灯，摄像头1493等。例如：处理器1410可以通过I2C接口耦合触摸传感器1480K，使处理器1410与触摸传感器1480K通过I2C总线接口通信，实现电子设备1400的触摸功能。
I2S接口可以用于音频通信。在一些实施例中，处理器1410可以包含多组I2S总线。处理器1410可以通过I2S总线与音频模块1470耦合，实现处理器1410与音频模块1470之间的通信。在一些实施例中，音频模块1470可以通过I2S接口向无线通信模块1460传递音频信号，实现通过蓝牙耳机接听电话的功能。
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块1470与无线通信模块1460可以通过PCM总线接口耦合。在一些实施例中，音频模块1470也可以通过PCM接口向无线通信模块1460传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器1410与无线通信模块1460。例如：处理器1410通过UART接口与无线通信模块1460中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块1470可以通过UART接口向无线通信模块1460传递音频信号，实现通过蓝牙耳机播放音乐的功能。
MIPI接口可以被用于连接处理器1410与显示屏1494，摄像头1493等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器1410和摄像头1493通过CSI接口通信，实现电子设备1400的拍摄功能。处理器1410和显示屏1494通过DSI接口通信，实现电子设备1400的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器1410与摄像头1493，显示屏1494，无线通信模块1460，音频模块1470，传感器模块1480等。GPIO接口还可以被配置为I14C接口，I14S接口，UART接口，MIPI接口等。
USB接口1430是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口1430可以用于连接充电器为电子设备1400充电，也可以用于电子设备1400与外围设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如AR设备等。
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备1400的结构限定。在本申请另一些实施例中，电子设备 1400也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块1440用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块1440可以通过USB接口1430接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块1440可以通过电子设备1400的无线充电线圈接收无线充电输入。充电管理模块1440为电池1442充电的同时，还可以通过电源管理模块1441为电子设备供电。
电源管理模块1441用于连接电池1442，充电管理模块1440与处理器1410。电源管理模块1441接收电池1442和/或充电管理模块1440的输入，为处理器1410，内部存储器1421，显示屏1494，摄像头1493，和无线通信模块1460等供电。电源管理模块1441还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块1441也可以设置于处理器1410中。在另一些实施例中，电源管理模块1441和充电管理模块1440也可以设置于同一个器件中。
电子设备1400的无线通信功能可以通过天线1，天线2，移动通信模块1450，无线通信模块1460，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。电子设备1400中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块1450可以提供应用在电子设备1400上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块1450可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块1450可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块1450还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块1450的至少部分功能模块可以被设置于处理器1410中。在一些实施例中，移动通信模块1450的至少部分功能模块可以与处理器1410的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器1470A，受话器1470B等)输出声音信号，或通过显示屏1494显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器1410，与移动通信模块1450或其他功能模块设置在同一个器件中。
无线通信模块1460可以提供应用在电子设备1400上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system， GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块1460可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块1460经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器1410。无线通信模块1460还可以从处理器1410接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，电子设备1400的天线1和移动通信模块1450耦合，天线2和无线通信模块1460耦合，使得电子设备1400可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
电子设备1400通过GPU，显示屏1494，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏1494和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器1410可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏1494用于显示图像，视频等。显示屏1494包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备1400可以包括1个或N个显示屏1494，N为大于1的正整数。
电子设备1400可以通过ISP，摄像头1493，视频编解码器，GPU，显示屏1494以及应用处理器等实现拍摄功能。
ISP用于处理摄像头1493反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头1493中。
摄像头1493用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元 件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备1400可以包括1个或N个摄像头1493，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备1400在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备1400可以支持一种或多种视频编解码器。这样，电子设备1400可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备1400的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口1420可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备1400的存储能力。外部存储卡通过外部存储器接口1420与处理器1410通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器1421可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部存储器1421可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备1400使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器1421可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。处理器1410通过运行存储在内部存储器1421的指令，和/或存储在设置于处理器中的存储器的指令，执行电子设备1400的各种功能应用以及数据处理。
电子设备1400可以通过音频模块1470，扬声器1470A，受话器1470B，麦克风1470C，耳机接口1470D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块1470用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块1470还可以用于对音频信号编码和解码。在一些实施例中，音频模块1470可以设置于处理器1410中，或将音频模块1470的部分功能模块设置于处理器1410中。
扬声器1470A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备1400可以通过扬声器1470A收听音乐，或收听免提通话。
受话器1470B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备1400接听电话或语音信息时，可以通过将受话器1470B靠近人耳接听语音。
麦克风1470C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨 打电话或发送语音信息时，用户可以通过人嘴靠近麦克风1470C发声，将声音信号输入到麦克风1470C。电子设备1400可以设置至少一个麦克风1470C。在另一些实施例中，电子设备1400可以设置两个麦克风1470C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备1400还可以设置三个，四个或更多麦克风1470C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
耳机接口1470D用于连接有线耳机。耳机接口1470D可以是USB接口1430，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
压力传感器1480A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器1480A可以设置于显示屏1494。压力传感器1480A的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器1480A，电极之间的电容改变。电子设备1400根据电容的变化确定压力的强度。当有触摸操作作用于显示屏1494，电子设备1400根据压力传感器1480A检测所述触摸操作强度。电子设备1400也可以根据压力传感器1480A的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令。当有触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。
陀螺仪传感器1480B可以用于确定电子设备1400的运动姿态。在一些实施例中，可以通过陀螺仪传感器1480B确定电子设备1400围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器1480B可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器1480B检测电子设备1400抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消电子设备1400的抖动，实现防抖。陀螺仪传感器1480B还可以用于导航，体感游戏场景。
气压传感器1480C用于测量气压。在一些实施例中，电子设备1400通过气压传感器1480C测得的气压值计算海拔高度，辅助定位和导航。
磁传感器1480D包括霍尔传感器。电子设备1400可以利用磁传感器1480D检测翻盖皮套的开合。在一些实施例中，当电子设备1400是翻盖机时，电子设备1400可以根据磁传感器1480D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。
加速度传感器1480E可检测电子设备1400在各个方向上(一般为三轴)加速度的大小。当电子设备1400静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态，应用于横竖屏切换，计步器等应用。
距离传感器1480F，用于测量距离。电子设备1400可以通过红外或激光测量距离。在一些实施例中，拍摄场景，电子设备1400可以利用距离传感器1480F测距以实现快速对焦。
接近光传感器1480G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。电子设备1400通过发光二极管向外发射红外光。电子设备1400使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定电子设备1400附近有物体。当检测到不充分的反射光时，电子设备1400可以确定电子设备1400附近没有物体。电子设备1400可以利用接近光传感器1480G检测用户手持电子设备1400贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器1480G也可用于皮套模式，口袋模式自动解锁与锁屏。
环境光传感器1480L用于感知环境光亮度。电子设备1400可以根据感知的环境光亮度自适应调节显示屏1494亮度。环境光传感器1480L也可用于拍照时自动调节白平衡。环境光传感器1480L还可以与接近光传感器1480G配合，检测电子设备1400是否在口袋里，以防误触。
指纹传感器1480H用于采集指纹。电子设备1400可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。
温度传感器1480J用于检测温度。在一些实施例中，电子设备1400利用温度传感器1480J检测的温度，执行温度处理策略。例如，当温度传感器1480J上报的温度超过阈值，电子设备1400执行降低位于温度传感器1480J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，电子设备1400对电池1442加热，以避免低温导致电子设备1400异常关机。在其他一些实施例中，当温度低于又一阈值时，电子设备1400对电池1442的输出电压执行升压，以避免低温导致的异常关机。
触摸传感器1480K，也称“触控器件”。触摸传感器1480K可以设置于显示屏1494，由触摸传感器1480K与显示屏1494组成触摸屏，也称“触控屏”。触摸传感器1480K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏1494提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器1480K也可以设置于电子设备1400的表面，与显示屏1494所处的位置不同。
骨传导传感器1480M可以获取振动信号。在一些实施例中，骨传导传感器1480M可以获取人体声部振动骨块的振动信号。骨传导传感器1480M也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器1480M也可以设置于耳机中，结合成骨传导耳机。音频模块1470可以基于所述骨传导传感器1480M获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器1480M获取的血压跳动信号解析心率信息，实现心率检测功能。
按键1490包括开机键，音量键等。按键1490可以是机械按键。也可以是触摸式按键。电子设备1400可以接收按键输入，产生与电子设备1400的用户设置以及功能控制有关的键信号输入。
马达1491可以产生振动提示。马达1491可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对 应不同的振动反馈效果。作用于显示屏1494不同区域的触摸操作，马达1491也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。
指示器1492可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
SIM卡接口1495用于连接SIM卡。SIM卡可以通过插入SIM卡接口1495，或从SIM卡接口1495拔出，实现和电子设备1400的接触和分离。电子设备1400可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口1495可以支持Nano SIM卡，Micro SIM卡，SIM卡等。同一个SIM卡接口1495可以同时插入多张卡。所述多张卡的类型可以相同，也可以不同。SIM卡接口1495也可以兼容不同类型的SIM卡。SIM卡接口1495也可以兼容外部存储卡。电子设备1400通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，电子设备1400采用eSIM，即：嵌入式SIM卡。eSIM卡可以嵌在电子设备1400中，不能和电子设备1400分离。
具体实现中，本申请还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，其中，在所述程序运行时控制所述计算机可读存储介质所在设备执行上述实施例中的部分或全部步骤。所述的存储介质可为磁碟、光盘、只读存储记忆体(英文：read-only memory，简称：ROM)或随机存储记忆体(英文：random access memory，简称：RAM)等。
具体实现中，本申请实施例还提供了一种计算机程序产品，所述计算机程序产品包含可执行指令，当所述可执行指令在计算机上执行时，使得计算机执行上述方法实施例中的部分或全部步骤。
本申请实施例中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示单独存在A、同时存在A和B、单独存在B的情况。其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项”及其类似表达，是指的这些项中的任意组合，包括单项或复数项的任意组合。例如，a，b和c中的至少一项可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。
本领域普通技术人员可以意识到，本文中公开的实施例中描述的各单元及算法步骤，能够以电子硬件、计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本发明所提供的几个实施例中，任一功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的 理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，简称ROM)、随机存取存储器(random access memory，简称RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本发明的具体实施方式，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。本发明的保护范围应以所述权利要求的保护范围为准。

Claims (15)

  1. 一种基于画中画的图像处理方法，应用于电子设备，其特征在于，包括：
    接收画中画拍摄模式下的视频拍摄操作；
    根据所述视频拍摄操作拍摄视频图像，其中，所述视频图像包含第一图像画面及至少一个第二图像画面，所述第一图像画面至少部分包围所述至少一个第二图像画面，所述第一图像画面和所述至少一个第二图像画面分别是至少两路视频数据的帧图像合成的视频图像；
    接收第一调整操作，所述第一调整操作用以调整所述第二图像画面的位置和/或大小；
    根据所述第一调整操作，实时调整所述视频图像的合成策略，所述合成策略是用以合成所述第一图像画面及至少一个第二图像画面的显示位置及显示大小的信息；
    根据调整后的所述合成策略，将所述至少两路视频数据的帧图像合成待显示的视频图像。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    将拍摄过程中，实时调整的所述视频图像进行编码生成视频文件；存储所述视频文件。
  3. 根据权利要求1所述的方法，其特征在于，所述第一调整操作包括缩放操作或拖动操作中的至少一种。
  4. 根据权利要求3所述的方法，其特征在于，
    在所述第一调整操作为缩放操作时，所述根据所述第一调整操作，调整所述视频图像的合成策略包括：
    根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，其中，所述辅图像画面是所述至少一个第二图像画面；
    根据所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略。
  5. 根据权利要求3所述的方法，其特征在于，
    在所述第一调整操作为拖动操作时，所述根据所述第一调整操作，调整视频图像的合成策略包括：
    根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置；
    根据所述辅图像画面在第一图像画面上的叠加位置，调整合成策略。
  6. 根据权利要求4所述的方法，其特征在于，所述根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小包括：
    在所述第一调整操作的初始触控点为单个触控点，且落入所述辅图像画面的调整范围时，确定所述第一调整操作为缩放操作；
    根据第一调整操作的触控轨迹，确定所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
  7. 根据权利要求4所述的方法，其特征在于，所述根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小包括：
    在所述第一调整操作的初始触控点为至少两个触控点时，根据所述至少两个触控 点的触控轨迹，确定所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小。
  8. 根据权利要求5所述的方法，其特征在于，所述根据所述第一调整操作，确定辅图像画面在第一图像画面上的叠加位置包括：
    根据所述第一调整操作的触控轨迹，确定所述辅图像画面在第一图像画面上的叠加位置。
  9. 根据权利要求4所述的方法，其特征在于，所述根据所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略包括：
    根据所述辅图像画面的大小，在所述辅图像画面的尺寸小于预设尺寸阈值时，根据所述辅图像画面在第一图像画面上的叠加位置及辅图像画面的大小，调整合成策略，其中，所述预设尺寸阈值为拼接显示模式下，所述第一图像画面与所述辅图像画面拼接显示的尺寸。
  10. 根据权利要求9所述的方法，其特征在于，所述方法还包括：
    在所述辅图像画面的尺寸不小于预设尺寸阈值时，将所述合成策略更新为拼接显示模式的合成策略。
  11. 根据权利要求1所述的方法，其特征在于，所述根据所述视频拍摄操作录制视频图像包括：
    根据所述视频拍摄操作获取至少两路视频数据；
    将所述至少两路视频数据的帧图像按照画中画显示模式的合成策略，合成视频图像。
  12. 根据权利要求11所述的方法，其特征在于，所述将所述至少两路视频数据的帧图像按照画中画显示模式的合成策略，合成视频图像包括：
    根据所述画中画显示模式的合成策略，确定每路视频数据的帧图像对应的显示位置及显示大小；
    根据每路视频数据的帧图像对应的显示大小调整每路视频数据的帧图像的大小；
    根据调整后的每路视频数据的帧图像确定每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息；
    根据所述每路视频数据的帧图像对应的显示位置、每路视频数据的帧图像的纹理信息及纹理信息对应的位置信息，将所述至少两路视频数据的帧图像合成为视频图像。
  13. 一种电子设备，其特征在于，包括用于存储计算机程序指令的存储器和用于执行程序指令的处理器，其中，当该计算机程序指令被所述处理器执行时，触发所述电子设备执行权利要求1-12任一项所述的方法。
  14. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质包括存储的程序，其中，在所述程序运行时控制所述计算机可读存储介质所在设备执行权利要求1-12中任意一项所述的方法。
  15. 一种计算机程序产品，其特征在于，所述计算机程序产品包含可执行指令，当所述可执行指令在计算机上执行时，使得计算机执行权利要求1-12中任意一项所述的方法。
PCT/CN2022/078618 2021-06-16 2022-03-01 基于画中画的图像处理方法、设备、存储介质和程序产品 WO2022262313A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110668551.5 2021-06-16
CN202110668551.5A CN113596319A (zh) 2021-06-16 2021-06-16 基于画中画的图像处理方法、设备、存储介质和程序产品

Publications (1)

Publication Number Publication Date
WO2022262313A1 true WO2022262313A1 (zh) 2022-12-22

Family

ID=78244004

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/078618 WO2022262313A1 (zh) 2021-06-16 2022-03-01 基于画中画的图像处理方法、设备、存储介质和程序产品

Country Status (2)

Country Link
CN (1) CN113596319A (zh)
WO (1) WO2022262313A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116320218A (zh) * 2023-05-24 2023-06-23 深圳金智凌轩视讯技术有限公司 基于嵌入式计算机平台的多路视频合成分析处理管理系统

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113596319A (zh) * 2021-06-16 2021-11-02 荣耀终端有限公司 基于画中画的图像处理方法、设备、存储介质和程序产品
CN114286177A (zh) * 2021-12-28 2022-04-05 北京快来文化传播集团有限公司 视频拼接方法、装置及电子设备
WO2023142959A1 (zh) * 2022-01-30 2023-08-03 华为技术有限公司 多机位拍摄系统的拍摄方法、设备、存储介质和程序产品
CN117201955A (zh) * 2022-05-30 2023-12-08 荣耀终端有限公司 视频拍摄方法、装置、设备和存储介质
CN115756175B (zh) * 2023-01-06 2023-05-16 山东维创精密电子有限公司 一种基于虚拟现实数据的数据处理系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014153396A1 (en) * 2013-03-20 2014-09-25 Covidien Lp System and method for enhancing picture-in-picture display for imaging devices used for surgical procedures
CN110716780A (zh) * 2019-09-23 2020-01-21 广州视源电子科技股份有限公司 界面的显示方法和装置
CN110784735A (zh) * 2019-11-12 2020-02-11 广州虎牙科技有限公司 一种直播方法、装置、移动终端、计算机设备和存储介质
CN112954218A (zh) * 2019-03-18 2021-06-11 荣耀终端有限公司 一种多路录像方法及设备
CN113596319A (zh) * 2021-06-16 2021-11-02 荣耀终端有限公司 基于画中画的图像处理方法、设备、存储介质和程序产品

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105632452A (zh) * 2014-12-01 2016-06-01 冠捷投资有限公司 避免显示器中子母画面或并排画面失真的方法
CN112578978B (zh) * 2020-12-14 2024-04-23 努比亚技术有限公司 一种窗口切换控制方法、设备及计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014153396A1 (en) * 2013-03-20 2014-09-25 Covidien Lp System and method for enhancing picture-in-picture display for imaging devices used for surgical procedures
CN112954218A (zh) * 2019-03-18 2021-06-11 荣耀终端有限公司 一种多路录像方法及设备
CN110716780A (zh) * 2019-09-23 2020-01-21 广州视源电子科技股份有限公司 界面的显示方法和装置
CN110784735A (zh) * 2019-11-12 2020-02-11 广州虎牙科技有限公司 一种直播方法、装置、移动终端、计算机设备和存储介质
CN113596319A (zh) * 2021-06-16 2021-11-02 荣耀终端有限公司 基于画中画的图像处理方法、设备、存储介质和程序产品

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116320218A (zh) * 2023-05-24 2023-06-23 深圳金智凌轩视讯技术有限公司 基于嵌入式计算机平台的多路视频合成分析处理管理系统
CN116320218B (zh) * 2023-05-24 2023-08-29 深圳金智凌轩视讯技术有限公司 基于嵌入式计算机平台的多路视频合成分析处理管理系统

Also Published As

Publication number Publication date
CN113596319A (zh) 2021-11-02

Similar Documents

Publication Publication Date Title
CN110072070B (zh) 一种多路录像方法及设备、介质
US11669242B2 (en) Screenshot method and electronic device
US20220337742A1 (en) Camera switching method for terminal, and terminal
WO2022262313A1 (zh) 基于画中画的图像处理方法、设备、存储介质和程序产品
US20220247857A1 (en) Full-screen display method for mobile terminal and device
CN113422903B (zh) 拍摄模式切换方法、设备、存储介质
CN113473005B (zh) 拍摄中转场动效插入方法、设备、存储介质
CN115473957A (zh) 一种图像处理方法和电子设备
WO2022007862A1 (zh) 图像处理方法、系统、电子设备及计算机可读存储介质
CN113596321B (zh) 转场动效的生成方法、设备和存储介质
WO2022257474A1 (zh) 图像的预测方法、电子设备和存储介质
CN114489533A (zh) 投屏方法、装置、电子设备及计算机可读存储介质
WO2022001258A1 (zh) 多屏显示方法、装置、终端设备及存储介质
CN110248037B (zh) 一种身份证件扫描方法及装置
US20230353862A1 (en) Image capture method, graphic user interface, and electronic device
CN114339429A (zh) 音视频播放控制方法、电子设备和存储介质
CN114257920B (zh) 一种音频播放方法、系统和电子设备
CN113593567B (zh) 视频声音转文本的方法及相关设备
CN113805983A (zh) 调整窗口刷新率的方法及电子设备
CN113923351B (zh) 多路视频拍摄的退出方法、设备和存储介质
CN113596320B (zh) 视频拍摄变速录制方法、设备、存储介质
WO2022062985A1 (zh) 视频特效添加方法、装置及终端设备
EP4206865A1 (en) Brush effect picture generation method, image editing method and device, and storage medium
CN113852755A (zh) 拍摄方法、设备、计算机可读存储介质及程序产品
CN112527220B (zh) 一种电子设备显示方法及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22823805

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE