WO2023226634A1 - 一种拍摄方法及电子设备 - Google Patents

一种拍摄方法及电子设备 Download PDF

Info

Publication number
WO2023226634A1
WO2023226634A1 PCT/CN2023/089350 CN2023089350W WO2023226634A1 WO 2023226634 A1 WO2023226634 A1 WO 2023226634A1 CN 2023089350 W CN2023089350 W CN 2023089350W WO 2023226634 A1 WO2023226634 A1 WO 2023226634A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
subject
frame
target
shooting
Prior art date
Application number
PCT/CN2023/089350
Other languages
English (en)
French (fr)
Inventor
王宁
徐荣跃
李智琦
郝智翔
陈代挺
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Priority to EP23810708.0A priority Critical patent/EP4383742A4/en
Priority to US18/691,192 priority patent/US20240422424A1/en
Publication of WO2023226634A1 publication Critical patent/WO2023226634A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
    • H04M1/724—User interfaces specially adapted for cordless or mobile telephones
    • H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
    • H04M1/724—User interfaces specially adapted for cordless or mobile telephones
    • H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
    • H04M1/7243—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages
    • H04M1/72439—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages for image or video messaging
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/62—Control of parameters via user interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H04N23/635—Region indicators; Field of view indicators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6811—Motion detection based on the image signal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M2250/00—Details of telephonic subscriber devices
    • H04M2250/52—Details of telephonic subscriber devices including functional features of a camera

Definitions

  • the present application relates to the field of image processing technology, and in particular, to a photographing method and electronic equipment.
  • This application provides a shooting method and electronic equipment that can track the shooting subject in the video so that the shooting subject is always in a target position such as the optimal visual position to highlight the shooting subject and the shooting content.
  • this application provides a shooting method, which can be applied to an electronic device, where the electronic device includes a camera, and the camera is used to collect original images.
  • the electronic device starts a camera application.
  • the electronic device displays a preview image and a subject image highlighting the target shooting subject on the recording interface.
  • the preview image and the main image are obtained according to the original image processing, the target shooting subject is at the target position on the main image, and the framing range of the main image is within the framing range of the preview image.
  • the electronic device displays both the preview image and the main body image in the recording interface.
  • the preview image is an image with a larger viewing range actually captured by the camera, so the content of the picture is richer.
  • the viewing range of the main image is within the viewing range of the preview image, that is, the main image is an image with a smaller viewing range.
  • the main image displays part of the screen content in the preview image. This allows the subject image to highlight the target subject and shooting content related to the target subject.
  • Subject images can be used to generate either still pictures or video files.
  • the target subject in the static picture is in a visually optimal position, so that the composition of the captured image is more beautiful and can highlight the target subject, thereby presenting a visually optimal display effect.
  • the target subject in the video file always remains at the target position. In this way, the user only needs to keep the target subject within the viewing range of the preview image, and can record the video without the need to adjust the target subject's position accurately in real time. Therefore, the difficulty of shooting is reduced and the user’s shooting experience is improved.
  • the target photographing subject remains at a target position on the subject image.
  • the target subject moves, the target subject is always located in the optimal visual position and is highlighted. This reduces the difficulty of shooting and improves the user's shooting experience.
  • the smoothness of the background change of the target subject in the subject image is greater than the smoothness of the background change of the target subject in the preview image.
  • the background in the subject image is smoother, which means the background transition is smoother.
  • the background outside the target subject in the main image will not change drastically when the movement of the target subject, especially violent movement, causes a sudden change in the center point of the target subject. In this way, the background is less prone to ghosting and the background is clearer.
  • the background change span of the main image in adjacent frames is small, which can better avoid jumps and flickers in the recorded video caused by drastic changes in the background. Therefore, the smoothness of the subject image is greater, and the visual effect presented by the subject image is better.
  • the target photographing subject includes at least one subject image
  • the subject image includes at least one subject.
  • one target subject corresponds to one subject image
  • one target subject corresponds to multiple subject images
  • the subject images are located outside the area of the preview image or partially overlap with the area of the preview image.
  • the target shooting subject and/or the target position are determined in response to the user's instruction operation or automatically determined by the electronic device.
  • the method further includes: the electronic device detects switching of the target shooting The operation of the subject is to display the subject image corresponding to the switched target shooting subject on the recording interface; and/or the operation of switching the target position is detected, and the target shooting subject is displayed on the recording interface to be the subject corresponding to the switched target position. image.
  • the electronic device can switch the target subject in various ways (such as active switching and passive switching) at the beginning of recording and during the recording process.
  • the electronic device can switch the current target shooting subject to the target shooting subject specified by the user according to the switching operation performed by the user during the recording process, and then the electronic device displays the subject image corresponding to the specified shooting subject on the recording interface.
  • the electronic device can also automatically determine the target subject during the recording process, and keep the target subject at the target position in the main image. In this way, even if the user's shooting skills are limited, the electronic device can assist the user in taking photos that stand out. The video of the shooting subject and the shooting content is reduced, thereby reducing the difficulty of shooting and improving the quality of the video captured.
  • the determination of the target position of the electronic device and the switching method can refer to the multiple switching methods of the target shooting subject mentioned above.
  • the electronic device can default to the initial target position of the target subject (such as the middle position of the image).
  • the electronic device can switch the target subject to the user-specified position according to the switching operation performed by the user.
  • Target location (such as the golden section point of the image). Then in the subject image displayed on the recording interface after switching, the target subject is always located at the golden section of the image.
  • the electronic device can display selectable target locations in the preview image, Indicates that the location can be used as a target location so that the user can perform a switching operation by touching the target location in the preview image.
  • the electronic device displays the i-th frame preview image and the subject image corresponding to the target shooting subject on the recording interface, where i is a positive integer greater than 1, including: the electronic device displays the i-th frame according to the i-th Frame original image, get the i-th frame preview image.
  • the electronic device smoothes the first position information of the target shooting subject in the i-th frame preview image according to the first position information of the target shooting subject in the i-1th frame preview image, and obtains the second position information of the target shooting subject. .
  • the electronic device crops the i-th frame preview image according to the second position information to obtain a subject image corresponding to the target shooting subject.
  • the i-th frame preview image and the subject image corresponding to the target shooting subject are displayed on the recording interface.
  • the electronic device uses the first position information of the i-1th frame preview image to smoothly process the first position information of the i-th frame preview image, so that the first position information of the i-th frame preview image both refers to the target.
  • the actual position of the subject is combined with the first position information of the i-1th frame preview image (the "predicted value" of the position at the historical moment). Therefore, when the target subject's movement changes such as opening arms, waving, jumping, etc., causing the actual position of the target subject to change drastically, the target shot in the i-th frame preview image can be adjusted through the "predicted value" of the position at the historical moment.
  • the actual location of the subject This not only increases the smoothness of adjacent subject images, but also reduces jumps and flickers in recorded videos caused by drastic changes in the background. Therefore, the quality of the video captured is improved.
  • the electronic device determines the first position of the target shooting subject in the i-th frame preview image based on the first position information of the target shooting subject in the i-1 th frame preview image.
  • the information is smoothed to obtain the second position information of the target shooting subject, including: the electronic device obtains the first position information of the target shooting subject in the i-1th frame preview image and the first position in the i-th frame preview image.
  • the first position difference between messages Based on the first position difference, it is determined that according to the first position information of the target shooting subject in the i-1th frame preview image, the first position information in the i-th frame preview image is smoothed to obtain the target shooting subject's first position information. Two position information, wherein the second position difference between the second position information and the first position information in the i-1th frame preview image is less than or equal to the first position difference.
  • the electronic device achieves smooth processing by reducing the degree of change of the first position information. That is to say, the electronic device obtains the displacement amplitude (i.e., the first position difference) of the target subject between the i-1th frame preview image and the i-th frame preview image, and uses the displacement amplitude to determine whether it is necessary to The frame preview image is subjected to smoothing processing.
  • the electronic device uses the first position information (the position "predicted value" of the historical moment) in the i-1th frame preview image to smooth the i-th frame preview image.
  • the first location information in (actual location at the current moment).
  • the displacement amplitude of the target subject ie, the second position difference
  • the change span of the background of the target subject is also smaller, and the background is smoother.
  • obtaining the preview image of the i-th frame based on the original image of the i-th frame includes: the electronic device performs format processing on the original image of the i-th frame to obtain the first image of the i-th frame. Then the first image of the i-th frame of the electronic device is the i-th frame preview image.
  • the electronic device can also perform jitter displacement compensation processing on the first image of the i-th frame to obtain the first output image of the i-th frame, and the first output image of the i-th frame is the preview image of the i-th frame.
  • the preview image can be the first image without anti-shake processing.
  • the first image is obtained directly from the original image collected by the camera through format processing, so it is also called a live image. Displaying live images enables the screen to More real and natural.
  • the preview image may also be the first output image that has undergone anti-shake processing.
  • the first output image compensates for the frame shift caused by camera shake, so the overall image is clearer and more stable. In this way, by selectively displaying the first image or the first output image on the recording interface, the user's diverse shooting needs can be met and the user's shooting experience can be improved.
  • performing jitter displacement compensation processing on the first image of the i-th frame to obtain the first output image of the i-th frame includes: the electronic device obtains the first image of the i-th frame relative to the first output image of the i-th frame.
  • shake displacement compensation processing is performed on the picture in the first image of the i-th frame to obtain the first output image of the i-th frame.
  • any frame or multiple first images before the first image of the i-th frame can be used as a reference image when the electronic device performs anti-shake processing on the first image of the i-th frame. Then the obtained picture jitter displacement information is more accurate.
  • this image shake displacement information for shake displacement compensation processing, a more accurate anti-shake effect can be achieved.
  • the pixels are not disturbed by jitter and can remain in a stable position. Therefore, the picture composed of pixels can achieve a clear and stable presentation effect.
  • obtaining the picture jitter displacement information of the i-th frame first image relative to any one or more frames of the first image before the i-th frame first image includes: detecting electronic equipment jitter amplitude. According to the jitter amplitude, the picture jitter displacement information is determined, where the picture jitter displacement information includes the position of each pixel in the first image of the i-th frame relative to the corresponding pixel in any one or more frames of the first image before the i-th frame. Offset.
  • the mobile phone's acceleration sensor and gyroscope sensor can be used to detect the shaking amplitude of the mobile phone.
  • the shake amplitude of the electronic device is detected to determine the shake amplitude of the first image (ie, the screen shake displacement information).
  • This implementation method is simpler and more efficient, and the image processing speed is improved.
  • the method further includes: performing downsampling processing on the first image of the i-th frame to obtain a second image of the i-th frame.
  • the method further includes: the electronic device obtains the first position information from the i-th frame preview image. The target shooting subject is detected in the second image of the i frame, and the third position information of the target shooting subject is obtained. Then the electronic device maps the third position information of the target shooting subject to the i-th frame preview image, and obtains the first position information of the target shooting subject located in the i-th frame preview image.
  • subject tracking processing and image anti-shake processing are two parallel processing tasks, and the electronic device fuses the results of the two tasks to obtain the second position information.
  • Electronic equipment adopts this method of parallel processing first and then fusing the results, which can reduce the waiting time of a single task. Furthermore, the utilization of processor resources of the electronic device can be improved, memory consumption can be reduced, and the processing speed of obtaining the first position information of the target shooting subject in the preview image of the i-th frame can be improved.
  • cropping the i-th frame preview image according to the second position information to obtain a subject image corresponding to the target shooting subject includes: the electronic device determines the minimum size of the target shooting subject according to the second position information. Encircling rectangular box. The cropping area is determined based on the minimum enclosing rectangular frame, where the cropping area includes the minimum enclosing rectangular frame. Then the electronic device previews the image in the i-th frame to obtain the image corresponding to the cropping area, and obtains the subject image corresponding to the target shooting subject.
  • the minimum circumscribed rectangular frame is the minimum bounded rectangle including the outer contour of the target subject, and the minimum circumscribed rectangular frame can be determined by the coordinate values of the top/bottom/left/right coordinate points on the outer contour.
  • the resulting cropped area includes the outer contour of the target subject, so that the cropped subject image can fully display the target subject.
  • the method further includes: the electronic device reduces or enlarges the image corresponding to the cropping area to obtain a preset display The subject image corresponding to the target shooting subject of the specification, wherein the preset display specification is the preset screen ratio of the electronic device.
  • the electronic device adjusts the specifications of the cropped image to meet the actual specifications of the output.
  • the main body image output according to the preset display specifications is displayed more clearly on the screen, and the main body of the adjacent frame The image specifications are unified and the display effect is better.
  • the target position includes a middle position or a golden section position.
  • the main image obtained by composing the electronic device according to the middle position and the golden section is not only more beautiful, but also can highlight the target subject and the video shooting content, thereby achieving the best visual effect, making the video or vlog presentation better, and improving Improve the user’s shooting and viewing experience.
  • the method further includes: the electronic device detects the end of the recording operation and generates a video file, where the video file includes a main body image, or the video file includes a main body image and a preview image.
  • the video frame image in the video file can keep the target shooting subject always in the visual position. Optimal position to highlight the target subject and content. Not only that, the interference caused by camera shake is reduced in the video file, and the picture transition between adjacent subject images and video frame images is more stable. It can also smooth the center point of the subject so that the deviation of the subject on the subject image is small, and the subject image and video frame images are less prone to ghosting and the background is clearer.
  • the method further includes: displaying a shooting preview interface after starting the camera application, wherein the shooting preview interface includes a preview image, or the shooting preview interface includes a preview image and a subject image.
  • the electronic device can preview the rendering effect in real time when recording starts, so that the user can know the actual effect of the video recording in advance.
  • users are shooting and creating, they can adjust the shooting parameters as needed in the preview interface, so that after the electronic device enters the recording interface, the video effect will be better.
  • this application provides an electronic device, including: a camera for capturing images, a screen for displaying the above preview interface and recording interface, one or more processors, memories, and one or more computer programs, wherein One or more computer programs are stored in the memory, the one or more computer programs include instructions.
  • the electronic device is caused to execute the method of the first aspect and its possible design manner.
  • the present application provides a computer-readable storage medium.
  • the computer-readable storage medium may include computer instructions.
  • the electronic device causes the electronic device to execute the first aspect and its possible design methods. Methods.
  • the present application provides a computer program product.
  • the computer program product When the computer program product is run on a computer, it causes the computer to execute the method of the first aspect and its possible design methods.
  • the electronic device of the second aspect, the computer storage medium of the third aspect, and the computer program product of the fourth aspect provided above are all used to execute the corresponding methods provided above. Therefore, what they can achieve For beneficial effects, please refer to the beneficial effects in the corresponding methods provided above and will not be described again here.
  • Figure 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
  • Figure 2 is a schematic diagram of the software structure of an electronic device provided by an embodiment of the present application.
  • Figure 3 is a flow chart of a shooting method provided by an embodiment of the present application.
  • Figure 4 is a schematic diagram of two types of users performing the first operation provided by the embodiment of the present application.
  • Figure 5 is a schematic diagram of a set of target locations provided by an embodiment of the present application.
  • Figure 6 is a flow chart of a smoothing processing method provided by an embodiment of the present application.
  • Figure 7 is a schematic diagram of a set of second interfaces provided by an embodiment of the present application.
  • Figure 8 is a schematic diagram of a subject image in the first application scenario provided by the embodiment of the present application.
  • Figure 9 is a schematic diagram of a subject image in the second application scenario provided by the embodiment of the present application.
  • Figure 10 is a schematic diagram of a subject image in the third application scenario provided by the embodiment of the present application.
  • Figure 11 is a schematic diagram of a subject image in the fourth application scenario provided by the embodiment of the present application.
  • Figure 12 is a schematic diagram of another subject image in the fourth application scenario provided by the embodiment of the present application.
  • Figure 13 is a schematic polyline diagram showing changes in the position of a photographed subject over time provided by an embodiment of the present application
  • Figure 14 is a schematic diagram of the first shooting device provided by the embodiment of the present application.
  • Figure 15 is a schematic diagram of the second shooting device provided by the embodiment of the present application.
  • Figure 16 is a schematic diagram of a third photographing device provided by an embodiment of the present application.
  • first and second are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
  • subject tracking technology helps the user in keeping the location information of the subject at the most visual level when shooting with the camera. The effect of superior location.
  • subject tracking technology there are several problems with subject tracking technology:
  • this technology can only be used in scenarios where the camera is fixed (such as using a camera tripod to fix the camera) to better assist users in shooting videos with the subject stably in the middle of the frame.
  • the camera Once the user holds the camera to record a video, the camera will be interfered by the shaking of the user's hand, resulting in a deviation in the subject position of the subject.
  • This kind of subject position deviation caused by camera shake cannot be solved by subject tracking technology, which leads to the problem that in videos shot using subject tracking technology, the subject cannot always be in the optimal visual position due to camera shake.
  • subject tracking technology only plays an auxiliary role when users are shooting.
  • This technology requires users to adjust the video screen displayed on the interface according to the location of the subject. It is difficult for users to adjust the video picture by adjusting the camera's shooting position and shooting angle in real time, so as to accurately keep the subject in the video picture at the optimal visual position. Therefore, subject tracking technology is difficult to shoot, which will cause users to Poor shooting experience.
  • the electronic devices include a display screen and a camera.
  • the electronic device starts the camera application and enters the video shooting mode of tracking the subject.
  • the electronic device processes the original image collected by the camera and displays the processed image on the recording interface.
  • the electronic device performs shake displacement compensation processing based on the original image to eliminate the subject position deviation caused by camera shake, and smoothes the subject's position based on the original image to reduce the impact of sudden changes in the subject's position on the subject's position. .
  • the subject in the image displayed on the recording interface has undergone shake compensation processing and is not disturbed by camera shake. It can be kept at the target position and highlighted, achieving a stable picture.
  • the embodiment of the present application can also smooth the center point of the photographed subject so that the deviation of the photographed subject is small, ghosting is less likely to occur, and the background is clearer.
  • the embodiment of the present application does not require the user to adjust the video screen displayed on the interface in real time according to the position of the shooting subject, and can keep the shooting subject at the target position, thereby reducing the difficulty of shooting and improving the user's shooting experience.
  • the embodiments of the present application can not only stably and clearly highlight the shooting content and the shooting subject when shooting videos, but also improve the shooting quality of the video, and the video presentation effect is better. It can also reduce the user's operations to adjust the picture, reduce the difficulty of shooting, and improve the user's shooting experience.
  • the shooting method provided by the embodiment of the present application can be applied to electronic devices equipped with cameras and display screens, where the electronic devices can be mobile phones, tablet computers, notebook computers, ultra-mobile personal computers (UMPC), Netbooks, handheld computers (Personal Digital Assistant, PDA), cameras Video cameras, digital cameras, surveillance and other equipment.
  • FIG. 1 shows a schematic structural diagram of an electronic device 100 provided by an embodiment of the present application.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2.
  • USB universal serial bus
  • Mobile communication module 150 wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, And subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • subscriber identification module subscriber identification module, SIM
  • the above-mentioned sensor module 180 may include a pressure sensor, a gyroscope sensor 180A, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor 180B, an ambient light sensor, a bone conduction sensor, etc. .
  • the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 .
  • the electronic device 100 may include more or fewer components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor neural network processor (neural-network processing unit, NPU), and/or Micro control unit (micro controller unit, MCU), etc.
  • application processor application processor, AP
  • modem processor a graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • NPU baseband processor neural network processor
  • MCU Micro control unit
  • different processing units can be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the electronic device 100 .
  • the controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • the processor 110 may also be provided with a memory for storing instructions and data.
  • the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
  • processor 110 may include one or more interfaces.
  • Interfaces can include integrated circuit (inter-integrated circuit, I2C) interface, serial peripheral interface (serial peripheral interface, SPI), integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation) , PCM) interface, universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, Subscriber identity module (SIM) interface, and/or universal serial bus (USB) interface, etc.
  • I2C integrated circuit
  • SPI serial peripheral interface
  • I2S integrated circuit built-in audio
  • I2S integrated circuit sound
  • pulse code modulation pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous receiver/transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM Subscriber identity module
  • USB universal serial bus
  • the interface connection relationships between the modules illustrated in the embodiments of the present application are only schematic illustrations and do not constitute a structural limitation of the electronic device 100 .
  • the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
  • the charging management module 140 is used to receive charging input from the charger.
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 receives input from the battery 142 and/or the charge management module 140 for the processor 110, internal memory 121, external memory, display screen 194, and camera. 193, and wireless communication module 160 and so on.
  • the power management module 141 and the charging management module 140 may also be provided in the same device.
  • the wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.
  • the mobile communication module 150 can provide solutions for wireless communication including 2G/3G/4G/5G applied on the electronic device 100 .
  • the wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as Wi-Fi network), Bluetooth (blue tooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (FM), NFC, infrared technology (infrared, IR) and other wireless communication solutions.
  • WLAN wireless local area networks
  • Bluetooth blue tooth, BT
  • global navigation satellite system global navigation satellite system
  • FM frequency modulation
  • NFC infrared technology
  • infrared infrared, IR
  • the electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like.
  • the GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 194 is used to display images, videos, etc.
  • the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
  • the electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.
  • the ISP is used to process the data fed back by the camera 193.
  • Camera 193 is used to capture still images or videos (also called raw images).
  • the ISP is also used to process the original image collected by the camera into a first image and a second image, and shunt the first image and the second image to different processors for anti-shake processing and subject tracking processing respectively.
  • the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
  • NPU is a neural network (NN) computing processor.
  • NN neural network
  • the NPU can realize applications such as intelligent cognition of the electronic device 100, such as subject recognition, anti-shake processing, subject position smoothing processing, etc.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. Such as saving music, videos, etc. files in external memory card.
  • Internal memory 121 may be used to store computer executable program code, which includes instructions.
  • the processor 110 executes instructions stored in the internal memory 121 to execute various functional applications and data processing of the electronic device 100 .
  • the internal memory 121 may include a program storage area and a data storage area. Among them, the stored program area can store an operating system, at least one application program required for a function (such as a sound playback function, an image playback function, etc.).
  • the storage data area may store data created during use of the electronic device 100 (such as audio data, phone book, etc.).
  • the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), etc.
  • the electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. For example, music playback, recording, voice interaction, etc.
  • Gyro sensor 180A also known as gyroscope and angular velocity sensor, can measure the rotation and deflection of the mobile phone.
  • the gyro sensor 180A is used to detect the shake amplitude of the mobile phone, where the shake amplitude is used by the processor 110 to determine the screen shake displacement, and to compensate the offset of the video screen according to the screen shake displacement.
  • Touch sensor 180B is also called “touch panel (TP)".
  • the touch sensor 180B is used to detect a touch operation acting on or near the touch sensor 180B.
  • the touch sensor can pass the detected touch operation to the application processor to determine Touch event type.
  • the mobile phone 100 can provide visual output related to the touch operation through the display screen 194 .
  • the touch sensor 180B and the display screen are combined to form the above-mentioned touch screen.
  • the touch sensor 180B may also be disposed on the surface of the electronic device 100 at a location different from that of the display screen 194 .
  • the buttons 190 include a power button, a volume button, etc.
  • the motor 191 can generate vibration prompts.
  • the indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, or may be used to indicate messages, missed calls, notifications, etc.
  • the SIM card interface 195 is used to connect a SIM card.
  • FIG. 2 is a software structure block diagram of the mobile phone provided by an embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has clear roles and division of labor.
  • the layers communicate through software interfaces.
  • the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime (Android Runtime) and system libraries, and kernel layer.
  • the application layer can include a series of application packages, which can include camera, calendar, map, video, music, short message, gallery, call, navigation, Bluetooth, WLAN and other applications.
  • application packages can include camera, calendar, map, video, music, short message, gallery, call, navigation, Bluetooth, WLAN and other applications.
  • the application framework layer provides an application programming interface (API) and programming framework for applications in the application layer.
  • API application programming interface
  • the application framework layer includes some predefined functions.
  • the application framework layer can include: the interface corresponding to the camera application and the interface management service (activity manager service, AMS) module.
  • AMS interface management service
  • AMS uniformly schedules the interface activities of the above applications.
  • the application framework layer can also include window managers, content providers, view systems, phone managers, resource managers, notification managers, etc.
  • a window manager is used to manage window programs.
  • the window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.
  • Content providers are used to store and retrieve data and make this data accessible to applications.
  • Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
  • the view system includes visual controls, such as controls that display text, controls that display pictures, etc.
  • a view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
  • the phone manager is used to provide communication functions of the mobile phone 100 .
  • call status management including connected, hung up, etc.
  • the resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc.
  • the notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, etc.
  • the notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text messages are prompted in the status bar, a tone is sounded, the phone vibrates, the indicator light flashes, etc.
  • Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for the scheduling and management of the Android system.
  • the core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.
  • the application layer and application framework layer run in virtual machines.
  • the virtual machine executes the java files of the application layer and application framework layer into binary files.
  • the virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.
  • the system library can include multiple functional modules.
  • surface manager surface manager
  • media libraries Media Libraries
  • 3D graphics processing libraries for example: OpenGL ES
  • 2D graphics engines for example: SGL
  • the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc.
  • the media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing.
  • 2D Graphics Engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer at least includes hardware drivers, which are used to drive the hardware.
  • the kernel layer includes: camera driver, display driver (display driver), graphics processor driver (graphics processing unit driver, GPU driver), etc.
  • the camera driver is used to start the camera to respond to the user's operation.
  • the kernel layer can also include audio drivers and sensor drivers.
  • the following is an example of the workflow of a mobile phone based on a shooting method provided by this application.
  • the icon of the camera application in the application layer can be displayed on the display screen of the mobile phone.
  • the mobile phone starts the camera application.
  • the camera application calls the camera driver in the kernel layer to start the camera in response to the recording operation.
  • the camera collects continuous original images, and the collected original images are processed by the ISP into a first image and a second image, and the first image and the second image are shunted to different processors 110 to process them respectively.
  • the first image is subjected to subject tracking processing, and the second image is subjected to anti-shake processing.
  • the processor 110 combines the subject tracking result with the anti-shake processing result to obtain an image that has undergone subject tracking processing and anti-shake processing, and then smoothes the subject position in the image to finally obtain the subject image.
  • the subject image is displayed on the display.
  • the subject image can keep the subject at the target position without the user having to adjust the video screen displayed on the interface in real time according to the subject's position, and through anti-shake and smoothing, the shooting content and subject can be stably and clearly highlighted when shooting video. Improve the video shooting quality and the video presentation effect is better. It also reduces the user's operations to adjust the picture and reduces the difficulty of shooting, thereby improving the user's shooting experience.
  • Embodiments of the present application provide a shooting method, which can be applied to electronic devices.
  • the electronic devices include a display screen and a camera.
  • the shooting method may include:
  • the mobile phone detects the first operation performed by the user, and the first operation is used to instruct starting the camera application.
  • the first operation includes: the user touching the icon of the camera application on the mobile phone desktop, or the user's voice instruction, or the user's gesture operation.
  • the first operation is when the user touches the icon of the camera application on the desktop of the mobile phone.
  • the mobile phone also includes a touch sensor.
  • the touch sensor and the display screen are combined into a touch screen, and the image content is presented through the touch screen.
  • the mobile phone displays the icon of the camera application on the graphical user interface (GUI).
  • GUI graphical user interface
  • the embodiments of this application are not limited to the camera application corresponding to the camera icon on the mobile phone desktop, but can also be an application with a camera function integrated with other applications, such as chat software, shopping software or short video social software integrated Applications for video recording, etc.
  • the user when the first operation is the user's voice instruction, as shown in (b) of Figure 4, the user says such as “Open the camera”, “I want to record”, “ Enter video shooting mode” and other commands to instruct the phone to launch the camera application.
  • the user when the first operation is a user gesture operation, the user can draw a gesture operation on the GUI of the mobile phone indicating starting the camera application, wherein the gesture operation can be an initial setting or a circle customized by the user. trajectories, wavy line trajectories, cross-hatch trajectories, etc.
  • the mobile phone After the user instructs the first operation in the above manner, the mobile phone detects the first operation, and then executes S302.
  • the mobile phone In response to the first operation, the mobile phone starts the camera application and enters the video shooting mode of tracking the subject.
  • the camera application that can realize the video shooting mode of tracking the subject can be an existing camera application embedded with the program and function of tracking the subject, or it can be a new application program with the video shooting mode of tracking the subject. Embodiments of the present application There are no restrictions on this.
  • the mobile phone enters multiple video shooting modes for tracking the subject.
  • the mobile phone automatically enters the video shooting mode for tracking the subject after launching the camera application.
  • the video shooting mode of tracking the subject selected by the user is detected, and the mobile phone enters this mode.
  • the shooting subject Different from the regular shooting mode, in the video shot in the subject-tracking video shooting mode, the shooting subject always automatically appears at the highlighted target position in the video screen. In this way, the mobile phone can automatically stabilize the shooting content and the shooting subject when shooting the video. And it is clearly highlighted in the video, thereby improving the shooting quality and presentation effect of the video, and improving the user's shooting experience.
  • the visually optimal position that can highlight the shooting content and the shooting subject is represented by a target position, and the target position includes an initial setting position or a user-defined position.
  • the initial setting position or the position customized by the user includes: the center of the image or the golden section.
  • the target position includes the center of the image, as shown in (a) of Figure 5 . Then in the video shooting mode that tracks the subject, the subject will always be in the center of the frame in the captured video.
  • the target position includes the golden section, as shown in (b) of Figure 5 . If the initial set position or the user-defined position is the golden section, the subject will always be at the golden section of the screen during shooting.
  • the mobile phone displays the first interface, and detects the second operation performed by the user on the first interface.
  • the second operation is used to instruct video shooting.
  • the first interface (also called the preview interface) is a shooting preview interface in the video shooting mode of tracking the subject.
  • the image displayed on the first interface is obtained based on the original image collected by the camera before the user performs the second operation.
  • the second operation is used to instruct the camera to start recording video (including recording vlog, etc.).
  • the second operation includes: the user touches the shooting button in the first interface, or the user speaks a voice command such as "start shooting", or the user performs a gesture operation.
  • the user touches the shooting button in the first interface or the user speaks a voice command such as "start shooting", or the user performs a gesture operation.
  • the mobile phone After the user instructs the second operation in the above manner, the mobile phone detects the second operation and executes S304.
  • the mobile phone displays the second interface on the display screen.
  • the second interface (also called the recording interface) is the recording interface where the mobile phone starts recording video after responding to the above-mentioned second operation.
  • the camera continuously collects original images
  • the mobile phone processes the original images, and displays the processed images on the second interface.
  • the second interface includes a subject image processed based on the original image, or the second interface also includes a first image or a first output image obtained based on the original image processing (the first image and/or the first output image are also called preview images).
  • the second interface may display the original image, or the second interface may not display any image.
  • the second interface displays the processed image, such as the above-mentioned main body image and the first image or the first output image.
  • the mobile phone processes the original image and displays the processed image on the second interface, which may include S305-S311.
  • the mobile phone acquires multiple original images collected by the camera, and processes each original image into a first image and a second image.
  • the camera continuously collects original images, and the mobile phone processes the original images into the first image.
  • the original image may be a raw image collected directly by a camera, or a raw image collected and synthesized by multiple cameras.
  • raw can be understood as "unprocessed", that is to say, the above raw image is the original image directly output by the camera sensor. Because it has not been processed and edited, it still retains the original wide color gamut internal color. , the image quality is higher.
  • the mobile phone inputs the original image into the image processing unit, and the image processing unit processes the original image into a first image and a second image.
  • the first image is used for image anti-shake processing
  • the second image is used for subject tracking processing.
  • the mobile phone displays the first image in the second interface. Displaying the first image (also known as live image) in the second interface can make the picture more real and natural, and retaining the jitter effect of the picture can make the video viewer immersed in the scene, thereby improving the video shooting and viewing experience.
  • the first image has the same size as the original image and is both the original size
  • the second image is a smaller image obtained by downsampling the first image.
  • the size is represented by M pixels * N pixels, that is to say, the number of pixels of the second image is smaller than the number of pixels of the first image, and the resolution of the second image is lower than the resolution of the first image.
  • the second image is used for subject detection with less calculation and faster processing. faster.
  • the size of the first image may be smaller than the size of the original image.
  • the size of the second image may be greater than or equal to the size of the second image.
  • the size of the second image is smaller than the second video size.
  • the embodiments of the present application do not limit the sizes of the first image and the second image.
  • the first image and the second image may both be brightness, chroma, and saturation (YUV) images
  • the YUV image refers to an image obtained using the YUV encoding method.
  • YUV is a color coding method used by European television systems.
  • a three-tube color camera can be used to capture images, and then the obtained color image signals are separated, amplified and corrected respectively to obtain RGB, and then passed through a matrix conversion circuit
  • the brightness signal Y and two color difference signals R-Y (i.e. U) and B-Y (i.e. V) are obtained.
  • the transmitting end encodes the three signals of brightness and color difference respectively to obtain the YUV image.
  • the imaging quality is high, the video is clear, and the details are good.
  • the first image and the second image can also be bitmaps, (Bitmap, BMP), joint photographic experts group (JPGE) images, etc.
  • the embodiment of the present application is for the first image and the second image. There are no restrictions on the format.
  • the image processing unit After obtaining the first image and the second image, the image processing unit allocates the first image and the second image to different processors for parallel processing, as shown in S306 to S309 below.
  • the mobile phone performs shake displacement compensation processing on the first image to obtain the first output image.
  • the method for the mobile phone to perform anti-shake processing on the first image may include S1-S2. That is to say, S306 includes S1-S2.
  • S1 The mobile phone determines the offset of the first image.
  • the picture jitter of the first image originates from the shift in the position of the pixels in the first image (the positions can be represented by coordinates), so the offset of the first image can be determined by the relative displacement of each pixel in the first image.
  • the difference is determined. For example, if the first image processed by the mobile phone is used as the first image of the current frame, and the image collected in the previous frame of the current frame is the first image of the previous frame, then the relative pixel displacement represents the pixel points corresponding to the same photographed subject.
  • the displacement difference between the position of the first image in the current frame and the position of the first image in the previous frame From this, the displacement differences of all pixel points in each frame of the first image can be obtained, recorded as a set of relative pixel displacements.
  • the relative pixel displacement of the first image of the second frame is s1
  • the relative pixel displacement of the first image of the Nth frame is sn-1
  • the first image of the Nth frame N groups of relative pixel displacements will be obtained correspondingly.
  • the offset of the first image of N frames can be determined through N groups of relative pixel displacements.
  • each group of relative pixel displacements can be stored in a warp map.
  • the mobile phone can directly compensate for the displacement difference of the first image due to jitter through the warp map. value, the specific compensation method is shown in S2.
  • S2 The mobile phone obtains the first output image based on the first image and offset.
  • the first output image is an output image obtained by compensating the first image with an offset (such as a relative pixel displacement in a warp map).
  • the first image can be cropped so that the position of the pixel corresponding to the same subject in the first image of the current frame is the same as the position of the first image of the previous frame. Specifically, when the position of the pixel point is higher than the position of the first image of the current frame relative to the position of the first image of the previous frame, the cropping frame is the upper position of the first image, then the removed part is the upper position of the first image.
  • this embodiment can determine the image content to be cropped and the portion retained after cropping through the offset, so that the pixel positions of the cropped first output image of the current frame and the first output image of the previous frame are consistent. In this way, by compensating for the positional offset of the pixels, jitter compensation for the jittery image of the camera is achieved.
  • the mobile phone displays the first output image that has undergone shake compensation processing on the second interface instead of the first image. This allows the subject in the first output image to remain at the target position without being affected by camera shake, thereby achieving a stable image effect.
  • anti-shake processing can be achieved through Electronic Image Stabilization (EIS) technology.
  • EIS uses the mobile phone's acceleration sensor and gyroscope sensor to detect the shake amplitude of the mobile phone, determines the screen shake displacement based on the shake amplitude, and determines the screen shake displacement based on the screen. The jitter displacement compensates for the offset of the first image.
  • anti-shake processing can also be achieved through an optical image stabilizer (Optical Image Stabilizer, OIS). The embodiments of this application do not limit the anti-shake method.
  • the mobile phone detects the first position information of the photographed subject from the second image.
  • the specific method for the mobile phone to detect the first position information of the photographed subject from the second image can refer to the above embodiments and tracking algorithms in conventional technologies such as pedestrian re-identification algorithm (re-identification, ReID).
  • re-identification re-identification, ReID
  • S307 and S306 can be processed in parallel.
  • the subject detection and tracking unit in the mobile phone is used to perform S307
  • the anti-shake unit in the mobile phone is used to perform S306.
  • the time for processing the first image and the second image will be more flexible, thus improving the processing speed of the system.
  • S306 and S307 are two steps that can be processed independently, in other embodiments, after the mobile phone executes S305, it can first execute S307 and then execute S306, or after the mobile phone executes S305, it can execute S306 first and then execute S307. . That is to say, this application does not limit the execution order of anti-shake processing and subject tracking processing.
  • the mobile phone maps the first position information of the photographing subject to the first output image, and obtains the second position information of the photographing subject in the first output image.
  • This step determines the position of the subject from the first output image to eliminate camera shake interference, so that the position change of the subject is only affected by the subject itself. For example, after eliminating camera shake interference, when the position of the photographed subject does not change (such as when a pet is stationary), the second position information obtained between two adjacent first output images remains unchanged or nearly unchanged. , when the photographed subject changes, the obtained second position information between the two adjacent first output images only changes due to the position change of the pet's movement, and is not affected by camera shake factors.
  • the embodiment of the present application maps the first position information obtained through the subject detection process in S307 to the first output image obtained through the anti-shake process in S306 through S308, thereby comparing the anti-shake results and the subject tracking results. Fusion is performed to obtain the second position information.
  • This method of first parallel processing and then fusing the results can reduce performance and power consumption pressure on mobile phones and increase the processing speed of mobile phone processors.
  • the mobile phone After obtaining the first output image, the mobile phone smoothes the subject in the first output image, and the mobile phone executes S309.
  • the mobile phone determines whether the position information of the photographic subject in the first output image of the current frame is consistent with the position information in the first output image of the previous frame based on the second position information. If it is determined that the position of the photographic subject is inconsistent, In this case, the mobile phone smoothes the position information of the shooting subject in the first output image of the current frame based on the historical position information to obtain the third position information of the shooting subject in the first output image.
  • the historical position information includes the position information of the shooting subject in any one or more frames of the first output image before the current frame.
  • the position information includes the coordinate position of the coordinate frame, where the coordinate position includes the coordinate point, or the coordinate position may also include the size of the coordinate frame.
  • the mobile phone smoothes the subject so that the smoothness of the background change of the subject in the first output image is greater than the smoothness of the background change of the target subject in the first output image/first image before processing. Smoothness.
  • the mobile phone obtains the coordinates of the target frame and the warp information (also called offset) output by the EIS, and uses the warp information of the EIS to convert the target frame to image coordinates after anti-shake processing. system to obtain the second location information.
  • the mobile phone obtains the historical position information of the previous frame image, and determines whether the current target frame is approximately stationary by whether the second position information is consistent with the historical position information. If they are consistent, it means that the position of the subject has not changed.
  • Set the position of the current target frame to the same position as the historical position of the previous frame image, and output the smoothed position of the current target frame in the EIS output map coordinate system (second position information ).
  • the changed subject position is smoothed, and the smoothed position of the current target frame in the EIS output map coordinate system is output (third location information).
  • the smooth position it is determined whether it is necessary to output the coordinate position of the smooth position of the target frame in the EIS output map. If so, cropping is performed based on the smooth position so that the cropped image takes into account both anti-shake and smoothness. If not, use the EIS warp information to convert the smoothed position into the coordinate system of the first image before anti-shake processing, so that the cropped image realizes the overall subject position change of the first output image from the first frame to the Nth frame. Smooth effect.
  • smoothing processing includes: removing outliers, filtering processing, etc.
  • filtering may include high speed filtering or Kalman filtering.
  • the smoothing process is Kalman filtering as an example. Kalman filtering is calculated based on the "predicted value" at the historical moment (i.e., historical location information) and the "real value" at the current moment (i.e., the second location information). Obtain the "predicted value” at the current moment (ie, the third location information). Among them, the third position information is predicted by referring to the second position information at the current moment and the "historical position information" at the historical moment.
  • the Kalman filter will smooth the second position information with the historical position information, reduce the change amplitude of the second position information, and obtain the smoothed third location information. Since the displacement between the third position information and the historical position information is smaller than the displacement between the second position information and the historical position information, the smoothing process can avoid things such as arm opening, waving, jumping, etc. that occur during the movement of the photographed subject.
  • the morphological changes resulting in sudden changes in the center point have an impact on the quality of the first image, and thus, this processing improves the shooting quality of the video.
  • the mobile phone first performs subject detection processing and anti-shake processing, and then fuses the two results to obtain the second position information of the subject in the first output image after anti-shake processing. , and then the phone performs smoothing processing.
  • the mobile phone can first smooth the first position information after subject detection processing, and then perform anti-shake processing.
  • the mobile phone can first perform anti-shake processing, then perform subject detection processing in the first output image obtained by anti-shake, and finally perform smoothing processing.
  • the mobile phone obtains the shooting subject in the first image, and determines the target shooting subject from the shooting subjects.
  • S310 further includes:
  • the mobile phone displays the subject frame of the photographed subject on the first image on the second interface.
  • the subject frame is used to identify the subject that can be switched by the user.
  • the identification method includes: enclosing the head area of the subject with a pair of square brackets (also known as square brackets), or the trackable outside of the subject. Outline plus stroke effect, or use a rectangular frame to indicate the subject that can be tracked.
  • the mobile phone detects the third operation performed by the user, wherein the third operation is used to select at least one main body frame when the first image contains multiple main body frames, and the mobile phone determines the target shooting subject from the shooting subject according to the third operation. main body.
  • the mobile phone displays the subject frame of each subject, and determines the target subject specified by the user based on at least one subject frame selected by the user. Therefore, during the video recording process, if the target shooting subject changes (for example, the mobile phone detects the user's operation of switching the target shooting subject), the subject image displayed on the second interface will also change accordingly, so that the changed target shooting subject It can be kept at the target position so that the focus is highlighted, and the subject image of the target subject is both smooth and anti-shake, resulting in good shooting effects.
  • the above-mentioned operation for the user to switch the target shooting subject may be that the user successively selects the subject frames of different shooting subjects during the recording process.
  • the user first selects the subject frame of the father's head, and the second interface displays the subject image of the father who is kept at the target position and is highlighted; when the user selects the subject frame of the younger brother's head, the second interface displays the younger brother's head.
  • the highlighted subject image remains at the target position, allowing switching to the target subject.
  • S310 further includes: the mobile phone acquires the shooting subject in the first image.
  • the mobile phone takes the subject that is at the target position at the first moment as the target subject.
  • the mobile phone automatically determines the target shooting subject from the shooting subjects.
  • the mobile phone After determining the target shooting subject, the mobile phone executes S311.
  • the mobile phone crops out the subject image corresponding to the target shooting subject in the first output image according to the third location information, and displays the subject image in the second interface.
  • the mobile phone crops out the area where the target subject is located after anti-shake processing and smoothing processing, and obtains the subject image.
  • the target subject remains at the target position and is highlighted.
  • the target position may include the middle position of the picture, the golden section position, or other positions where the photographed subject is highlighted.
  • the mobile phone can display and track the target subject so that the target subject is in a prominent position in the picture.
  • this step not only improves the stability of the picture through shake position compensation processing, but also uses smoothing processing to smooth the position of the subject moving at a high speed to achieve a smooth picture effect.
  • the second interface displayed by the mobile phone is shown in Figure 7, as shown in (a) of Figure 7.
  • the subject image in the second interface can be displayed in the area of the first image, as shown in (b) of Figure 7 ), the second interface can be displayed outside the area of the first image, as shown in (c) in Figure 7.
  • the number of main images is 1 to N, and N is the number of tracking preset by the camera application or the number of tracking set by the user. .
  • the first image includes at least one photographic subject, and the one to be highlighted among the photographic subjects is the target photographic subject.
  • the target photographing subject there may be one or more target photographing subjects highlighted in the first image, and each subject image may include one or more target photographing subjects.
  • the target photographing subject The position situations include many, for example: one target shooting subject is at the target position, or two target shooting subjects are distributed on both sides (or both sides) of the target position, or both target shooting subjects are at the target position.
  • the number of subject images displayed by the mobile phone in the second interface corresponds to the number of subject boxes selected by the user on the first interface, and the number of subject boxes and subject images may correspond to one or more. In some embodiments, the number of subject images may be greater than the number of subject frames, or equal to the number of subject frames, or smaller than the number of subject frames. And each subject image can display one subject or multiple subjects.
  • the number of subject images is one, and the subject image displays a target subject corresponding to a subject frame selected by the user.
  • the first image includes multiple characters, from left to right, they are sister, father, brother, brother, and sister respectively.
  • a pair of brackets are displayed in the head area of the father and brother in the second interface.
  • the finger or stylus, the same below
  • the characters in the area can be displayed on the second interface.
  • the second interface Take a main image of the subject. For example, if your finger touches the bracketed area on the father's head, the second interface will display the subject image of the father in the middle of the image; when your finger touches the bracketed area of the younger brother's head, the second interface will display the subject of the younger brother in the middle of the image. image.
  • the number of subject images is one, and the subject image displays subjects corresponding to at least two subject frames selected by the user.
  • the second interface displays the main body image including the father and the younger brother, in which the position of the father or the younger brother in the main image remains at the target position so that is highlighted.
  • the method for the user to select multiple body frames includes: using multiple fingers to touch different body frames; or long pressing one of the body frames on the display screen and sliding to another body frame.
  • the second interface displays a subject image including the father and the younger brother.
  • the position of the father or younger brother in the main image is located in the middle of the image, as shown in Figure 9 (a)
  • the father is located at the target position (the middle of the image); or the area between the father and the younger brother in the main image is located in the middle of the image.
  • Figure 9 (b) the father is located in the middle of the main image
  • the younger brother is located at the golden section of the main image, as shown in Figure 9 (c).
  • each subject image displays the subject corresponding to a subject frame selected by the user.
  • one subject image displays the father's subject image
  • the other subject image displays the younger brother's subject image.
  • the father and the younger brother are respectively at the target positions of their corresponding subject images (the target positions in Figure 10 are the father and the younger brother respectively. pin in the middle of the image).
  • At least one subject image includes multiple subject frames.
  • each subject image shows the same subject.
  • the difference between the images displayed in the two subject images may be that the positions of the photographed subjects are different.
  • the father is located in the middle of the image
  • the younger brother is located at the golden section of the image.
  • the area between dad and brother in the second subject image is in the middle of the image.
  • the subject positions displayed in different subject images are different, which can provide users with more shooting composition choices, thereby improving the user's shooting experience.
  • each subject image shows a different subject.
  • the first main body image shows the image of the younger sister
  • the second main body image shows the images of the father and younger brother. That is to say, in this embodiment, the three shooting subjects of sister, father and brother can be highlighted at the same time, so that these three shooting subjects can be stably, smoothly and prominently displayed on the screen while taking into account video anti-shake and smoothing.
  • the above-mentioned S309 and S311 may be replaced by S5 and S6.
  • the mobile phone determines whether the position of the shooting subject in the current frame in the first image is stationary relative to the position of the previous frame based on the second position information. If it is determined that the position of the shooting subject is not stationary, the mobile phone determines the position of the current frame based on the historical position information. Perform smoothing processing to obtain fourth position information of the subject in the first image.
  • the mobile phone crops out a subject image corresponding to the target shooting subject in the first image according to the fourth position information, and displays the subject image in the second interface, where the shooting subject is located at the target position in the main image.
  • the fourth position information obtained in this embodiment is the position of the shooting subject in the first image without anti-shake processing. That is, the mobile phone crops the smoothed subject from the first image before anti-shake processing to obtain the subject image.
  • position tracking and position smoothing are performed on the subject, but the image jitter interference caused by the shaking of the mobile phone is not eliminated.
  • the subject image presented in this way basically achieves the purpose of placing the subject in the middle of the video, and compared with S309 , S311, the subject image is more realistic and can meet the diverse shooting needs of users.
  • the shooting preview interface which is the first interface, can also display processed images on the mobile phone.
  • the mobile phone After entering the first interface, the mobile phone starts executing S305-306 to obtain the first image and the first output image.
  • the mobile phone can display the first output image on the first interface.
  • the mobile phone can display the first image on the first interface.
  • the first image also known as the live image
  • the mobile phone displays the first output image after anti-shake processing on the first interface, so that the displayed picture is always clear and stable. In this way, by selectively displaying the first image or the first output image on the first interface, the user's diverse shooting needs can be met and the user's shooting experience can be improved.
  • the target shooting subject in the shooting preview state, similar to the video recording process, can also be determined and switched, and the subject image corresponding to the target shooting subject can also be displayed on the first interface.
  • the subject image the target shooting subject is located The best visual position, thereby highlighting the target subject and shooting content, and improving the quality of the picture. visual expression effect.
  • the first interface may display the original image collected by the camera before the user performs the second operation.
  • the first interface may also display a tracking box, wherein the original image or the first output image may be displayed in the tracking box. image or no image is displayed.
  • the mobile phone can display the viewfinder frame on the first interface, and display the viewfinder frame and the above-mentioned tracking frame on the second interface. If the mobile phone processes and obtains the first image, the first output image, and the subject image, any one of the original image, the first image, and the first output image is displayed in the viewfinder frame, and the subject image is displayed in the tracking frame. If the mobile phone does not process the above-mentioned first image, first output image or subject image, the original image or no image may be displayed in the viewfinder frame and the tracking frame.
  • the mobile phone detects the third operation performed by the user on the second interface, and the third operation is used to indicate the end of video shooting.
  • the mobile phone After the user instructs the second operation in the above manner, the mobile phone detects the second operation and executes S313.
  • the mobile phone In response to the third operation, the mobile phone generates the recorded N frames of subject images into video files.
  • N frames are the number of subject images acquired by the mobile phone from responding to the second operation to responding to the third operation.
  • the mobile phone can save the video file in the gallery.
  • the mobile phone in response to the third operation, produces the recorded N frames of main body images and N frames of first images displayed on the second interface as video files.
  • the mobile phone in response to the third operation, produces the recorded N frames of main body images and N frames of first output images displayed on the second interface as video files.
  • the video frame image in the video file can keep the target shooting subject always at the visually optimal position. , to highlight the target subject and the shooting content; it can also reduce the interference caused by camera shake, make the transition between adjacent subject images and video frame images more stable, and can also smooth the center point of the subject so that the subject is on the subject image The degree of deviation is small, the subject image and the video frame image are less likely to produce ghosting, and the background is clearer.
  • the mobile phone in response to the third operation, produces the recorded N frames of first images or N frames of first output images displayed on the second interface as a video file.
  • the main body image is displayed during the recording process on the mobile phone, but the video file obtained after the mobile phone ends recording does not include the main body image. This way the subject image serves to stand out during the recording process.
  • the horizontal axis represents time
  • the vertical axis represents the positional displacement of the photographed subject.
  • the position of the shooting subject moves with time, and the motion trajectory of the shooting subject is shown in curve a in Figure 13.
  • the camera will shake when shooting, and the camera shake trajectory is shown as curve b in Figure 13. Therefore, in the video recorded without processing, the positional displacement of the subject is affected by both the movement of the subject and the movement of the camera, which is represented by curve c in Figure 13.
  • the moving image of the subject is always kept in the visually optimal position, as shown by curve d in Figure 13.
  • the mobile phone first trusts the first The subject image obtained by smoothing the information and then performing anti-shake processing will fluctuate around the target position and cannot be maintained at the target position.
  • the position offset of the shooting subject is represented by curve e in Figure 13.
  • embodiments of the present application provide a shooting method.
  • the mobile phone determines the target shooting subject from the shooting subjects.
  • the mobile phone tracks the target shooting subject and performs anti-shake and smoothing on the target shooting subject.
  • the processing method ensures that the target subject is always displayed in the visual optimal position of the picture. This not only keeps the target subject at the target position during the video shooting process and is highlighted, but also reduces the interference caused by camera shake and achieves the best picture quality.
  • the stabilization effect can also smooth the center point of the subject so that when the target subject moves, the smoothness of the background change of the target subject in the subject image is greater than the smoothness of the background change of the target subject in the preview image.
  • the subject image It is less likely to produce ghosting and the background will be clearer.
  • the video file produced from the video frame image including N frames of subject images can not only highlight the target subject and the shooting content; it can also reduce the interference caused by camera shake, and the transition between adjacent subject images and video frame images is smoother Stable; it can also smooth the center point of the subject so that the deviation of the subject in the subject image is small, the subject image and the video frame image are less likely to produce ghosting, and the background is clearer. Therefore, the presentation effect of the captured video files is better. In addition, it can keep the subject at the target position without the user adjusting the video picture displayed on the interface in real time according to the subject's position, thereby reducing the difficulty of shooting and improving the user's shooting experience.
  • the functionality of shooting has been increased and it is more convenient to use.
  • Even ordinary users who have not learned photography, art, etc., the subject they want to highlight in the video can be in the middle of the screen or in the screen.
  • the embodiment of the present application also provides a first shooting device.
  • the shooting device includes an inertial measurement unit (Inertial Measurement Unit, IMU), a subject detection and tracking unit, a smoothing unit, a first image processing unit, Anti-shake deformation compensation unit and image cropping unit, etc.
  • IMU Inertial Measurement Unit
  • the shooting device performs the following steps to implement the shooting method provided by the above embodiment:
  • the IMU detects the camera shake amplitude to obtain the warp map, and inputs the warp map into the anti-shake deformation compensation unit and smoothing unit.
  • the warp map stores the coordinate offset of the grid position in the first image of the adjacent frame. (The coordinate offset is equivalent to the relative pixel displacement of S2 mentioned above).
  • the mobile phone obtains a raw image (12M) with a size of 12 million pixels mega (12M), and processes the raw image into a YUV (12M) image with a constant size and a YUV (YUV) image with a smaller tiny size through the first image processing unit. (tiny)) image, where the YUV (12M) image is equivalent to the first image in the above S305, and the YUV (tiny) image is equivalent to the second image in the above S305.
  • the mobile phone inputs the YUV (12M) image into the anti-shake deformation compensation (EIS warp) unit.
  • the unit inputs the body frame coordinates into the warp map for interpolation to obtain the body frame coordinates compensated by shake displacement.
  • the body frame coordinates are the shooting
  • the subject is mapped to the subject coordinates in the image after stabilization. (The subject coordinates are equivalent to the second position information after jitter displacement compensation in S308 above), and output the YUV (4K) image with a pixel resolution of 4096*2160 and 4K to the image cropping unit.
  • the mobile phone inputs the YUV (tiny) image into the subject detection and tracking unit to identify and obtain the coordinates and size of the subject frame, and inputs the coordinates and size of the subject frame into the smoothing unit to obtain the coordinates and size of the cropping frame after smoothing the coordinates of the shooting subject.
  • the coordinates and size of the cropping frame are also the third position information in S309 above), and the coordinates and size of the cropping frame are input into the image cropping unit.
  • the image cropping unit crops the YUV (High Definition, FHD) image on the YUV (4K) image based on the coordinates and size of the cropping frame.
  • the (YUV (Full High Definition, FHD) image is the one in S311 above. Main body image), and encode the YUV (FHD) image (Video encoder) and display (display).
  • the shooting device performs subject detection and tracking on the first image, camera anti-shake supplementation, and subject position smoothing, so that the moving image of the subject is always kept in the visually optimal position.
  • the main body detection and tracking unit and the anti-shake deformation compensation unit work in parallel, so that the performance and power consumption of the mobile phone are small and the processing speed is fast.
  • the embodiment of the present application also provides a second shooting device.
  • the shooting device includes an inertial measurement unit (IMU), a subject detection and tracking unit, a smoothing unit, a second image processing unit, and an anti-shake unit. Cropping warp map unit and anti-shake deformation compensation unit.
  • IMU inertial measurement unit
  • subject detection and tracking unit a subject detection and tracking unit
  • smoothing unit a smoothing unit
  • second image processing unit a second image processing unit
  • anti-shake unit Cropping warp map unit and anti-shake deformation compensation unit.
  • the mobile phone inputs the YUV (tiny) image into the subject detection and tracking unit to identify and obtain the coordinates and size of the subject frame, and inputs the coordinates and size of the subject frame into the smoothing unit to obtain the coordinates and size of the cropping frame after coordinate smoothing of the subject (cropping
  • the frame coordinates and size are the third position information in S309 above), and the cropping frame coordinates and size are input into the anti-shake cropping warp map unit.
  • EIS warp crops the YUV (12M) image based on the new warp map (the cropping process is equivalent to the above S2) to obtain the YUV (FHD) image, (the YUV (FHD) image is the main image in the above S311), and Encode and display YUV (FHD) images.
  • the anti-shake cropping warp map unit image directly outputs the coordinate offset after smoothing and anti-shaking, so that the EIS warp only needs to be cropped once to obtain the YUV (FHD) image. (Equivalent to the main body image in S311 described above). Compared with the previous embodiment, the cropping step during anti-shake processing is omitted, so the processing speed is improved.
  • Embodiments of the present application also provide a third shooting device.
  • the shooting device includes an inertial measurement unit (Inertial Measurement Unit, IMU), a subject detection and tracking unit, a smoothing unit, a first image processing unit, and an anti-shake unit. Deformation compensation unit and image cropping unit.
  • IMU Inertial Measurement Unit
  • the mobile phone inputs the YUV (12M) image into the anti-shake deformation compensation (EIS warp) unit, obtains the first output image after anti-shake deformation compensation for the YUV (12M) image, and downsamples the first output image.
  • EIS warp anti-shake deformation compensation
  • Obtain the YUV small size image (YUV tiny) image then input the YUV tiny image into the subject detection and tracking unit, obtain the coordinates and size of the body frame and enter it into the smoothing unit.
  • the smoothing unit smoothes the body frame coordinates and size to obtain the cropping frame and size.
  • the image cropping unit crops the YUV (4K) image based on the cropping frame and size, and outputs the YUV (FHD) image.
  • the shooting equipment is sensitive to YUV (12M) images.
  • the anti-shake deformation compensation process and the subject recognition process are performed in sequence, so that the two processes are executed sequentially.
  • the shooting device can directly perform subject detection and tracking in the first output image after anti-shake deformation compensation.
  • the interpolation step of mapping the subject coordinates into the first output image is reduced, so the processing speed is improved.
  • Embodiments of the present application provide an electronic device, which may include a memory and one or more processors.
  • the memory is used to store computer program code, which includes computer instructions.
  • the processor executes the computer instructions, the electronic device can perform each function or step performed by the mobile phone in the above method embodiment.
  • the structure of the electronic device may refer to the structure of the electronic device 100 shown in FIG. 1 .
  • Embodiments of the present application also provide a computer-readable storage medium.
  • the computer-readable storage medium includes computer instructions.
  • the electronic device When the computer instructions are run on an electronic device, the electronic device causes the electronic device to perform the steps performed by the mobile phone in the above method embodiment. Each function or step.
  • Embodiments of the present application also provide a computer program product.
  • the computer program product When the computer program product is run on a computer, it causes the computer to perform various functions or steps performed by the mobile phone in the above method embodiments.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of modules or units is only a logical function division.
  • there may be other division methods for example, multiple units or components may be The combination can either be integrated into another device, or some features can be omitted, or not implemented.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated.
  • the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed to multiple different places. . Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.
  • the above integrated units can be implemented in the form of hardware or software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium.
  • the technical solutions of the embodiments of the present application are essentially or contribute to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium , including several instructions to cause a device (which can be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of this application.
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Studio Devices (AREA)

Abstract

本申请提供一种拍摄方法及电子设备，该方法包括：电子设备启动相机应用。电子设备检测到视频录制操作后，在录制界面上显示预览图像和突出显示目标拍摄主体的主体图像。预览图像和主体图像根据原始图像处理获得，目标拍摄主体在主体图像上处于目标位置，主体图像的取景范围在预览图像的取景范围之内。这样用户只需要将目标拍摄主体保持在预览图像的取景范围内，就能在不需要实时、精确调整目标拍摄主体位置的情况下进行视频录制，因此降低了拍摄难度，提高了用户的拍摄体验。

Description

一种拍摄方法及电子设备
本申请要求于2022年5月27日提交国家知识产权局、申请号为202210594819.X、发明名称为“一种主体追踪方法及设备”的中国专利申请的优先权，以及于2022年7月22日提交国家知识产权局、申请号为202210871133.0、发明名称为“一种拍摄方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种拍摄方法及电子设备。
背景技术
用户录制视频或视频网络日志(video log，vlog)时，一般较为关注拍摄主体，希望拍摄主体始终位于视觉最优位置(如图像中心或黄金分割处等)。然而，当拍摄主体运动时，拍摄主体的位置发生变化，录制的视频或者vlog便无法实现用户希望的拍摄主体始终位于视觉最优位置的效果，视频或vlog的呈现效果较差，影响用户拍摄和观看体验。
发明内容
本申请提供一种拍摄方法及电子设备，能够追踪视频中的拍摄主体，使得拍摄主体始终处于视觉最优位置等目标位置，以突出拍摄主体以及拍摄内容。
第一方面，本申请提供一种拍摄方法，该方法可以应用于电子设备，其中电子设备包括摄像头，摄像头用于采集原始图像，该方法中，电子设备启动相机应用。电子设备检测到视频录制操作后，在录制界面上显示预览图像和突出显示目标拍摄主体的主体图像。预览图像和主体图像根据原始图像处理获得，目标拍摄主体在主体图像上处于目标位置，主体图像的取景范围在预览图像的取景范围之内。
本申请中，电子设备在录制界面中既显示了预览图像，还显示了主体图像。预览图像为摄像头实际采集的较大的取景范围的图像，因此画面内容更丰富，而主体图像在取景范围在预览图像的取景范围之内，即主体图像为较小的取景范围的图像。那么主体图像显示的是预览图像中的部分画面内容。这样主体图像便可突出显示目标拍摄主体及与目标拍摄主体相关的拍摄内容。
主体图像既可用于生成静态图片，也可用于生成视频文件。其中，静态图片中目标拍摄主体处于视觉最优位置，这样拍摄的图像构图更为美观，能突出目标拍摄主体，从而呈现视觉最优的显示效果。而视频文件中目标拍摄主体始终保持在目标位置，这样用户只需要将目标拍摄主体保持在预览图像的取景范围内，就能在不需要实时、精确调整目标拍摄主体位置的情况下进行视频录制，因此降低了拍摄难度，提高了用户的拍摄体验。
结合第一方面，在一种可能的设计方式中，在目标拍摄主体发生运动的过程中，目标拍摄主体保持在所述主体图像上的目标位置。
该设计方式下，用户不需要精确调整，就能在目标拍摄主体发生运动时，让目标拍摄主体始终位于视觉最优位置从而被突显，因此降低了拍摄难度，提高了用户的拍摄体验。
结合第一方面，在一种可能的设计方式中，在目标拍摄主体发生运动的过程中，主体图像中目标拍摄主体背景变化的平滑度，大于预览图像中目标拍摄主体背景变化的平滑度。
该设计方式下，主体图像中的背景的平滑度更大，意味着背景过渡更为平滑。那么，相较于预览图像，主体图像中的目标拍摄主体之外的背景，在目标拍摄主体运动特别是剧烈运动造成目标拍摄主体中心点突变时不会剧烈变化。这样背景不易产生虚影，那么背景更加清晰。且相邻帧的主体图像的背景变化跨度小，能够更好避免因背景剧烈变化带来的录制的视频中画面产生跳跃和闪烁的情况。因此，主体图像的平滑度更大，主体图像所呈现的视觉效果更好。
结合第一方面，在一种可能的设计方式中，目标拍摄主体包括至少一个，主体图像包括至少一个。其中，一个目标拍摄主体对应一个主体图像，或者一个目标拍摄主体对应多个主体图像，主体图像位于预览图像的区域外或者与预览图像的区域部分重叠。
该设计方式下，录制界面上显示的主体图像的数量以及主体图像中所突显的目标拍摄主体的数量是不限制的。那么在视频录制过程中，可以根据拍摄目的和构图需求进行调整，从而满足多样化拍摄需要，提高用户的拍摄体验。另外，在录制界面中主体图像与预览图像的位置关系也可以多样化调整，以提高拍摄功能性和趣味性。
结合第一方面，在一种可能的设计方式中，目标拍摄主体和/或目标位置为响应于用户的指示操作确定的或者电子设备自动确定的，该方法还包括：电子设备检测到切换目标拍摄主体的操作，在录制界面上显示切换后的目标拍摄主体对应的主体图像；和/或检测到切换目标位置的操作，在所述录制界面上显示目标拍摄主体位于切换后的目标位置对应的主体图像。
该设计方式下，录制开始时刻以及录制过程中，电子设备都可以对目标拍摄主体进行多种方式的切换(如主动切换和被动切换)。比如，电子设备可以在录制过程中根据用户执行的切换操作，将当前目标拍摄主体切换为用户所指定的目标拍摄主体，然后电子设备将指定的拍摄主体所对应的主体图像显示在录制界面。这样，便可达到更好的交互体验。再比如，电子设备还可以在录制过程自动确定目标拍摄主体，并且在主体图像中将目标拍摄主体保持在画面的目标位置，这样即使用户的拍摄技巧有限，电子设备也能辅助用户拍摄出能够突出拍摄主体和拍摄内容的视频，从而降低了拍摄难度，也提高了拍摄得到的视频质量。
该设计方式下，电子设备目标位置的确定以及切换方式可以参考上述目标拍摄主体进行多种切换的方式。例如，电子设备可以默认目标拍摄主体的初始目标位置(如图像中间位置)，在录制前的预览过程或者录制过程中，电子设备可以根据用户执行的切换操作，将目标拍摄主体切换至用户指定的目标位置(如图像的黄金分割点)。那么切换后录制界面显示的主体图像中，目标拍摄主体始终位于图像的黄金分割点。
在一种可能的设计方式下，电子设备可以将可选的目标位置显示在预览图像中， 用以指示该位置可作为目标位置，这样用户便可在通过触摸预览图像中的目标位置，从而执行切换操作。
结合第一方面，在一种可能的设计方式中，电子设备在录制界面上显示第i帧预览图像和目标拍摄主体对应的主体图像，i为大于1的正整数，包括：电子设备根据第i帧原始图像，获取第i帧预览图像。电子设备根据目标拍摄主体在第i-1帧预览图像中的第一位置信息，对目标拍摄主体在第i帧预览图像中的第一位置信息进行平滑处理，得到目标拍摄主体的第二位置信息。然后电子设备根据第二位置信息裁剪第i帧预览图像，得到目标拍摄主体对应的主体图像。并在录制界面上显示第i帧预览图像和目标拍摄主体对应的主体图像。
该设计方式下，电子设备以第i-1帧预览图像的第一位置信息，来平滑处理第i帧预览图像的第一位置信息，使得第i帧预览图像的第一位置信息既参考了目标拍摄主体的实际位置，又结合了第i-1帧预览图像的第一位置信息(历史时刻的位置“预测值”)。从而能够在目标拍摄主体运动过程中发生如张臂、挥手、蹦跳等形态变化，导致目标拍摄主体的实际位置剧烈变化时，通过历史时刻的位置“预测值”调整第i帧预览图像中目标拍摄主体的实际位置。这样既增加了相邻主体图像的画面流畅度，还能够减少因背景剧烈变化带来的录制的视频中画面产生跳跃和闪烁的情况。故提高了拍摄得到的视频质量。
结合第一方面，在一种可能的设计方式中，电子设备根据目标拍摄主体在第i-1帧预览图像中的第一位置信息，对目标拍摄主体在第i帧预览图像中的第一位置信息进行平滑处理，得到目标拍摄主体的第二位置信息，包括：电子设备获取目标拍摄主体分别在第i-1帧预览图像中的第一位置信息和在第i帧预览图像中的第一位置信息之间的第一位置差值。基于第一位置差值，确定根据目标拍摄主体在第i-1帧预览图像中的第一位置信息，对第i帧预览图像中的第一位置信息进行平滑化处理，得到目标拍摄主体的第二位置信息，其中，第二位置信息与第i-1帧预览图像中的第一位置信息之间的第二位置差值小于或等于第一位置差值。
该设计方式下，电子设备通过减少第一位置信息的变化程度，从而实现平滑处理。也就是说，电子设备获取目标拍摄主体在第i-1帧预览图像和第i帧预览图像之间的位移幅度(即第一位置差值)，并以该位移幅度，确定是否需要对第i帧预览图像进行平滑处理，在确定需要进行平滑处理的情况下，电子设备以第i-1帧预览图像中的第一位置信息(历史时刻的位置“预测值”)，平滑第i帧预览图像中的第一位置信息(当前时刻的实际位置)。使得经平滑处理后，目标拍摄主体的位移幅度(即第二位置差值)更小，那么目标拍摄主体的背景的变化跨度也就更小，背景更为平滑化。
结合第一方面，在一种可能的设计方式中，根据第i帧原始图像，获取第i帧预览图像，包括：电子设备对第i帧原始图像进行格式处理，得到第i帧第一图像。然后电子设备第i帧第一图像即为第i帧预览图像。或者，电子设备还可以对第i帧第一图像进行抖动位移补偿处理，得到第i帧第一输出图像，第i帧第一输出图像即为第i帧预览图像。
该设计方式下，预览图像可以是未经防抖处理的第一图像。第一图像由摄像头采集的原始图像直接经过格式处理得到，因此又称实况图像。显示实况图像能够使画面 更真实和自然。预览图像也可以是经过防抖处理的第一输出图像。第一输出图像通过补偿因相机抖动造成的画面偏移，因此画面整体更为清晰和稳定。如此，通过在录制界面有选择的显示第一图像或者第一输出图像的方式能够满足用户多样化的拍摄需要，提高用户的拍摄体验。
结合第一方面，在一种可能的设计方式中，对第i帧第一图像进行抖动位移补偿处理，得到第i帧第一输出图像，包括：电子设备获取第i帧第一图像相对于第i帧第一图像之前的任一或多帧第一图像的画面抖动位移信息。根据画面抖动位移信息，对第i帧第一图像中的画面进行抖动位移补偿处理，得到第i帧第一输出图像。
该设计方式下，第i帧第一图像之前任一帧或多帧第一图像都可以作为电子设备对第i帧第一图像进行防抖处理时的参考图像。那么得到的画面抖动位移信息更为准确。采用该画面抖动位移信息进行抖动位移补偿处理，就能达到更准确地防抖效果。经防抖得到的第一输出图像中，像素点不受抖动干扰，能够保持在稳定位置，因此像素点所构成的画面可以达到清晰和稳定的呈现效果。
结合第一方面，在一种可能的设计方式中，获取第i帧第一图像相对于第i帧第一图像之前的任一或多帧第一图像的画面抖动位移信息，包括：检测电子设备的抖动幅度。根据抖动幅度，确定画面抖动位移信息，其中，画面抖动位移信息包括第i帧第一图像中的每个像素点相对于第i帧前的任一或多帧第一图像中的对应像素点的偏移量。
其中，可以采用手机加速度传感器和陀螺仪传感器检测手机的抖动幅度。
该设计方式下，采用检测电子设备抖动幅度的方式来确定第一图像的画面抖动幅度(即画面抖动位移信息)，这样实现方式更为简单高效，提高了图像处理速度。
结合第一方面，在一种可能的设计方式中，该方法还包括：对第i帧第一图像进行下采样处理，得到第i帧第二图像。在根据目标拍摄主体在第i-1帧预览图像中的第一位置信息，对目标拍摄主体在第i帧预览图像中的第一位置信息进行平滑处理之前，该方法还包括：电子设备从第i帧第二图像中检测得到目标拍摄主体，并获取目标拍摄主体的第三位置信息。然后电子设备将目标拍摄主体的第三位置信息映射到第i帧预览图像中，得到目标拍摄主体位于第i帧预览图像中的第一位置信息。
该设计方式下，主体追踪处理和图像防抖处理为两个并行处理的任务，电子设备对两个任务的结果进行融合，得到第二位置信息。电子设备采用这样先并行处理，再融合结果的方式，可以减少单个任务的等待时间。且可以提高对电子设备的处理器资源的利用率，减少内存消耗，从而提高了获取第i帧预览图像中目标拍摄主体的第一位置信息的处理速度。
结合第一方面，在一种可能的设计方式中，根据第二位置信息裁剪第i帧预览图像，得到目标拍摄主体对应的主体图像，包括：电子设备根据第二位置信息确定目标拍摄主体的最小外接矩形框。根据最小外接矩形框确定裁剪区域，其中，裁剪区域包含最小外接矩形框。然后电子设备在第i帧预览图像获取裁剪区域对应的图像，得到目标拍摄主体对应的主体图像。
其中，最小外接矩形框为包括目标拍摄主体的外轮廓的最小边界矩形，最小外接矩形框可以通过外轮廓上最上/下/左/右的坐标点的坐标值确定。
该设计方式下，得到的裁剪区域包含了目标拍摄主体的外轮廓，使得裁剪后的主体图像能完整显示目标拍摄主体。
结合第一方面，在一种可能的设计方式中，在第i帧预览图像获取裁剪区域对应的图像之后，该方法还包括：电子设备将裁剪区域对应的图像缩小或放大，得到具有预设显示规格的目标拍摄主体对应的主体图像，其中，预设显示规格为预设的电子设备的屏幕占比。
该设计方式下，电子设备对裁剪后的图像进行规格的调整，以满足输出的实际规格的需求，这样按照预设显示规格输出的主体图像在屏幕中显示地更为清楚，且相邻帧主体图像的规格统一，显示效果更好。
结合第一方面，在一种可能的设计方式中，目标位置包括中间位置或黄金分割点位置。
电子设备按照中间位置和黄金分割点进行构图得到的主体图像，不仅更为美观，而且能突出目标拍摄主体及视频拍摄内容，从而实现视觉最优效果，使视频或vlog的呈现效果更好，提高了用户的拍摄和观看体验。
结合第一方面，在一种可能的设计方式中，该方法还包括：电子设备检测到结束录制操作，便生成视频文件，其中视频文件包括主体图像，或者视频文件包括主体图像和预览图像。
该设计方式下，由于视频文件中的视频帧图像所包括的主体图像是目标拍摄主体兼顾防抖和平滑且位于目标位置的图像，因而视频文件中的视频帧图像能保持目标拍摄主体始终位于视觉最优位置，以突显该目标拍摄主体以及拍摄内容。不仅如此，视频文件中减少相机抖动造成的干扰，相邻主体图像和视频帧图像间的画面过渡更为稳定，还能够平滑拍摄主体的中心点使主体图像上拍摄主体的偏离程度小，主体图像和视频帧图像不易产生虚影且背景更加清晰。
结合第一方面，在一种可能的设计方式中，该方法还包括：在启动相机应用后显示拍摄预览界面，其中，拍摄预览界面包括预览图像，或者拍摄预览界面包括预览图像和主体图像。
该设计方式中，电子设备可以将开始录制时的呈现效果先实时预览，这样可以让用户提前知道视频录制出来的实际效果。用户进行拍摄创作时，在预览界面便可根据需要调节拍摄参数，从而使电子设备进入录制界面后，得到视频效果更好。
第二方面，本申请提供一种电子设备，包括：用于采集图像摄像头、用于显示上述预览界面和录制界面的屏幕、一个或多个处理器、存储器、以及一个或多个计算机程序，其中一个或多个计算机程序被存储在存储器中，一个或多个计算机程序包括指令。当该指令被电子设备执行时，使得电子设备执行如第一方面及其可能的设计方式的方法。
第三方面，本申请提供一种计算机可读存储介质，该计算机可读存储介质可以包括计算机指令，当计算机指令在电子设备上运行时，使得电子设备执行如第一方面及其可能的设计方式的方法。
第四方面，本申请提供一种计算机程序产品，当计算机程序产品在计算机上运行时，使得计算机执行如第一方面及其可能的设计方式的方法。
可以理解地，上述提供的第二方面的电子设备，第三方面的计算机存储介质，以及第四方面的计算机程序产品均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
附图说明
图1为本申请实施例提供的一种电子设备的硬件结构示意图；
图2为本申请实施例提供的一种电子设备的软件结构示意图；
图3为本申请实施例提供的一种拍摄方法的流程图；
图4为本申请实施例提供的两种用户执行第一操作的示意图；
图5为本申请实施例提供的一组目标位置的示意图；
图6为本申请实施例提供的一种平滑处理方法的流程图；
图7为本申请实施例提供的一组第二界面的示意图；
图8为本申请实施例提供的第一种应用场景下的主体图像的示意图；
图9为本申请实施例提供的第二种应用场景下的主体图像的示意图；
图10为本申请实施例提供的第三种应用场景下的主体图像的示意图；
图11为本申请实施例提供的第四种应用场景下的一种主体图像的示意图；
图12为本申请实施例提供的第四种应用场景下的另一种主体图像的示意图；
图13为本申请实施例提供的拍摄主体位置随时间变化的折线示意图；
图14为本申请实施例提供的第一种拍摄设备的示意图；
图15为本申请实施例提供的第二种拍摄设备的示意图；
图16为本申请实施例提供的第三种拍摄设备的示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。其中，在本申请实施例的描述中，除非另有说明，“/”表示或的意思，例如，A/B可以表示A或B；本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
在用户录制视频或vlog时，一般更加关注拍摄主体，希望拍摄主体始终位于视觉最优位置(如图像中心或黄金分割处等)。然而录制时拍摄主体通常不会一直保持静止，因此拍摄主体在图像上就难以保持在视觉最优位置，录制的视频或者vlog便无法达到用户想要的拍摄效果。
为此，可以采用将相机的拍摄角度跟随拍摄主体一起运动的方式，以尽量使得拍 摄主体能够处于视觉最优位置，这样有助于强调拍摄主体以及拍摄内容，提高画面的视觉表达效果。然而在相机的拍摄角度跟随拍摄主体运动的过程中，很难保证相机与拍摄主体的运动频率一致，因此常出现相机运动过快或者过慢，导致拍摄主体偏离画面中心、拍摄主体的背景产生虚影等问题。
此外，还可以考虑采用主体追踪技术识别并突显拍摄界面中的拍摄主体的位置信息，主体追踪技术通过突显位置信息从而达到用户手持摄像头拍摄时，辅助用户尽量将拍摄主体的位置信息保持在视觉最优位置的效果。然而，主体追踪技术存在几点问题：
首先，该技术只有应用在相机固定(如采用相机三脚架固定相机)的场景下才能较好辅助用户拍摄出拍摄主体稳定处于画面中间的视频。一旦用户手持相机录制视频，相机会受到用户手部的抖动的干扰，从而造成拍摄主体的主体位置偏差。这种因相机抖动造成的主体位置偏差无法采用主体追踪技术解决，这就导致采用主体追踪技术拍摄的视频，拍摄主体受相机抖动无法始终处于视觉最优位置的问题。
其次，拍摄过程中，若拍摄主体的形态发生改变(如张臂，挥手，蹦跳等)，则可能影响拍摄主体的中心点发生突变，从而造成拍摄的视频中拍摄主体偏离画面中心或者拍摄主体的背景产生虚影等问题。这种拍摄主体的中心点突变对画面呈现质量造成影响的问题也无法通过主体追踪技术解决。
另外，主体追踪技术在用户拍摄时起到的只是辅助作用，该技术需要用户根据主体位置实施调整界面显示的视频画面。用户很难通过实时地调整相机的拍摄位置和拍摄角度来调整视频画面，从而精确地将拍摄的视频画面中拍摄主体保持在视觉最优位置，因此主体追踪技术的拍摄难度高，会造成用户较差的拍摄体验。
基于此，本申请实施例提供一种拍摄方法，该方法可以应用于电子设备，电子设备包括显示屏和摄像头。其中，电子设备启动相机应用，进入追踪主体的视频拍摄模式。该模式下，在开始录制视频后，电子设备对摄像头采集到的原始图像进行处理，并将处理后的图像显示在录制界面上。其中，电子设备基于原始图像进行抖动位移补偿处理，来消除因相机抖动造成的主体位置偏差，并基于原始图像对拍摄主体的位置进行平滑处理，以减少拍摄主体位置突变对拍摄主体的位置造成影响。这样，显示在录制界面上的图像中的拍摄主体经过了抖动补偿处理，不受相机抖动干扰，能保持在目标位置从而被重点突出，达到了画面稳定的效果。
而且，本申请实施例还能够平滑拍摄主体的中心点使拍摄主体的偏离程度小，不易产生虚影，背景更加清晰。
此外，本申请实施例不需要用户根据拍摄主体位置实时调整界面显示的视频画面，就能将拍摄主体保持在目标位置，从而降低了拍摄难度，提高了用户的拍摄体验。
综上，本申请实施例在拍摄视频时不仅能够稳定且清晰地突显拍摄内容和拍摄主体，提高了视频的拍摄质量，视频的呈现效果较好。还可以减少用户调整画面的操作，降低拍摄难度，提高用户的拍摄体验。
本申请实施例提供的拍摄方法可以适用于设置有摄像头和显示屏的电子设备，其中，该电子设备可以为手机、平板电脑、笔记本电脑、超级移动个人计算机(Ultra-mobile Personal Computer，UMPC)、上网本、掌上电脑(Personal Digital Assistant，PDA)、摄 像机、数码相机、监控等设备。请参考图1，其示出了本申请实施例提供的一种电子设备100的结构示意图。该电子设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。
其中，上述传感器模块180可以包括压力传感器，陀螺仪传感器180A，气压传感器，磁传感器，加速度传感器，距离传感器，接近光传感器，指纹传感器，温度传感器，触摸传感器180B，环境光传感器，骨传导传感器等。
可以理解的是，本申请实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器神经网络处理器(neural-network processing unit，NPU)，和/或微控制单元(micro controller unit，MCU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，串行外设接口(serial peripheral interface，SPI)，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
可以理解的是，本申请实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块140用于从充电器接收充电输入。电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，外部存储器，显示屏194，摄像头 193，和无线通信模块160等供电。在另一些实施例中，电源管理模块141和充电管理模块140也可以设置于同一个器件中。
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks，WLAN)(如Wi-Fi网络)，蓝牙(blue tooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，NFC，红外技术(infrared，IR)等无线通信的解决方案。
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。ISP用于处理摄像头193反馈的数据。摄像头193用于捕获静态图像或视频(又称原始图像)。本申请实施例中，ISP还用于将摄像头采集的原始图像处理为第一图像和第二图像，并将第一图像和第二图像分流给不同处理器分别进行防抖处理和主体追踪处理。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备100的智能认知等应用，例如：主体识别，防抖处理，主体位置平滑处理等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行电子设备100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如音频数据，电话本等)。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音、语音交互等。
陀螺仪传感器180A又称陀螺仪、角速度传感器，可以对手机转动、偏转的动作进行测量。本申请实施中，陀螺仪传感器180A用于检测手机的抖动幅度，其中，抖动幅度用于处理器110确定画面抖动位移，并根据画面抖动位移对视频画面进行偏移量的补偿。
触摸传感器180B，也称“触控面板(TP)”。触摸传感器180B用于检测作用于其上方或附近的触摸操作。例如用户使用手指、触控笔等任何适合的物体或附件在触摸屏上或触摸屏附近的操作(如用户按压“拍摄”按钮)，触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。并且手机100可以通过显示屏194提供与触摸操作相关的视觉输出。在本申请实施例中，触摸传感器180B和显示屏组合成上述触摸屏。在另一些实施例中，触摸传感器180B也可以设置于电子设备100的表面，与显示屏194所处的位置不同。
按键190包括开机键，音量键等。马达191可以产生振动提示。指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。SIM卡接口195用于连接SIM卡。
以电子设备100为手机为例，图2是本申请实施例提供的手机的一种软件结构框图。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android Runtime)和系统库，以及内核层。
如图2所示，应用程序层可以包括一系列应用程序包，应用程序包可以包括相机，日历、地图、视频、音乐、短消息、图库、通话、导航、蓝牙、WLAN等应用程序。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
如图2所示，应用程序框架层可以包括：相机应用对应的接口、界面管理服务(activity manager service，AMS)模块。其中，AMS统一调度上述应用程序的界面Activity。
应用程序框架层还可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供手机100的通信功能。例如通话状态的管理(包括接通，挂断等)。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，手机振动，指示灯闪烁等。
Android Runtime包括核心库和虚拟机。Android Runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
如图2所示，系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。
2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。内核层中至少包括硬件的驱动，用于驱动硬件工作。
如图2所示，内核层中包括：摄像头驱动、显示驱动(display driver)、图形处理器驱动(graphics processing unit driver，GPU driver)等。
其中，摄像头驱动用于启动摄像头响应用户的操作。内核层还可以包含音频驱动，传感器驱动。
下面结合本申请提供的一种拍摄方法，示例性说明手机的工作流程。应用程序层中相机应用的图标可以显示于手机的显示屏上，当相机应用的图标被用户手指接触，位于显示屏上的触摸传感器180B检测到触摸操作时，手机启动相机应用。相机应用调用内核层中的摄像头驱动，来启动摄像头响应录制操作。在视频录制的过程中，摄像头采集连续的原始图像，所采集的原始图像经过ISP处理为第一图像和第二图像，并将第一图像和第二图像分流给不同处理器110，来分别对第一图像进行主体追踪处理，对第二图像进行防抖处理。然后，处理器110将主体追踪结果与防抖处理结果相结合，从而得到经过主体追踪处理和防抖处理的图像，再对该图像中的主体位置进行平滑处理，最终得到主体图像。手机开始录制视频后，在显示屏上显示该主体图像， 主体图像能够在用户不根据拍摄主体位置实时调整界面显示的视频画面的情况下将拍摄主体保持在目标位置，而且通过防抖和平滑可以在拍摄视频时稳定且清晰地突显拍摄内容和拍摄主体，提高视频的拍摄质量，视频的呈现效果较好。还减少用户调整画面的操作，降低拍摄难度，从而提高用户的拍摄体验。
本申请实施例提供一种拍摄方法，该方法可以应用于电子设备，电子设备包括显示屏和摄像头。以电子设备是手机为例，如图3所示，该拍摄方法可以包括：
S301、手机检测到用户执行的第一操作，第一操作用于指示启动相机应用。
例如，第一操作包括：用户触摸手机桌面上的相机应用的图标的操作，或者用户的语音指示，或者用户手势的操作。
在一些实施例中，第一操作是用户触摸手机桌面上的相机应用的图标的操作时，手机还包括触摸传感器，触摸传感器和显示屏组合成触摸屏，图像内容通过触摸屏呈现。如图4中的(a)所示，手机在图形用户界面(graphical user interface，GUI)上显示相机应用的图标。用户手指触摸该相机应用的图标，以指示手机启动相机应用。
需要说明的是，本申请实施例不局限于手机桌面上的相机图标对应的相机应用，也可以是其他应用集成的具有相机功能的应用，比如聊天软件、购物软件或短视频社交软件里集成的用于录像的应用等。
在一些实施例中，当第一操作是用户的语音指示时，如图4的(b)所示，用户在距离手机预设范围内说出如“打开相机”、“我要录制”、“进入视频拍摄模式”等命令，以指示手机启动相机应用。
在一些实施例中，当第一操作是用户手势的操作时，用户可以在手机的GUI上划一个指示启动相机应用的手势操作，其中，该手势操作可以为初始设置或由用户自定义的圆圈轨迹、波浪线轨迹、交叉线轨迹等。
用户以上述方式作出第一操作的指示后，手机检测到第一操作，然后执行S302。
S302、手机响应于第一操作，启动相机应用，进入追踪主体的视频拍摄模式。
需要说明的是，能实现追踪主体的视频拍摄模式的相机应用可以是嵌入追踪主体程序和功能的现有相机应用，也可以是具备追踪主体的视频拍摄模式的新的应用程序，本申请实施例对此并无限制。
手机进入追踪主体的视频拍摄模式包括多种，在一些实施例中，手机启动相机应用后自动进入追踪主体的视频拍摄模式。在另一些实施例中，手机启动相机应用后，检测到用户选择的追踪主体的视频拍摄模式，手机进入该模式。
不同于常规拍摄模式，追踪主体的视频拍摄模式下拍摄出的视频，拍摄主体始终自动出现在视频画面被重点突出的目标位置，这样，手机在拍摄视频时便可以自动将拍摄内容和拍摄主体稳定且清晰地在视频中突显出来，从而提高了视频的拍摄质量和呈现效果，提高了用户的拍摄体验。
本步骤中，能够凸显拍摄内容和拍摄主体的视觉最优位置以目标位置来表示，目标位置包括初始设置位置或由用户自定义的位置。其中，初始设置位置或由用户自定义的位置包括：图像中心或者黄金分割处。
一些实施例中，目标位置包括图像中心，如图5的(a)所示。那么在追踪主体的视频拍摄模式下，拍摄的视频中主体始终处于画面中心。
另一些实施例中，目标位置包括黄金分割处，如图5的(b)所示。若初始设置位置或由用户自定义的位置是黄金分割处，则拍摄过程中，拍摄主体始终处于画面的黄金分割点。
S303、手机显示第一界面，手机检测到用户在第一界面执行的第二操作，该第二操作用于指示拍摄视频。
其中，第一界面(又称预览界面)为追踪主体的视频拍摄模式下的拍摄预览界面。第一界面显示的图像根据摄像头在用户执行第二操作前采集的原始图像得到。
第二操作用于指示摄像头启动录制视频(包括录制vlog等)。第二操作包括：用户触摸第一界面中的拍摄按钮，或者用户说出的如“开始拍摄”的语音指令，或者用户手势的操作。该第二操作可参见上述S301中的第一操作的介绍，在此不做累赘说明。
用户以上述方式作出第二操作的指示后，手机检测到第二操作，并执行S304。
S304、手机响应于第二操作，在显示屏上显示第二界面。
第二界面(又称录制界面)为手机响应上述第二操作后，开始录制视频的录制界面。录制视频过程中摄像头连续采集原始图像，手机对原始图像进行处理，并将处理得到的图像显示于第二界面。第二界面包括基于原始图像处理得到的主体图像，或者第二界面还包括基于原始图像处理得到的第一图像或第一输出图像(第一图像和/或第一输出图像又称预览图像)。
一些实施例中，在录制视频过程中，手机处理得到图像之前，第二界面可以显示原始图像，或者第二界面可以不显示任何图像。在手机处理得到图像后，第二界面显示处理得到的图像，如上述主体图像以及第一图像或第一输出图像。
其中，在录制视频的过程中，手机对原始图像进行处理，并将处理得到的图像显示于第二界面的过程可以包括S305-S311。
S305、手机获取摄像头采集的多个原始图像，将每个原始图像处理为第一图像和第二图像。
录制视频过程中摄像头连续采集原始图像，手机将原始图像处理为第一图像。其中，原始图像可以是摄像头直接采集得到的原始(raw)图，也可以是多个摄像头采集并合成的raw图。其中，raw可理解为“未经加工”，也就是说，上述raw图是由摄像头的传感器直接输出的原图，因为其尚未被处理和编辑，因此仍保留原有的宽色域的内部色彩，图像的质量更高。
该步骤中，手机将原始图像输入图像处理单元，由图像处理单元将原始图像处理为第一图像和第二图像。其中，第一图像被用于进行图像防抖处理，第二图像被用于进行主体追踪处理。在一些实施例中，手机将第一图像显示在第二界面中。以第一图像(又称实况图像)显示在第二界面中能够使画面更真实和自然，且保留画面的抖动效果可以让视频观看者身临其境，从而提高了视频的拍摄和观看体验。
在一些实施例中，第一图像与原始图像的尺寸相同，都为原始尺寸，第二图像是对第一图像进行降采样得到的尺寸较小的图像。其中，尺寸由M像素*N像素表示，也就是说，第二图像的像素个数小于第一图像的像素个数，第二图像的分辨率低于第一图像的分辨率。如此，第一图像所呈现的视频质量更高，画面更清晰，用于防抖处理中对于抖动位移的获取精度更高。第二图像用于主体检测中的计算量更小，处理速 度更快。当然，在一些实施例中，第一图像的尺寸可以小于原始图像的尺寸。或者第二图像的尺寸可以大于、等于第二图像的尺寸。或者在第一图像的尺寸小于原始图像的尺寸的情况下，第二图像的尺寸小于第二视频尺寸。本申请实施例对第一图像及第二图像的尺寸并无限制。
示例性的，第一图像和第二图像可同为亮度色度饱和度(YUV)图，YUV图是指采用YUV编码方法得到的图像。其中YUV是被欧洲电视系统所采用的一种颜色编码方法，通常可以采用三管彩色摄影机进行取像，然后把取得的彩色图像信号经分色、分别放大校正后得到RGB，再经过矩阵变换电路得到亮度信号Y和两个色差信号R-Y(即U)、B-Y(即V)，最后发送端把亮度和色差三个信号分别进行编码，得到YUV图。采用YUV图的好处在于，成像质量高，视频清晰，细节表现好。当然，第一图像、第二图像还可以为位图，(Bitmap，BMP)、联合图像专家小组规范(joint photographic experts group，JPGE)图等，本申请实施例对第一图像和第二图像的格式并无限制。
在得到第一图像和第二图像之后，图像处理单元将第一图像和第二图像分配给不同处理器从而并行处理，如下面S306至S309。
S306、手机对第一图像进行抖动位移补偿处理，得到第一输出图像。
在一些实施例中，手机对第一图像进行防抖处理的方法可以包括S1-S2。也就是说，S306包括S1-S2。
S1：手机确定第一图像的偏移量。
第一图像的画面抖动来源于第一图像中的像素点的位置(位置可以由坐标表示)发生偏移，因此第一图像的偏移量可以由第一图像中的每个像素点的相对位移差值确定。示例性的，以手机在处理的第一图像作为当前帧第一图像，在当前帧图像的上一帧采集到的为上一帧第一图像，那么相对像素位移表示同一拍摄主体对应的像素点在当前帧第一图像的位置相对于上一帧第一图像的位置的位移差值。由此每一帧第一图像都可以得到所有像素点的位移差值，记为一组相对像素位移。如首帧第一图像的相对像素位移为s0＝0，第二帧第一图像的相对像素位移为s1……第N帧第一图像的相对像素位移为sn-1，则N帧第一图像会对应得到N组相对像素位移。通过N组相对像素位移便可确定N帧第一图像的偏移量。
在一些实施例中，为方便数据存储和查询，可以将每组相对像素位移存储在变形地图(warp map)中，这样，手机便可直接通过warp map补偿因抖动导致的第一图像的位移差值，具体的补偿方法如S2所示。
S2：手机根据第一图像和偏移量得到第一输出图像。
其中，第一输出图像为第一图像经偏移量(如warp map中的相对像素位移)的补偿处理得到的输出图像。
在一些实施例中，可以通过裁剪第一图像使得同一拍摄主体对应的像素点在当前帧第一图像的位置与上一帧第一图像的位置相同。具体的，当像素点位置在当前帧第一图像的位置相对于上一帧第一图像的位置偏上时，裁剪框为第一图像中偏上位置，那么被去掉的部分是第一图像中的下部图像，这样裁剪后图像的拍摄主体仍处于当前帧第一图像中与上一帧第一图像相同或接近相同的位置；反之，当像素点位置在当前帧第一图像的位置相对于上一帧第一图像的位置偏向下时，裁剪框为第一图像中的偏 下位置，被去掉的部分是第一图像中的上部图像。这样，本实施例便可通过偏移量确定需要裁剪的图像内容和裁剪后保留的部分，从而使裁剪得到的当前帧第一输出图像与上一帧第一输出图像的像素点位置保持一致。如此，通过对像素点的位置偏移量进行补偿，便实现了对摄像头的抖动画面进行抖动补偿。
在一些实施例中，手机以经过抖动补偿处理的第一输出图像代替上述第一图像显示于第二界面。使得第一输出图像中的拍摄主体能够不受相机抖动影响，保持在目标位置，从而达到画面稳定效果。
在一些实施例中，可以通过电子防抖技术(Electric Image Stabilization，EIS)实现防抖处理，EIS采用手机加速度传感器和陀螺仪传感器检测手机的抖动幅度，根据抖动幅度确定画面抖动位移，并根据画面抖动位移对第一图像的偏移量进行补偿。此外，还可以通过光学图像稳定器(Optical Image Stabilizer，OIS)实现防抖处理，本申请实施例对防抖方法不做限制。
S307、手机从第二图像中检测得到拍摄主体的第一位置信息。
其中，手机从第二图像中检测得到拍摄主体的第一位置信息的具体方法，可以参考上述实施例和常规技术中的追踪算法如行人重识别算法(re-identification，ReID)，本申请实施例这里不予赘述。
在该实施例中，S307和S306可以并行处理，例如采用手机中的主体检测与追踪单元执行S307，同时采用手机中的防抖单元执行S306。该设计方式下，处理第一图像和第二图像的时间会更为灵活，因此能提高系统的处理速度。
由于S306和S307是可以独立处理的两个步骤，因此在另一些实施例中，手机执行S305后，可以先执行S307，然后执行S306，或者，手机执行S305后，可以先执行S306，然后执行S307。即本申请对防抖处理和主体追踪处理的执行顺序不做限制。
手机采用上述方法进行防抖处理以及检测到第一位置信息之后，执行S308。
S308、手机将拍摄主体的第一位置信息映射到第一输出图像中，得到拍摄主体在第一输出图像中的第二位置信息。
该步骤从第一输出图像中确定拍摄主体的位置，以消除相机抖动干扰，使拍摄主体的位置变化只受到拍摄主体自身的影响。例如，消除相机抖动干扰后，当拍摄主体不发生位置变化(如宠物静止不动时)时，所得到的相邻两张第一输出图像之间的第二位置信息保持不变或者接近不变，当拍摄主体发生变化时，所得到的相邻两张第一输出图像之间的第二位置信息只因宠物运动的位置变化而变化，不受相机抖动因素变化。
值得一提的是，本申请实施例通过S308将S307中经主体检测处理得到的第一位置信息映射到S306中经防抖处理得到的第一输出图像中，从而对防抖结果和主体追踪结果进行融合，得到第二位置信息。这样先并行处理，再融合结果的方式，对于手机性能消耗而言，可以减少性能功耗压力，提高手机处理器的处理速度。
得到第一输出图像之后，手机对第一输出图像中的拍摄主体进行平滑处理，手机执行S309。
S309、手机根据第二位置信息确定拍摄主体在当前帧第一输出图像中的位置信息相对于上一帧第一输出图像中的位置信息是否一致，在确定拍摄主体的位置不一致的 情况下，手机根据历史位置信息对当前帧第一输出图像中的拍摄主体的位置信息进行平滑处理，得到拍摄主体在第一输出图像中的第三位置信息。
其中，历史位置信息包括第一输出图像在当前帧之前的任一或多帧图像中拍摄主体的位置信息。位置信息包括坐标框的坐标位置，其中，坐标位置包括坐标点，或者坐标位置还可以包括坐标框大小。
该步骤中，手机通过对拍摄主体进行平滑处理，使得到的第一输出图像中的拍摄主体的主体背景变化的平滑度，大于处理前第一输出图像/第一图像中目标拍摄主体背景变化的平滑度。
示例性的，平滑处理如图6所示，手机通过获取目标框的坐标与EIS输出的warp信息(又称偏移量)，利用EIS的warp信息将目标框换算到防抖处理后的图像坐标系，得到第二位置信息。手机获取上一帧图像的历史位置信息，通过第二位置信息与历史位置信息是否一致确定目前目标框是否近似静止。若一致，说明拍摄主体的位置没有变化，将当前目标框的位置与上一帧图像的历史位置设为相同位置，并输出当前目标框在EIS输出图坐标系中的平滑位置(第二位置信息)。若不一致，说明相对于前一帧，当前帧第一输出图像中拍摄主体的位置有变化，那么对有变化的主体位置进行平滑处理，输出当前目标框在EIS输出图坐标系中的平滑位置(第三位置信息)。在得到平滑位置之后，判断是否需要输出目标框的平滑位置在EIS输出图中的坐标位置，若是，则基于平滑位置进行裁剪，使得裁剪出的图像兼顾防抖和平滑。若否，则利用EIS warp信息将平滑位置换算到防抖处理前的第一图像的坐标系中，使得裁剪出的图像实现第一输出图像从首帧到第N帧的整体的主体位置变化较为平滑的效果。
其中，平滑处理包括：剔除异常值、滤波处理等。在一些实施例中，滤波可以包括高速滤波或者卡尔曼滤波。本实施例中以平滑处理是卡尔曼滤波处理为例，卡尔曼滤波是以历史时刻的“预测值”(即历史位置信息)与当前时刻的“真实值”(即第二位置信息)，计算得到当前时刻的“预测值”(即第三位置信息)。其中，第三位置信息同时参考了当前时刻的第二位置信息以及历史时刻的“历史位置信息”预测得到，因此若第二位置信息相比于历史位置信息发生显著变化时(如画面中物体快速经过，使得第二位置信息与历史位置信息之间的位移量过大)，卡尔曼滤波便会以历史位置信息对第二位置信息进行平滑处理，减少第二位置信息的变化幅度，得到平滑后的第三位置信息。由于第三位置信息与历史位置信息之间的位移量小于第二位置信息与历史位置信息之间的位移量，因此平滑处理可以避免在拍摄主体运动过程中发生的如张臂、挥手、蹦跳等导致中心点突变的形态变化对第一图像质量的影响，从而，该处理提高了视频的拍摄质量。
值得一提的是，上述S306-S309中，手机先进行主体检测处理和防抖处理，然后将二者结果相融合，得到拍摄主体在防抖处理后的第一输出图像中的第二位置信息，然后手机再进行平滑处理。而一些实施例中，手机可以在主体检测处理之后先对第一位置信息进行平滑化处理，再进行防抖处理，关于该实施例的介绍，可以参见下面对第二种拍摄设备及对应的拍摄方法的说明。此外，一些实施例中，手机可以先进行防抖处理，再在防抖得到的第一输出图像中进行主体检测处理，最后进行平滑处理，关于该实施例的介绍，可以参见下面对第三种拍摄设备及对应的拍摄方法的说明。也就 是说，手机执行主体检测处理、防抖处理和平滑处理的处理顺序可以根据实际情况来选择，本申请实施例对此并无限制。
S310、手机获取第一图像中的拍摄主体，从拍摄主体中确定目标拍摄主体。
其中，手机获取第一图像中的拍摄主体的步骤，可以参见上述实施例和常规技术中的追踪算法如行人重识别算法(re-identification，ReID)，本申请实施例这里不予赘述。
在一些实施例中，S310进一步包括：
S3、手机在第二界面的第一图像上显示拍摄主体的主体框。
其中，主体框用于对用户可切换拍摄主体进行标识，该标识的方式包括：以一对中括号(又称方括号)围出拍摄主体的头部区域，或者为可追踪的拍摄主体的外轮廓加描边特效，或者以矩形框指示可以追踪的拍摄主体。
S4、手机检测到用户执行的第三操作，其中，第三操作用于在第一图像中包含多个主体框的情况下选择至少一主体框，手机根据第三操作从拍摄主体中确定目标拍摄主体。
在上述S3-S4中，手机显示每个拍摄主体的主体框，并根据用户选择的至少一主体框确定用户所指定的目标拍摄主体。因此在录制视频的过程中，如果目标拍摄主体发生变化(例如手机检测到用户切换目标拍摄主体的操作)，则第二界面上显示的主体图像也相应发生变化，以使得变化后的目标拍摄主体位于能保持在目标位置从而被重点突出，且目标拍摄主体所在的主体图像兼顾平滑和防抖，呈现出的拍摄效果好。上述用户切换目标拍摄主体的操作可以是录制过程中用户先后选择了不同拍摄主体的主体框。示例性的，视频录制过程中用户先选择爸爸头部的主体框，第二界面显示爸爸保持在目标位置从而被突出显示的主体图像；当用户选择弟弟头部的主体框，第二界面显示弟弟保持在目标位置从而被突出显示的主体图像，这样便可实现对目标拍摄主体的切换。
在另一些实施例中，S310进一步包括：手机获取第一图像中的拍摄主体。手机将第一时刻处于目标位置的拍摄主体作为目标拍摄主体。本实施例中，手机自动从拍摄主体中确定目标拍摄主体。第一时刻为开始录制视频时，或者第一时刻为开始录制视频后的t时刻，(0<t<＝300，毫秒ms)t可以为10ms，20ms，50ms，100ms，200ms等。
手机在确定目标拍摄主体之后，执行S311。
S311、手机根据第三位置信息在第一输出图像中裁剪出目标拍摄主体对应的主体图像，并在第二界面中显示主体图像。
本步骤中，手机将经过防抖处理和平滑处理后的目标拍摄主体所在区域裁剪出来，得到主体图像。裁剪出的主体图像中目标拍摄主体保持在目标位置从而被突出显示。其中，目标位置可以包括画面中间位置或者黄金分割点位置或者其他使得拍摄主体被重点突出的位置。这样，当用户想要在拍摄画面中突显目标拍摄主体时，手机可以显示并追踪该目标拍摄主体，使目标拍摄主体在画面中处于重点突出位置。如此，本步骤既通过抖动位置补偿处理，提高了画面的稳定性；又通过平滑处理，对移动速度剧烈的拍摄主体的进行位置平滑，达到画面流畅的效果。
在一些实施例中，手机显示的第二界面参见图7，如图7中的(a)所示，第二界面中主体图像可以显示于第一图像的区域内，如图7中的(b)所示，第二界面可以显示于第一图像的区域外，如图7中的(c)所示，主体图像为1到N张，N为相机应用预设追踪数量或者用户设置的追踪数量。
该实施例中，第一图像包括至少一拍摄主体，拍摄主体中需要被突出显示的为目标拍摄主体。其中，第一图像中被突出显示的目标拍摄主体可以为一个或者多个，每个主体图像可以包括一个或多个目标拍摄主体，在主体图像包括多个目标拍摄主体的情况下，目标拍摄主体的位置情况包括多种，例如：其中一个目标拍摄主体的位置处于目标位置，或者，两个目标拍摄主体分布在目标位置两侧(或两方)，或者两个目标拍摄主体都处于目标位置。
在一些实施例中，手机在第二界面中显示的主体图像的数量与用户在第一界面的选中主体框数量相对应，主体框和主体图像的数量可以对应为一个或者多个。在一些实施例中，主体图像的数量可以大于主体框的数量，或者等于主体框的数量，或者小于主体框的数量。且每个主体图像可显示一个主体或多个主体。
在第一种应用场景下，主体图像的数量为一个，且主体图像显示的用户选择的一个主体框所对应的目标拍摄主体。
示例性的，如图8所示，第一图像中包括多个人物，从左到右分别是姐姐、爸爸、弟弟、哥哥、妹妹。其中，第二界面中爸爸和弟弟的头部区域显示有一对中括号，当手指(或触控笔，下同)触摸到中括号所在区域，即可在第二界面中显示该区域内人物为拍摄主体的主体图像，如手指触摸爸爸头部的中括号区域，第二界面显示爸爸位于图像中间的主体图像；当手指触摸弟弟头部的中括号区域，第二界面显示弟弟位于图像中间的主体图像。
在第二种应用场景下，主体图像的数量为一个，且主体图像显示用户选择的至少两个主体框所对应的拍摄主体。在该场景下，当用户选择爸爸头部的中括号区域以及弟弟头部的中括号区域，第二界面显示包括爸爸和弟弟的主体图像，其中主体图像中爸爸或弟弟的位置保持在目标位置从而被突出显示。
其中，用户选择多个主体框的方法包括：采用多个手指触摸不同的主体框；或者在显示屏上长按其中一主体框，并滑动至另一主体框。
示例性的，用户的两根手指接触爸爸头部的中括号区域以及弟弟头部的中括号区域时，第二界面显示包括爸爸和弟弟的主体图像。其中，主体图像中爸爸或弟弟的位置位于图像中间位置，如图9的(a)所示，爸爸位于目标位置(图像中间位置)；或者主体图像中爸爸和弟弟之间的区域位于图像中间位置，如图9的(b)所示；又或者，爸爸位于主体图像的图像中间位置，弟弟位于主体图像的黄金分割点位置，如图9的(c)所示。
在第三种应用场景下，主体图像的数量有多个，且每个主体图像显示用户选择的一个主体框所对应的拍摄主体。
示例性的，如图10所示，当用户选择爸爸头部的中括号区域以及弟弟头部的中括号区域，一个主体图像中显示爸爸的主体图像，另一主体图像中显示弟弟的主体图像，且爸爸和弟弟分别在其对应的主体图像的目标位置(图10中目标位置是爸爸和弟弟分 别在图像中间的位置)。
在第四种应用场景下，主体图像的数量有多个，且至少一个主体图像内包括多个主体框。
该场景下的在一些实施例中，每个主体图像显示同一拍摄主体。其中，两个主体图像内显示的图像区别可以是：拍摄主体的位置不同。如图11所示，主体图像的数量有两个，且两个主体图像都显示爸爸和弟弟的图像，其中，第一个主体图像中爸爸位于图像中间位置，弟弟位于图像的黄金分割点处。第二个主体图像中爸爸和弟弟之间的区域位于图像中间。不同主体图像内显示的拍摄主体位置不同，可以为用户提供更多的拍摄构图选择，从而提高了用户的拍摄体验。
该场景下的另一些实施例中，每个主体图像显示不同拍摄主体。如图12所示，主体图像的数量有两个，第一个主体图像显示妹妹的图像，第二个主体图像显示爸爸和弟弟的图像。也就是说，该实施例中可以同时突出妹妹、爸爸和弟弟三个拍摄主体，使这三个拍摄主体都在兼顾视频防抖和平滑的情况下稳定、流畅且突出的显示于屏幕中。
在一些实施例中，上述S309、S311可以由S5、S6替代。
S5、手机根据第二位置信息确定第一图像中的拍摄主体在当前帧的位置相对于上一帧的位置是否静止，若确定拍摄主体的位置不静止，手机根据历史位置信息对当前帧的位置进行平滑处理，得到拍摄主体在第一图像中的第四位置信息。
S6、手机根据第四位置信息在第一图像中裁剪出目标拍摄主体对应的主体图像，并在第二界面中显示主体图像，其中，拍摄主体位于主体图像中的目标位置。
区别于上述S309和S311中获取的第三位置信息，本实施例获取的第四位置信息是拍摄主体在未经防抖处理的第一图像中的位置。即，手机将经过平滑处理后的拍摄主体在防抖处理前的第一图像中裁剪出来，以得到主体图像。在该实施例中，对拍摄主体进行位置追踪以及位置平滑处理，但并未消除手机抖动带来的图像抖动干扰，这样呈现的主体图像基本实现拍摄主体位于视频中间的目的，且相比于S309、S311，主体图像更为真实，能够满足用户多样化的拍摄需求。
在一些实施例中，与录制视频过程中第二界面显示处理后的图像类似，在拍摄预览过程中，拍摄预览界面即第一界面，也可以显示手机处理后的图像。例如，手机在进入第一界面后，开始执行S305-306，得到第一图像和第一输出图像。手机可以将第一输出图像显示在第一界面。或者，手机可以将第一图像显示在第一界面。其中，第一图像(又称实况图像)，显示实况图像能够使画面更真实和自然，且一定程度上可以反映实际拍摄环境，例如，当用户在风雪天气的室外拍摄路过的行人时，保留画面的抖动效果可以让视频观看者身临其境。而手机在第一界面显示经过防抖处理的第一输出图像，使得呈现出的画面始终清晰和稳定。如此，通过在第一界面有选择的显示第一图像或者第一输出图像的方式能够满足用户多样化的拍摄需要，提高用户的拍摄体验。
在另一些实施例中，在拍摄预览状态下，与录制视频过程类似，也可以确定和切换目标拍摄主体，第一界面上也可以显示目标拍摄主体对应的主体图像，主体图像中目标拍摄主体位于视觉最优位置，从而突出目标拍摄主体以及拍摄内容，提高画面的 视觉表达效果。
在另一些实施例中，在拍摄预览过程中，第一界面可以显示摄像头在用户执行第二操作前采集到的原始图像。
在另一些实施例中，在拍摄预览过程中，第一界面除了显示上述原始图像或者第一图像或者第一输出图像外，还可以显示追踪框，其中，追踪框内可以显示原始图像或者第一图像或者不显示任何图像。
在一些实施例中，手机可以在第一界面显示取景框，在第二界面显示取景框和上述追踪框。若手机处理得到第一图像、第一输出图像、主体图像，则取景框内显示原始图像、第一图像、第一输出图像的任一种，追踪框内显示主体图像。若手机未处理得到上述第一图像、第一输出图像或主体图像，取景框和追踪框中可以显示原始图像或者不显示任何图像。
S312、手机检测到用户在第二界面执行的第三操作，该第三操作用于指示结束拍摄视频。
其中，第三操作可参见上述S301中的第一操作的介绍，在此不做累赘说明。用户以上述方式作出第二操作的指示后，手机检测到第二操作，并执行S313。
S313、手机响应于第三操作，将录制的N帧主体图像制作为视频文件。
其中，N帧为手机从响应于第二操作到响应于第三操作之间所获取的主体图像的数量，手机可以将视频文件保存在图库中。
在一些实施例中，手机响应于第三操作，将录制的第二界面显示的N帧主体图像和N帧第一图像制作为视频文件。
在一些实施例中，手机响应于第三操作，将录制的第二界面显示的N帧主体图像和N帧第一输出图像制作为视频文件。
这样，由于视频文件中的视频帧图像所包括的主体图像是目标拍摄主体兼顾防抖和平滑且位于目标位置的图像，因而视频文件中的视频帧图像能保持目标拍摄主体始终位于视觉最优位置，以突显该目标拍摄主体以及拍摄内容；而且能够减少相机抖动造成的干扰，相邻主体图像和视频帧图像间的画面过渡更为稳定，还能够平滑拍摄主体的中心点使主体图像上拍摄主体的偏离程度小，主体图像和视频帧图像不易产生虚影且背景更加清晰。
在一些实施例中，手机响应于第三操作，将录制的第二界面显示的N帧第一图像或者N帧第一输出图像制作为视频文件。
在该实施例中，手机录制过程中显示主体图像，但手机结束录制后得到的视频文件中不包括主体图像。这样主体图像的作用是在录制过程中突显。
下面结合图13介绍本申请实施例所实现的效果，在图13中，横轴表示时间，纵轴表示拍摄主体的位置位移量。在拍摄过程中，拍摄主体的位置随时间发生运动，拍摄主体运动轨迹如图13的曲线a所示。并且拍摄时相机会发生抖动，相机抖动轨迹如图13中的曲线b所示。因此未经过处理所录制的视频，拍摄主体的位置位移量既受到主体运动的影响，又受到相机运动的影响，由图13中的曲线c表示。对录制的视频经过主体追踪、主体位置防抖处理和平滑处理后，使拍摄主体运动图像始终保持在视觉最优位置，如图13中的曲线d所示。另外，手机在主体检测处理之后先对第一位置信 息进行平滑化处理，再进行防抖处理得到的主体图像会在目标位置周围发生波动，无法维持在目标位置，其中拍摄主体的位置偏移量如图13的曲线e表示。
综上，本申请实施例提供了一种拍摄方法，在使用手机拍摄视频的情况下，手机从拍摄主体中确定目标拍摄主体，手机通过追踪目标拍摄主体，并对目标拍摄主体进行防抖和平滑处理的方式使将该目标拍摄主体始终显示在画面的视觉最优位置，这样不仅能够在拍摄视频的过程中将目标拍摄主体保持在目标位置从而被突出显示，减少相机抖动造成的干扰，达到画面稳定效果，还能够平滑拍摄主体的中心点使得，在目标拍摄主体发生运动的过程中，主体图像中目标拍摄主体背景变化的平滑度，大于预览图像中目标拍摄主体背景变化的平滑度，主体图像中不易产生虚影，背景更加清晰。那么由包括N帧主体图像的视频帧图像制作得到的视频文件不仅可以突显该目标拍摄主体以及拍摄内容；而且能够减少相机抖动造成的干扰，相邻主体图像和视频帧图像间的画面过渡更为稳定；还能够平滑拍摄主体的中心点使主体图像上拍摄主体的偏离程度小，主体图像和视频帧图像不易产生虚影且背景更加清晰。因此拍摄得到的视频文件的呈现效果更好。此外还能够在用户不根据拍摄主体位置实时调整界面显示的视频画面的情况下将拍摄主体保持在目标位置从而降低了拍摄难度，提高了用户的拍摄体验。而且对用户而言，拍摄的功能性增加，使用时更方便，即使是未学过摄影、美术等知识的普通用户，拍摄出的视频中所想要突显的拍摄主体也能在画面中间或者画面的黄金分割点位置，这样拍摄并显示在第二界面中的视频更能让人的注意力集中在拍摄主体上。
另外，本申请实施例还提供第一种拍摄设备，如图14所示，该拍摄设备包括惯性测量单元(Inertial Measurement Unit，IMU)、主体检测与追踪单元、平滑单元、第一图像处理单元、防抖变形补偿单元及图像裁剪单元等。该拍摄设备执行以下步骤，以实现上述实施例提供的拍摄方法：
S1401、IMU检测相机抖动幅度，从而得到warp map，并将warp map输入防抖变形补偿单元和平滑单元中，其中warp map中存储了网格位置在相邻帧第一图像中的坐标偏移量(坐标偏移量相当于上述S2的相对像素位移)。
S1402、手机获取大小为12百万像素mega(12M)的raw图(12M)，经过第一图像处理单元将raw图处理为尺寸不变的YUV(12M)图和较小tiny尺寸的YUV(YUV(tiny))图，其中，YUV(12M)图相当于上述S305中的第一图像，YUV(tiny)图相当于上述S305中的第二图像。
S1403、手机将YUV(12M)图输入防抖变形补偿(EIS warp)单元中，该单元将主体框坐标输入warp map中进行插值，得到经抖动位移补偿的主体框坐标，该主体框坐标为拍摄主体映射到防抖后的图像中的主体坐标。(主体坐标相当于上述S308中经抖动位移补偿后的第二位置信息)，并输出分辨率大小为4096*2160的像素分辨率，4K的YUV(4K)图至图像裁剪单元。
S1404、然后手机将YUV(tiny)图输入主体检测与追踪单元识别得到主体框坐标与尺寸，并将主体框坐标与尺寸输入平滑单元中，得到拍摄主体的坐标经平滑后的裁剪框坐标和尺寸(裁剪框坐标和尺寸也就是上述S309中的第三位置信息)，并将裁剪框坐标和尺寸输入图像裁剪单元中。
S1405、最后，由图像裁剪单元基于裁剪框坐标和尺寸在YUV(4K)图上裁剪出YUV(高清，FHD)图，(YUV(全高清，Full High Definition，FHD)图也就是上述S311中的主体图像)，并将YUV(FHD)图进行编码(Video encoder)和展示(display)。
在上述S1401-S1405描述的方案中，拍摄设备对第一图像进行主体检测与追踪，相机防抖补充以及主体位置平滑处理，从而使拍摄主体运动图像始终保持在视觉最优位置。其中主体检测与追踪单元与防抖变形补偿单元并行工作，这样手机的性能功耗压力小，且处理速度快。
本申请实施例还提供第二种拍摄设备，如图15所示，该拍摄设备包括惯性测量单元(Inertial Measurement Unit，IMU)、主体检测与追踪单元、平滑单元、第二图像处理单元、防抖裁剪warp map单元及防抖变形补偿单元。
与上述实施例中的拍摄系统不同的是，该设计方式下，手机执行上述S1402之后，执行S1413-S1415，以实现上述实施例提供的拍摄方法：
S1413、手机将YUV(tiny)图输入主体检测与追踪单元识别得到主体框坐标与尺寸，并将主体框坐标与尺寸输入平滑单元中，得到拍摄主体经坐标平滑后的裁剪框坐标和尺寸(裁剪框坐标和尺寸也就是上述S309中的第三位置信息)，并将裁剪框坐标和尺寸输入防抖裁剪warp map单元中。
S1414、在防抖裁剪warp map单元中将裁框坐标和尺寸输入warp map进行插值，得到兼顾防抖和平滑的拍摄主体在YUV(tiny)图中的坐标位置，使该坐标位置兼有裁剪和偏移功能，得到新的warp map，并输入EIS warp中。
S1415、EIS warp基于新的warp map对YUV(12M)图进行裁剪(裁剪处理相当于上述S2)，得到YUV(FHD)图，(YUV(FHD)图也就是上述S311中的主体图像)，并将YUV(FHD)图进行编码和展示。
在上述S1401-S1402及S1413-1415描述的方案中，防抖裁剪warp map单元图像直接输出兼顾平滑和防抖后的坐标偏移量，使得EIS warp只做一次裁剪便可得到YUV(FHD)图(相当于上述S311中的主体图像)。相比于上一实施例，省略了在防抖处理时的裁剪步骤，因此处理速度提高。
本申请实施例还提供第三种拍摄设备，如图16所示，该拍摄设备包括惯性测量单元(Inertial Measurement Unit，IMU)、主体检测与追踪单元、平滑单元、第一图像处理单元、防抖变形补偿单元及图像裁剪单元。
与上述实施例不同的是，该设计方式下，手机执行上述S1402之后，执行S1423-S1425，以实现上述实施例提供的拍摄方法：
S1423、手机将YUV(12M)图输入防抖变形补偿(EIS warp)单元中，得到对YUV(12M)图经过防抖变形补偿后的第一输出图像，并对第一输出图像进行下采样，得到YUV小尺寸图(YUV tiny)图，然后将YUV tiny图输入主体检测和追踪单元中，获取主体框坐标与尺寸并输入平滑单元中。
S1424、然后由平滑单元将主体框坐标与尺寸进行平滑处理，得到裁剪框和尺寸。
S1425、图像裁剪单元基于裁剪框和尺寸对YUV(4K)图作裁剪，输出YUV(FHD)图。
在上述S1401-S1402及S1423-S1425描述的方案中，拍摄设备对YUV(12M)图 依次进行防抖变形补偿处理和主体识别处理，使得两个处理过程顺序执行。这样拍摄设备可以直接在经过防抖变形补偿后的第一输出图像中进行主体检测和追踪。相比于上述两个实施例，减少了将主体坐标映射到第一输出图像中的插值步骤，因此处理速度提高。
本申请实施例提供了一种电子设备，该电子设备可以包括：存储器和一个或多个处理器。该存储器用于存储计算机程序代码，该计算机程序代码包括计算机指令。当处理器执行计算机指令时，电子设备可执行上述方法实施例中手机执行的各个功能或者步骤。该电子设备的结构可以参考图1所示的电子设备100的结构。
本申请实施例还提供一种计算机可读存储介质，该计算机可读存储介质包括计算机指令，当所述计算机指令在电子设备上运行时，使得该通电子设备执行上述方法实施例中手机执行的各个功能或者步骤。
本申请实施例还提供一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机执行上述方法实施例中手机执行的各个功能或者步骤。
通过以上实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上内容，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任 何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (18)

  1. 一种拍摄方法，应用于电子设备，所述电子设备包括摄像头，所述摄像头用于采集原始图像，其特征在于，所述方法包括：
    启动相机应用；
    检测到视频录制操作后，在录制界面上显示预览图像和突出显示目标拍摄主体的主体图像；所述预览图像和所述主体图像根据所述原始图像处理获得，所述目标拍摄主体处于主体图像上的目标位置，所述主体图像的取景范围在所述预览图像的取景范围之内。
  2. 根据权利要求1所述的方法，其特征在于，在所述目标拍摄主体发生运动的过程中，所述目标拍摄主体保持在所述主体图像上的所述目标位置。
  3. 根据权利要求1或2所述的方法，其特征在于，在所述目标拍摄主体发生运动的过程中，所述主体图像中所述目标拍摄主体背景变化的平滑度，大于所述预览图像中所述目标拍摄主体背景变化的平滑度。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述目标拍摄主体包括至少一个，所述主体图像包括至少一个；其中，一个所述目标拍摄主体对应一个所述主体图像，或者一个所述目标拍摄主体对应多个所述主体图像，所述主体图像位于所述预览图像的区域外或者与所述预览图像的区域部分重叠。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述目标拍摄主体和/或所述目标位置为响应于用户的指示操作确定的或者所述电子设备自动确定的，所述方法还包括：
    检测到切换目标拍摄主体的操作，在所述录制界面上显示切换后的目标拍摄主体对应的主体图像；和/或
    检测到切换目标位置的操作，在所述录制界面上显示所述目标拍摄主体位于切换后的目标位置对应的主体图像。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，在所述录制界面上显示第i帧预览图像和所述目标拍摄主体对应的主体图像，i为大于1的正整数，包括：
    根据第i帧原始图像，获取所述第i帧预览图像；
    根据所述目标拍摄主体在第i-1帧预览图像中的第一位置信息，对所述目标拍摄主体在所述第i帧预览图像中的第一位置信息进行平滑处理，得到所述目标拍摄主体的第二位置信息；
    根据所述第二位置信息裁剪所述第i帧预览图像，得到所述目标拍摄主体对应的主体图像；
    在所述录制界面上显示所述第i帧预览图像和所述目标拍摄主体对应的主体图像。
  7. 根据权利要求6所述的方法，其特征在于，所述根据所述目标拍摄主体在第i-1帧预览图像中的第一位置信息，对所述目标拍摄主体在所述第i帧预览图像中的第一位置信息进行平滑处理，得到所述目标拍摄主体的第二位置信息，包括：
    获取所述目标拍摄主体分别在所述第i-1帧预览图像中的第一位置信息和在所述第i帧预览图像中的第一位置信息之间的第一位置差值；
    基于所述第一位置差值，确定根据所述目标拍摄主体在所述第i-1帧预览图像中 的所述第一位置信息，对所述第i帧预览图像中的第一位置信息进行平滑化处理，得到所述目标拍摄主体的第二位置信息，其中，所述第二位置信息与所述第i-1帧预览图像中的第一位置信息之间的第二位置差值小于或等于所述第一位置差值。
  8. 根据权利要求6或7所述的方法，其特征在于，所述根据第i帧原始图像，获取所述第i帧预览图像，包括：
    对所述第i帧原始图像进行格式处理，得到第i帧第一图像；
    所述第i帧第一图像即为所述第i帧预览图像；或者，对所述第i帧第一图像进行抖动位移补偿处理，得到第i帧第一输出图像，所述第i帧第一输出图像即为所述第i帧预览图像。
  9. 根据权利要求8所述的方法，其特征在于，所述对所述第i帧第一图像进行抖动位移补偿处理，得到第i帧第一输出图像，包括：
    获取所述第i帧第一图像相对于所述第i帧第一图像之前的任一或多帧第一图像的画面抖动位移信息；
    根据所述画面抖动位移信息，对所述第i帧第一图像中的画面进行抖动位移补偿处理，得到第i帧第一输出图像。
  10. 根据权利要求9所述的方法，其特征在于，获取所述第i帧第一图像相对于所述第i帧第一图像之前的任一或多帧第一图像的画面抖动位移信息，包括：
    检测所述电子设备的抖动幅度；
    根据所述抖动幅度，确定画面抖动位移信息，其中，所述画面抖动位移信息包括所述第i帧第一图像中的每个像素点相对于所述第i帧前的任一或多帧第一图像中的对应像素点的偏移量。
  11. 根据权利要求8-10中任一项所述的方法，其特征在于，所述方法还包括：
    对所述第i帧第一图像进行下采样处理，得到第i帧第二图像；
    在所述根据所述目标拍摄主体在第i-1帧预览图像中的第一位置信息，对所述目标拍摄主体在所述第i帧预览图像中的第一位置信息进行平滑处理之前，所述方法还包括：
    从所述第i帧第二图像中检测得到目标拍摄主体，并获取所述目标拍摄主体的第三位置信息；
    将所述目标拍摄主体的所述第三位置信息映射到所述第i帧预览图像中，得到所述目标拍摄主体位于所述第i帧预览图像中的所述第一位置信息。
  12. 根据权利要求6-11中任一项所述的方法，其特征在于，根据所述第二位置信息裁剪所述第i帧预览图像，得到所述目标拍摄主体对应的主体图像，包括：
    根据所述第二位置信息确定所述目标拍摄主体的最小外接矩形框；
    根据所述最小外接矩形框确定裁剪区域，其中，所述裁剪区域包含所述最小外接矩形框；
    在所述第i帧预览图像获取所述裁剪区域对应的图像，得到所述目标拍摄主体对应的主体图像。
  13. 根据权利要求12所述的方法，其特征在于，在所述第i帧预览图像获取裁剪区域对应的图像之后，所述方法还包括：
    将所述裁剪区域对应的图像缩小或放大，得到具有预设显示规格的所述目标拍摄主体对应的主体图像，其中，所述预设显示规格为预设的所述电子设备的屏幕占比。
  14. 根据权利要求1-13中任一项所述的方法，其特征在于，所述目标位置包括中间位置或黄金分割点位置。
  15. 根据权利要求1-14中任一项所述的方法，其特征在于，所述方法还包括：
    检测到结束录制操作；
    生成所述视频文件，其中所述视频文件包括所述主体图像和/或所述预览图像。
  16. 根据权利要求1-15中任一项所述的方法，其特征在于，所述方法还包括：
    在启动相机应用后显示拍摄预览界面，其中，所述拍摄预览界面包括所述预览图像，或者所述拍摄预览界面包括所述预览图像和所述主体图像。
  17. 一种电子设备，其特征在于，包括：
    摄像头，用于采集图像；
    屏幕，用于显示界面；
    一个或多个处理器；
    存储器；
    以及一个或多个计算机程序，其中所述一个或多个计算机程序被存储在所述存储器中，所述一个或多个计算机程序包括指令；当所述指令被所述电子设备执行时，使得所述电子设备执行如权利要求1-16中任一项所述的方法。
  18. 一种计算机可读存储介质，其特征在于，包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行如权利要求1-16中任一项所述的方法。
PCT/CN2023/089350 2022-05-27 2023-04-19 一种拍摄方法及电子设备 WO2023226634A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23810708.0A EP4383742A4 (en) 2022-05-27 2023-04-19 PHOTOGRAPHY METHOD AND ELECTRONIC DEVICE
US18/691,192 US20240422424A1 (en) 2022-05-27 2023-04-19 Shooting method and electronic device

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210594819.X 2022-05-27
CN202210594819 2022-05-27
CN202210871133.0A CN116074620B (zh) 2022-05-27 2022-07-22 一种拍摄方法及电子设备
CN202210871133.0 2022-07-22

Publications (1)

Publication Number Publication Date
WO2023226634A1 true WO2023226634A1 (zh) 2023-11-30

Family

ID=86172127

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/089350 WO2023226634A1 (zh) 2022-05-27 2023-04-19 一种拍摄方法及电子设备

Country Status (4)

Country Link
US (1) US20240422424A1 (zh)
EP (1) EP4383742A4 (zh)
CN (1) CN116074620B (zh)
WO (1) WO2023226634A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117676315A (zh) * 2024-02-02 2024-03-08 深圳市积加创新技术有限公司 一种基于高像素相机形成预览双画面的方法

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102017603A (zh) * 2008-04-28 2011-04-13 松下电器产业株式会社 摄像装置
US20160212358A1 (en) * 2011-11-14 2016-07-21 Sony Corporation Information processing apparatus, method, and non-transitory computer-readable medium
CN108900790A (zh) * 2018-06-26 2018-11-27 努比亚技术有限公司 视频图像处理方法、移动终端及计算机可读存储介质
CN110072070A (zh) * 2019-03-18 2019-07-30 华为技术有限公司 一种多路录像方法及设备
CN111010590A (zh) * 2018-10-08 2020-04-14 传线网络科技（上海）有限公司 一种视频裁剪方法及装置
CN112261218A (zh) * 2020-10-21 2021-01-22 维沃移动通信有限公司 视频控制方法、视频控制装置、电子设备和可读存储介质
CN113766129A (zh) * 2021-09-13 2021-12-07 维沃移动通信（杭州）有限公司 录像方法、装置、电子设备及介质
CN114598819A (zh) * 2022-03-16 2022-06-07 维沃移动通信有限公司 视频录制方法、装置和电子设备

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9781350B2 (en) * 2015-09-28 2017-10-03 Qualcomm Incorporated Systems and methods for performing automatic zoom
KR102726608B1 (ko) * 2016-03-24 2024-11-07 엘지전자 주식회사 디스플레이 장치 및 그의 동작 방법
WO2020107266A1 (zh) * 2018-11-28 2020-06-04 深圳市大疆创新科技有限公司 手持云台及其拍摄控制方法
CN110896451B (zh) * 2019-11-20 2022-01-28 维沃移动通信有限公司 预览画面显示方法、电子设备及计算机可读存储介质
CN111083557B (zh) * 2019-12-20 2022-03-08 浙江大华技术股份有限公司 一种视频录播控制方法及装置
CN113676655B (zh) * 2020-05-15 2022-12-27 华为技术有限公司 拍摄方法、装置、移动终端及芯片系统
CN114430457B (zh) * 2020-10-29 2024-03-08 北京小米移动软件有限公司 拍摄方法、装置、电子设备和存储介质
CN112954220A (zh) * 2021-03-03 2021-06-11 北京蜂巢世纪科技有限公司 图像预览方法及装置、电子设备、存储介质

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102017603A (zh) * 2008-04-28 2011-04-13 松下电器产业株式会社 摄像装置
US20160212358A1 (en) * 2011-11-14 2016-07-21 Sony Corporation Information processing apparatus, method, and non-transitory computer-readable medium
CN108900790A (zh) * 2018-06-26 2018-11-27 努比亚技术有限公司 视频图像处理方法、移动终端及计算机可读存储介质
CN111010590A (zh) * 2018-10-08 2020-04-14 传线网络科技（上海）有限公司 一种视频裁剪方法及装置
CN110072070A (zh) * 2019-03-18 2019-07-30 华为技术有限公司 一种多路录像方法及设备
CN112261218A (zh) * 2020-10-21 2021-01-22 维沃移动通信有限公司 视频控制方法、视频控制装置、电子设备和可读存储介质
CN113766129A (zh) * 2021-09-13 2021-12-07 维沃移动通信（杭州）有限公司 录像方法、装置、电子设备及介质
CN114598819A (zh) * 2022-03-16 2022-06-07 维沃移动通信有限公司 视频录制方法、装置和电子设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117676315A (zh) * 2024-02-02 2024-03-08 深圳市积加创新技术有限公司 一种基于高像素相机形成预览双画面的方法
CN117676315B (zh) * 2024-02-02 2024-06-11 深圳市积加创新技术有限公司 一种基于高像素相机形成预览双画面的方法

Also Published As

Publication number Publication date
CN116074620B (zh) 2023-11-07
US20240422424A1 (en) 2024-12-19
CN116074620A (zh) 2023-05-05
EP4383742A1 (en) 2024-06-12
EP4383742A4 (en) 2025-01-01

Similar Documents

Publication Publication Date Title
CN114157804B (zh) 一种长焦拍摄的方法及电子设备
WO2022068537A1 (zh) 一种图像处理方法及相关装置
US11831977B2 (en) Photographing and processing method and electronic device
CN115191110B (zh) 一种视频拍摄方法与电子设备
WO2021244455A1 (zh) 一种图像内容的去除方法及相关装置
WO2021213477A1 (zh) 多路录像的取景方法、图形用户界面及电子设备
US12200349B2 (en) Video shooting method and electronic device
WO2022228259A1 (zh) 一种目标追踪方法及相关装置
US20230353864A1 (en) Photographing method and apparatus for intelligent framing recommendation
CN117119285B (zh) 一种拍摄方法
US20240007736A1 (en) Photographing method and electronic device
WO2023226634A1 (zh) 一种拍摄方法及电子设备
US20240414432A1 (en) Shooting method and related device
CN117119276B (zh) 一种水下拍摄方法及电子设备
CN116055868B (zh) 一种拍摄方法及相关设备
CN115484394B (zh) 一种隔空手势的引导使用方法及电子设备
US20240064397A1 (en) Video Shooting Method and Electronic Device
CN115484393B (zh) 一种异常提示方法及电子设备
CN115484392B (zh) 一种拍摄视频的方法及电子设备
CN116761082B (zh) 图像处理方法及装置
WO2025025591A1 (zh) 一种拍摄方法及电子设备
CN119653225A (zh) 一种拍照方法及折叠屏设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23810708

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023810708

Country of ref document: EP

Effective date: 20240305

NENP Non-entry into the national phase

Ref country code: DE