WO2021042364A1

WO2021042364A1 - Method and device for taking picture

Info

Publication number: WO2021042364A1
Application number: PCT/CN2019/104674
Authority: WO
Inventors: 赵杨; 那柏林; 孙新江; 刘理; 仇芳; 黄枭
Original assignee: 华为技术有限公司
Priority date: 2019-09-06
Filing date: 2019-09-06
Publication date: 2021-03-11
Also published as: CN112771612A; CN112771612B

Abstract

Provided in the present application are a method and device for taking a picture. The method comprises: determining a first snap photography mode among multiple preset snap photography modes on the basis of multiple captured pictures; and, using an evaluation strategy corresponding to the first snap photography mode, or an evaluation strategy corresponding to first snap photography category in the first snap photography mode, determining a snap-photographed picture corresponding to the first snap photography mode among multiple captured pictures to be selected, where the evaluation strategy used is one among the multiple preset evaluation strategies. As such, the snap-photographed picture being outputted is more in line with the actual photographed scene, thus favoring the acquisition of an ideal snap photography effect.

Description

Method and device for photographing images

Technical field

The present application relates to the field of image processing technology, and more specifically, to a method and apparatus for capturing images.

Background technique

Smart capture is an important photographing function of current smart terminals. The smart terminal with the smart capture function enabled can score multiple frames of images to be selected based on certain scoring rules, thereby selecting the highest-scoring frame of image and recommending it to the user. However, when the smart terminal is scoring the selected images, the scoring rules are single, and often focus on facial information and ignore other information. Even if no human face is detected in the image to be selected, it is only based on the optical flow information of the two frames before and after as the scoring basis. This scoring mechanism is not universal, and the recommended optimal frames may not be ideal. For example, for images that include high-speed moving objects, the above methods cannot recommend wonderful images at the moment of movement to users, and the capture effect is not ideal. Therefore, there is an urgent need to develop a flexible capture solution suitable for more scenes.

Summary of the invention

The present application provides a method and device for capturing an image, so that the captured image is more in line with the actual shooting scene.

In the first aspect, a method of capturing an image is provided. The method includes: determining the first capture mode among the preset multiple capture modes according to the captured multi-frame images; using the evaluation strategy corresponding to the first capture mode to determine the first capture mode among the captured multi-frame images to be selected A capture frame image corresponding to a capture mode; the evaluation strategy is one of a variety of preset evaluation strategies.

Therefore, in the embodiment of the present application, by presetting multiple different capture modes and corresponding evaluation strategies, the capture mode can be determined according to the actual shooting scene. And an evaluation strategy corresponding to the first capture mode can be selected from a plurality of preset evaluation strategies, so as to use the evaluation strategy to determine the captured frame image. Therefore, the captured image obtained is more in line with the actual shooting scene, which is conducive to obtaining an ideal capture effect, the flexibility is improved and it is suitable for more scenes.

In combination with the first aspect, in some implementations of the first aspect, the aforementioned multiple capture modes include one or more of the following: facial expression capture mode, group photo capture mode, sports capture mode, multiplayer sports capture mode, pet capture mode And landscape capture mode.

It can be seen that a variety of different capture modes and their corresponding evaluation strategies are preset to apply to different shooting scenes, so that the captured images obtained conform to the actual shooting scenes.

It should be understood that the capture modes listed above are only examples and should not constitute any limitation to this application. This application does not limit the content specifically included in the above-mentioned multiple capture modes.

With reference to the first aspect, in some implementations of the first aspect, each of the multiple capture modes corresponds to at least one of the foregoing preset multiple evaluation strategies, and each evaluation strategy includes One or more scoring parameters used for image scoring and the mode weight of each scoring parameter. The using the evaluation strategy corresponding to the first capture mode to determine the captured frame image corresponding to the first capture mode among the captured multiple frames to be selected includes: using at least one evaluation corresponding to the first capture mode An evaluation strategy in one or more of the scoring parameters and the mode weight of each scoring parameter to calculate the score of each frame of the image to be selected in the multiple frames of images to be selected; according to the multiple scores of the multiple frames of images to be selected, Determine the captured frame image corresponding to the first capture mode among the multiple frames of images to be selected.

Therefore, different scoring parameters can be assigned to different capture modes, and different weights can be applied to each scoring parameter. Therefore, the scoring results obtained by scoring the same image based on different capture modes are different. After the first capture mode is determined, the evaluation strategy corresponding to the first capture mode is selected to score multiple frames of images to be selected, so as to determine the captured frame image. The captured frame image obtained in this way is combined with the evaluation strategy corresponding to the first capture mode, so it can better meet the requirements of the first capture mode and conform to the actual shooting scene.

With reference to the first aspect, in some implementations of the first aspect, the captured frame image has the highest score among multiple frames to be selected.

After scoring the image to be selected based on the evaluation strategy corresponding to the first capture mode, the selected image with the highest score is the image selected from the multiple frames of images to be selected that best meets the requirements of the first capture mode, of course. It best suits the actual shooting scene.

With reference to the first aspect, in some implementations of the first aspect, different evaluation strategies corresponding to different capture modes include the same scoring parameters, and different evaluation strategies include different mode weights.

The different evaluation strategies mentioned here include different mode weights. Specifically, it may mean that the mode weights applied to the same scoring parameter in different evaluation strategies are different. Moreover, when the evaluation strategy includes multiple scoring parameters, different evaluation strategies impose different mode weights on at least one scoring parameter.

That is to say, in different capture modes, combined with different needs of different capture modes, different weights can be applied to the same scoring parameter. For example, in sports capture mode and expression capture mode, different weights can be applied to the scoring parameter of expression intensity. In the sports capture mode, a lower weight can be applied, and in the expression capture mode, a higher weight can be applied. Conversely, different weights can also be applied to the scoring parameter of posture height. In the sports capture mode, a higher weight can be applied, while in the expression capture mode, a lower weight can be applied.

Therefore, different evaluation strategies corresponding to different capture modes may respectively include different mode weights corresponding to the same scoring parameter.

It should be understood that the above examples are only for ease of understanding, and should not constitute any limitation on this application.

With reference to the first aspect, in some implementations of the first aspect, each capture mode includes one or more capture categories, and each capture category corresponds to an evaluation strategy; at least one corresponding to the first capture mode In an evaluation strategy, each evaluation strategy includes one or more scoring parameters corresponding to the first snapping mode, a mode weight of each scoring parameter, and a category weight corresponding to one snapping category.

The determining the first snapping mode among the preset multiple snapping modes further includes: determining the first snapping category in the first snapping mode according to multiple frames of images.

The one or more scoring parameters and the mode weight of each scoring parameter in one of the at least one evaluation strategy corresponding to the first snapping mode are used to calculate each frame of the image to be selected in the multiple frames of image to be selected The scoring includes: using one or more scoring parameters corresponding to the first snapshot mode and the mode weight of each scoring parameter, as well as the category weight of each scoring parameter corresponding to the first snapshot category, to calculate multiple frames to be selected The score of each frame of the image to be selected.

In order to find the captured frame image that best fits the shooting scene, this application not only proposes an evaluation strategy corresponding to the capture mode, assigns different scoring parameters and mode weights to different capture modes, and further proposes the corresponding to the capture category in the capture mode The category weight. That is, the different capture categories in the capture mode are further refined, and the details of the different capture categories are further weighted. As a result, the selected captured frame images can not only meet the requirements of the first capture mode, but also take into account the capture category of the subject, and find out the images that can better present the wonderful moments of the capture.

With reference to the first aspect, in some implementations of the first aspect, different evaluation strategies corresponding to different capture categories include the same scoring parameters, and different evaluation strategies include different category weights.

The category weights included in the different evaluation strategies mentioned here are different. Specifically, it may mean that the category weights applied to the same scoring parameter in different evaluation strategies are different. Moreover, when the evaluation strategy includes multiple scoring parameters, different evaluation strategies apply different category weights to at least one scoring parameter.

That is to say, under different capture categories in the same capture mode, combined with different needs of different capture categories, different weights can be applied to the same scoring parameter. For example, in the sports capture mode, jumping and shooting focus on the action differently. Jumping pays more attention to the bending angle of the leg. Therefore, when the capture category is jumping, the bending angle of the leg is given a higher weight; and Shooting is more concerned with the bending angle of the arm, so when the capture category is shooting, a higher weight is applied to the bending angle of the arm.

Therefore, different evaluation strategies corresponding to different capture categories may respectively include different category weights corresponding to the same scoring parameter.

Different weights can be applied to the scoring parameter of expression intensity. In the sports capture mode, a lower weight can be applied, and in the expression capture mode, a higher weight can be applied. Conversely, different weights can also be applied to the scoring parameter of posture height. In the sports capture mode, a higher weight can be applied, while in the expression capture mode, a lower weight can be applied.

With reference to the first aspect, in some implementations of the first aspect, the method further includes: invoking at least one detection model corresponding to the first capture mode to perform image recognition on multiple frames of images to be selected to output a recognition result; Determine the value of one or more scoring parameters based on the recognition result.

In this embodiment of the present application, at least one detection model is used to perform image recognition on the image to be selected. The at least one detection model may include, for example, one or more of a face attribute detection model, a human frame detection model, a scene recognition model, a posture point estimation model, and an action detection model. Through these detection models, different attention points of the image can be detected, and the value of each scoring parameter can be determined according to the recognition result.

The above-mentioned detection model may be obtained through machine learning training, for example. In a possible design, the aforementioned detection model may be a model embedded in a neural network processing unit (NPU). This application does not limit this.

Optionally, when the first capture mode is a motion capture mode or a multi-person motion capture mode, the at least one detection model includes a pose estimation model and a motion detection model.

Optionally, the first capture mode is an expression capture mode or a group photo capture mode, and the at least one detection model includes a face attribute detection model.

It should be understood that the detection models corresponding to the different capture modes listed above are only examples, and should not constitute any limitation to this application. For example, in the case of different capture modes, the same multiple detection models can also be called to perform image recognition on the images to be selected. In the scoring process, different weights can be applied to each scoring parameter according to different capture modes, so as to achieve similar effects.

With reference to the first aspect, in some implementations of the first aspect, the first capture mode is determined from the preset multiple capture modes according to the captured multi-frame images, including: in the video mode or the preview mode, According to the multiple frames of images, the first capture mode is determined among multiple functional capture modes.

That is to say, the method provided in this application can not only be applied in the smart capture mode, but also can run synchronously with other modes. For example, in the video recording mode, if it is detected that the image meets the trigger condition of the first capture mode, the smart capture mode can be run in the background at the same time to enter the first capture mode. For another example, in the preview mode, if it is detected that the image meets the trigger condition of the first capture mode, the first capture mode can be automatically activated. Therefore, the device can automatically switch between multiple modes, which is conducive to obtaining an ideal captured frame image.

With reference to the first aspect, in some implementations of the first aspect, the determining the first capture mode among the preset multiple capture modes according to the captured multi-frame images includes: comparing the captured images based on the first frame rate The mode detection is performed on multiple frames of images to determine the first capture mode among the preset multiple capture modes. The calling at least one detection model corresponding to the first capture mode to perform image recognition on multiple frames of images to be selected includes: calling at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected at a second frame rate Perform image recognition; wherein the first frame rate is less than the second frame rate.

In other words, before determining the first capture mode, mode detection can be performed based on a lower frame rate. This method can be applied to the above-mentioned video mode or preview mode. Before entering the smart capture mode, you can use a lower frame rate for mode detection. Once the first capture mode is determined, that is, the smart capture mode is entered, a higher frame rate can be used for image recognition. Therefore, it is possible to perform mode detection based on a low frame rate before entering the smart capture mode, saving power consumption caused by a high frame rate.

Of course, in the embodiments of the present application, pattern detection and image recognition can also be performed at the same frame rate. For example, in the smart capture mode, the mode is determined at a higher frame rate. And after entering the first capture mode, image recognition is still performed in a higher mode. This application does not limit this.

In addition, this application does not limit the specific value of the frame rate.

With reference to the first aspect, in some implementations of the first aspect, after the first capture mode is determined, the method further includes: determining a second capture mode based on the newly captured multi-frame images, where the second capture mode is One of the multiple snapping modes is different from the first snapping mode; switching to the second snapping mode.

After entering the first capture mode, the newly captured image can also be continuously detected. When it is detected that the newly captured image meets the trigger condition of another capture mode (such as the second capture mode), the second capture mode can be automatically switched to. Optionally, the switching to the second capture mode includes: switching to the second capture mode when the running time of the first capture mode exceeds a preset protection period.

In order to avoid switching back and forth between multiple capture modes and avoid accidental touches, the protection period can also be preset for each capture mode. During the protection period, even if it is detected that the newly captured image meets the trigger condition of another capture mode, the mode switching is not performed. After the protection period is exceeded, if it is detected that the newly captured image meets the trigger condition of another capture mode, the mode can be switched.

In a second aspect, an image capturing apparatus is provided, which includes various modules or units for executing the method in any one of the possible implementation manners of the first aspect.

In a third aspect, a device for capturing images is provided, including a processor, a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the device for waking up the screen executes the first On the one hand and its various possible implementations are the methods of capturing images.

Optionally, there are one or more processors and one or more memories.

Optionally, the memory may be integrated with the processor, or the memory and the processor may be provided separately.

In a fourth aspect, an electronic device is provided, and the electronic device includes the image capturing apparatus as described in the second or third aspect.

In a fifth aspect, a computer program product is provided. The computer program product includes: a computer program (also called code, or instruction), which when the computer program is executed, causes a computer to execute any one of the above-mentioned first aspects. One of the possible implementation methods.

In a sixth aspect, a computer-readable medium is provided, and the computer-readable medium stores a computer program (also called code, or instruction) when it runs on a computer or at least one processor, so that the computer or the At least one processor executes the method in any possible implementation manner of the first aspect.

In a seventh aspect, a chip system is provided, and the chip system includes a processor for supporting the chip system to implement the functions involved in any possible implementation manner of the first aspect.

Description of the drawings

FIG. 1 is a schematic diagram of an electronic device provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of a method for capturing an image provided by an embodiment of the present application;

Figure 3 is a schematic diagram of a mobile phone interface provided by an embodiment of the present application;

FIG. 4 is a schematic flowchart of a method for capturing an image according to another embodiment of the present application;

FIG. 5 is a schematic flowchart of a method for capturing an image provided by another embodiment of the present application;

FIG. 6 is a schematic flowchart of a method for capturing an image according to still another embodiment of the present application;

Fig. 7 is a schematic block diagram of an image capturing apparatus provided by an embodiment of the present application.

detailed description

The technical solution in this application will be described below in conjunction with the accompanying drawings. The method of capturing images provided by the embodiments of the application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR)/virtual reality (VR) devices, notebook computers, and ultra-mobile personal For electronic devices such as a computer (ultra-mobile personal computer, UMPC), netbook, and personal digital assistant (personal digital assistant, PDA), the embodiments of this application do not limit the specific types of electronic devices. The image capturing apparatus provided in the embodiments of the present application may be configured in the various electronic devices listed above, or may be the various electronic devices listed above. This application does not limit this.

Illustratively, FIG. 1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110. The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), a neural network processing unit (NPU), an application processor (AP), a modem processor, and a graphics processing unit (graphics processing unit, GPU), image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), and one or more of the baseband processor. Among them, the different processing units may be independent devices or integrated in one or more processors.

For example, the controller may be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

NPU is a neural-network (NN) processor. By learning from the structure of biological neural network, such as the transfer mode between human brain neurons, it can quickly process input information and can continuously learn by itself. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face detection, human frame detection, scene detection, gesture point detection, motion detection, and so on.

In the embodiments of the present application, one or more detection models may be embedded in the NPU, such as the face attribute detection model, pose estimation model, motion detection model, human frame detection model, and scene detection model described below. one or more. Each detection model can be obtained by algorithm training based on machine learning. For example, it can be obtained by training based on support vector machines (VSM), convolutional neural networks (convolutional neural networks, CNN), or recurrent neural networks (RNN). It should be understood that this application does not limit the specific training method.

Each detection model can correspond to a processor in the NPU; or, each detection model can correspond to a processing unit in the NPU, and the functions of multiple detection models can be integrated into a processor through multiple processing units to fulfill. This application does not limit this.

In addition, the NPU may also have a communication connection with one or more other processors in the processor 100. For example, the NPU may have a communication connection with the GPU, ISP, and application processor. This application does not limit this.

Optionally, the electronic device 100 further includes a memory 120. The memory 120 may be used to store computer executable program code, where the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function. The data storage area can store data (such as audio data, phone book, etc.) created during the use of the electronic device 100. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

In some possible embodiments, a memory may be provided in the processor 110. For example, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or used cyclically. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

Of course, the memory may also exist independently of the processor 110, such as the memory 120 as shown in the figure. This application does not limit this.

Optionally, the electronic device 100 further includes a transceiver 130. In addition, in order to make the functions of the electronic device 100 more complete, the electronic device 100 may also include one or more of the input unit 160, the display unit 170, the audio circuit 180, the camera 190, and the sensor 101. The audio The circuit can also be coupled to a speaker 182, a microphone 184, and the like.

The electronic device 100 implements a display function through a GPU, a display unit 170, an application processor, and the like. The GPU is an image processing microprocessor, which is connected to the display unit 170 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

The display unit 170 is used to display images, videos, and the like. The display unit 170 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). emitting diode (AMOLED), flexible light-emitting diode (FLED), mini LED (MiniLED), micro LED (MicroLED), micro OLED (Micro-OLED) or quantum dot light emitting diode, QLED) etc. In some embodiments, the electronic device 100 may include one or more display units 170.

The electronic device 100 may implement a shooting function through an ISP, a camera 190, a video codec, a GPU, a display unit 160, an application processor, and the like.

The ISP is used to process the data fed back by the camera 190. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and is converted into an image visible to the naked eye. ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 190.

The camera 190 is used to capture still images or dynamic videos. The object generates an optical image through the lens and is projected to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard image signals in RGB, YUV and other formats. In some embodiments, the electronic device 100 may include one or more cameras 190.

For example, in the method for capturing images provided in the present application, the camera 190 may be used to capture images and display the captured images in the capture interface. The photosensitive element converts the collected optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for related image processing.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.

The application processor outputs a sound signal through an audio device (such as a speaker 182, etc.), or displays an image or video through the display unit 170. Optionally, the aforementioned electronic device 100 may further include a power supply 150 for providing power to various devices or circuits in the terminal device.

It should be understood that the electronic device 100 shown in FIG. 1 can implement each process of the method embodiments shown in FIG. 2 and FIG. 4 to FIG. 6. The operations and/or functions of each module or unit in the electronic device 100 are respectively for implementing the corresponding processes in the foregoing method embodiments. For details, please refer to the descriptions in the above method embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

It should be understood that FIG. 1 is only for ease of understanding, and exemplarily shows each module or unit in an electronic device and the connection relationship between the modules or units, but this should not constitute any limitation to the application. This application does not limit the modules and units specifically included in the electronic device and their mutual connection relationship.

The method provided by the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. FIG. 2 is a schematic flowchart of an image capturing method 200 provided by an embodiment of the present application. The method 200 provided in FIG. 2 may be executed by an electronic device or a processor in the electronic device. In the following, for ease of description, an electronic device is used as an execution subject to describe the embodiments of the present application.

Each step in the method 200 shown in FIG. 2 will be described in detail below. As shown in the figure, the method 200 may include step 210 to step 240. In step 210, according to the captured multiple frames of images, a first capture mode is determined among a plurality of preset capture modes.

Specifically, the electronic device may periodically detect the captured image, and according to the detection result, determine the capture mode suitable for the current shooting. For the convenience of distinction and description, the capture mode applicable to the current shooting is recorded as the first capture mode. It should be understood that the image captured by the electronic device can be stored in the cache. The cache, for example, may be a part of the storage space in the camera module of the electronic device, or may exist independently of the camera module, which is not limited in this application.

The electronic device can continuously obtain multiple frames of images from the cache. The multiple frames of images can be input to one or more detection models. The electronic device can call one or more detection models to detect the multi-frame images, and based on the detection results output by the detection models, determine the first capture mode from a plurality of preset capture modes. Based on the determination of the first capture mode, the electronic device may enable the first capture mode.

Among them, as an example and not a limitation, the preset multiple capture modes include one or more of the following: facial expression capture mode, group photo capture mode, sports capture mode, multiplayer sports capture mode, pet capture mode, and landscape capture mode.

The aforementioned multiple capture modes are all defined as smart capture modes in the embodiments of the present application. In other words, the above-mentioned first capture mode is a smart capture mode. The electronic device can enter the smart capture mode in advance, and then determine the first capture mode among the preset multiple capture modes according to the captured multi-frame images; it can also be in the photo mode or video mode, based on the detection of the captured images , Automatically determine the first capture mode. This application does not limit this.

The multiple preset capture modes described herein may specifically refer to that the electronic device pre-stores the evaluation strategy corresponding to each capture mode in the multiple capture modes. When the electronic device determines to use one of the capture modes, it can call the corresponding evaluation strategy to evaluate the captured multiple frames of images to be selected. Hereinafter, the process of evaluating multiple frames of images to be selected by the electronic device using the evaluation strategy will be described in detail in conjunction with step 220, which is omitted here.

The specific process of determining the first capture mode based on multiple frames of images will be described in detail below through specific examples.

For example, in the photo mode or video mode, the electronic device can call one or more of the face attribute detection model, the human frame detection model, and the scene recognition model to detect the captured multi-frame images, and according to each The detection result output by the detection model determines the first capture mode.

Current smart terminals usually have camera functions and video recording functions. The above-mentioned face attribute detection model, human frame detection model, and scene recognition model are configured in the smart terminal. Therefore, in the photo mode or the video mode, the electronic device can detect the image by calling an existing model to determine the first capture mode.

For another example, in the smart capture mode, the electronic device can detect the captured multi-frame images by calling one or more of the face attribute detection model, the pose estimation model, and the motion detection model, and output according to each detection model The detection result determines the first capture mode.

The face attribute detection model, the human frame detection model, the scene recognition model, the pose estimation model, and the action detection model listed above may all be models obtained by training based on machine learning algorithms. Based on different functions, different models are defined. Among them, based on different functions, face attribute detection models can be further divided into face feature point detection models, open and closed eyes detection models, and so on. This application does not limit this. The names of the detection models are merely examples for ease of understanding, and this application does not exclude the possibility of replacing the detection models listed above with other names to achieve the same or similar functions.

It should be understood that the specific functions of these detection models are also implemented by the processor executing its corresponding computer instructions. This application does not limit the number and form of processors used to implement the above detection models. In a possible design, the aforementioned face attribute detection model, human frame detection model, scene recognition model, pose estimation model, and motion detection model can all be embedded in the NPU.

The determination of the first capture mode may be determined based on the detection result of a certain detection model on the image, or it may be determined based on the detection result of the image by multiple detection models. When the electronic device calls multiple detection models to determine the first capture mode, the multiple detection models can run simultaneously or alternately. The electronic device can comprehensively consider the detection results of the image based on multiple detection models. When the detection result of the image by the one or more detection models meets the trigger condition of a certain capture mode, the capture mode can be determined as the first capture mode. The following uses a number of examples to illustrate the specific process of the electronic device determining the first capture mode by calling one or more detection models.

Optionally, the electronic device may call the face attribute detection model to detect the face in the image, and determine the first capture mode according to the detection result. The face attribute detection model can be specifically obtained through machine learning algorithm training. When the face attribute detection model detects a face in an image, it can detect each feature point of the face. The electronic device can exclude passers-by entry scenes and sports scenes based on the detected face position and depth information. In this case, the detection result of the image by the face attribute detection model satisfies the trigger condition of the expression capture mode, and it can be determined that the first capture mode is the expression capture mode.

When the face attribute detection model detects multiple faces in the image, the electronic device can exclude passersby entry scenes and sports scenes based on the position and depth information of the faces. In this case, the detection result of the image by the face attribute detection model satisfies the trigger condition of the group photo capture mode, and it can be determined that the first capture mode is the group photo capture mode.

Optionally, the electronic device may call the scene recognition model to detect the shooting scene of the image, and determine the first capture mode according to the detection result. The scene recognition model can be specifically obtained by training based on multiple predefined scenes through a machine learning algorithm. When the scene detected by the scene recognition model in the image is one of a plurality of predefined motion scenes, the motion scene is output. The electronic device can determine that the shooting object is in a motion state according to the motion scene detected by the scene recognition model. In this case, the detection result of the scene recognition model on the image satisfies the trigger condition of the motion capture mode, and it can be determined that the first capture mode is the motion capture mode. By way of example and not limitation, the aforementioned sports scene may include: a court (for example, including a basketball court, a football field, etc.), a swimming pool, or a running track.

Further optionally, the electronic device may also call the scene recognition model and the human frame detection model to detect the image, and synthesize the output results of the scene recognition model and the human frame detection model to determine the first capture mode. For example, when the scene in the image is detected as a predefined motion scene through the scene recognition model, and multiple human frames are detected in the image through the human frame detection model, the scene recognition model and the human frame detection model detect the image The trigger condition of the multi-person sports capture mode is satisfied, and the first capture mode can be determined to be the multi-person sports capture mode.

Optionally, the electronic device may call the human body frame detection model to detect the human body frame in the image. The human frame detection model can be obtained through machine learning algorithm training to be used to detect the human frame in the image. The electronic device can also call other motion area detection algorithms to determine the motion area in the image. For example, the motion area in the image can be determined based on optical flow information and the like. This application does not limit this. When the overlap area between the motion area and the human body frame in the two frames of the image is large, if the ratio of the overlap area to the entire human body frame is higher than a certain preset threshold, the motion area in the image can be determined to be foreground motion , Not background motion or relative camera motion. In this case, the degree of overlap between the human body frame and the motion area in the image meets the triggering condition of the motion capture mode, and it can be determined that the first capture mode is the motion capture mode. Further, the electronic device may also determine that the first capture mode is a multi-person motion capture mode when there are multiple human body frames detected by the human body frame detection model.

Optionally, the electronic device may call the pose estimation model to detect multiple pose points of the human body in the image. The pose estimation model can be obtained by training based on a plurality of predefined pose points (or feature points) through a machine learning algorithm. The foregoing multiple predefined posture points include, for example, head, shoulders, neck, elbows, hips, legs, knees, ankles, and so on. For the sake of brevity, I will not list them all here. The pose estimation model can be used to detect multiple pose points in the image and determine the coordinate information of each pose point.

In an implementation manner, the coordinate information of each posture point may be represented by the two-dimensional coordinates of the corresponding pixel point in the image. For example, the pixel (u, v) represents the u-th row and v-th column of the pixel in the two-dimensional image. In another implementation manner, the coordinate information of each posture point can be represented by the three-dimensional coordinates of the corresponding pixel in the image. For example, the pixel (u, v) can further carry depth information d, and the three-dimensional coordinates of the pixel can be expressed as (u, v, d). The depth information is used to indicate the distance between the pixel and the camera. It should be understood that expressing the coordinate information of the posture point by the two-dimensional coordinates (u, v) or the three-dimensional coordinates (u, v, d) of the pixel point is only a possible implementation manner, and should not constitute any limitation in this application.

The pose estimation model can be based on multiple pose points of the human body, and the human body frame can be further estimated. On the same frame of image, the human skeleton frame can be obtained by connecting multiple posture points. According to the coordinate information of the multiple posture points, the position and size of the human body frame can be estimated. Therefore, the electronic device can detect the coincidence degree of the determined motion region according to the human body frame estimated by the pose estimation model and the motion region algorithm, thereby determining whether the triggering condition of the motion capture mode is satisfied. The specific method is similar to the method introduced above in conjunction with the human frame detection model. For brevity, it will not be repeated here.

In another implementation manner, the electronic device can determine whether the photographed object is in a moving state according to the coordinate information of each posture point in the multiple frames of images before and after. As the subject is in motion, the coordinate information of some or all of the posture points will change relatively in the multiple frames before and after the image. Through the relative changes of the coordinate information of these posture points, the changes of the posture points of the human body under the motion state can be obtained. Therefore, when the electronic device determines that the trigger condition of the motion capture mode is satisfied according to the coordinate information of the multiple pose points in each frame of the image detected by the pose estimation model, it can be determined that the first capture mode is the motion capture mode.

Further optionally, the electronic device can call the posture estimation model and the motion detection model to identify the motion category of the object in the image. As described above, the pose estimation model can be used to determine multiple pose points in an image and the coordinate information of each pose point. The coordinate information of multiple posture points in each frame of image can be input to the motion detection model to determine the motion category of the subject. The action detection model can be obtained by training based on multiple predefined action categories through a machine learning algorithm. The motion detection model can determine the motion category of the subject based on the training sample and the coordinate changes of the above-mentioned posture points. The action category includes, for example: running, jumping, shooting, kicking, rock climbing, swimming, diving, skating, etc.

If the motion detection model determines that the coordinate change of the posture point is the same or approximately the same as the coordinate change of the posture point under a certain predefined action category, the action category can be determined as the action category of the subject. The motion detection model can output the motion category of the subject. For example, when the motion detection model detects that the human body performs a specific action (such as the action categories listed above) in the image, the electronic device can determine that the image meets the trigger condition of the motion capture mode, thereby determining the first capture mode It is a sports capture mode.

A number of examples for the electronic device to determine the first capture mode are listed above in combination with the functions of each model, but it should be understood that these examples should not constitute any limitation to this application. Multiple models can also be used in combination, and the first capture mode suitable for the current shooting is determined based on the trigger conditions predefined for various capture modes.

The electronic device can also call corresponding models in sequence according to the priority of multiple capture modes. As an example and not a limitation, the sports capture mode has a higher priority than the expression capture mode. Based on the priority ranking of the multiple capture modes listed above, the electronic device may sequentially call the detection models based on the detection models corresponding to the different capture modes to determine the first capture mode.

For example, the motion capture mode can be determined by calling the human frame detection model, or the scene recognition model, or the pose estimation model, or the pose estimation model and the motion detection model to detect the captured multiple frames of images. The expression capture mode can be determined by calling the multi-frame images captured by the face attribute detection model to detect. It should be understood that the relationships between the various modes and models listed here are only examples, and should not constitute any limitation on this application. As mentioned above, a capture mode can be determined by the detection results of multiple models. In other words, a capture mode can be determined by calling multiple models to detect multiple frames of images captured. This application does not limit the models corresponding to various capture modes.

Since the motion capture mode has a higher priority than the face attribute detection model, the electronic device may first call the human frame detection model or the scene recognition model. When it is determined that the captured multi-frame images meet the triggering condition of the motion capture mode, it can be directly determined that the first capture mode is the motion capture mode. The electronic device can no longer call the face attribute detection model, thereby saving the time for determining the capture mode and at the same time saving power consumption caused by the running of the model.

It should be understood that, in conjunction with an example of priority sorting, the process of the electronic device sequentially calling models according to the priority sorting of multiple capture modes is described in detail here. However, this is only an example for ease of understanding, and should not constitute any limitation to this application. This application does not limit the priority ranking among multiple capture modes.

In step 220, the evaluation strategy corresponding to the first capture mode is used to determine the captured frame image corresponding to the first capture mode from among the captured images to be selected.

Among them, the evaluation strategy is one of a variety of preset evaluation strategies. Each evaluation strategy can be used to define a rule or method for determining the captured frame image among multiple frames to be selected. The captured frame image corresponding to the first capture mode may specifically refer to an image that is determined based on the first capture mode to best show the wonderful moments of the first capture mode. For example, if the first capture mode is a sports capture mode, the captured frame image may be, for example, the image that best reflects the wonderful action moment of the subject among the captured multiple frames to be selected. For another example, the first capture mode is an expression capture mode, such as a smiling face capture, the captured frame image may be, for example, the image at the moment when the subject's smile is the brightest among the captured multiple frames to be selected. For another example, if the first capture mode is a group photo capture mode, the captured frame image may be, for example, the best moment of the expression, demeanor, or composition of each subject in the captured images to be selected.

It should be noted that the multiple frames of images to be selected herein and the multiple frames of images described in step 210 above may not overlap each other, or partially overlap. The overlap mentioned here may specifically mean that a certain frame of image in step 210 and a certain frame of to-be-selected image in step 220 are the same frame of image, or in other words, are images captured at the same time point. For example, have the same time stamp. For example, the multi-frame image to be selected may be a multi-frame continuous image captured after the multi-frame image captured in step 210, or it may further include at least part of the multi-frame image captured in step 210. Multi-frame images included. For another example, the multiple frames of images to be selected may be multiple frames of discontinuous images after the multiple frames of images captured in step 210. For another example, the multi-frame image to be selected may be a multi-frame image that is after the multi-frame image captured in step 210 and is not continuous with the multi-frame image.

Specifically, the captured multi-frame images in step 210 may include, for example, preview images captured before the user performs a photographing operation or a video recording operation. Normally, after turning on the camera, the electronic device enters the photo mode by default. Although the user did not perform the photographing operation, the preview image can still be seen through the photographing interface. The above-mentioned multi-frame images may include multi-frame preview images captured in the photographing mode. Thereafter, the electronic device may trigger the first capture mode based on the user's manual adjustment or the detection result of the detection model. That is, the electronic device enters the smart capture mode. In the smart capture mode, although the user may not perform a photo operation, multiple frames of preview images can still be captured. The above-mentioned multi-frame images may also include multi-frame preview images captured after entering the smart capture mode. The electronic device can save the above-mentioned multi-frame images in a buffer for subsequent use, for example, in step 210 to determine the first capture mode.

In addition, in the camera mode or the smart capture mode, because the user has not performed a camera operation, a preview image can still be captured for determining the first capture mode. In the embodiment of the present application, the mode in which the camera is in the camera mode, smart capture mode, or other camera mode before performing the camera operation may be referred to as the preview mode. The user can observe the photographed subject in the preview mode, so as to select the appropriate time to perform the photographing operation. It should be understood that the preview mode is not necessarily before the photographing operation, and the mode in between two consecutive photographing operations may also be referred to as the preview mode.

The multiple frames of images to be selected in step 220 may be, for example, N frames of images before and after the moment the shutter is pressed, and N is a positive integer. For example, if N is 10, the multiple frames of images to be selected may be the first 10 frames of images and the last 10 frames of images at the moment when the shutter is pressed, a total of 20 frames of images. In this case, the multiple frames of images to be selected may not overlap with the multiple frames of images described in step 210, or there may be some overlaps. This may depend on the length of time between the time when the shutter is pressed and the time when the electronic device enters the first capture mode, the value of N, and so on. For example, if the value of N is large, and the user takes a photo after entering the first capture mode, the multiple frames of images to be selected may include part or all of the multiple frames of images described in step 210. This application does not limit the relationship between the multiple frames of images described in step 210 and the multiple frames of images to be selected in step 220. Alternatively, the multiple frames of images to be selected may also be the first N frames of images or the last N frames of images at the moment when the shutter is pressed, which is not limited in the embodiment of the present application.

It should be understood that the user can press the shutter by clicking the shooting control in the user interface or other buttons for controlling the shooting during the shooting process, and the specific operation of the user pressing the shutter is not limited in this application. It should also be understood that the value of N above is only an example for ease of understanding, and should not constitute any limitation to this application. This application does not limit the specific value of N.

In addition, the multi-frame images described in step 210 may also include video images saved in the video recording mode. For example, if the user uses the video mode to record video after turning on the camera function. When the user performs video recording through the camera, although the photographing operation may not be performed, in fact, the video image may be a sequence of continuous multi-frame images. The electronic device can save these images in the cache for subsequent use, for example, to determine the first capture mode in step 210 and to determine the captured frame image corresponding to the first capture mode in step 220.

In the video recording mode, the multiple frames of images to be selected in step 220 may be, for example, the first N frames of images, the last N frames of images, or the previous and next N frames of images at the moment the shutter is pressed, or they may be based on the provided in the embodiment of the present application. The scoring result selected by the evaluation strategy exceeds the preset threshold image. This application does not limit this.

In an implementation manner, the captured frame image corresponding to the first capture mode may be an optimal frame image determined from a plurality of captured images to be selected. For example, based on the evaluation strategy corresponding to the first capture mode, multiple captured images to be selected may be scored, and the image with the highest score may be selected as the optimal frame image.

Optionally, before step 220, the method 200 further includes:

Step 230: Perform image recognition on the captured multiple frames of images to be selected to output a recognition result;

Step 240: Determine the value of each scoring parameter based on the recognition result.

Specifically, the electronic device can call one or more of the above detection models to perform image recognition on the captured multiple frames of images to be selected. The electronic device can use the evaluation strategy corresponding to the first capture mode to score the captured images to be selected according to the recognition result of the image to be selected, so that the capture frame corresponding to the first capture mode can be determined according to the scoring result image.

Optionally, the above step 210 may specifically include: determining the first capture mode among multiple preset capture modes based on the first frame rate according to the captured multi-frame images. Optionally, step 230 may specifically include: performing image recognition on the captured multiple frames of images to be selected based on the second frame rate to output the recognition result. Wherein, the first frame rate may be equal to or less than the second frame rate. This is related to the shooting mode used by the electronic device. The following will be described in detail in conjunction with the embodiments shown in FIG. 4 to FIG. 6.

The following describes in detail the specific process of using the evaluation strategy corresponding to the first capture mode to determine the captured frame image corresponding to the first capture mode among the captured multiple frames to be selected.

Optionally, each of the multiple capture modes corresponds to at least one of the preset multiple evaluation strategies, and each evaluation strategy includes one or more scoring parameters for image scoring and each The mode weight of the scoring parameter. Step 220 may specifically include: using one or more scoring parameters in one of the at least one evaluation strategy corresponding to the first capture mode and the mode weight of each scoring parameter to calculate the captured multiple frames of images to be selected The score of each frame of the image to be selected in the frame; and according to the multiple scores of the multiple frames of images to be selected, the captured frame image corresponding to the first capture mode is determined among the multiple frames of images to be selected.

Specifically, the aforementioned multiple capture modes can correspond to multiple preset evaluation strategies. Each capture mode can correspond to one or more of the preset multiple evaluation strategies. Among them, each evaluation strategy may include a scoring parameter set corresponding to the first snapshot mode. Each scoring parameter set may include one or more scoring parameters. For example, the scoring parameter corresponding to the sports capture mode may include one or more of the following: posture stretch, posture height, and so on. For another example, the scoring parameters corresponding to the facial expression capture mode may include: facial expression strength, facial occlusion, eyes open and closed, and face angle.

Among them, posture stretch can also be called posture stretch, which can specifically refer to the degree of bending of the limbs and the relative distance to the trunk. The posture stretch can be obtained by weighting the angles of the body joints, and the angle parameters of the body joints that are strongly related to the human body movement can be pre-set. The degree of posture stretch can be determined by, for example, a posture estimation model and a motion detection model. Corresponding to different joints, the degree of posture stretch may further include, for example, the angles of the various joints of the human body, such as but not limited to, wrist joint angles, elbow joint angles, arm bending angles, leg bending angles, and knee joints. Angle, ankle angle, etc. For the sake of brevity, I will not list them all here. In the embodiment of the present application, the included angle of each joint may exist as a scoring parameter. The posture stretch can be understood as a generalization of the included angle of each joint point. The posture height may specifically refer to the height position of the center height of the body in the image. The expression intensity can specifically indicate the intensity of a certain expression when the subject expresses it. The expression intensity can be determined by, for example, a face attribute detection model, or it can be calculated by feature points. The expression intensity can be obtained by weighting the various local features of the face. The expression intensity may further include, for example, the size of the mouth corners grinning, the degree of raising the corners of the mouth, and the degree of opening and closing of the eyes. For the sake of brevity, I will not list them all here. In the embodiment of the present application, each local feature included in the expression intensity listed above may exist as a scoring parameter. The intensity of expression can be understood as a generalization of the above-mentioned local features. Eyes open and closed specifically refers to whether the subject has closed eyes. Eye opening and closing can be determined by, for example, a face attribute detection model. Face occlusion specifically refers to whether the face of the subject is occluded and the degree of occlusion. Facial occlusion can be calculated by feature points, for example. The face angle specifically refers to whether the face of the subject is tilted and the tilt angle. The face angle can be determined by, for example, a face attribute detection model.

In addition to the scoring parameters listed above, in different capture modes, the scoring parameters can also include sharpness, exposure, and composition. The sharpness can specifically refer to the sharpness of the shadow lines and their boundaries on the image. Sharpness is a parameter used to describe image quality. Exposure can specifically refer to the process in which the photosensitive element of the camera receives external light and then forms an image. The amount of external light received by the photosensitive element directly affects the brightness of the photo. According to the light receiving degree of the photosensitive element, it can be roughly divided into three situations: underexposure, correct exposure and overexposure. Composition can specifically refer to the process of identifying and organizing elements to produce a harmonious photo. The composition may specifically include, but is not limited to, the rule of thirds composition (or called the Jiugongge composition), the symmetrical composition, the frame composition, etc., which are not limited in this application.

It should be understood that the scoring parameters corresponding to the capture modes listed above are only examples, and should not constitute any limitation to this application. This application does not limit the specific content and names of the scoring parameters corresponding to each capture mode. Optionally, the scoring parameters included in different evaluation strategies corresponding to the same capture mode may be the same. However, in different capture modes, the scoring parameters included in different evaluation strategies are not necessarily the same.

Optionally, different evaluation strategies corresponding to different capture modes include the same scoring parameters, and different evaluation strategies include different mode weights.

In other words, in different capture modes, the mode weights corresponding to the same scoring parameter in different evaluation strategies can be different. In other words, different evaluation strategies corresponding to different capture modes may respectively include different mode weights corresponding to the same scoring parameter.

For example, the scoring parameters included in the evaluation strategy corresponding to the sports capture mode can include posture height, posture stretch, clarity, exposure, and composition; or, the scoring parameters included in the evaluation strategy corresponding to the sports capture mode can also include posture. Height, posture stretch, expression intensity, eyes open and closed, face occlusion, face angle, clarity, exposure and composition, but the mode weights applied to expression strength, eyes open and closed, face occlusion, and face angle are smaller. For example, it is zero, or close to zero. Therefore, the essence of the two statements is the same.

It should be understood that the scoring parameters listed above are only examples and should not constitute any limitation to this application. As long as the scoring parameter corresponding to the motion capture mode includes any one of the scoring parameters of the posture height and the posture stretch, it should fall within the protection scope of the embodiment of the present application. The scoring parameters corresponding to the sports capture mode may also include, for example, rotation, etc. The application does not limit the scoring parameters corresponding to the sports capture mode and the mode weights thereof.

For another example, the scoring parameters included in the evaluation strategy corresponding to the expression capture mode may include expression strength, eyes open and closed, face occlusion, face angle, clarity, exposure, and composition; or, in the evaluation strategy corresponding to the expression capture mode The included scoring parameters can also include expression strength, open and closed eyes, face occlusion, face angle, posture height, posture stretch, clarity, exposure and composition, but the mode weights applied to posture height and posture stretch respectively are smaller , For example, zero, or close to zero. Therefore, the essence of these two statements is the same.

It should be understood that the scoring parameters listed above are only examples and should not constitute any limitation to this application. As long as the scoring parameters corresponding to the expression capture mode include any of the scoring parameters of expression strength, eyes open and closed, face occlusion, and face angle, it should fall within the protection scope of this application. This application does not limit the scoring parameters corresponding to the facial expression capture mode and the mode weights.

From the above list of scoring parameters and mode weights in different capture modes, it can be seen that corresponding to different capture modes, the same scoring parameter has different mode weights. For example, the posture height and posture stretch listed above are given a higher weight in the evaluation strategy corresponding to the motion capture mode, and a lower weight is applied to the evaluation strategy corresponding to the expression capture mode. , Or no weight (weight 0); for example, the expression intensity, eyes open and closed, face occlusion, and face angle listed above are given a higher weight in the evaluation strategy corresponding to the expression capture mode, and In the evaluation strategy corresponding to the sports capture mode, a lower weight is applied, or no weight is applied (weight 0).

In addition, the mode weights of definition, composition, and exposure have little relationship with the capture mode, so the same mode weights can be defined in different capture modes. It should be understood that the above description is only for the convenience of understanding in combination with different capture modes to explain different scoring parameters and their mode weights, but this should not constitute any limitation to this application.

It should be noted that the evaluation strategy corresponding to the snapshot mode may be predefined. Once the evaluation strategy corresponding to the first capture mode is determined, the scoring parameter used when scoring the image to be selected and the mode weight of each scoring parameter are determined.

For example, the score of the image to be selected can be evaluated by the formula

to make sure. Among them, G represents the scoring result, G>0; I represents the number of scoring parameters, I≥1 and is an integer; i represents the i-th scoring parameter in I scoring parameters, 1≤i≤I, and i is an integer ; T _i represents the i-th numerical score parameter, T _{_i} ≥0; α _i weight of the i schematic rates weight parameter, α _i ≥0.

In the first snapping mode, the score for each frame of image may be the result of weighting the scores of each scoring parameter _{based on the scoring parameter T i} corresponding to the first snapping mode and the mode weight α _{i of each scoring parameter} . When a certain capture mode corresponds to multiple evaluation strategies, the electronic device can select one of the multiple evaluation strategies for scoring, for example, select the default evaluation strategy, or the default evaluation strategy, general evaluation Strategy, etc.; or, the multiple evaluation strategies can also correspond to multiple capture categories, and the electronic device can also select the corresponding assessment strategy based on the capture categories determined by the previous detection models of multiple frames of images.

For ease of understanding, the following first describes in detail the process of scoring images to be selected using an evaluation strategy corresponding to the first capture mode. An evaluation strategy corresponding to the first capture mode may be, for example, the default evaluation strategy described above.

The following takes different capture modes as examples to describe in detail the specific process of the electronic device determining the captured frame image corresponding to the first capture mode from multiple frames of images to be selected.

First, the electronic device can call the detection model corresponding to the first capture mode to perform image recognition on the image to be selected to obtain the recognition result.

For example, if the first capture mode is a motion capture mode, the model corresponding to the first capture mode may include, for example, a pose estimation model and a motion detection mode model.

The electronic device can call the pose estimation model to perform image recognition on the captured multiple frames of images to be selected. The pose estimation model can perform image recognition on each frame of the image to be selected, and obtain the coordinate information of multiple pose points in each frame of the image to be selected. The coordinate information of each posture point in each frame of the image to be selected is the recognition result output by the posture estimation model. Since the specific process of determining the coordinate information of each posture point by the posture estimation model has been explained in step 210 above, for the sake of brevity, it will not be repeated here.

Optionally, the electronic device can also call an action detection model to perform image recognition on multiple frames of images to be selected. The action detection model can be combined with the pose estimation model to identify the action category of the subject. The action category recognized according to each frame of the image to be selected may be the recognition result output by the action detection model. Since the specific process of determining the action category by the action detection model has been explained in step 210 above, for the sake of brevity, it will not be repeated here.

The action detection model can specifically indicate the recognized action category through an action type index or other information that can be used to uniquely indicate an action category. This application does not limit the specific form of the recognition result output by the motion detection model.

It should be understood that in this embodiment, the evaluation strategy does not necessarily correspond to the action category. Therefore, the electronic device may call the motion detection model to recognize the action category, or may not call the motion detection model, which is not limited in this application.

For another example, if the first capture mode is an expression capture mode, the model corresponding to the first capture mode is a face attribute detection model. The electronic device can call the face attribute detection model to perform image recognition on the captured multiple frames of images to be selected. The face attribute detection model can be based on face attributes, for example, including expression categories (such as happy, angry, sad, funny, etc.), eyes closed, feature points, age, etc., to establish a classification model of face attributes, as described above The facial feature point detection model, the open and closed eyes detection model, etc., are used to perform image recognition on each frame of the image to be selected, and output information such as the expression category of the subject, whether the eyes are closed, whether the face is blocked, and age. In other words, the facial expression category, whether the eyes are closed, whether the face is blocked, and the age of the subject in each frame of the image to be selected are the recognition results output by the face attribute detection model based on the recognition of each frame of the image to be selected.

The face attribute detection model can detect the expression of the subject based on a variety of pre-trained expression categories. When it is determined that the expression of the subject belongs to any one of a plurality of pre-trained expression categories, the expression category may be determined as the expression category of the subject. In addition, different expression categories can correspond to different priorities. When there are multiple expression categories determined by the face attribute detection model, the expression category with a higher priority can be sorted according to a pre-defined priority, and the expression category with a higher priority is determined as the expression category of the subject.

After the face attribute detection model completes the image recognition of the image to be selected, it can output the recognition result. Specifically, for the expression category, it can be indicated by an expression type index or other information that can be used to uniquely indicate a kind of expression category; for open and closed eyes, for example, the binary values "0" and "1" can be used to indicate " Eyes closed" and "eyes open"; for feature points, the position of each feature point can be indicated by coordinate information; for age, it can be indicated by a specific numerical value.

It should be understood that, in this embodiment, since the evaluation strategy does not necessarily correspond to the expression category, the face attribute detection model may not output the expression category. The electronic device can determine the strength of expression, whether the eyes are closed, whether the face is occluded, and other information according to the position and age of each feature point in each frame of the image to be selected, and use the evaluation strategy corresponding to the first capture mode to determine each frame to be selected. Select the image for scoring.

It should also be understood that the above only shows several possible implementations for indicating the detection result for ease of understanding, and should not constitute any limitation to the application. This application does not limit the specific indication method of the detection result. It should also be understood that the scoring parameters corresponding to the facial expression capture modes listed above are only examples, and should not constitute any limitation in this application. As long as the scoring parameters corresponding to the expression capture mode include any of the parameters of expression intensity, face occlusion, eyes opening and closing, and face angle, it should fall within the protection scope of this application. Of course, when the first capture mode is a group photo capture mode, the scoring parameter may also include one or more of expression strength, face occlusion, eyes open and closed, and face angle. For brevity, the following text will not be repeated.

It should be noted that when the electronic device calls one or more detection models for image recognition, it can call the detection model corresponding to the first snapping category to perform image recognition, as in the example above. The capture category may be a specific sub-mode or classification in the capture mode obtained by further dividing the capture mode. The electronic device can also call multiple predefined detection models for image recognition, such as calling a face attribute detection model, a posture estimation model, and an action detection model for image recognition. Based on different capture modes, the mode weights applied to each scoring parameter are different. Therefore, although multiple detection models are called, the electronic device will apply different modes based on the capture mode when scoring based on the results of image recognition. Weights. Therefore, it will not affect the final selection results. Therefore, this application does not limit the specific model used for image recognition.

After obtaining the recognition result of the image to be selected, the electronic device can obtain the value of each scoring parameter based on the recognition result. For example, if the first capture mode is a sports capture mode, the electronic device can determine parameters such as the height of the human skeleton, the angle of each joint, and the like according to the coordinate information of the posture point.

The electronic device can determine the value of the scoring parameter of the posture height based on the height of the human skeleton. For example, the center point of the human skeleton, the highest point of the human skeleton, etc. can be used as the value of the posture height. It should be understood that when the electronic device selects a certain point (such as the center point of the human skeleton) as the value of the posture height, the same point is selected for all the images to be selected to determine the value of the posture height.

The electronic device can determine the posture stretch based on the included angle of each joint point. Specifically, the posture stretch can be determined by the curvature of each joint point of the human body, so the posture stretch can include the value of the curvature of each joint point. As mentioned above, the value of posture stretch can be weighted by the included angle of each joint point. The weight of the included angle of each joint point may be predefined, that is, the weight of the mode described above. Of course, the determination of the value of each scoring parameter can also be determined in other ways, which is not determined in this application. For example, an existing algorithm, such as an action intensity algorithm, can be used to determine the value of each scoring parameter.

It should be understood that the posture height and posture stretch are only one possible way of expression, and this application does not exclude the possibility of expressing the same or similar meaning through other possible expression ways. For example, posture stretch can also be replaced by motion range. For another example, if the first capture mode is an expression capture mode, the electronic device may determine the value of each scoring parameter according to multiple feature points of the human face.

Taking the scoring parameter of expression intensity as an example, the electronic device can determine the value of expression intensity according to multiple feature points. Specifically, the expression intensity may be determined by various local features of the human face, such as, but not limited to, the size of the mouth grinning, the degree of raising the corners of the mouth, and the degree of opening and closing of the eyes. Therefore, the value of expression intensity can include the value of each local feature. As mentioned earlier, the expression intensity can be obtained by weighting the values of the various local features of the human face. The weight of each local feature can be pre-defined, that is, the mode weight described above.

Among them, for the degree of eye opening and closing, for example, the value can be determined by the ratio of the vertical distance and the horizontal distance of the eye. For the size of the mouth grin, for example, the value can be determined by the ratio of the sum of the distance between the upper and lower lips of the mouth and the horizontal distance between the corners of the mouth and the distance between the eyes. Regarding the degree of uplift of the corner of the mouth, for example, the value can be determined by the distance between the horizontal line of the corner of the mouth and the lower lip of the mouth.

Taking the scoring parameter of face occlusion as an example, the electronic device can determine whether any feature points are missing according to the detected multiple feature points, and if so, it can be considered that the face of the subject is occluded. For face occlusion, for example, the value can be determined by the ratio of the detected feature points to the predefined feature points.

In addition to the expression intensity and facial occlusion listed above, the electronic device can also determine values for scoring parameters such as eyes open and closed, and face angles. For example, the electronic device can also call an existing algorithm, such as an expression intensity algorithm, to determine a value for each scoring parameter.

In addition, in different capture modes, the electronic device can further load scoring parameters such as sharpness, exposure, and composition, and determine the value of each scoring parameter. Among them, for different capture modes, the value of the scoring parameter of composition can be performed based on different composition methods. For example, in the expression capture mode, you can load the Jiugongge composition; in the group photo capture mode, you can load the symmetrical composition; in the landscape capture mode, you can load the horizontal line composition. Therefore, the definition of the mode weight of the scoring parameter of composition can be defined based on different capture modes.

Take the group photo capture mode as an example. Since the symmetrical composition can be loaded, the distance from the center of all people in the image to the center of the screen and the distance between two adjacent people can be calculated. Weights are applied to these two distances respectively to obtain a weighted sum. The weighted sum can be used as the value of the scoring parameter of the composition.

It should be understood that the specific methods for determining the value of each scoring parameter listed above are only examples, and should not constitute any limitation in this application. Since the method for determining the value of each scoring parameter can refer to the prior art, for the sake of brevity, an example is not described here.

It should be noted that when the scoring parameter is weighted by multiple parameter values, the parameter values can be normalized to the same magnitude, and then weighted. After determining the value of each scoring parameter, the electronic device can use the evaluation strategy corresponding to the first capture mode to determine the score of each frame of the image to be selected.

As mentioned earlier, each capture mode can correspond to one or more evaluation strategies, and each evaluation strategy defines the scoring parameters and the mode weights. In other words, in each evaluation strategy, the scoring parameters and their mode weights are preset. After the evaluation strategy is determined, the scoring parameters and the weight of the model can be determined. As long as the electronic device substitutes the value of the scoring parameter determined for each frame of the image to be selected, the score for each frame of the image to be selected can be obtained.

In this embodiment, the electronic device can use an evaluation strategy corresponding to the first capture mode to substitute the value of the scoring parameter previously determined for each frame of the image to be selected to calculate the score of each frame of the image to be selected. The scoring of each frame of the image to be selected by the electronic device can be based on the formula listed above, for example

Calculated. The meaning of each parameter in the formula has been explained above, for the sake of brevity, it will not be repeated here. When the scores of multiple scoring parameters are of different magnitudes, the scores of each scoring parameter can also be normalized to the same magnitude.

It should be understood that the formula for calculating the score listed here is only an example, and should not constitute any limitation to the application. This application does not exclude the possibility of using other calculation methods to calculate the score of the image to be selected. After determining the score of each frame of the image to be selected based on the above formula, the electronic device may determine the captured frame image corresponding to the first capture mode according to the score of each frame of the image to be selected. Here, the captured frame image corresponding to the first capture mode may be, for example, a frame with the highest score determined based on the scores of multiple frames of images to be selected. In other words, among the multiple frames of images to be selected, the score of the captured frame image corresponding to the first capture mode is higher than the score of any frame of the image to be selected except the captured frame image. The electronic device may, after determining the captured frame image corresponding to the first capture mode, save the captured frame image in the electronic device, or output to the display unit of the electronic device or the like. This application does not limit this.

In another implementation, each capture mode can correspond to multiple evaluation strategies. The multiple evaluation strategies in each capture mode can further correspond to multiple capture categories. For example, one or more capture categories corresponding to the sports capture mode may include at least one of the following: shooting, running, jumping, swimming, kicking, etc. The one or more capture categories corresponding to the facial expression capture mode include at least one of the following: happy, angry, sad, funny, etc. It should be understood that the above listing of the capture categories corresponding to each capture mode is only an example, and should not constitute any limitation to this application. This application does not limit the specific capture category corresponding to each capture mode.

In the embodiment of the present application, each type of capture may correspond to an evaluation strategy. In at least one evaluation strategy corresponding to the first snapping mode, each evaluation strategy includes one or more scoring parameters corresponding to the first snapping mode, the mode weight of each scoring parameter, and the mode weight corresponding to one snapping category. Category weight. In other words, in the first snapshot mode, each scoring parameter can be defined with a mode weight. Under each capture category corresponding to the first capture mode, each scoring parameter can be further defined with a category weight.

Optionally, different evaluation strategies corresponding to different capture categories include the same scoring parameters, and the different evaluation strategies include different category weights.

The category weights included in the different evaluation strategies mentioned here are different. Specifically, it may mean that the category weights applied to the same scoring parameter in different evaluation strategies are different. Moreover, when the evaluation strategy includes multiple scoring parameters, different evaluation strategies apply different category weights to at least one scoring parameter. In other words, different evaluation strategies corresponding to different capture categories may respectively include different category weights corresponding to the same scoring parameter.

For the convenience of description, the capture category determined by the electronic device according to the captured multiple frames of images is the first capture category. Based on the difference in the first capture category, the category weights corresponding to the same scoring parameter in the same capture mode are also different.

to make sure. Among them, β _i represents the category weight of the i-th scoring parameter, and β _i ≥0. The meanings of T _i and α _i have been explained above, for the sake of brevity, they will not be repeated here.

In the same snapshot mode, for the same scoring parameter, even if the mode weight is the same, the category weight can be different. For example, the first capture mode is a sports capture mode. The corresponding scoring parameters in the sports capture mode may include: posture height, posture stretch, and so on. The sports capture mode can include shooting, diving, swimming, running and other action categories.

When the first action category is shooting, the category weight of the posture height is higher than the category weight of the posture stretch degree, and in the posture stretch degree, the category weight of the elbow joint angle and the arm bending angle is higher than other categories (such as knee joint angle). , Ankle joint angle, etc.) have high category weights. When the first action category is diving, the category weight of the knee joint angle is higher than the category weight of the posture height, and the category weight of the posture height is higher than the category weight of the arm bending angle. When the first action category is swimming, the category weight of the leg bending angle is higher than the category weight of the arm bending angle, and the category weight of the arm bending angle is higher than the category weight of the posture height. When the first action category is the default category, the larger the joint angle, the more violent the human body action, which can be considered as a wonderful moment. Therefore, the category weight of the leg bending angle and the arm bending angle is higher than that of the posture height category. It should be understood that these examples are only for ease of understanding and should not constitute any limitation to the application. This application does not limit the distribution of the category weight of each scoring parameter under each capture category.

Optionally, the above step 210 may further include: determining the first capture category in the first capture mode according to the captured multiple frames of images. The above-mentioned step 220 may further include: using one or more scoring parameters corresponding to the first snapping mode and the mode weight of each scoring parameter, and the category weight of each scoring parameter corresponding to the first snapping category, to calculate The score of each frame of the image to be selected in the multi-frame image to be selected.

That is to say, in addition to determining the first capture mode, the electronic device may further determine the first capture category by calling the motion detection model or the face attribute detection model. The following examples illustrate the specific process of determining the first capture category based on the first capture mode, and scoring multiple frames of images to be selected according to the evaluation strategy corresponding to the first capture category.

Firstly, the electronic device may determine the first capture category based on the first capture mode.

For example, if the first capture mode is a motion capture mode, the electronic device can at least call the pose estimation model and the motion detection model to perform image recognition on the image to be selected. As mentioned above, the action detection model can recognize the action category based on the coordinate information of multiple posture points in each frame of the image to be selected input by the posture estimation model. The action detection model can construct the coordinate changes of each posture point during the movement process based on the coordinate information of the multiple posture points in each frame of the image to be selected to determine the first action category.

Since the specific process of determining the action category by the action detection model has been explained in step 210 above, for the sake of brevity, it will not be repeated here. It should be noted that if the action category determined by the action detection model according to the coordinate information of the posture point in the image to be selected does not belong to the predefined action category, the action category can be determined as the default category, or the default category. In the embodiment of the present application, when the action detection model does not detect a specific action category, the action category can be classified as a default category. The motion detection model can output the action category of the subject as the default category, more specifically, the default category in the sports capture mode.

If the first capture mode is an expression capture mode, the electronic device can at least call the face attribute detection model to perform image recognition on the image to be selected. As mentioned above, the face attribute detection model can determine the expression category based on the analysis of the feature points of each frame of the image to be selected.

Since the specific process of determining the facial expression category by the face attribute detection model has been explained in step 210 above, for the sake of brevity, it will not be repeated here. It should be noted that, if the expression category determined by the face attribute detection model according to the feature points in the image to be selected does not belong to the predefined action category, the expression category may be determined as the default category, or the default category. In the embodiment of the present application, when the face attribute detection model does not detect a specific expression category, the expression category may be classified as a default category. The face attribute detection model can output the expression category of the subject as the default category, more specifically, the default category in the expression capture mode. The determination of the value of each scoring parameter by the electronic device has been described in detail above in conjunction with an example, and for the sake of brevity, it will not be repeated here.

After the first capture category in the first capture mode is determined, the electronic device can use the corresponding evaluation strategy to substitute various scoring parameters to determine the score of each frame of the image to be selected. In this embodiment, since the capture category is further refined for the first capture mode, when the electronic device weights the value of each scoring parameter, it can further combine the attention to different details of the capture category to apply a category weight to each scoring parameter , To calculate a score that is more in line with the first snap category in the first snap mode.

For example, if the first capture mode is sports capture mode, and the first capture category is shooting, then the category weight can be applied to the elbow joint angle and the arm bending angle than other (such as knee joint angle, ankle joint angle, etc.) The category weight of the parameter is high, and the category weight applied to the attitude height can be higher than the category weight of the attitude stretch degree.

In this embodiment, the electronic device can use the evaluation strategy corresponding to the first capture category in the first capture mode, and substitute the previous value based on the scoring parameter determined for each frame of the image to be selected to calculate each frame of the image to be selected Rating. The scoring of each frame of the image to be selected by the electronic device can be based on the formula listed above, for example

It should be understood that the formula for calculating the score listed here is only an example, and should not constitute any limitation to the application. This application does not limit the possibility of using other calculation methods to calculate the score of the image to be selected.

After determining the score of each frame of the image to be selected based on the above formula, the electronic device may determine the captured frame image corresponding to the first capture mode according to the score of each frame of the image to be selected. Here, the captured frame image corresponding to the first capture mode may be, for example, a frame with the highest score determined based on the scores of multiple frames of images to be selected. In other words, among the multiple frames of images to be selected, the score of the captured frame image corresponding to the first capture mode is higher than the score of any frame of the image to be selected except the captured frame image.

The electronic device may, after determining the captured frame image corresponding to the first capture mode, save the captured frame image in the electronic device, or output to the display unit of the electronic device or the like. This application does not limit this. It should be noted that, in the same capture mode, corresponding to different capture categories, the category weight of some scoring parameters can also be defined as zero, or close to zero. When the category weight of a certain scoring parameter is defined as zero or close to zero, it can also be understood that the scoring parameter corresponding to the action category does not include the scoring parameter. From this perspective, the scoring parameters included in multiple evaluation strategies corresponding to the same capture mode are not necessarily the same. It should be understood that the multiple examples listed above for determining the score for each frame of the image to be selected are only for ease of understanding, and should not constitute any limitation to this application. Based on the same concept, those skilled in the art can make changes or substitutions to the above steps to achieve the same effect. This application does not limit the specific process for the electronic device to calculate the score for each frame of the image to be selected.

As mentioned above, the above-mentioned multi-frame images to be selected are the first N frames, the last N frames, or the front and back N frames at the moment when the electronic device presses the shutter, or a frame or frame whose score exceeds the preset threshold during the recording process. Multi-frame images. These images are stored in the electronic device, and can also be sent to the display unit and presented to the user. This application does not limit this.

In one implementation, each frame of image may correspond to a time stamp. After determining the captured frame image corresponding to the first capture mode, the electronic device can search for and obtain an image matching the time stamp of the captured frame image from the camera module. After the camera module encodes the image, it can be pushed to the display unit and presented to the user. At the same time, the image captured based on the user's photographing operation can also be saved in the user's photo album after being encoded. In order to facilitate the user to distinguish, the captured frame image and/or the frame of the image actually captured by the user can also be distinguished by marks, for example, leave a "best moment" or other similar on the captured frame image. Mark, or leave a different mark on the above two frames of images to show distinction.

In fact, the electronic device may also preprocess the captured multi-frame images before determining the first capture mode based on the captured multi-frame images, so as to obtain a more accurate estimation result. That is to say, the image input to the above one or more models may be the original image or the image after preprocessing, which is not limited in this application.

Optionally, the method 200 further includes: performing image preprocessing on the captured multiple frames of images. Specifically, the image processing module in the electronic device, such as ISP, can perform image preprocessing on the image. Image preprocessing may include, for example, cropping the size of the image of each frame to meet the input size of the pose estimation model; for example, it may include preprocessing the image of each frame, such as subtracting the mean and normalizing. , Data enhancement (such as rotation, etc.) processing. This application does not limit the specific content and implementation methods of image processing. The pre-processed image can meet the input size of the aforementioned multiple models, and at the same time, the diversity of the data can be enhanced to prevent the model from overfitting.

Based on the technical solution described above, by presetting a variety of different capture modes and their corresponding evaluation strategies, the capture mode can be determined according to the actual shooting scene. An evaluation strategy corresponding to the first capture mode can be selected from a plurality of preset evaluation strategies, and the capture frame image can be determined by using the evaluation strategy. For example, the introduction of scoring parameters such as posture stretch and posture height in the sports capture mode and multi-person sports capture mode, and the introduction of scoring parameters such as expression strength, open and closed eyes, face occlusion, and face angle in the expression capture mode and multi-person photo mode. , So that the determined capture frame image corresponding to the first capture mode can be selected based on the corresponding evaluation strategy, and higher mode weights are applied to the different scoring parameters concerned by different capture modes, so that the captured images are more realistic The shooting scene is conducive to obtaining the ideal capture effect, the flexibility is improved and it is suitable for more scenes.

In addition, the technical solution provided by the present application may further determine the weight of each scoring parameter based on different capture categories. For example, because different action categories focus on different focuses, among the multiple scoring parameters corresponding to the sports capture mode, the category weights of the scoring parameters configured for different action categories are different. In the motion capture mode, multiple frames of images to be selected are scored based on the category weights transmitted by each score corresponding to the action category, which is conducive to obtaining an ideal captured frame image. Compared with recommending a captured frame image based on optical flow information, the solution provided in this application pays more attention to the action itself, so that a better capturing effect can be obtained. Because the electronic device continuously detects the captured multi-frame images. The image captured by the electronic device changes based on the camera running time. Once it is detected that the image meets the triggering condition of another capture mode different from the first capture mode, it may switch to another capture mode.

Optionally, the method further includes: determining a second capture mode based on the newly captured multi-frame images, where the second capture mode is a capture mode that is different from the first capture mode among the foregoing preset multiple capture modes; And switch to the second snapshot mode. In order to avoid the ping-pong effect caused by frequent switching of electronic devices between multiple capture modes, a protection period can be set for each capture mode. The duration of the guard period can be a predefined value. The protection period of each capture mode can be of the same duration or different durations, which is not limited in this application. Therefore, before the electronic device switches to the second capture mode, it can be determined whether the running time of the first capture mode exceeds the preset protection period. If the operating time of the first capture mode is still within the protection period, regardless of whether the detection of the newly captured multi-frame images meets the trigger conditions of the second capture mode, the mode will not be switched, and the mode will remain in the first capture mode. run. However, if the running duration of the first capture mode exceeds the duration range of the protection period, it can be switched to the second switching mode based on the detection of the newly captured multi-frame images.

When a user uses an electronic device to take a photo, he can first turn on the camera in the electronic device and manually select a shooting mode. Figure 3 shows an example of a mobile phone interface. (A) in FIG. 3 shows the interface content 301 displayed by the system on the screen of the mobile phone when the mobile phone is in an unlocked state. The interface content 301 may include multiple icons, and the multiple icons may correspond to multiple applications (applications, apps), such as Alipay, Weibo, photo album, camera, WeChat, etc., which are not listed here.

If the user wants to take a photo or video with a mobile phone, he can first start the camera application by clicking the "camera" icon 302 on the user interface. The mobile phone interface display after starting the camera application may be as shown in (b) in FIG. 3, and this interface may be referred to as the shooting interface of the camera. The shooting interface may include a viewing frame 303, an album icon 504, a shooting control 305, and the like. Among them, the viewfinder frame 303 is used to obtain a photographed preview image, and can display the preview image in real time. It should be understood that the preview images described above are not necessarily saved in the album. However, in the embodiment of the present application, the preview image may be stored in the cache of the mobile phone or in other storage units, which is not limited in the present application.

The album icon 304 is used to quickly enter the album. When the mobile phone detects that the user clicks on the album icon, it can display the photos or videos that have been taken on the screen. The shooting control 305 can be used to take photos or videos. If the camera is in the photographing mode, when the mobile phone detects that the user clicks on the photographing control, the mobile phone performs the photographing operation and saves the photographed photos, which is the photographing stream described above. If the camera is in the video recording mode, when the mobile phone detects that the user clicks the shooting control, the mobile phone performs the video shooting operation; when the mobile phone detects that the user clicks the shooting control again, the video shooting ends. The phone can save the recorded video. In one implementation, the video can be saved by continuous multiple frames of images, that is, the video stream described above.

In addition, the shooting interface may also include a functional control 306 for setting a shooting mode, such as the portrait mode, the photo mode, the video mode, the panoramic mode, etc. shown in (b) of FIG. 3. The user can switch the shooting mode by clicking the function control. Optionally, the shooting interface may further include a camera rotation control 307, for example, as shown in (b) of FIG. 3. The camera rotation control 307 can be used to control the switching of the front camera and the rear camera.

It should be understood that FIG. 3 is only for ease of understanding, and a mobile phone is taken as an example to illustrate in detail the process of the user opening the camera function or other functions through operations. However, it should be understood that the mobile phone interface shown in FIG. 3 is only an example, and should not constitute any limitation to this application. Different operating systems and different brands of mobile phones may have different mobile phone interfaces. In addition, the embodiments of the present application can also be applied to other electronic devices that can be used for taking pictures except for mobile phones. And the interface shown in the figure is only an example, and should not constitute any limitation to this application.

In order to better understand the method provided by this application, a detailed description will be given below in conjunction with several specific embodiments. FIG. 4 is a schematic flowchart of a method for capturing an image according to another embodiment of the present application. In the method shown in FIG. 4, the user can manually adjust the camera mode to the smart capture mode. In response to the user's operation, the electronic device enters the smart capture mode. In other words, the embodiment shown in FIG. 4 mainly describes a method for an electronic device in the smart capture mode to capture an image.

It should be understood that the method shown in FIG. 4 may be executed by an electronic device or a processor in the electronic device. In step 401, the captured multi-frame images are periodically detected at a high frame rate. Specifically, the electronic device can continuously obtain multiple frames of images captured by the electronic device from the cache, and call the aforementioned detection models (such as the face attribute detection model, the pose estimation model, and the motion detection model) to cycle the captured images. Sexual detection to determine the first capture mode suitable for the current shooting. In the smart capture mode, the electronic device can use a higher frame rate for detection, such as 30 frames per second. Each detection model can be run alternately to reduce power consumption. It should be understood that, before exiting the smart capture mode, step 401 may run through the entire process from step 401 to step 410. It should also be understood that the electronic device calling the detection model to detect multiple frames of images is still being executed by the electronic device. Therefore, in this embodiment, for the sake of brevity, the process of calling the detection model to detect images will not be specifically described.

In step 402, it is determined that the trigger condition of the first capture mode is satisfied. Specifically, the electronic device can determine that a trigger condition of a certain capture mode is satisfied based on the detection of multiple frames of images. For example, the trigger condition of the first snapshot mode is satisfied. At this time, step 403 can be executed to enable the first capture mode. When the image does not meet the trigger condition of any one of the capture modes, step 401 can be continued to perform periodic detection of the captured multi-frame images at a high frame rate.

As the above method 200 has listed different trigger conditions for different capture modes, and combined the face attribute detection model, the pose estimation model and the motion detection model, the trigger condition for determining whether a certain capture mode is satisfied is described in detail. For the sake of brevity, I won't repeat them here. It should be noted that the specific capture mode enabled by the electronic device has no effect on the captured image. Electronic equipment only uses corresponding evaluation strategies to evaluate and recommend images based on specific capture modes.

In step 404, the electronic device may keep running in the first capture mode, and perform periodic detection of the newly captured image at a high frame rate. The electronic device may keep running in the first capture mode until it is determined that the trigger condition of another capture mode (for example, referred to as the second capture mode) is satisfied according to the detection of the newly captured image. It should be noted that step 404 is provided for the convenience of describing the following embodiments, and does not mean that the electronic device has performed a new operation. After enabling the first capture mode in step 403, the electronic device can always keep running in the first capture mode, and continuously perform periodic detection of newly captured images at a high frame rate. It should also be noted that after the first capture mode is enabled, the electronic device can run the first capture mode in the background without prompting the user through the shooting interface, so the user may not perceive it; the first capture mode can also be run in the foreground , The user can perceive through the shooting interface. This application does not limit this.

In step 405, it is determined that the trigger condition of the second snapshot mode is satisfied. Since the camera continues to operate, the electronic device can continuously detect the newly captured image. If it is not detected that the newly captured multi-frame image satisfies the trigger condition of another capture mode (such as the second capture mode), that is, it is determined that the trigger condition of the second capture mode is not met, step 404 can be performed to keep in the first capture mode. Mode, while continuously detecting the newly captured images periodically at a high frame rate.

If it is detected that the newly captured multi-frame images meet the trigger condition of the second capture mode, step 406 may be executed to determine whether the running time of the first capture mode exceeds the preset protection period. If the running time of the first capture mode does not exceed the preset protection period, step 404 may be executed. If the running time of the first capture mode exceeds the preset protection period, step 407 may be executed to enable the second capture mode. That is, the electronic device switches to the second capture mode. After enabling the second capture mode, the electronic device keeps running in the second capture mode until it is determined that the trigger condition of another capture mode (for example, the third capture mode) is satisfied according to the newly captured image. At the same time, the newly captured images are periodically detected at a high frame rate. For the sake of brevity, the figure does not show the step of determining that the electronic device meets the triggering condition of the third capture mode. However, it can be understood that when the trigger condition of the third capture mode is satisfied, the operation performed by the electronic device may be similar to the operation performed when the trigger condition of the third capture mode is satisfied in the figure. For the sake of brevity, it is not here. Go into details again.

It should be understood that the first capture mode and the second capture mode are different capture modes, the third capture mode and the second capture mode are different capture modes, but the first capture mode and the third capture mode may be the same capture mode, It can also be a different capture mode, which is not limited in this application. Regardless of whether it is switched to the second capture mode, the electronic device runs in the smart capture mode and continuously detects the captured images at a high frame rate.

If the electronic device detects the user's photographing operation, for example, the user clicks on the photographing control to perform the photographing operation, step 408 may be executed, and in response to the user's photographing operation, the photograph is taken and the image is saved. The saved image can be presented to the user through the display unit after subsequent processing such as encoding. If the electronic device does not detect the user's photographing operation, it can continue to run in the smart capture mode, and continuously capture new images for periodic detection at a high frame rate. It is understandable that the photographing operation may be an operation performed in the first snapshot mode, or it may be an operation performed in the second snapshot mode, depending on whether the electronic device switches to the second snapshot before performing the photographing operation mode. This application does not limit this.

In step 409, based on the currently running snapshot mode, a corresponding evaluation strategy is used to score the multi-frame images to be selected. Specifically, the electronic device may score the first N frames of images, the next N frames of images, or the previous and next N frames of images at the moment the shutter is pressed based on the user's photographing operation. Here the value of N can be predefined. This application does not limit the specific value of N. If N=20, 20 frames (ie, 40 frames in total) images before and after the moment the shutter is pressed can be scored.

The electronic device can call the corresponding detection model to perform image recognition on multiple frames of images according to the currently running capture mode, and output the recognition result. The electronic device can score each frame of the image to be selected according to the detection result. Optionally, the models related to the motion capture mode and the multi-person motion capture mode may include, for example, a pose estimation model and a motion detection model. Optionally, the detection model related to the facial expression capture mode and the group photo mode may include, for example, a face attribute detection model. The face attribute model may include, but is not limited to, a face feature point detection model, an open and closed eye model, etc., for example. This application does not limit this.

Assuming that the currently running capture mode is a motion capture mode, the electronic device can call the pose estimation model and the motion detection model to perform image recognition on the captured images to be selected, so as to obtain the coordinate information of the pose points and the action category. The electronic device can determine an evaluation strategy according to the motion capture mode and the action category, and use the evaluation strategy to score each frame of captured images to be selected.

Assuming that the currently running capture mode is the expression capture mode, the electronic device can call the face attribute detection model to perform image recognition on the captured images to be selected to obtain one of expression strength, eyes open and closed, face occlusion, and face angle Or multiple recognition results. The recognition result may include an expression category and related parameters that can be used to characterize one or more of expression strength, eye opening and closing, facial occlusion, and face angle. The electronic device can determine an evaluation strategy according to the expression capture mode and expression category, and use the evaluation strategy to score each captured image to be selected.

In step 410, the captured frame image corresponding to the currently running capture mode is determined. The electronic device can determine the captured frame image corresponding to the currently running capture mode according to the score of each frame to be selected. The electronic device may further obtain an image matching the timestamp from the cache, process the image, and present it to the user through the display unit.

Thereafter, the electronic device can clean up the cache space occupied during scoring, and release the occupied cache space. Optionally, the user can exit the smart capture mode through manual adjustment, or exit the camera function directly. In response to the user's operation, the electronic device also exits the smart capture mode.

Optionally, the electronic device that exits the smart capture mode can still remain in the photo mode, and can perform periodic detection of the image at a low frame rate. Optionally, the electronic device that exits the camera function can stop acquiring images and stop detecting. Each detection model can also be stopped.

It should be understood that the above is only for ease of understanding, and only the first capture mode and the second capture mode are shown, but this should not constitute any limitation to the present application. As the operating time of the camera increases, the electronic device can continuously acquire newly captured images and continuously detect the newly captured images. Therefore, as long as the electronic device does not have a camera function, some or all of the above steps 404 to 410 can be executed in a loop. What needs to be noted is that after switching the first capture mode to the second capture mode, the second capture mode becomes the new first capture mode.

It should also be understood that FIG. 4 shows an example in which the method for capturing an image provided by an embodiment of the present application is applied to a specific scene. The steps in the figure are only shown for ease of understanding. Each step in the flowchart does not necessarily have to be performed. For example, some steps can be skipped, or some steps can be combined. The execution order of each step is not fixed, nor is it limited to that shown in FIG. 4. The execution sequence of each step should be determined by its function and internal logic.

FIG. 5 is a schematic flowchart of a method for capturing an image according to another embodiment of the present application. In the method shown in FIG. 5, the user can open the smart capture mode without manual operation. For example, the user can set the shooting mode to the camera mode. The electronic device can be based on periodic detection, and can automatically turn on the smart capture mode. In other words, the embodiment shown in FIG. 5 mainly describes a method for an electronic device in a photographing mode to photograph an image.

It should be understood that the method shown in FIG. 5 may be executed by an electronic device or a processor in the electronic device. In step 501, the captured image is periodically detected at a low frame rate. Specifically, the electronic device can continuously obtain multiple frames of images captured by the electronic device from the cache, and call the aforementioned detection models (such as the face attribute detection model, the human frame detection model, and the scene recognition model) to perform periodic detection to determine Whether the trigger condition for entering a certain capture mode is met. In the camera mode, the electronic device can use a lower frame rate for detection, such as 15 frames per second, to save power consumption. It should be understood that, before entering the smart capture mode, such as step 503, step 501 can be performed continuously. After exiting the smart capture mode, such as after step 509, step 501 can also be continued. It should also be understood that the electronic device calling the detection model to detect multiple frames of images is still being executed by the electronic device. Therefore, in this embodiment, for the sake of brevity, the process of calling the detection model to detect images will not be specifically described.

In step 502, it is determined that the trigger condition of the first capture mode is satisfied. Specifically, the electronic device can determine that a trigger condition of a certain capture mode is satisfied based on the detection of multiple frames of images. For example, the trigger condition of the first snapshot mode is satisfied. In the above method 200, different triggering conditions have been listed for different capture modes, and combined with the face attribute detection model, the human frame detection model and the scene recognition model, the trigger condition for determining whether a certain capture mode is satisfied is described in detail. For the sake of brevity, I won't repeat them here.

If it is determined that the detected multi-frame images do not meet the triggering condition of a certain capture mode, the operation in the photographing mode can be continued, that is, step 501 is continued. If it is determined that the detected multiple frames of images meet the trigger condition of the first capture mode, step 503 may be executed to enter the smart capture mode, and the first capture mode is enabled. Enabling the first capture mode means that the electronic device is switched to the smart capture mode. Therefore, entering the smart capture mode and enabling the first capture mode refer to the same operation. In addition, the electronic device may also perform periodic detection of the newly captured image at a high frame rate in step 503. In other words, the detection of the image by the electronic device switches from a low frame rate to a high frame rate. For example, the electronic device can detect images at a frame rate of 30 frames per second. It should be understood that before the electronic device exits the smart capture mode, step 501 may be continuously performed to perform periodic detection of newly captured images at a high frame rate.

In step 504, the electronic device keeps running in the first snapshot mode. The electronic device may keep running in the first capture mode until it is determined that the trigger condition of another capture mode (for example, referred to as the second capture mode) is satisfied according to the detection of the newly captured image. It should be noted that step 504 is set for the convenience of describing the following embodiments, and does not mean that the electronic device has performed a new operation. After enabling the first capture mode in step 503, the electronic device can always keep running in the first capture mode, and continuously perform periodic detection of newly captured images at a high frame rate.

In step 505, it is determined that the trigger condition of the second snapshot mode is satisfied. Since the camera continues to operate, the electronic device can continuously detect the newly captured image. If it is not detected that the newly captured image satisfies the trigger condition of another capture mode (such as the second capture mode), that is, it is determined that the trigger condition of the second capture mode is not satisfied, step 504 may be performed to maintain the first capture mode. At the same time, the image is continuously inspected periodically at a high frame rate. If it is determined that it is detected that the newly captured image meets the trigger condition of the second capture mode, it may be considered whether to switch to the second capture mode.

In order to avoid the ping-pong effect caused by frequent switching of electronic devices between multiple capture modes, a protection period can be set for each capture mode. The duration of the protection period may be a predefined value, and step 506 is executed to determine whether the running duration of the first capture mode exceeds the protection period. If the running time of the first capture mode does not exceed the protection period, that is, the electronic device detects that the image meets the trigger condition of the second capture mode within the protection period, then step 504 may be performed to maintain the first capture mode. The electronic device kept in the first capture mode can still continue to periodically detect the image at a high frame rate.

If the operating time of the first capture mode exceeds the protection period, that is, the electronic device detects that the newly captured image meets the trigger condition of the second capture mode outside the protection period, then step 507 may be executed to enable the second capture mode , In other words, switch from the first capture mode to the second capture mode. The electronic device with the second capture mode enabled can also continuously perform periodic detection of newly captured images at a high frame rate. After the second capture mode is enabled, the electronic device can keep running in the second capture mode until it is determined that the trigger condition of another capture mode (for example, the third capture mode) is satisfied according to the newly captured image. For the sake of brevity, the figure does not show the step of determining that the trigger condition of the third capture mode is satisfied. However, it can be understood that when the trigger condition of the third capture mode is satisfied, the operation performed by the electronic device may be similar to the operation performed when the trigger condition of the third capture mode is satisfied in the figure. For the sake of brevity, it is not here. Go into details again. Regardless of whether it is switched to the second capture mode, as long as the electronic device is running in the smart capture mode, the captured image can be continuously detected at a high frame rate.

In step 508, it is determined whether a photographing operation is detected within a preset time period. Since the detection of the newly captured image in the smart capture mode is a high frame rate detection, the power consumption is relatively high. In order to reduce power consumption, the electronic device can automatically exit the smart capture mode if it does not detect a photographing operation for a long time. The electronic device that exits the smart capture mode can return to the photo mode. The user may not perceive that the electronic device exits the smart capture mode.

Specifically, within a preset period of time after the first capture mode or the second capture mode is activated, if no photographing operation is detected, step 509 may be executed to exit the smart capture mode and return to the photographing mode. Furthermore, step 501 and the steps thereafter can be repeated until the user exits the camera function. The duration of the preset time period may be a predefined value. For example, when the electronic device activates the first capture mode in step 503 or the second capture mode in step 507, it can start timing. For example, a timer is started, and the running time of the timer may be, for example, the aforementioned preset time period. When the user's photographing operation is not detected within the preset time period, for example, the timer runs overtime, step 509 may be executed to exit the smart capture mode and return to the photographing mode. If the running time of the timer has not reached, it can continue to run in the smart capture mode. If the user's photographing operation is detected within the preset time period, step 510 may be performed, and in response to the user's photographing operation, the photograph is taken and the image is saved.

In step 511, based on the currently running snapshot mode, a corresponding evaluation strategy is used to score multiple frames of images to be selected. In step 512, the captured frame image corresponding to the currently running capture mode is determined. The specific process from step 510 to step 512 is the same as the specific process from step 406 to step 408 in the above embodiment. Since step 406 to step 408 have been described in detail above, for the sake of brevity, it will not be repeated here. Thereafter, if the user performs an operation to exit the camera, the electronic device can exit the camera function in response to the user's operation. The electronic device that exits the camera function stops acquiring images and stops detecting. Each detection model can also be stopped.

It should be understood that the duration of the guard period in step 506 and the duration of the preset period described in step 508 may be the same or different, which is not limited in this application. If the two are the same, you can share a timer; if they are different, you can use their own independent timers. Of course, counting by a timer is only one possible implementation, and should not constitute any limitation to this application.

It should be understood that part of the steps and descriptions in FIG. 5 can refer to the related descriptions in FIG. It should also be understood that FIG. 5 shows an example in which the method for capturing an image provided by an embodiment of the present application is applied to a specific scene. The steps in the figure are only shown for ease of understanding. Each step in the flowchart does not necessarily have to be performed. For example, some steps can be skipped, such as step 505 to step 507, or some steps can be combined, such as step 503 and step 504. The order of execution of each step is not fixed, nor is it limited to that shown in FIG. 5. The execution sequence of each step should be determined by its function and internal logic.

FIG. 6 is a schematic flowchart of a method for capturing an image according to another embodiment of the present application. In the method shown in FIG. 6, the user can open the smart capture mode without manual operation. For example, the user can set the shooting mode to the video recording mode. The electronic device can be based on periodic detection, and can automatically turn on the smart capture mode. In other words, the embodiment shown in FIG. 6 mainly describes a method for an electronic device in a video recording mode to capture an image.

It should be understood that the method shown in FIG. 6 may be executed by an electronic device or a processor in the electronic device. In step 601, the captured multi-frame images are periodically detected at a low frame rate. In step 602, it is determined that the trigger condition of the first snapshot mode is satisfied. If the electronic device determines that the trigger condition of the first capture mode is satisfied according to the detection of the captured multi-frame images, step 603 may be executed to enable the first capture mode while recording. The first snapshot mode may be a mode running in the background. Judging from the shooting interface, the electronic device is still continuing to record video. Enabling the first capture mode means that the electronic device has enabled the smart capture mode. In other words, the video mode and smart capture mode run in parallel. In addition, the electronic device starts to periodically detect newly captured images at a high frame rate. That is to say, the detection of the image by the electronic device switches from a low frame rate to a high frame rate. For example, the electronic device can detect the preview image at a frame rate of 30 frames per second. And before the electronic device exits the smart capture mode, it can continuously perform periodic detection of newly captured images at a high frame rate.

It should also be understood that the electronic device calling the detection model to detect multiple frames of images is still being executed by the electronic device. Therefore, in this embodiment, for the sake of brevity, the process of calling the detection model to detect images will not be specifically described. In step 604, the electronic device keeps running the first snapshot mode in the background. The electronic device may keep running the first capture mode in the background until it is determined that the trigger condition of another capture mode (for example, referred to as the second capture mode) is satisfied according to the detection of the newly captured image. It should be noted that step 604 is set for the convenience of describing the following embodiments, and does not mean that the electronic device has performed a new operation. After enabling the first capture mode in step 603, the electronic device can always keep running the first capture mode in the background, and continuously perform periodic detection of newly captured images at a high frame rate.

In step 605, it is determined that the trigger condition of the second snapshot mode is satisfied. Since the camera continues to operate, the electronic device can continuously detect the newly captured image. If it is not detected that the newly captured image meets the trigger condition of another capture mode (such as the second capture mode), that is, it is determined that the trigger condition of the second capture mode is not met, step 604 can be performed to keep the first capture running in the background Mode, while continuously detecting the newly captured images periodically at a high frame rate. If it is detected that the newly captured image meets the trigger condition of the second capture mode, it may be considered whether to switch to the second capture mode.

In order to avoid the ping-pong effect caused by frequent switching of electronic devices between multiple capture modes, a protection period can be set for each capture mode. The duration of the protection period may be a predefined value, and step 606 is executed to determine whether the running duration of the first capture mode exceeds the protection period. If the running time of the first capture mode does not exceed the protection period, that is, the electronic device detects that the newly captured image meets the trigger condition of the second capture mode within the protection period, then step 604 may be performed to keep the first capture mode mode. The electronic device remaining in the first capture mode can still continue to periodically detect the newly captured image at a high frame rate. If the running time of the first capture mode exceeds the protection period, that is, the electronic device detects that the image meets the trigger condition of the second capture mode outside the protection period, step 607 may be executed to enable the second capture mode, or in other words , Switch from the first capture mode to the second capture mode. When the electronic device has enabled the second capture mode, the electronic device can also continuously perform periodic detection of newly captured images at a high frame rate. After the second capture mode is enabled, the electronic device can keep running the second capture mode until it is determined that the trigger condition of another capture mode (for example, the third capture mode) is satisfied according to the newly captured image. For the sake of brevity, the figure does not show the step of determining that the trigger condition of the third capture mode is satisfied. However, it can be understood that when the trigger condition of the third capture mode is satisfied, the operation performed by the electronic device may be similar to the operation performed when the trigger condition of the third capture mode is satisfied in the figure. For the sake of brevity, it is not here. Go into details again. Regardless of whether it is switched to the second capture mode, as long as the smart capture mode is still running in the background of the electronic device, the captured image can be continuously detected at a high frame rate.

In step 608, it is determined whether a photographing operation is detected within a preset time period. Since the detection of the newly captured image in the smart capture mode is a high frame rate detection, the power consumption is relatively high. In order to reduce power consumption, the electronic device can automatically exit the smart capture mode if it does not detect a photographing operation for a long time. Electronic devices that exit the smart capture mode can still continue to record. The user may not perceive that the electronic device exits the smart capture mode. In the preset time period after the second snapping mode is activated, if no photographing operation is detected, step 609 can be executed to exit the smart snapping mode, and the video recording mode remains running. Furthermore, step 601 and the steps thereafter can be repeated until the user exits the camera function. Within a preset period of time after the first capture mode or the second capture mode is activated, if a photographing operation is detected, step 610 may be executed, and in response to the user's photographing operation, the photograph is taken and the image is saved.

In step 611, based on the currently running snapshot mode, a corresponding evaluation strategy is used to score multiple frames of images to be selected. In step 612, the captured frame image corresponding to the currently running capture mode is determined.

In another implementation manner, after the first capture mode is enabled in step 603 or the second capture mode is enabled in step 607, the electronic device can continuously perform image recognition and scoring on each frame of image, and when the score result exceeds the preset When the threshold is set, this frame of image is recommended to the user. When there are more than one frame of images that exceed the preset threshold, the image with the highest score can be recommended to the user. In this case, step 610 to step 612 may not be performed. In this case, the images to be selected may refer to all the images acquired by the electronic device during the recording process.

It should be understood that the steps in FIG. 6 are similar to the steps described above in conjunction with FIG. 5, and the specific process can be referred to the relevant description above. For the sake of brevity, details are not repeated here. It should also be understood that FIG. 6 shows an example in which the method for capturing an image provided by an embodiment of the present application is applied to a specific scene. The steps in the figure are only shown for ease of understanding. Each step in the flowchart does not necessarily have to be performed. For example, some steps can be skipped, such as step 605 to step 607, step 610 to step 612, or some steps can be combined, such as step 603 And step 604. The order of execution of each step is not fixed, nor is it limited to that shown in FIG. 6. The execution sequence of each step should be determined by its function and internal logic.

The foregoing describes the method for capturing images provided in the embodiments of the present application in combination with multiple possible scenarios. However, it should be understood that these scenarios should not constitute any limitation to the scenarios applicable to this application. When the electronic device can manually adjust the specific capture mode to be selected, the preview of the scoring parameter corresponding to the capture mode based on the motion capture mode, the expression capture mode, the multi-person sports capture mode, and the group photo capture mode provided in the embodiments of the present application The process of image scoring and optimal frame recommendation can also be used separately.

For example, in the sports capture mode, based on the scoring parameters and mode weights corresponding to the sports capture mode, including posture height, posture stretch, etc., the images to be selected are evaluated and recommended. Moreover, the category weight of each scoring parameter can be further determined according to the action category, so as to determine the captured frame images matching different action categories. The captured frame image thus determined pays more attention to the details of the action, and it is more likely to find the wonderful image at the moment of movement and recommend it to the user. Therefore, the captured image is more in line with the shooting scene, and the capture effect is better.

Based on the technical solution described above, by presetting multiple different capture modes and their corresponding one or more evaluation strategies, different scoring parameters and mode weights can be used to score the selected images according to different capture modes. For example, in sports capture mode and multi-person sports capture mode, the introduction of posture stretch, posture height, body occlusion and other scoring parameters, in the expression capture mode, multi-person photo mode, the introduction of expression intensity, eyes closed, face occlusion, and face angle The scoring parameters can be selected so that the captured frame images recommended to the user can be selected based on the evaluation strategies corresponding to different capturing modes.

In addition, the technical solution provided by the present application may further determine the category weight of each scoring parameter based on different capture categories. For example, because different capture categories focus on different focuses, among the multiple scoring parameters corresponding to the capture mode, the same scoring parameter configured for different capture categories has different category weights. Therefore, it is conducive to obtain an ideal captured frame image.

Especially in the sports capture mode, a higher mode weight is assigned to the scoring parameters such as the posture height and posture stretch that express the details of the action. And the same scoring parameter configured for different action categories can have different category weights according to the different attention details to different action categories. Compared with recommending the optimal frame based on optical flow information, the solution provided in this application pays more attention to the action itself, so that a better motion capture effect can be obtained.

Above, the method for capturing images provided by the embodiments of the present application has been described in detail with reference to FIGS. 2 to 6. Hereinafter, the image capturing apparatus provided by the embodiment of the present application will be described in detail with reference to FIG. 7.

FIG. 7 is a schematic block diagram of an image capturing apparatus 700 provided by an embodiment of the present application. As shown in FIG. 7, the apparatus 700 may include a mode determining unit 710 and a captured frame determining unit 720.

Specifically, the mode determining unit 710 is configured to determine the first capturing mode among the preset multiple capturing modes according to the captured multi-frame images; the capturing frame determining unit 720 is configured to use the first capturing mode corresponding to the The evaluation strategy is to determine the captured frame image corresponding to the first capture mode among the captured multiple frames to be selected; the evaluation strategy is one of a plurality of preset evaluation strategies.

Optionally, the multiple capture modes include one or more of the following: facial expression capture mode, group photo capture mode, sports capture mode, multi-person sports capture mode, pet capture mode, and landscape capture mode. Optionally, each of the multiple capture modes corresponds to at least one evaluation strategy, and each evaluation strategy includes one or more scoring parameters for image scoring and a mode weight of each scoring parameter.

Optionally, the capture frame determination unit 720 is configured to use one or more scoring parameters in one of at least one of the multiple preset evaluation strategies corresponding to the first capture mode and each of the evaluation strategies. The weight of the mode weight of each scoring parameter is used to calculate the score of each frame of the image to be selected in the multi-frame image to be selected; and used to determine the score of the image in the multi-frame image to be selected according to the multiple scores of the image to be selected. A snapshot frame image corresponding to a snapshot mode.

Optionally, the captured frame image has the highest score among the multiple frames of images to be selected. Optionally, different evaluation strategies corresponding to different capture modes include the same scoring parameters, and different evaluation strategies include different mode weights.

Optionally, each capture mode includes one or more capture categories, and each capture category corresponds to an evaluation strategy; among at least one evaluation strategy corresponding to the first capture mode, each evaluation strategy includes One or more scoring parameters corresponding to the first snapping mode, the mode weight of each scoring parameter, and the category weight corresponding to one snapping category. Optionally, the mode determining unit 710 is further configured to determine the first capturing category in the first capturing mode according to the multi-frame images; the capturing frame determining unit 720 is configured to use one corresponding to the first capturing mode Or multiple scoring parameters and the mode weight of each scoring parameter, and the category weight of each scoring parameter corresponding to the first snapped category, and calculating the score of each frame of the image to be selected in the multiple frames of image to be selected. Optionally, different evaluation strategies corresponding to different capture categories include the same scoring parameters, and different evaluation strategies include different category weights.

Optionally, the capture frame determination unit 720 is further configured to call at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected to output a recognition result; and to determine the recognition result based on the recognition result. The value of one or more scoring parameters.

Optionally, when the first capture mode is a motion capture mode or a multi-person motion capture mode, the at least one detection model includes a pose estimation model and a motion detection model. Optionally, the first capture mode is an expression capture mode or a group photo capture mode, and the at least one detection model includes a face attribute detection model.

Optionally, the mode determining unit 710 is further configured to perform mode detection on the captured multi-frame image based on the first frame rate to determine the first capturing mode among the preset multiple capturing modes; the capturing frame determining unit 720 is also used to call at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected at a second frame rate; wherein, the first frame rate is less than the second frame rate.

Specifically, the apparatus 700 may include a unit for executing the method executed by the electronic device in the embodiment of the method 200 in FIG. 2. The mode determining unit 710 may be used to perform step 210 in the above method 200, and the captured frame determining unit 720 may be used to perform step 220 to step 240 in the above method 200. In addition, the device 700 may also include one or more detection models. In a specific implementation process, the mode determination unit 710 may call the one or more detection models for image detection; the captured frame determination unit 720 may also call the one or more detection models for image recognition.

The apparatus 700 can also be used to execute the method executed by the electronic device in the embodiments in FIGS. 4 to 6. In addition, each unit in the device 700 and other operations and/or functions described above are used to implement the corresponding processes of the embodiments in FIG. 4 to FIG. 6, respectively. For the sake of brevity, I will not go into details here.

It should be understood that the image capturing apparatus 700 may correspond to at least part of the electronic device in the method embodiment according to the embodiment of the present application. For example, the apparatus 700 may be the electronic device, or a component in the electronic device, such as a chip or a chip system. Specifically, the functions implemented by the image capturing apparatus 700 may be implemented by one or more processors executing corresponding programs.

The application also provides an electronic device or its internal device 700. The electronic device or apparatus 700 may include one or more processors to implement the functions of the image capturing apparatus 700 described above. The one or more processors may, for example, include or execute the mode determination unit, the capture frame determination unit, and one or more detection models described in the above embodiments. The one or more processors may correspond to the processor 110 in the electronic device 100 shown in FIG. 1, for example. The mode determination unit, the capture frame determination unit, and one or more detection models may be software, hardware, or a combination thereof. The software may be executed by a processor, and the hardware may be embedded in the processor.

Optionally, the electronic device or device 700 further includes one or more memories. The one or more memories are used to store computer programs and/or data, such as images captured by a camera. The one or more memories may correspond to the memory 120 in the electronic device 100 shown in FIG. 1, for example. Optionally, the electronic device may also include a camera, a display unit, and the like. The camera may correspond to the camera 190 in the electronic device 100 shown in FIG. 1, for example. The display unit may, for example, correspond to the display unit 170 in the electronic unit 100 shown in FIG. 1. The processor may obtain the computer program stored in the memory to execute the method flow involved in the above embodiment. The memory further includes the one or more preset detection models, so that the processor can obtain the one or more detection models from the memory.

The present application also provides a computer storage medium that stores computer instructions that, when run on an electronic device, cause the electronic device to execute the above-mentioned related method steps to implement the image capturing method in the above-mentioned embodiment. The computer storage medium may, for example, correspond to the memory 120 in the electronic device 100 shown in FIG. 1. The mode determination unit, the capture frame determination unit, and one or more detection models involved in the embodiment of FIG. 7 may exist in the form of software and be stored in the computer storage medium.

The present application also provides a computer program product, which can be stored in the computer storage medium, and when the computer program product runs on a computer, the computer is caused to execute the above-mentioned related steps to implement the image capturing method in the above-mentioned embodiment.

Among them, the image capturing device, electronic device, computer storage medium, computer program product, or chip provided in the embodiments of the present application are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the above The beneficial effects of the corresponding methods provided in the article will not be repeated here.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in the embodiments disclosed in this specification can be implemented by computer software, electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A method for photographing an image, characterized in that it comprises:

According to the captured multi-frame images, determine the first capture mode among the preset multiple capture modes;

Use the evaluation strategy corresponding to the first capture mode to determine the captured frame image corresponding to the first capture mode among the captured multiple frames to be selected, and the evaluation strategy is one of the preset multiple evaluation strategies Kind of.
The method of claim 1, wherein the multiple capture modes include one or more of the following: facial expression capture mode, group photo capture mode, sports capture mode, multiplayer sports capture mode, pet capture mode, and landscape Snapshot mode.
The method of claim 1 or 2, wherein each of the multiple capture modes corresponds to at least one of the preset multiple evaluation strategies, and each evaluation strategy includes One or more scoring parameters used for image scoring and the mode weight of each scoring parameter; and

The using the evaluation strategy corresponding to the first capture mode to determine the captured frame image corresponding to the first capture mode among the captured multiple frames to be selected includes:

Using one or more scoring parameters in one of the at least one evaluation strategy corresponding to the first capture mode and the mode weight of each scoring parameter to calculate each frame of the multi-frame to-be-selected image Scoring of selected images;

According to multiple scores for the multiple frames of images to be selected, a captured frame image corresponding to the first capture mode is determined among the multiple frames of images to be selected.
The method according to claim 3, wherein the captured frame image has the highest score among the multiple frames to be selected.
The method according to claim 3 or 4, wherein different evaluation strategies corresponding to different capture modes include the same scoring parameters, and the different evaluation strategies include different mode weights.
The method according to any one of claims 3 to 5, wherein each capture mode includes one or more capture categories, and each capture category corresponds to an evaluation strategy; in comparison with the first capture mode In the at least one corresponding evaluation strategy, each evaluation strategy includes one or more scoring parameters corresponding to the first snapping mode, a mode weight of each scoring parameter, and a category weight corresponding to one snapping category; and

The determining the first capture mode among the preset multiple capture modes further includes: determining the first capture category in the first capture mode according to the multiple frames of images;

The use of one or more scoring parameters in one of the at least one evaluation strategy corresponding to the first capture mode and the mode weight of each scoring parameter are used to calculate each of the multiple frames of images to be selected. The scoring of the frame to be selected, including:

Using one or more scoring parameters corresponding to the first snapping mode, the mode weight of each scoring parameter, and the category weight of each scoring parameter corresponding to the first snapping category, calculate the multi-frame waiting The score of each frame of the image to be selected in the selection image.
7. The method according to claim 6, wherein different evaluation strategies corresponding to different capture categories include the same scoring parameters, and the different evaluation strategies include different category weights.
The method according to any one of claims 3 to 7, wherein the method further comprises:

Calling at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected to output a recognition result;

The value of the one or more scoring parameters is determined based on the recognition result.
The method of claim 8, wherein when the first capture mode is a motion capture mode or a multi-person motion capture mode, the at least one detection model includes a pose estimation model and a motion detection model; or

The first capture mode is an expression capture mode or a group photo capture mode, and the at least one detection model includes a face attribute detection model.
The method according to claim 8 or 9, wherein the determining the first capture mode among the preset multiple capture modes according to the captured multi-frame images comprises:

Performing mode detection on the captured multi-frame images based on the first frame rate, so as to determine the first capturing mode among the preset multiple capturing modes;

The invoking at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected includes:

Calling at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected at a second frame rate;

Wherein, the first frame rate is less than the second frame rate.
An image capturing device, characterized in that it comprises:

The mode determining unit is configured to determine the first capture mode among the preset multiple capture modes according to the captured multi-frame images;

The capture frame determination unit is configured to use an evaluation strategy corresponding to the first capture mode to determine a capture frame image corresponding to the first capture mode among the captured multiple frames to be selected, and the evaluation strategy is a pre-selection One of the multiple evaluation strategies for setting up.
The device of claim 11, wherein the multiple capture modes include one or more of the following: facial expression capture mode, group photo capture mode, sports capture mode, multiplayer sports capture mode, pet capture mode, and landscape capture mode. Snapshot mode.
The device according to claim 11 or 12, wherein each of the multiple capture modes corresponds to at least one of the preset multiple evaluation strategies, and each evaluation strategy includes One or more scoring parameters used for image scoring and the mode weight of each scoring parameter;

The capture frame determination unit is specifically configured to use one or more scoring parameters in one of the at least one evaluation strategy corresponding to the first capture mode and the mode weight of each scoring parameter to calculate the The score of each frame of the images to be selected in the multiple frames of images to be selected; and used to determine the image corresponding to the first capture mode in the multiple frames of images to be selected according to the multiple scores of the images to be selected Snap a frame image.
The device of claim 13, wherein the captured frame image has the highest score among the multiple frames to be selected.
The device according to claim 13 or 14, wherein different evaluation strategies corresponding to different capture modes include the same scoring parameters, and the different evaluation strategies include different weights.
The device according to any one of claims 13 to 15, wherein each capture mode includes one or more capture categories, and each capture category corresponds to an evaluation strategy; in comparison with the first capture mode In the at least one corresponding evaluation strategy, each evaluation strategy includes one or more scoring parameters corresponding to the first snapping mode, a mode weight of each scoring parameter, and a category weight corresponding to one snapping category;

The mode determining unit is further configured to determine a first capture category in the first capture mode according to the multiple frames of images;

The capture frame determination unit is further configured to use one or more scoring parameters corresponding to the first capture mode, the mode weight of each scoring parameter, and the value of each scoring parameter corresponding to the first capture category. The category weight is used to calculate the score of each frame of the image to be selected in the multiple frames of image to be selected.
16. The device of claim 16, wherein different evaluation strategies corresponding to different capture categories include the same scoring parameters, and different evaluation strategies include different category weights.
The device according to any one of claims 13 to 17, wherein the device further comprises one or more detection models;

The capture frame determination unit is further configured to call at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected to output a recognition result; and to determine the recognition result based on the recognition result. The value of one or more scoring parameters.
The device of claim 18, wherein when the first capture mode is a motion capture mode or a multi-person motion capture mode, the at least one detection model includes a pose estimation model and a motion detection model; or

The first capture mode is an expression capture mode or a group photo capture mode, and the at least one detection model includes a face attribute detection model.
The device according to claim 18 or 19, wherein the mode determining unit is specifically configured to perform mode detection on the captured multi-frame image based on the first frame rate, so as to perform mode detection on the preset multiple capture modes Determine the first snapshot mode in the middle;

The capture frame determination unit is specifically configured to call at least one detection model corresponding to the first capture mode to perform image recognition on the multiple frames of images to be selected at a second frame rate;

Wherein, the first frame rate is less than the second frame rate.
A device for photographing an image, comprising a processor for calling and running a computer program from a memory to execute the method according to any one of claims 1 to 10.
A computer-readable storage medium, characterized by comprising a computer program, which when running on an electronic device or a processor, causes the electronic device or the processor to execute as described in any one of claims 1 to 10 The method described.