WO2024239792A1 - 运动抓拍方法及电子设备 - Google Patents

运动抓拍方法及电子设备 Download PDF

Info

Publication number
WO2024239792A1
WO2024239792A1 PCT/CN2024/083747 CN2024083747W WO2024239792A1 WO 2024239792 A1 WO2024239792 A1 WO 2024239792A1 CN 2024083747 W CN2024083747 W CN 2024083747W WO 2024239792 A1 WO2024239792 A1 WO 2024239792A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
electronic device
frame
exposure
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/083747
Other languages
English (en)
French (fr)
Inventor
吴天航
郑耀国
杨坤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honor Device Co Ltd
Original Assignee
Honor Device Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202310590214.8A external-priority patent/CN117692761B/zh
Priority claimed from CN202310595470.6A external-priority patent/CN117692781B/zh
Application filed by Honor Device Co Ltd filed Critical Honor Device Co Ltd
Publication of WO2024239792A1 publication Critical patent/WO2024239792A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders

Definitions

  • the present application relates to the field of electronic technology, and in particular to a motion capture method and an electronic device.
  • the present application provides a motion capture method and an electronic device.
  • the present application provides a motion capture method, which may include: an electronic device starts a first camera, displays a preview image on a display screen, the preview image comes from an image captured by the first camera, and the electronic device caches the latest captured multiple frames of images (also called cached frames).
  • the electronic device determines whether there is a moving object in the preview image. If there is a moving object, the exposure time of the first camera is reduced during preview, for example, the exposure time is reduced to a shorter than the light energy cycle.
  • the electronic device can detect a photo operation, such as a shutter button being pressed, and in response to the photo operation, the electronic device saves the photo, which may come from the cached frame.
  • the activation of the first camera may be triggered by an event that the user opens a camera application.
  • the exposure time of the first camera may be a first duration, which is relatively long and not suitable for capturing moving objects, and prone to motion smear.
  • the first duration may be greater than the light energy period and an integer multiple of the light energy period, such as 50ms.
  • the electronic device reduces the exposure time and enters the short exposure mode when it detects a moving object during preview, rather than reducing the exposure time after the user presses the shutter.
  • the cached preview frame has the effect of reducing exposure, and once a photo operation is detected, the electronic device can obtain the photo to be saved based on the cached preview frame without waiting for the short exposure time to take effect before the photo can be output, so the problems of slow imaging speed and shutter lag are improved.
  • the electronic device when reducing the exposure time of the first camera, may further detect the movement speed (also a relative speed) of the moving object in the preview image, and control the exposure reduction according to the movement speed: the faster the movement speed, the more the exposure time is reduced, and the shorter the reduced exposure time.
  • the movement speed can be determined by the pixel movement distance and the frame interval. For example, for two adjacent frames of images with a fixed frame interval, the greater the pixel movement distance of the moving object between the two frames, the greater the movement speed.
  • the method may further include: during the preview, applying different exposure values EV at different frame times to cache multiple frames of images with different brightness. Applying different exposure values EV at frame moments may specifically include: using the same exposure time but different exposure gains at different frame moments, or using different exposure times and different exposure gains at different frame moments, or using the same exposure gain but different exposure times at different frame moments.
  • the method may further include: the electronic device uses a reference frame and multiple frames of images with different exposure values EV to perform HDR fusion processing to obtain an HDR fused image frame; the reference frame is the image frame with the highest clarity in the preview buffer, and the multiple frames of images with different exposure values EV are selected from cached image frames outside the reference frame.
  • the electronic device saves the photo, which may specifically include: saving the HDR fused image frame as a photo.
  • the photo comes from the latest captured multiple frames of the cache, which specifically means here that the photo comes from the reference frame and multiple frames of images with different exposure values EV, and is generated by the fusion of the reference frame and multiple frames of images with different exposure values EV.
  • the method may further include: the electronic device uses the reference frame and the multi-frame preferred frame to perform multi-frame fusion noise reduction processing to obtain an image frame after the multi-frame fusion noise reduction processing; the reference frame is the image frame with the highest clarity in the preview buffer, and the multi-frame preferred frame is selected from the cached image frames other than the reference frame.
  • the electronic device saves the photo, which may specifically include: saving the image frame after the multi-frame fusion noise reduction processing as a photo.
  • the photo comes from the latest multi-frame image captured in the cache, which specifically means here: the photo comes from the reference frame and the multi-frame preferred frame, and is generated by the fusion of the reference frame and the multi-frame preferred frame.
  • the electronic device can quickly take out multiple frames of images used to generate the photo from the cache frames, and perform multi-frame image fusion, which can reduce image noise and enhance details, thereby improving image quality; moreover, for HDR shooting scenes, multiple brightness frames are additionally exposed and output, and multi-frame brightness fusion is implemented.
  • the electronic device may also start a flickering light source detection device to detect flickering light sources. Reducing the exposure time of the first camera during preview may specifically include: if a flickering light source is detected during preview, the electronic device controls the reduced exposure time to be an integer multiple of the light energy cycle, and the shortest exposure time is set to one light energy cycle; if no flickering light source is detected during preview, the electronic device sets the exposure time to less than the light energy cycle.
  • the flicker light source detection device may include: an ambient light sensor (ALS) and a multispectral sensor.
  • the ALS operates at a high frequency F. According to Shannon's theorem, when F is higher than twice the light energy period F', the energy change of the flicker light source can be restored by reading the data of the ALS, thereby confirming whether there is a flicker light source in the current environment.
  • the multispectral sensor can directly read the energy change information of the flicker light source.
  • the flickering light source detection device may also be another rolling shutter camera (referred to as a second camera) on the electronic device.
  • a second camera another rolling shutter camera
  • the second camera is not specially introduced for flickering light source detection, and the images it collects will not be displayed. It is usually used as a depth of field camera, a wide-angle camera or a macro camera, and its exposure time can be set to a second duration, which is shorter than the light energy cycle, for example, 9ms, so that the images collected by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
  • the second duration can be further set to meet the exposure time for detecting flickering light sources under a variety of operating frequencies, that is, less than the minimum light energy cycle among the light energy cycles of these multiple operating frequencies. Taking the two operating frequencies of 50Hz and 60Hz as examples, the second duration can be set to 7ms, which is shorter than the light energy cycle of a 60Hz light source (about 8.3 milliseconds).
  • the electronic device when the flickering light source detection device is a second camera, the electronic device starts the flickering light source detection device to perform flickering light source detection, which may specifically include: the electronic device performs image recognition on the image captured by the second camera, and determines whether the image content contains water ripple features. If so, it is determined that a flickering light source is detected; the detection result is used to control the exposure time of the first camera.
  • the electronic device before the electronic device starts the flashing light source detection device to perform flashing light source detection, the electronic device can also first determine that the ambient brightness is neither too bright nor too dark, wherein the brightness of the overbright environment is higher than a first high brightness, such as 10,000 lux, and the brightness of the overdark environment is lower than a first low brightness, such as 50 Lux.
  • a first high brightness such as 10,000 lux
  • a first low brightness such as 50 Lux.
  • the exposure duration of the first camera may be set by an automatic exposure control AEC; the electronic device determines that the ambient brightness is neither too bright nor too dark, which may specifically include: determining that the ambient brightness is not too bright based on the exposure duration of the first camera being greater than a third duration (such as 20ms), determining that the ambient brightness is not too dark based on the exposure duration of the first camera being less than a fourth duration (200ms), and the third duration being less than the fourth duration.
  • the third duration and the fourth duration are recorded in the exposure table of the first camera, for example, the third duration belongs to several shortest exposure times corresponding to high EV in the exposure table, and the fourth duration belongs to several longest exposure times corresponding to low EV in the exposure table.
  • the present application provides a method for detecting a flickering light source, which may include: the electronic device starts a first camera and displays an image captured by the first camera on a display screen, and the electronic device also starts a second camera and sets the exposure time of the second camera to a second duration, and the second duration is shorter than the light energy cycle.
  • the electronic device can perform image recognition on the image captured by the second camera to determine whether the image content contains water ripple features. If so, it is considered that there is a flickering light source in the shooting environment, and the exposure time of the first camera is controlled to be an integer multiple of the light energy cycle.
  • the image captured by the second camera will not be displayed and is only used for flickering light source detection.
  • the electronic device can control the exposure of the first camera according to the flickering light source detection result, and control the exposure time of the first camera to be an integer multiple of the light energy cycle to avoid water ripples in the preview image provided by the first camera.
  • the electronic device can specifically reduce the exposure time of the first camera to a small integer multiple such as one or two times the light energy period, that is, the reduced exposure time is controlled to be an integer multiple of the light energy period, so as to avoid the problem of water ripples in the imaging picture under the flickering light source.
  • the electronic device when reducing the exposure time of the first camera, can specifically control the reduced exposure time of the first camera to one light energy period.
  • the electronic device when starting the first camera, can set the exposure time of the first camera to a first duration, and the first duration is greater than the light energy period and is an integer multiple of the light energy period, for example 50ms. In this way, the user can avoid seeing the generation of water ripples at the beginning of the camera startup.
  • the electronic device before the electronic device starts the first camera, it further includes: detecting a user operation of the user opening a camera application. That is, the start of the first camera may be triggered by the event of the user opening the camera application.
  • the first camera may be a main camera, and the second camera is not a main camera.
  • the first camera may be multiple cameras.
  • the preview image displayed by the camera application on the display screen is composed of images simultaneously captured by the multiple cameras, so as to provide the user with the function of taking or recording photos with multiple cameras.
  • the second duration can be further set to meet the exposure time for detecting the flickering light source under multiple power frequencies, that is, less than the minimum light energy cycle among the light energy cycles of the multiple power frequencies. Taking 50 Hz and 60 Hz as examples, the second duration can be set to 7 ms, which is shorter than the light energy cycle of a 60 Hz light source (about 8.3 milliseconds).
  • the electronic device before the electronic device starts the second camera, it also includes: the electronic device determines that the ambient brightness is neither too bright nor too dark, wherein the brightness of the overbright environment is higher than a first high brightness, such as 10,000 lux, and the brightness of the overdark environment is lower than a first low brightness, such as 50 Lux.
  • a first high brightness such as 10,000 lux
  • a first low brightness such as 50 Lux.
  • the exposure duration of the first camera may be set by an automatic exposure control AEC; the electronic device determines that the ambient brightness is neither too bright nor too dark, which may specifically include: determining that the ambient brightness is not too bright based on the exposure duration of the first camera being greater than a third duration (such as 20ms), determining that the ambient brightness is not too dark based on the exposure duration of the first camera being less than a fourth duration (200ms), and the third duration being less than the fourth duration.
  • the third duration and the fourth duration are recorded in the exposure table of the first camera, for example, the third duration belongs to several shortest exposure times corresponding to high EV in the exposure table, and the fourth duration belongs to several longest exposure times corresponding to low EV in the exposure table.
  • the first camera is the first camera or the second camera is not fixed.
  • the following uses the first camera as the main camera as an example of various switching situations of the first camera.
  • the electronic device may also include a telephoto camera.
  • the method may further include: the electronic device detects a user operation of increasing the zoom ratio to the zoom range of the telephoto camera, and switches the first camera from the main camera to the telephoto camera.
  • the electronic device may also include a wide-angle camera.
  • the method may further include: the electronic device detects a user operation of reducing the zoom ratio to the zoom range of the wide-angle camera, and switches the first camera from the main camera to the wide-angle camera.
  • the main camera may specifically include a rear main camera and a front main camera.
  • the method may further include: the electronic device detects a user operation of flipping the camera, and switches the first camera from the rear main camera to the front main camera, or switches the first camera from the front main camera to the rear main camera.
  • the method further includes: the electronic device detects a user operation of switching the shooting mode to multi-channel photo taking or video recording, starts one or more cameras other than the main camera, and also adds them as the first camera.
  • the second camera when the first camera is switched, can be selected from the remaining cameras.
  • the second camera can be selected to switch to the original first camera, especially for the case where the electronic device only has two cameras, there is no need to start a new camera, and the response speed is faster.
  • the second camera can also be fixed, such as a depth of field camera.
  • the image data collected by this type of camera does not participate in the image processing.
  • the preview image is displayed and is only used as auxiliary data (such as depth data) for other cameras, such as depth data of a face, so it will not be occupied by the first camera due to the change of the shooting mode.
  • the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions.
  • the electronic device executes the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
  • an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
  • the present application provides a computer-readable storage medium comprising instructions.
  • the instructions When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
  • the present application provides a computer program product comprising instructions, which, when the computer program product is run on an electronic device, enables the electronic device to perform the method described in the first aspect and any possible implementation of the first aspect, or the method described in the second aspect and any possible implementation of the second aspect.
  • the electronic device provided in the third aspect, the chip system provided in the fourth aspect, the computer storage medium provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.
  • FIG1 shows a motion capture method provided by Embodiment 1 of the present application
  • FIG2 exemplarily shows the change of image quality before and after exposure reduction
  • FIG3 shows a motion capture method provided by Embodiment 2 of the present application
  • FIG4 shows a motion capture method provided in Embodiment 3 of the present application.
  • FIG5 exemplarily shows the working principle of a rolling shutter
  • FIG6 exemplarily shows that a preview image provided by a rolling shutter camera has a water ripple problem
  • FIG7 exemplarily shows that the exposure time of a rolling shutter camera is set to be twice as long as the light energy period to solve the water ripple problem
  • FIG8 shows the overall process of the flashing light source method provided by an embodiment of the present application.
  • FIG9 exemplarily shows the hardware structure of an electronic device according to an embodiment of the present application.
  • FIG10 exemplarily shows the software architecture of the electronic device according to an embodiment of the present application.
  • FIG. 11 shows a detailed implementation process of the flashing light source method provided in an embodiment of the present application based on the internal architecture of an electronic device.
  • the shutter is an important part of the camera, which is used to control the exposure time of the photosensitive film.
  • setting a faster shutter speed is the guarantee of successful capture, and the faster the object moves, the higher the shutter speed should be set.
  • a fast shutter speed means a short exposure time. If a faster shutter speed (shorter exposure time) is used with the aperture unchanged, the exposure gain needs to be increased at the same time to increase the camera's sensitivity to avoid a decrease in the brightness of the image.
  • Parameters such as aperture size, exposure time, and exposure gain can be called exposure parameters. There can be multiple sets of exposure parameters to achieve the same exposure value (EV).
  • motion capture mode or motion photo mode
  • the photo frame is the image frame at the moment the shutter button is pressed, but in reality, considering issues such as shutter lag, the photo frame is generally slightly later than the moment the shutter button is pressed.
  • this motion capture function still has some problems, as follows:
  • the imaging speed is slower than still photography. You need to wait for the new exposure value automatically calculated by the smart device to take effect before applying new exposure parameters to the photo frame or re-exposing and reproducing the image, which will lead to the problem of slower imaging speed.
  • This effect can mean that the camera needs to reconfigure and enable related components to adopt new exposure parameters, such as reconfiguring and enabling the shutter and its controller to take effect of the new exposure time, reconfiguring and enabling the exposure gain circuit (such as the sensor gain circuit, the signal processor gain circuit) and its controller to take effect of the new exposure gain.
  • Shutter lag is more serious than still photography.
  • Shutter lag refers to the delay from the moment the user presses the shutter button to the actual imaging moment. As mentioned in 1 above, since it is necessary to wait for the new exposure value to take effect, the time from the shutter button being pressed to the shutter starting to capture the image is longer.
  • the image quality is worse than that of still photography. Since the smart device automatically calculates a new faster shutter speed for motion capture, it is necessary to increase the exposure gain at the same time to avoid a decrease in the brightness of the image. However, increasing the exposure gain will introduce more color noise and brightness noise, reducing the image quality.
  • Shutters can be divided into global shutter and rolling shutter, and the two have different exposure methods.
  • the exposure method of the global shutter is: all pixels on the photosensitive film start to be exposed to light at the same time, and after the same exposure time, they end at the same time;
  • the exposure method of the rolling shutter is: each row of pixels on the photosensitive film is exposed row by row in a rolling manner, and the exposure time of each row is the same until all pixels are exposed.
  • the exposure start time and end time of different rows are different.
  • Taking pictures with a rolling shutter camera under certain light sources may also cause "water ripples" to appear in the image.
  • These light sources are often driven by alternating current.
  • the energy transmitted by alternating current is not constant, but changes with a fixed frequency (such as 50Hz or 60Hz), which causes periodic fluctuations in the light intensity of the light source.
  • the period followed is the light energy cycle corresponding to the fixed frequency (also called the industrial frequency).
  • the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz
  • the light energy cycle is 10 milliseconds, that is, the light source flickers 100 times in 1 second.
  • this light source is also called a flickering light source.
  • Fig. 1 shows a motion capture method provided by Embodiment 1.
  • the method can solve the problem that the imaging speed of motion capture is slower than that of still photography and the shutter lag is more serious than that of still photography.
  • the method may include:
  • S401 The electronic device starts a first camera.
  • the electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.
  • S402 may continue to be executed, and the preview stream from the first camera may continue to be displayed, so as to continuously provide a preview display to the user.
  • the electronic device may also provide a preview frame cache function to cache the latest few frames of images (hereinafter referred to as cache frames), so that when the user intends to take a photo, the image frames used to generate the photo can be quickly taken from the cache frames.
  • the latest cached multi-frame images may refer to the multi-frame preview images that are earlier than the photo time but closest to the photo time.
  • the photo time may refer to the time when the electronic device detects the photo operation, such as the time when the user presses the shutter button.
  • S403 During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S405; if there is no moving object, execute S404.
  • S404 The electronic device maintains normal exposure of the first camera during preview.
  • S404 is not an additional action to be performed.
  • the electronic device can achieve normal exposure of the first camera without changing the exposure time of the first camera (maintaining the first duration).
  • the electronic device reduces the exposure time of the first camera during preview, for example, reducing the exposure time to a value shorter than the light energy period.
  • the electronic device may detect a photo-taking operation, such as a shutter button being pressed.
  • the photo operation may also include other user operations, such as voice commands with specific content (such as the voice command "YOYO, please take a photo!), specific actions (such as jumping actions), specific expressions (such as laughing), etc.
  • voice commands with specific content such as the voice command "YOYO, please take a photo!”
  • specific actions such as jumping actions
  • specific expressions such as laughing
  • the electronic device In response to the photo-taking operation, the electronic device saves the photo, which may be from a cache frame.
  • the electronic device reduces the exposure time and enters the short exposure mode when it detects a moving object during preview, rather than reducing the exposure time after the user presses the shutter.
  • the cached preview frame has the effect of reducing exposure, and once the photo operation is detected, the electronic device can obtain the photo to be saved based on the cached preview frame, without having to wait for the short exposure time to take effect before the photo is output, so the problems of slow imaging speed and shutter lag are improved.
  • FIG2 exemplarily shows the change in image quality before and after exposure reduction.
  • the 1st and 2nd frames are images at the initial normal exposure.
  • the exposure time is 30ms.
  • the rotating windmill in the preview screen has motion ghosting, and the image is blurred.
  • Frames 51 and 52 are images captured after the exposure time is reduced during preview. The exposure time is reduced to 10ms.
  • the rotating windmill in the preview screen no longer has motion ghosting, and the image is clear.
  • Frames 53 and 54 are images when the user presses the shutter button. Since the shutter pressing operation causes the phone to shake significantly, the image appears blurred overall.
  • Frame 55 is the image captured after the photo operation. At this time, the device returns to stability, the exposure time is 10ms, and the image is clear overall.
  • the electronic device can generate a photo based on the image frames with reduced exposure effect cached during preview (such as the 51st frame and the 52nd frame), which not only avoids the motion ghosting problem but also can quickly generate photos.
  • the specific implementation of determining whether there is a moving object in the preview image may include: detecting the moving object by using the optical flow method between adjacent preview frames, and if the pixel movement area reaches a certain area threshold, it is considered that there is a moving object in the preview image.
  • the motion detected in this way is the relative motion between the camera and the photographed object, which can be divided into three cases: case one, the camera remains stationary and the photographed object is in motion; case two, the photographed object remains stationary and the camera is in motion; case three, the photographed object and the camera are both moving, but the speeds of the two are different.
  • the motion mentioned in these three cases refers to the motion relative to the ground, which can be called absolute motion.
  • the electronic device can also detect whether the electronic device itself (camera) is in motion through motion sensors such as gyroscopes and accelerometers. Once the electronic device is detected to be in motion, it can be considered that there is a moving object in the preview image. Because when the camera is in motion, the object being photographed is likely to be moving in the preview image, and the special case that both the object being photographed and the camera are moving at the same speed is excluded.
  • motion sensors such as gyroscopes and accelerometers.
  • the electronic device when it reduces the exposure time of the first camera, it can further detect the movement speed (also a relative speed) of the moving object in the preview image, and control the exposure reduction according to the movement speed: the faster the movement speed, the more the exposure time is reduced, and the shorter the exposure time after reduction.
  • the movement speed can be determined by the pixel movement distance and the frame interval. For example, for two adjacent frames of images with a fixed frame interval, the greater the pixel movement distance of the moving object between the two frames, the greater the movement speed.
  • Fig. 3 shows a motion capture method provided by Embodiment 2. Based on the technical problem solved by Embodiment 1, the method provided by Embodiment 2 can further improve the problem that the imaging quality is inferior to that of still photography.
  • the method may include:
  • S501 The electronic device starts a first camera.
  • S502 The electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.
  • S503 During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S505; if there is no moving object, execute S504.
  • S504 The electronic device maintains normal exposure of the first camera during preview.
  • S505 The electronic device reduces the exposure time of the first camera during preview.
  • the preview may specifically be a high dynamic range (HDR) preview, in which case the electronic device may also apply different EVs at different frame times to obtain multiple cache frames with different brightness (different EVs).
  • the electronic device may obtain multiple preview frames with different EVs by changing the exposure gain and exposure time. For example, the same exposure time but different exposure gains may be used at different frame times to achieve different EVs, or different exposure times and different exposure gains may be used at different frame times to achieve different EVs, or the same exposure gain but different exposure times may be used at different frame times to achieve different EVs.
  • the electronic device may detect a photo-taking operation, such as a shutter button being pressed.
  • the electronic device determines whether the preview is an HDR preview. If not, execute S508-S511. If so, execute S512-S515.
  • the electronic device may select a reference frame from the cached preview frames.
  • the cached preview frames refer to the latest cached multi-frame images, which are multi-frame images that are earlier than the photographing time but closest to the photographing time.
  • the reference frame may be the one with the highest definition among the multiple preview frames cached.
  • the multiple frames selected in S509 are aligned according to the reference frame to eliminate the influence of motion on definition and enhance details.
  • the algorithm used for the alignment of multiple frames may be a motion estimation and motion compensation algorithm commonly used in video coding and decoding technology, that is, the entire image is divided into a series of non-overlapping blocks, and the minimum value of the sum of squared difference (SSD) or the sum of absolute difference (SAD) between each block and each block of the reference frame is calculated to estimate the position or motion vector corresponding to the current block in the reference frame, and then compensation is performed based on this.
  • SSD sum of squared difference
  • SAD sum of absolute difference
  • the electronic device can analyze the image clarity of the multiple preview frames based on a clarity operator (such as a sobel operator), and can further consider the device jitter factor, shutter time, etc. at the multiple preview frame moments to select a reference frame.
  • the device jitter factor can be determined based on motion data collected by motion sensors such as gyroscopes and accelerometers. It is essentially a factor that reflects the degree of device (camera) jitter. The larger it is, the more severe the device jitter is, which will cause image blur and be detrimental to image clarity. The shorter the shutter time, the less severe the motion smear problem is, and the more conducive to image clarity.
  • the electronic device can specifically run an algorithm model to consider the impact of one or more of the clarity operator, device jitter factor, and shutter time on image clarity.
  • the algorithm model can be a neural network model.
  • the electronic device may select a multi-frame image for multi-frame fusion noise reduction from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.
  • the electronic device may perform multi-frame fusion noise reduction (such as multi-frame time domain noise reduction) using the multi-frame images selected for multi-frame fusion noise reduction, and enhance details based on the reference frame to obtain a fused image frame.
  • multi-frame fusion noise reduction such as multi-frame time domain noise reduction
  • the electronic device saves the fused image frames as photos.
  • the saved photos mentioned in S407 in the first embodiment come from the cache frame, which specifically means that the saved photos come from the reference frame selected in S508 and the multiple frames of images selected in S509, and are generated by the fusion of them.
  • the electronic device may select a reference frame from the cached preview frames.
  • the electronic device may select multiple image frames with different exposure values EV from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.
  • the premise for implementing S512 is that, in order to implement HDR fusion, the electronic device reduces the exposure time in the preview while also changing the exposure gain and exposure time to obtain multiple preview frames with different EVs. For example, under the premise of shortening the exposure time relative to the first duration, the same exposure time but different exposure gains are used at different frame moments to achieve different EVs, or different exposure times and different exposure gains are used at different frame moments to achieve different EVs, or the same exposure gain but different exposure times are used at different frame moments to achieve different EVs.
  • S514 The electronic device uses the selected reference frame and multiple frames of images with different EVs to perform HDR fusion to obtain fused image frames.
  • HDR fusion also includes multi-frame image alignment based on the reference frame for multiple frames of images with different EVs, and then multi-frame fusion noise reduction for the multiple frames.
  • the difference between the two is that HDR fusion can also perform brightness fusion (also known as exposure fusion) based on multiple frames of images with different brightness (different EVs).
  • S515 The electronic device saves the image frame after HDR fusion processing as a photo.
  • the saved photos mentioned in S407 in the first embodiment come from the cache frame, which specifically means that the saved photos come from the reference frame selected in S512 and the multi-frame images with different EVs selected in S513, and are generated by their fusion.
  • the electronic device in Example 2 can quickly take out multiple frames of images used to generate photos from the cache frame, and perform multi-frame image fusion, which can reduce image noise and enhance details, thereby improving image quality; moreover, for HDR shooting scenes, multiple brightness frames are additionally exposed and output, and multi-frame brightness fusion is implemented.
  • the algorithm effect parameters and ISP parameters of the HDR algorithm or multi-frame fusion are specially adapted, and the effect is debugged for high exposure gain and short exposure time.
  • the algorithm effect parameters can be, for example, noise reduction intensity, contrast intensity, sharpening intensity, fusion ratio, ghost threshold, motion threshold, matching threshold, fusion pyramid size and other common fusion, noise reduction, and dynamic range algorithm adjustable parameters (parameters may vary depending on the algorithm prototype).
  • ISP debugging parameters can include automatic exposure, automatic focus, automatic white balance parameters (also known as 3A parameters), lens shading correction (LSC), gamma (GAMMA), dynamic range correction (DRC), YUV noise reduction (YNR) and other image debugging effect parameters.
  • YNR is a common YUV noise reduction parameter of ISP, which reduces the noise impact caused by high exposure gain during motion by targeted adjustment. In this way, frame selection, image fusion algorithm and parameter separation are used to compensate for the noise caused by the increase in exposure gain, while enhancing the image effect.
  • FIG4 shows a motion capture method provided by Embodiment 3.
  • the first camera may be a rolling shutter camera, and under a flickering light source, water ripples may appear in the preview image provided by the first camera.
  • the method provided by Embodiment 3 can further solve the water ripple (banding) problem.
  • the method may include:
  • S601 The electronic device starts a first camera.
  • the electronic device displays a preview image on a display screen, where the preview image comes from an image captured by the first camera.
  • S603 During preview, the electronic device determines whether there is a moving object in the preview image. If there is a moving object, execute S604; if there is no moving object, execute S605.
  • S604 The electronic device maintains normal exposure of the first camera during preview.
  • S601 - S604 may refer to steps S501 - S504 in the second embodiment, which will not be described in detail here.
  • S605 The electronic device reduces the exposure time of the first camera during preview.
  • the electronic device can reduce the exposure time while taking into account the water ripple problem.
  • the implementation of S605 may specifically include S6051-S6055:
  • the flickering light source detection device may include: an ambient light sensor (ALS) and a multi-spectral sensor.
  • the ALS operates at a high frequency F. According to Shannon's theorem, when F is higher than twice the light energy period F', The energy change of the flickering light source can be restored by reading the ALS data, thereby confirming whether there is a flicker light source in the current environment.
  • the multispectral sensor can directly read the energy change information of the flickering light source.
  • dedicated flicker detectors such as high-sampling-rate ambient light sensors and multi-spectral flicker detectors can be used to detect whether there is a flickering light source in the shooting environment, and cooperate with the camera to solve the water ripple problem.
  • flicker detectors such as high-sampling-rate ambient light sensors and multi-spectral flicker detectors can be used to detect whether there is a flickering light source in the shooting environment, and cooperate with the camera to solve the water ripple problem.
  • such devices require additional costs and design space.
  • the flickering light source detection device may also be another rolling shutter camera (referred to as a second camera) on the electronic device.
  • a second camera another rolling shutter camera
  • the second camera is not specially introduced for flickering light source detection. It is usually used as a depth of field camera, a wide-angle camera or a macro camera. Its exposure time can be set to a second duration, which is shorter than the light energy cycle, for example, 9ms. Therefore, the image captured by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
  • the second duration can be further set to meet the exposure time for detecting flickering light sources under a variety of operating frequencies, that is, less than the minimum light energy cycle among the light energy cycles of these multiple operating frequencies. Taking 50Hz and 60Hz as examples, the second duration can be set to 7ms, which is shorter than the light energy cycle of a 60Hz light source (about 8.3 milliseconds).
  • the exposure time referred to by normal exposure can be set to an integer multiple of the light energy cycle, which can avoid the user seeing water ripples when previewing during normal exposure.
  • the specific implementation of S6053 may include: the electronic device performs image recognition on the image captured by the second camera to determine whether the image content contains water ripple features, and if so, it is considered that there is a flicker light source in the shooting environment.
  • the image captured by the second camera will not be sent for display and is only used for flicker detection, so the user will not perceive it.
  • expectation may mean: reducing the exposure time according to the moving speed of the moving object, the image quality in HDR and other scenarios.
  • the exposure time is usually reduced to a very short time, such as 5 milliseconds, which is shorter than the light energy cycle.
  • the reduced exposure time can be controlled to be an integer multiple of the light energy cycle (such as 1 times, 2 times, etc.).
  • the shortest exposure time can be set to 1 times the light energy cycle, such as 10ms.
  • the electronic device can also judge whether the ambient brightness is appropriate, such as the indoor lighting environment, based on the exposure time set by the automatic exposure control (AEC) for the first camera, and then determine that flicker detection is required. This is because the exposure time of the photosensor set by the AEC can reflect the brightness of the shooting environment to a certain extent.
  • AEC automatic exposure control
  • Flicker detection is generally not required in shooting environments that are too bright or too dark. This is because, in environments that are too bright, the main concern is overexposure, and the exposure time needs to be significantly reduced, making it difficult to take into account the water ripple problem. In environments that are too dark, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise underexposure problems may easily occur.
  • too bright or too dark can be defined with reference to the exposure table on the electronic device 100. For example, the several shortest exposure times corresponding to high EV in the exposure table are determined as exposure times for environments that are too bright. Once the AEC exposure time belongs to these shortest exposure times, it is considered that the environment is too bright.
  • the electronic device can also collect ambient light through an ambient light brightness sensor to determine whether it is a shooting environment that is too bright or too dark, and the present application does not limit its technical implementation.
  • the electronic device may detect a photo-taking operation, such as a shutter button being pressed.
  • the electronic device determines whether the preview is an HDR preview. If not, execute S608-S611. If so, execute S612-S615.
  • the electronic device may select a reference frame from the cached preview frames.
  • the electronic device may select a multi-frame image for multi-frame fusion noise reduction from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.
  • the electronic device may perform multi-frame fusion noise reduction (such as multi-frame time domain noise reduction) using the multi-frame images selected for multi-frame fusion noise reduction, and enhance details based on the reference frame to obtain a fused image frame.
  • multi-frame fusion noise reduction such as multi-frame time domain noise reduction
  • S611 The electronic device saves the fused image frame as a photo.
  • the electronic device may select a reference frame from the cached preview frames.
  • the electronic device may select multiple image frames with different exposure values EV from the remaining cache frames, where the remaining cache frames refer to cache frames other than the reference frame.
  • S614 The electronic device uses the selected reference frame and multiple frames of images with different EVs to perform HDR fusion to obtain fused image frames.
  • S615 The electronic device saves the image frame after HDR fusion processing as a photo.
  • S606-S615 can refer to steps S506-S515 in the second embodiment, which will not be repeated here.
  • Taking pictures with a rolling shutter camera under certain light sources may cause "banding" to appear in the image.
  • These light sources are often driven by alternating current.
  • the energy transmitted by alternating current is not constant, but changes with a fixed frequency (such as 50Hz or 60Hz), which causes periodic fluctuations in the light intensity of the light source.
  • the period followed is the light energy cycle corresponding to the fixed frequency (also called the industrial frequency).
  • the light intensity fluctuation frequency of a 50Hz AC light source is 100Hz
  • the light energy cycle is 10 milliseconds, that is, the light source flickers 100 times in 1 second.
  • this light source is also called a flickering light source.
  • the exposure time is less than the light energy cycle of the light source
  • the exposure start and end times of each row of pixels on the photosensitive film are different, the accumulated light energy received by each row of pixels within the same exposure time is different, and light and dark stripes will appear on the image.
  • the light energy received by the first row of pixels on the photosensitive film is the light energy accumulated from time 0 to time t2
  • the light energy received by the nth row of pixels on the photosensitive film is the light energy accumulated from time t1 to 5 milliseconds.
  • the light energy received by the nth row is less than the light energy received by the first row.
  • the light energy received by the first row of pixels on the photosensitive film is the light energy accumulated from time t1 to 5 milliseconds.
  • the light energy received by the pixels in the nth row on the photosensitive film is the light energy accumulated from time t3 to time t5, while the light energy received by the pixels in the nth row on the photosensitive film is the light energy accumulated from time t4 to time t6.
  • the light energy received by the first row is less than the light energy received by the nth row.
  • the rows with large light energy appear as bright stripes, while the rows with small light energy appear as dark stripes, as shown in Figure 6.
  • the alternating light and dark stripes are water ripples, which is also called flicker.
  • users can also observe the rolling of water ripples.
  • the exposure time is set to an integer multiple of the light energy cycle. As shown in Figure 7, the light energy cycle is 10 milliseconds, and the exposure time can also be set to 10 milliseconds, that is, 1 times the light energy cycle.
  • Detection of flickering light sources is the key to solving the water ripple problem.
  • One detection method is to use an image recognition algorithm to identify the water ripple features in the image content, confirm that there are water ripples in the image, and then adjust the exposure time to avoid water ripples. However, before the water ripples are eliminated, users will still see the water ripples, which is a poor user experience.
  • Another detection method is to use a dedicated flicker detector such as a high-sampling rate ambient light sensor, a multi-spectral flicker detector, etc. to detect whether there is a flickering light source in the shooting environment, and cooperate with the camera to solve the water ripple problem.
  • a dedicated flicker detector such as a high-sampling rate ambient light sensor, a multi-spectral flicker detector, etc.
  • the embodiment of the present application provides a flickering light source detection method, which can detect the flickering light source without introducing additional dedicated detection devices.
  • the electronic device implementing the method has two or more rolling shutter cameras.
  • a camera used to detect flickering light sources is selected from the remaining cameras.
  • the camera used for preview display is called the first camera
  • the camera used for flickering light source detection also known as flicker detection
  • the second camera is not specially introduced for flickering light source detection, and it is usually used as a depth of field camera, a wide-angle camera, or a macro camera.
  • the shutters of the first camera and the second camera are both rolling shutters.
  • FIG8 shows the overall process of the flickering light source detection method provided by the embodiment of the present application.
  • the electronic device starts a first camera.
  • the activation of the first camera may be triggered by the event that the user opens the camera application.
  • the shooting mode of the camera application may be a normal shooting mode
  • the zoom ratio is 1x
  • the first camera may be a rear main camera
  • the second camera is selected from the remaining cameras.
  • the rear main camera may be the one with the highest pixel on the electronic device, and its focal length is usually between the focal length of the wide-angle camera and the focal length of the telephoto camera.
  • the exposure time of the first camera when starting the first camera, can be set to a first duration, and the first duration is greater than the light energy cycle and is an integer multiple of the light energy cycle, such as 50ms, so that the user can avoid seeing the generation of water ripples at the beginning of the camera startup.
  • the exposure time of the first camera in the normal shooting mode can be relatively long, such as 50ms, 100ms and other high multiples of the light energy cycle.
  • the first duration of this exposure time can be set by the ISP for the first camera through the AEC.
  • the light energy cycle may not be limited to the light energy cycle under the 50Hz operating frequency, but may also be the light energy cycle under the 60Hz operating frequency, or the light energy cycle under other operating frequencies.
  • the scenario of starting the first camera is not limited to the scenario where the user opens the camera application.
  • the scenario of starting the first camera can also be other scenarios that require starting the camera to collect images, such as video call scenarios, scan code payment, identity verification, etc.
  • the electronic device can start the rear main camera to scan the payment QR code.
  • the first camera is the rear main camera.
  • the electronic device can activate the front main camera and the front depth camera to scan and obtain facial feature data for user identity verification.
  • the first camera can be the front main camera.
  • a video call scenario the electronic device can activate the front main camera to capture the portrait image of the user in front of the screen.
  • the image captured by the first camera will be sent to the display so as to be used as the preview image displayed on the display screen by the camera application.
  • the electronic device displays the image captured by the first camera on a display screen.
  • the first camera can be a single camera, in which case the preview image displayed on the display screen by the camera application only comes from this camera.
  • the first camera can also be multiple cameras, in which case the preview image displayed on the display screen by the camera application is composed of images simultaneously captured by the multiple cameras, so as to provide the user with the function of taking photos or recording videos with multiple cameras.
  • S702 may continue to be executed to continue to display the preview stream from the first camera, so as to continuously provide a preview display to the user.
  • the electronic device starts the second camera and sets the exposure time of the second camera to a second duration, which is shorter than the light energy cycle, for example 9ms, so that the image collected by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
  • the electronic device Before executing S703, the electronic device can also determine that the ambient brightness is appropriate, such as the indoor lighting environment, based on the exposure time set by the AEC for the first camera, and then determine that flicker detection is required. This is because the exposure time of the photosensitive sensor set by the AEC can reflect the brightness of the shooting environment to a certain extent.
  • Flicker detection is generally not required in a shooting environment that is too bright or too dark, because in a too bright environment, the main consideration is the overexposure problem, and the exposure time needs to be significantly reduced, and it is difficult to take into account the water ripple problem; in a too dark environment, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise the problem of underexposure may easily occur.
  • too bright or too dark can be defined with reference to the exposure table on the electronic device 100. For example, the shortest exposure times corresponding to the high EV in the exposure table are determined as the exposure time of the too bright environment. Once the AEC exposure time belongs to these shortest exposure times, it is considered that the environment is too bright.
  • the longest exposure times corresponding to the low EV in the exposure table are determined as the exposure time of the too dark environment. Once the AEC exposure time belongs to these longest exposure times, it is considered that the environment is too dark. That is, the reference exposure value for too bright or too dark can be derived from the exposure table. Then, if the AEC exposure time belongs to the remaining exposure time in the exposure table, it can reflect that the ambient brightness is neither too bright nor too dark.
  • the electronic device is not limited to analyzing the ambient brightness by the exposure time set by the AEC for the first camera. The electronic device can also collect ambient light through an ambient light brightness sensor to determine whether the shooting environment is too bright or too dark. This application does not impose any restrictions on its technical implementation.
  • the electronic device performs image recognition on the image captured by the second camera (not displayed), and determines whether the image content contains water ripple features. If so, the exposure time of the first camera is controlled to be an integer multiple of the light energy cycle.
  • the image captured by the second camera will not be displayed, but only used for flicker detection, so the user will not perceive it.
  • the flicker detection result may include: a flickering light source is detected, and a flickering light source is not detected. If the image content captured by the second camera contains water ripple features, it is considered that there is a flickering light source in the shooting environment. Thereafter, the electronic device can control the exposure of the first camera according to the flicker detection result, and control the exposure time of the first camera to be an integer multiple of the light energy cycle to avoid water ripples in the preview image provided by the first camera.
  • the electronic device may specifically reduce the exposure time of the first camera to one or two times the light energy period. That is, the exposure time after being reduced is controlled to be an integer multiple of the light energy cycle, so as to avoid the problem of water ripples in the imaging picture under the flickering light source.
  • the electronic device can default to the normal photo mode with a zoom ratio of 1.
  • the first camera can be the main camera, and the second camera can be selected from the remaining cameras.
  • the first camera can be switched from the main camera to other cameras.
  • the electronic device may switch the first camera from the main camera to the telephoto camera.
  • the electronic device may switch the first camera from the main camera to the wide-angle camera.
  • the electronic device may switch the first camera from the main camera to the macro camera.
  • the second camera can be switched to the original first camera, especially for the case where the electronic device has only two cameras, there is no need to start a new camera, and the response speed is faster.
  • the second camera can also be fixed, such as a depth of field camera.
  • the image data collected by this type of camera does not participate in the preview image display, but is only used as auxiliary data (such as depth of field data) for other cameras, such as depth data of the face, and therefore will not be occupied by the first camera due to changes in the shooting mode.
  • Electronic equipment can be equipped Or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and/or smart city devices, etc.
  • PDA personal digital assistants
  • AR augmented reality
  • VR virtual reality
  • AI artificial intelligence
  • wearable devices wearable devices
  • vehicle-mounted devices smart home devices and/or smart city devices, etc.
  • Fig. 9 exemplarily shows an electronic device 100 provided by an embodiment of the present application.
  • the electronic device 100 can solve the problem that the imaging speed of motion capture is slower than that of still photography and the shutter lag is more serious than that of still photography.
  • the electronic device 100 may include: a processor 110, a memory 120, a camera 130, a display screen 140, and a flickering light source detection device (not shown).
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors.
  • the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, and a processor 110.
  • I2C inter-integrated circuit
  • I2S inter-integrated circuit sound
  • PCM pulse code modulation
  • UART universal asynchronous receiver/transmitter
  • Asynchronous receiver/transmitter (UART) interface mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and/or universal serial bus (USB) interface, etc. These interfaces are used for data exchange between the processor 110 and peripherals.
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB universal serial bus
  • the memory 120 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).
  • the random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc.
  • the non-volatile memory can also store executable programs and store user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.
  • a storage unit can also be set in the processor 110, and the storage unit can be a high-speed cache storage unit, which can be used to save instructions or data that the processor 110 has just used or circulated.
  • the implementation code of the flickering light source detection method provided in the embodiment of the present application can be stored in the NVM.
  • the code can be loaded into the RAM.
  • the processor 110 can directly read the program code from the RAM to implement the flickering light source detection method provided in the embodiment of the present application.
  • image files such as photos and videos obtained by the user using the camera application can be written into the NVM for storage for the user to browse.
  • the image captured by the camera 130 can be sent to the display to provide a preview image.
  • the camera 130 may include: a lens, a photosensitive sensor and a flexible printed circuit board (FPCB) part.
  • the FPCB is responsible for connecting other components of the camera 130 with the processor 110, for example, transmitting the raw data output by the photosensitive sensor to the processor 110.
  • the shutter of the camera 130 is opened, and light enters and shines on the photosensitive sensor.
  • the photosensitive sensor converts the light signal into an electrical signal, and then further converts the electrical signal into a digital signal through analog digital conversion (ADC) to pass it to the ISP for processing.
  • ADC analog digital conversion
  • the ISP can perform the following processing on the output data of the photosensitive sensor: auto exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, removal of bad pixels, etc.
  • AEC automatic exposure control
  • AGC automatic gain control
  • AVB automatic white balance
  • the ISP can also be integrated into the camera 130
  • the display screen 140 may be used to display images captured by the camera 130.
  • the images processed by the ISP will be sent to the display screen 140 to show the user a preview of the images captured by the camera.
  • sending to the display screen means pushing the images captured by the camera to a frame buffer (FB) for storage.
  • FB frame buffer
  • the frame buffer is a storage space that may be located in the video memory or in the internal memory and is used to store rendering data that has been processed by the graphics card chip or is to be extracted.
  • the content of the frame buffer corresponds to the interface display on the display screen 140 and can be simply understood as a cache corresponding to the content displayed on the display screen 140. Modifying the content in the frame buffer is to modify the content on the display screen 140.
  • the flicker light source detection device can be used for flicker detection, and specifically may include an ambient light sensor (ALS) and a multispectral sensor.
  • the ALS operates at a high frequency F.
  • F ambient light sensor
  • the multispectral sensor can directly read the energy change information of the flicker light source.
  • the flicker light source detection device can also be a rolling shutter camera (called the second camera) on an electronic device. The image captured by the second camera will not be displayed and is only used for flicker detection, so the user will not perceive it.
  • the second camera is not specially introduced for flicker detection. It can usually be a depth of field camera, a wide-angle camera, a macro camera, etc., with lower resolution and lower power consumption. It is a non-primary camera, and the present application embodiment gives it new functions.
  • the second camera can also be one or more cameras.
  • the exposure time of the camera that provides the preview display can also be controlled by the automatic exposure control (AEC) function.
  • AEC automatically adjusts the exposure time according to the light intensity.
  • the automatic exposure control (AEC) provided by the ISP provides this adaptive capability. Therefore, the exposure time of the photosensor can reflect the brightness of the shooting environment to a certain extent.
  • the electronic device 100 may also store an exposure table to be used by the AEC.
  • the exposure table of a camera sets the exposure parameters (such as exposure time, exposure gain, and aperture size) to be used by the camera at different exposure values (EV), and may be debugged and recorded by engineers.
  • the electronic device 100 may further include: an audio module 150 , a speaker 150A, a receiver 150B, a microphone 150C, and an earphone interface 150D.
  • the electronic device can realize audio functions through the audio module 150, the speaker 150A, the receiver 150B, the microphone 150C, the headphone interface 150D, and the application processor (AP).
  • the audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals.
  • the audio module 150 can also be used to encode and decode audio signals.
  • the audio module 150 can be arranged in the processor 110, or some functional modules of the audio module 150 are arranged in the processor 110.
  • the speaker 150A also known as the "speaker", is used to convert audio electrical signals into sound signals.
  • the electronic device can listen to music or listen to hands-free calls through the speaker 150A.
  • the receiver 150B also known as the "earpiece” is used to convert audio electrical signals into sound signals.
  • the voice can be answered by placing the receiver 150B close to the human ear.
  • Microphone 150C also called “microphone” or “microphone” is used to convert sound signals into electrical signals.
  • the user can make a sound by approaching the microphone 150C with his mouth to input the sound signal into the microphone 150C.
  • the electronic device can be provided with at least one microphone 150C. In other embodiments, the electronic device can be provided with two microphones 150C, which can not only collect sound signals but also realize noise reduction function.
  • the electronic device can also be provided with three, four or more microphones 150C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.
  • the headphone jack 150D is used to connect wired headphones.
  • the headphone jack 150D can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the electronic device 100 may further include a sensor module 160 , which may specifically include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, and the like.
  • a sensor module 160 which may specifically include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, and the like.
  • the structure shown in FIG9 does not constitute a specific limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently.
  • the components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
  • FIG. 10 shows the main software architecture involved in detecting flickering light sources using another rolling shutter camera on the electronic device 100 .
  • the software architecture may include the following functions or modules: camera application (camera apk), camera Service (camera service), camera hardware abstraction layer (camera HAL), flicker detection (CameraFlickerDetector), camera VNDK and Kernel.
  • camera application camera apk
  • camera Service camera service
  • camera hardware abstraction layer camera HAL
  • flicker detection Camera VNDK
  • Kernel Kernel
  • the camera application (camera apk) can be located in the application layer.
  • the application layer can include a series of other applications, such as the gallery application.
  • the camera application can be mainly responsible for human-computer interaction, such as monitoring and responding to user operations such as selecting a shooting mode, adjusting the zoom ratio, and selecting a focus.
  • the camera service can be located in the application framework layer.
  • the camera service can include interfaces such as camera management and camera device, which can interact with the camera application through the application programming interface (API) and with the camera HAL through the HAL interface definition language (HIDL).
  • API application programming interface
  • HIDL HAL interface definition language
  • VNDK vendor native development kit
  • the camera hardware abstraction layer can be the part of the virtual hardware function about the camera in the hardware abstraction layer (HAL).
  • the hardware abstraction layer is located in the interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system, and the upper layer does not need to care about the underlying hardware implementation.
  • the camera HAL can communicate with the camera service through the HIDL interface, and can issue control operations for the camera through the standard hardware abstraction layer interface.
  • the camera HAL can include a camera startup (CamEntry) module and an automatic exposure control processor (AECProcessor) module.
  • the CamEntry module can be used as the entrance of the HAL, receiving instructions such as open, close, stream configuration (configurestream), and request from the camera service.
  • the AECProcessor module can provide ISP's AEC-related functions, such as providing the AEC parameters of the first camera (such as exposure time) to the CameraFlickerDetector, so that the CameraFlickerDetector can determine whether the current shooting environment requires flicker detection.
  • the flicker detection (CameraFlickerDetector) module can be located in the application framework layer (framework) or the application layer.
  • the CameraFlickerDetector module can obtain the AEC parameters (such as exposure time) of the first camera from the camera HAL through the HIDL interface, so as to decide whether to start the second camera for flicker detection.
  • the CameraFlickerDetector module can issue control operations to start the second camera and set the exposure time through the camera VNDK interface.
  • the CameraFlickerDetector module may include a CameraHelper module and an ImageReader module, wherein the CameraHelper module can be used to implement the call of functions such as opening (open), closing (close), stream configuration (configurestream), and request (request) of the camera service of the second camera through the call of the VNDK interface, and the ImageReader module is mainly used to receive and further process the images sent after the second camera is started.
  • the CameraHelper module can be used to implement the call of functions such as opening (open), closing (close), stream configuration (configurestream), and request (request) of the camera service of the second camera through the call of the VNDK interface
  • the ImageReader module is mainly used to receive and further process the images sent after the second camera is started.
  • the Vendor Native Development Kit (VNDK) is a collection of libraries for vendors to develop their own HALs (vendors).
  • the camera VNDK is the "own camera HAL" developed by the vendor of the second camera.
  • the camera VNDK can be used to receive the control operation of starting the second camera issued by the CameraFlickerDetector, and forward the control operation to the camera service.
  • the camera VNDK may include the ACameraManager module, the ACameraDevice module, the ACameraReader module, the ACaptureSession module, and the ACaptureRequest module.
  • the ACameraManager module provides the function of accessing the camera service from the vendor layer
  • the ACameraDevice module provides the function of accessing the camera device from the vendor layer
  • the ACameraReader module is responsible for reading image data, such as the data captured by the second camera
  • the ACaptureSession module provides the function of allowing the vendor to manage the frames captured by the camera
  • the ACaptureRequest module is responsible for the settings and output targets required for the camera to capture a single image.
  • the kernel part may include various hardware driver control modules, such as ISP, photosensor, DSP, AP and other driver controls.
  • FIG. 10 only shows the main software architecture involved in the flickering light source detection method provided in the embodiment of the present application, which is essentially only a part of the software architecture of the electronic device 100 .
  • FIG10 also shows a process of detecting flickering light sources using another rolling shutter camera, including:
  • the camera application calls the camera service, which authenticates the camera application. After authentication, the camera service calls the camera HAL to start the first camera through the camera HAL, and control the photosensor and ISP to output images. The image frame will be sent to the display to realize image preview, photo taking and other functions.
  • the electronic device 100 can start the second camera to perform detection.
  • the second camera is not specially introduced for flicker detection. It can usually be a depth of field camera, a wide-angle camera, or a macro camera, etc. It has a low resolution and low power consumption. It is a non-primary camera.
  • the embodiment of the present application gives it a new function.
  • the AECProcessor module in the camera HAL can pass the exposure time and other parameters of the first camera to the CameraFlickerDetector module through the HIDL interface.
  • the exposure time set by the AEC can reflect the brightness of the shooting environment. Therefore, the CameraFlickerDetector module can determine whether to perform flicker detection based on the exposure time.
  • the environment for flicker detection is usually an indoor scene.
  • Flicker detection is generally not required in a shooting environment that is too bright or too dark, because in a too bright environment, the main consideration is the overexposure problem, and the exposure time needs to be significantly reduced, which makes it difficult to take into account the problem of avoiding water ripples; in a too dark environment, there is generally no need to reduce the exposure time to less than the light energy cycle, otherwise the problem of underexposure is likely to occur.
  • too bright or too dark can be defined with reference to the exposure table on the electronic device 100.
  • the several shortest exposure times corresponding to the high EV in the exposure table are determined as the exposure time of the too bright environment. Once the actual exposure time belongs to these shortest exposure times, it is considered that the environment is too bright.
  • the several longest exposure times corresponding to the low EV in the exposure table are determined as the exposure time of the too dark environment. Once the actual exposure time falls within these longest exposure times, it is considered that the environment is too dark.
  • the CameraFlickerDetector module passes the control command to start the second camera to the camera VNDK.
  • the camera VNDK then passes the CameraFlickerDetector module to the camera service to start controlling the second camera.
  • the camera service After authenticating the CameraFlickerDetector module, the camera service calls the camera HAL again to start the second camera through the camera HAL, so that the second camera can be started from the camera HAL side without involving the application layer, so the user does not perceive it.
  • the CameraFlickerDetector module controls the exposure of the second camera so that the exposure time of the second camera is less than the light energy cycle. For example, if the exposure time is set to 9ms, the image collected by the second camera under a 50Hz or 60Hz flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
  • the CameraFlickerDetector module can also call the camera HAL to read the output image of the second camera, detect whether the image content has water ripple features through image recognition technology, and pass the detection results to the first camera.
  • the output image of the second camera is not displayed and is only used for flicker detection.
  • the first camera learns the flicker detection result, it can reduce the exposure time of the first camera to twice the light energy cycle, such as 10ms, after detecting a moving object in the preview image, so as to solve the problem of water ripples in the imaging image under the flickering light source.
  • the above software architecture newly introduces the CameraFlickerDetector module, and implements the interaction between the CameraFlickerDetector module and the camera HAL and camera service through VNDK and HIDL. It does not involve modifications to the camera application and various layers of the architecture, and has good portability, scalability, and maintainability.
  • FIG11 only illustrates the method by taking the photo or video preview provided by the camera application as an example, so that those skilled in the art can have a deeper understanding of the present application embodiment, and its details should not constitute a limitation on the protection scope of the present application.
  • an application processor may start a rear main camera.
  • the photo mode can be a normal photo mode, and the exposure time of the rear main camera is longer than the light energy cycle, for example, 50ms, which is a multiple of the light energy cycle.
  • the exposure time of the rear main camera can be set by the AEC of the ISP.
  • the display screen may display a system desktop, including a desktop icon of a camera application.
  • the user operation of opening the camera application may refer to the user clicking the desktop icon of the camera application.
  • the rear main camera transmits the captured raw image frame (RAW) to the ISP.
  • RAW captured raw image frame
  • the ISP can perform the following processing on the RAW image output by the rear main camera: automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction, removal of bad pixels, etc.
  • AEC automatic exposure control
  • AGC automatic gain control
  • AVB automatic white balance
  • the image processed by the ISP can be a color image, such as a YUV image.
  • S13.ISP can transmit the output YUV image frame of the rear main camera to the display screen.
  • S13 only illustrates the flow path of image data, and does not mean that the ISP directly sends data to the display screen. Its implementation includes the process of sending to the display.
  • the YUV image frame of the rear main camera can be cached in the frame buffer first.
  • the GPU renders the camera preview image, it reads the YUV image frame of the rear main camera from the frame buffer and displays it on the display screen.
  • the display screen displays the YUV image frame of the rear main camera.
  • S15.AP can obtain the exposure time set by AEC for the rear main camera from ISP.
  • the exposure time set by AEC can reflect the brightness of the shooting environment.
  • S16.AP can determine whether flicker detection is required based on the exposure time set by AEC for the rear main camera.
  • the environment for flicker detection is usually an indoor scene. Flicker detection is generally not required in a shooting environment that is too bright or too dark. For specific instructions, please refer to the previous article and will not be repeated here.
  • the application processor can start the depth of field camera as a second camera, and set the exposure time of the depth of field camera to be less than the light energy cycle, such as 9 milliseconds.
  • the exposure time of the second camera is less than the light energy cycle, the image captured by the second camera under the flickering light source will inevitably produce water ripples, and the flickering light source can be detected.
  • the AP may also directly start the second camera without determining whether to perform flicker detection, that is, S15-S16 may not be executed.
  • the depth of field camera can transmit the captured raw image frame (RAW) to the ISP.
  • RAW captured raw image frame
  • the ISP can perform a series of processing on the RAW image and output a YUV image frame.
  • the YUV image frame of the depth of field camera is not sent for display, that is, it will not be transmitted to the display screen for display.
  • S19.AP can obtain the YUV image frame of the depth of field camera from the ISP.
  • S20.AP can perform image recognition on the YUV image frame of the depth of field camera to detect whether the image content contains water ripple features to obtain flicker detection results. If the image content contains water ripple features, it is considered that there is a flickering light source in the shooting environment, otherwise, it is considered that there is no flickering light source in the shooting environment.
  • the AP may run the software architecture shown in FIG10 , especially the CameraFlickerDetector module, to implement flicker detection. As time passes, flicker detection may continue to be performed, especially during the period when the first camera is turned on.
  • the photo taking mode may be switched to the motion capture mode or the HDR mode, and the electronic device may set the exposure time of the first camera based on the flicker detection result to avoid the water ripple problem.
  • the AP learns that the camera mode is switched to the motion capture mode or the HDR mode.
  • S22.AP can reduce the exposure time of the rear main camera to meet the needs of motion capture mode or HDR mode. Specifically, it can reduce the exposure time to 1 times the light energy cycle to take into account the water ripple avoidance problem.
  • the exposure time is not limited to 1 times of the light energy cycle.
  • the exposure time can also be reduced to 2 times, 3 times, or other low multiples of the light energy cycle, depending on the exposure reduction requirements of the motion capture mode or HDR mode.
  • the first camera is not static.
  • the following steps illustrate the switching of the first camera in several typical scenarios (zooming, flipping the camera, multi-channel photo taking or video recording).
  • increasing the zoom ratio to the zoom range of the telephoto camera, or decreasing the zoom ratio to the zoom range of the wide-angle camera, can change the first camera that provides the preview image.
  • the application processor (AP) detects that the zoom ratio is increased to the zoom range of the telephoto camera.
  • the application processor (AP) can turn off the rear main camera.
  • the application processor may activate the telephoto camera and use the telephoto camera as the first camera.
  • the telephoto camera transmits the acquired raw image frame (RAW) to the ISP.
  • RAW acquired raw image frame
  • the ISP can perform a series of processing on the RAW image and output a YUV image frame.
  • S28.ISP can transmit the YUV image frame of the telephoto camera to the display.
  • S27 only illustrates the flow path of image data, and does not mean that the ISP directly sends data to the display screen. Its implementation includes the process of sending and displaying.
  • the display screen displays the YUV image frame of the telephoto camera.
  • the telephoto camera starts to provide a preview image, and the first camera used for preview display is switched from the rear main camera to the telephoto camera.
  • the implementation process of reducing the zoom ratio to the zoom range of the wide-angle camera is similar to that of changing the first camera. Please refer to the adaptive adjustment of S23-S28, which will not be expanded here.
  • the first camera can be switched from the rear main camera to the front main camera.
  • the application processor (AP) detects that the camera is flipped.
  • the application processor (AP) can turn off the rear main camera.
  • the application processor may start the front main camera and use the front main camera as the first camera.
  • the exposure time can be set to 1 times the light energy cycle when the front main camera is started. Please refer to S33.
  • the front main camera transmits the captured raw image frame (RAW) to the ISP.
  • RAW captured raw image frame
  • the ISP can perform a series of processing on the RAW image and output a YUV image frame.
  • S35.ISP can transmit the YUV image frame of the front main camera to the display.
  • S35 only illustrates the flow path of image data, and does not mean that the ISP directly sends the YUV image frame of the front main camera to the display screen. Its implementation includes the process of sending and displaying.
  • the display screen displays the YUV image frame of the front main camera.
  • the front main camera starts to provide photo preview images, and setting its exposure time based on the flicker detection results can also avoid water ripples in its image.
  • flipping the camera can also trigger the first camera to switch from the front main camera to the rear main camera.
  • the implementation process is similar and will not be expanded here.
  • one or more cameras other than the main camera are started and also added as the first camera.
  • the application processor detects that the camera mode is switched to the front and rear dual-channel camera mode.
  • the front and rear dual-channel photo mode is only one of the multi-channel photo modes.
  • the multi-channel photo mode can also include: the rear main camera, rear telephoto camera and rear wide-angle camera and other rear cameras taking photos at the same time. Multi-channel video recording is similar.
  • the application processor may activate the front main camera and add the front main camera as the first camera.
  • the first camera includes two cameras: a rear main camera and a front main camera.
  • the application processor (AP) does not turn off the rear main camera, and the YUV image frame of the rear main camera is not sent to the display independently, but needs to be spliced with the YUV image frame of the front main camera and sent to the display together.
  • the front main camera can also transmit the captured raw image frames (RAW) to the ISP.
  • RAW captured raw image frames
  • the ISP can also perform a series of processing on the RAW image and output the YUV image frame of the front main camera.
  • the YUV image frame of the front main camera needs to be combined with the YUV image frame of the rear main camera to form a spliced preview image to achieve the function of simultaneous photo preview of the front and rear cameras.
  • the S40.ISP can transmit the stitched image frames from the front and rear main cameras to the display.
  • S40 only illustrates the flow path of image data, and does not mean that the ISP directly sends the spliced image frame to the display screen, and its implementation includes the process of sending and displaying.
  • the display screen displays the spliced image frame.
  • the exposure time of the first camera can be set based on the flicker detection result to avoid water ripples in the imaging picture.
  • the electronic device can also switch the first camera in other scenarios. For example, when the photo mode is switched to "macro mode", the first camera can be switched to a macro camera.
  • the second camera when the first camera is switched, the second camera always uses the depth of field camera. Not limited to this, the second camera can be switched, and the embodiment of the present application does not limit its switching strategy. For example, for an electronic device with only two cameras (a front camera and a rear camera), when a flip camera is detected, the first camera switches to the front camera, and the second camera can be switched to the rear camera.
  • each camera is coupled to one ISP.
  • the electronic device may be configured with multiple ISPs, and multiple cameras may be coupled to different ISPs.
  • UI user interface
  • the term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user.
  • the user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML).
  • the interface source code is parsed and rendered on the terminal device, and finally presented as content that can be recognized by the user, such as pictures, text, buttons and other controls.
  • Controls also known as widgets, are basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text.
  • the properties and contents of controls in the interface are defined by tags or nodes.
  • XML specifies the controls contained in the interface through nodes such as ⁇ Textview>, ⁇ ImgView>, and ⁇ VideoView>.
  • a node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering.
  • many applications such as hybrid applications, usually include web pages in their interfaces.
  • a web page also known as a page, can be understood as a special control embedded in the application interface.
  • a web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc.
  • the web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser.
  • the specific content contained in a web page is also defined by tags or nodes in the web page source code.
  • HTML defines the elements and attributes of a web page through ⁇ p>, ⁇ img>, ⁇ video>, and ⁇ canvas>.
  • GUI graphical user interface
  • It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.
  • Each step in the above method embodiment provided by the present application can be completed by hardware integrated logic circuits in the processor or software instructions.
  • the execution is completed, or the execution is completed by a combination of hardware and software modules in the processor.
  • the present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.
  • the present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.
  • the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.
  • the chip system may be composed of the chip, or may include the chip and other discrete devices.
  • the processor in the chip system may be one or more.
  • the processor may be implemented by hardware or by software.
  • the processor may be a logic circuit, an integrated circuit, etc.
  • the processor may be a general-purpose processor implemented by reading software code stored in a memory.
  • the memory in the chip system may also be one or more.
  • the memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application.
  • the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively.
  • the embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.
  • the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.
  • FPGA field programmable gate array
  • ASIC application specific integrated circuit
  • SoC system on chip
  • CPU central processor unit
  • NP network processor
  • DSP digital signal processor
  • MCU microcontroller unit
  • PLD programmable logic device
  • the present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.
  • a computer program also referred to as code, or instruction
  • the present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction).
  • a computer program also referred to as code or instruction.
  • the computer program executes the method executed by the electronic device in any of the above embodiments.
  • a computer program product includes one or more computer instructions.
  • a computer program instruction When a computer program instruction is loaded and executed on a computer, a process or function according to the present application is generated in whole or in part.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
  • Computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.
  • a computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein.
  • Available media may be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state drive Solid State Disk), etc.
  • the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media.
  • the programs can include the processes of the above-mentioned method embodiments.
  • the aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

本申请实施例提供了运动抓拍方法及电子设备，电子设备在预览时检测到运动物体就降低曝光时间，进入短曝光模式，而不是在用户按下快门后才降低曝光时间。如此，缓存的预览帧便带有降曝效果，之后一旦检测到拍照操作，电子设备便可以基于缓存的预览帧获得要保存的照片，而无需等待短曝光时间生效后才能出图，因此运动抓拍时成像速度慢和快门迟滞的问题便得到改善。

Description

运动抓拍方法及电子设备
本申请要求于2023年05月23日提交中国专利局、申请号为202310590214.8、申请名称为“运动抓拍方法及电子设备”的中国专利申请，以及2023年05月23日提交中国专利局、申请号为202310595470.6、申请名称为“闪烁光源检测方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及电子技术领域，尤其涉及运动抓拍方法及电子设备。
背景技术
随着电子技术的发展，手机、平板电脑等电子设备一般都配置有摄像头。用户可以使用这样的电子设备去抓拍运动中的各种各样的对象，以记录下它们在运动中的精彩瞬间。但是，拍摄好运动中的物体，尤其是拍清楚运动中的物体，通常需要较复杂的曝光设置，而一般用户对相机曝光了解很少，操作难度高。
发明内容
本申请提供了运动抓拍方法及电子设备。
第一方面，本申请提供了运动抓拍方法，可包括：电子设备启动第一摄像头，在显示屏上显示预览图像，该预览图像来自第一摄像头采集的图像，并且电子设备缓存最新采集的多帧图像(又称为缓存帧)。在预览时，电子设备判断预览图像中是否有运动物体，若有运动物体，则在预览时降低第一摄像头的曝光时间，例如降低曝光时间至短于光能量周期。电子设备可检测到拍照操作，例如快门按键被按下，响应于拍照操作，电子设备保存照片，该照片可来自于缓存帧。
第一方面中，第一摄像头的启动可以是用户打开相机应用这一事件触发的。在启动第一摄像头时，第一摄像头的曝光时间可以为第一时长，相对较长，不适合抓拍运动中的物体，容易出现运动拖影。例如，第一时长可以大于光能量周期且为光能量周期的整数倍，如50ms。
通过第一方面的方法，电子设备在预览时检测到运动物体就降低曝光时间，进入短曝光模式，而不是在用户按下快门后才降低曝光时间。如此，缓存的预览帧便带有降曝效果，之后一旦检测到拍照操作，电子设备便可以基于缓存的预览帧获得要保存的照片，而无需等待短曝光时间生效后才能出图，因此成像速度慢和快门迟滞的问题便得到改善。
结合第一方面，在一些实施例中，电子设备在降低第一摄像头的曝光时间时，还可以进一步检测预览画面中运动物体的运动速度(也是一种相对速度)，并根据运动速度控制降曝：运动速度越快，曝光时间被降低的越多，降低后的曝光时间越短。运动速度可以通过像素移动距离和帧间隔来确定，例如对于固定帧间隔的相邻两帧图像，在这两帧图像之间运动物体的像素移动距离越大，运动速度就越大。
结合第一方面，在一些实施例中，若预览是高动态范围HDR预览，则方法还可以包括：在预览时，在不同帧时刻应用不同曝光值EV以缓存亮度不同的多帧图像。其中，在不同 帧时刻应用不同曝光值EV，具体可包括：在不同帧时刻采用相同的曝光时间但不同的曝光增益，或者在不同帧时刻采用不同的曝光时间且不同的曝光增益，或者在不同帧时刻采用相同曝光增益但不同曝光时间。
结合第一方面，在一些实施例中，若预览是HDR预览，则在电子设备保存照片之前，方法还可以包括：电子设备利用参考帧和曝光值EV不同的多帧图像进行HDR融合处理，得到HDR融合后的图像帧；参考帧是预览缓存区中清晰度最高的图像帧，曝光值EV不同的多帧图像是从参考帧之外的缓存的图像帧中选择出来的。如此，电子设备保存照片，具体可包括：将HDR融合后的图像帧保存为照片。照片来自于缓存的最新采集的多帧图像，在这里具体是指：照片来自参考帧和曝光值EV不同的多帧图像，是参考帧和曝光值EV不同的多帧图像融合产生的。
结合第一方面，在一些实施例中，若预览不是HDR预览，则在电子设备保存照片之前，方法还可以包括：电子设备利用参考帧和多帧优选帧进行多帧融合降噪处理，得到多帧融合降噪处理后的图像帧；参考帧是预览缓存区中清晰度最高的图像帧，多帧优选帧是从参考帧之外的缓存的图像帧中选择出来的。如此，电子设备保存照片，具体可包括：将多帧融合降噪处理后的图像帧保存为照片。照片来自于缓存的最新采集的多帧图像，在这里具体是指：照片来自参考帧和多帧优选帧，是参考帧和多帧优选帧融合产生的。
这样，待用户产生拍照意图时，电子设备可迅速从缓存帧中取出用于生成照片的多帧图像，并进行多帧图像融合，可以降低图像噪声和增强细节，提高了图像质量；而且，针对HDR拍摄场景，还额外进行了多个亮度帧的曝光出图，实施了多帧亮度融合。
结合第一方面，在一些实施例中，在预览时，电子设备还可以启动闪烁光源检测装置进行闪烁光源检测。在预览时降低第一摄像头的曝光时间，具体可包括：若在预览时检测到闪烁光源，则电子设备控制降低后的曝光时间是光能量周期的整数倍，曝光时间最短设置成一倍光能量周期；若在预览时未检测到闪烁光源，则电子设备将曝光时间至光能量周期以下。
其中，闪烁光源检测装置可以包括：环境光传感器(ALS)、多光谱传感器。其中，ALS运行在高频率F下，根据香农定理，当F高于光能量周期F’的两倍时，可以通过读取ALS的数据恢复闪烁光源的能量变化情况，从而确认当前环境是否存在flicker光源。多光谱传感器可直接读出闪烁光源的能量变化信息。
结合第一方面，在一些实施例中，闪烁光源检测装置也可以是电子设备上的另一个卷帘快门摄像头(称为第二摄像头)。
第二摄像头并非为了闪烁光源检测专门引入的，其采集的图像不会送显，通常它被用作景深摄像头、广角摄像头或者微距摄像头，其曝光时间可以被设置成第二时长，第二时长比光能量周期短，例如9ms，从而第二摄像头在50Hz或者60Hz闪烁光源下采集的图像必然会产生水波纹，闪烁光源能够被检测出。第二时长可进一步设定成满足多种工频下检测闪烁光源的曝光时间，即小于这多种工频的光能量周期中的最小光能量周期。以50Hz和60Hz这两种工频为例，第二时长可以设置成7ms，其值比60Hz光源的光能量周期(约8.3毫秒)还短。
结合第一方面，在一些实施例中，当闪烁光源检测装置是第二摄像头时，电子设备启动闪烁光源检测装置进行闪烁光源检测，具体可以包括：电子设备对第二摄像头采集的图像进行图像识别，判断图像内容中是否包含水波纹特征，若包含，则确定检测出闪烁光源；检测的结果用于控制第一摄像头的曝光时间。
结合第一方面，在一些实施例中，在电子设备启动闪烁光源检测装置进行闪烁光源检测之前，电子设备还可以先确定出环境亮度不是过亮环境，也不是过暗环境，其中，过亮环境的亮度高于第一高亮度，如10000勒克斯(Lux)，过暗环境的亮度低于第一低亮度，如50Lux。
结合第一方面，在一些实施例中，第一摄像头的曝光时长可以是自动曝光控制AEC设定的；电子设备确定出环境亮度不是过亮环境，也不是过暗环境，具体可包括：根据第一摄像头的曝光时长大于第三时长(如20ms)确定出环境亮度不是过亮环境，根据第一摄像头的曝光时长小于第四时长(200ms)确定出环境亮度不是过暗环境，第三时长小于第四时长。其中，第三时长、第四时长记录于第一摄像头的曝光表中，例如第三时长属于曝光表中高EV对应的几个最短曝光时间，第四时长属于曝光表中低EV对应的几个最长曝光时间。
第二方面，本申请提供了一种闪烁光源检测方法，可包括：电子设备启动第一摄像头，并在显示屏上显示第一摄像头采集的图像，电子设备还启动第二摄像头，并将第二摄像头的曝光时间设置成第二时长，第二时长比光能量周期短。电子设备可对第二摄像头采集的图像进行图像识别，判断其图像内容中是否包含水波纹特征，若包含，则认为拍摄环境中存在闪烁光源，控制第一摄像头的曝光时间是光能量周期的整数倍。第二摄像头采集的图像不会被送显，仅用于闪烁光源检测。
通过第二方面的方法，电子设备可以根据闪烁光源检测结果对第一摄像头进行曝光控制，控制第一摄像头的曝光时间是光能量周期的整数倍，以避免第一摄像头提供的预览图像中出现水波纹。
例如，当拍照模式从普通拍照模式切换到运动抓拍模式时，在降低曝光时间以减轻运动拖影问题时，电子设备可具体将第一摄像头的曝光时间降低成光能量周期的一倍或者两倍等小数值整数倍，即控制降低后的曝光时间是光能量周期的整数倍，以同时避免闪烁光源下成像画面中出现水波纹的问题。
结合第二方面，在一些实施例中，若第二摄像头采集的图像包含水波纹特征，则在降低第一摄像头的曝光时间时，电子设备具体可以控制降低后的第一摄像头的曝光时间至一倍光能量周期。
结合第二方面，在一些实施例中，在启动第一摄像头时，电子设备可以将第一摄像头的曝光时间设置成第一时长，第一时长大于光能量周期且为光能量周期的整数倍，例如50ms，这样在摄像头启动之初都能避免用户看到水波纹的产生。
结合第二方面，在一些实施例中，在电子设备启动第一摄像头之前，还包括：检测到用户打开相机应用的用户操作。也即，第一摄像头的启动可以是用户打开相机应用这一事件触发的。第一摄像头可以是主摄像头，第二摄像头不是主摄像头。
结合第二方面，在一些实施例中，第一摄像头可以是多个摄像头，此时相机应用展示于显示屏上的预览图像由这多个摄像头同时采集的图像拼接构成，以向用户提供多摄像头拍照或录像的功能。
结合第二方面，在一些实施例中，第二时长可进一步设定成满足多种工频下检测闪烁光源的曝光时间，即小于这多种工频的光能量周期中的最小光能量周期。以50Hz和60Hz这两种工频为例，第二时长可以设置成7ms，其值比60Hz光源的光能量周期(约8.3毫秒)还短。
结合第二方面，在一些实施例中，在电子设备启动第二摄像头之前，还包括：电子设备确定环境亮度不是过亮环境，也不是过暗环境，其中，过亮环境的亮度高于第一高亮度，如10000勒克斯(Lux)，过暗环境的亮度低于第一低亮度，如50Lux。
结合第二方面，在一些实施例中，第一摄像头的曝光时长可以是自动曝光控制AEC设定的；电子设备确定出环境亮度不是过亮环境，也不是过暗环境，具体可包括：根据第一摄像头的曝光时长大于第三时长(如20ms)确定出环境亮度不是过亮环境，根据第一摄像头的曝光时长小于第四时长(200ms)确定出环境亮度不是过暗环境，第三时长小于第四时长。其中，第三时长、第四时长记录于第一摄像头的曝光表中，例如第三时长属于曝光表中高EV对应的几个最短曝光时间，第四时长属于曝光表中低EV对应的几个最长曝光时间。
结合第二方面，在一些实施例中，第一摄像头、第二摄像头具体是哪个摄像头不是一成不变的。下面以第一摄像头原为主摄像头示例第一摄像头的多种切换情况。
电子设备还可以具备长焦摄像头。此时，在电子设备在显示屏上显示预览图像之后，方法还可以包括：电子设备检测到提高变焦倍率到长焦摄像头的变焦范围的用户操作，将第一摄像头从主摄像头切换成长焦摄像头。
电子设备还可以具备广角摄像头。此时，在电子设备在显示屏上显示预览图像之后，方法还可包括：电子设备检测到降低变焦倍率到广角摄像头的变焦范围的用户操作，将第一摄像头从主摄像头切换成广角摄像头。
主摄像头具体可以包括后置主摄像头和前置主摄像头。此时，在电子设备在显示屏上显示预览图像之后，方法还可包括：电子设备检测到翻转摄像头的用户操作，将第一摄像头从后置主摄像头切换成前置主摄像头，或者将第一摄像头从前置主摄像头切换成后置主摄像头。
在电子设备在显示屏上显示预览图像之后，方法还包括：电子设备检测到将拍摄模式切换到多路拍照或录像的用户操作，启动主摄像头之外的一个或多个摄像头，也添加用作第一摄像头。
结合第二方面，在一些实施例中，在第一摄像头发生切换时，第二摄像头便可以从余下摄像头中选择出。第二摄像头可以选择切换成原来的第一摄像头，尤其是针对电子设备只具有两个摄像头的情况，无需再去启动新的摄像头，响应速度更快。当然，实际应用中，也可以基于功耗等问题考虑将第二摄像头切换到功耗较低的摄像头，本申请实施例对此不作限制。第二摄像头也可以固定下来，例如景深摄像头，这类摄像头采集的图像数据不参 与成预览图像显示，仅用作其他摄像头的辅助数据(如景深数据)，例如人脸面部的深度数据，因而不会因拍照模式改变而被占用成第一摄像头。
第三方面，本申请提供了一种电子设备，该电子设备包括一个或多个处理器和一个或多个存储器；其中，一个或多个存储器与一个或多个处理器耦合，一个或多个存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当一个或多个处理器执行计算机指令时，使得电子设备执行如第一方面以及第一方面中任一可能的实现方式描述的方法，或者第二方面以及第二方面中任一可能的实现方式描述的方法。
第四方面，本申请实施例提供了一种芯片系统，该芯片系统应用于电子设备，该芯片系统包括一个或多个处理器，该处理器用于调用计算机指令以使得该电子设备执行如第一方面以及第一方面中任一可能的实现方式描述的方法，或者第二方面以及第二方面中任一可能的实现方式描述的方法。
第五方面，本申请提供一种计算机可读存储介质，包括指令，当上述指令在电子设备上运行时，使得上述电子设备执行如第一方面以及第一方面中任一可能的实现方式描述的方法，或者第二方面以及第二方面中任一可能的实现方式描述的方法。
第六方面，本申请提供一种包含指令的计算机程序产品，当上述计算机程序产品在电子设备上运行时，使得上述电子设备执行如第一方面以及第一方面中任一可能的实现方式描述的方法，或者第二方面以及第二方面中任一可能的实现方式描述的方法。
可以理解地，上述第三方面提供的电子设备、第四方面提供的芯片系统、第五方面提供的计算机存储介质、第六方面提供的计算机程序产品均用于执行本申请所提供的方法。因此，其所能达到的有益效果可参考对应方法中的有益效果，此处不再赘述。
附图说明
图1示出了本申请实施例一提供的运动抓拍方法；
图2示例性示出了降曝前后的图像质量变化；
图3示出了本申请实施例二提供的运动抓拍方法；
图4示出了本申请实施例三提供的运动抓拍方法；
图5示例性示出了卷帘快门的工作原理；
图6示例性示出了卷帘快门摄像头提供的预览图像存在水波纹问题；
图7示例性示出了将卷帘快门摄像头的曝光时间设置成光能量周期的一倍那么长以解决水波纹问题；
图8示出了本申请实施例提供的闪烁光源方法的总流程；
图9示例性示出了本申请实施例的电子设备的硬件结构；
图10示例性示出了本申请实施例的电子设备的软件架构；
图11示出了本申请实施例提供的闪烁光源方法基于电子设备内部架构的详细实现流程。
具体实施方式
本申请以下实施例中所使用的术语只是为了描述特定实施例的目的，而并非旨在作为对本申请的限制。
快门是摄像头的一个重要部件，用来控制感光片的曝光时间。抓拍运动物体时，设置较快的快门速度是抓拍成功的保证，并且被拍摄的物体移动速度越快，快门速度就要设置越高。快门速度快，意味着曝光时间短。在光圈不变的情况下，若使用较快的快门速度(较短的曝光时间)，则同时需要提高曝光增益以提高相机的感光度，避免成像画面亮度降低。光圈大小、曝光时间、曝光增益这些参数可以称为曝光参数。实现同一个曝光值(exposure value，EV)的曝光参数可以存在多套。
为了方便用户拍摄运动物体，很多智能设备设计了运动抓拍模式(或称为运动拍照模式)，在用户按下快门按键时会自动计算一个合适的快门速度应用于拍照帧，无需用户操心如何控制曝光。拍照帧会被保存为照片。理论上，拍照帧是按下快门按键时刻的图像帧，但实际上，考虑快门迟滞等问题，拍照帧的时刻一般稍晚于快门按键被按下时刻。但是，这种运动抓拍功能还存在一些问题，如下：
1.成像速度差于静物拍照。需要等待智能设备自动计算出的新曝光值生效后，才能对拍照帧应用新的曝光参数或者重新曝光再出图，这样会带来成像速度变慢的问题。该生效可以是指相机要重新配置并使能相关部件采用新的曝光参数，例如重配并使能快门及其控制器以生效新曝光时间、重配并使能曝光增益电路(如传感器增益电路、信号处理器增益电路)及其控制器以生效新曝光增益。
2.快门迟滞比静物拍照严重。快门迟滞是指用户按下快门的时刻到实际成像时刻的延迟。如上面1中所述，由于需要等待新曝光值生效，因此从快门按键被按下到快门启动捕获影像的时间更长。
3.成像质量差于静物拍照。由于智能设备为运动抓拍自动计算出了新的较快的快门速度，因此，需要同时提高曝光增益以避免成像画面亮度降低，但提高曝光增益会引入更多彩色噪声和亮度噪声，降低成像质量。
4.水波纹(banding)问题。利用卷帘快门摄像头在闪烁光源下拍照，若卷帘快门的曝光时间小于光能量周期，则成像画面中会出现曝光不均匀的现象，呈现“水波纹”。
快门可分为全局快门(global shutter)和卷帘快门(rolling shutter)，二者的曝光方式不一样。其中，全局快门的曝光方式是：感光片上所有像素同时开始感光，经历相同的感光时长后又同时结束感光；卷帘快门的曝光方式是：感光片上每行像素滚动式的逐行曝光，每行曝光时间相同，直到全部像素曝光完毕，不同行的曝光起始时间、结束时间不同。
利用卷帘快门摄像头拍摄运动中的物体可能导致成像画面中的运动物体出现拖影(简称运动拖影)，尤其是运动物体过快的时候，这种形变问题更严重。而且，快门速度越慢(曝光时间越长)，运动拖影问题越严重。解决该问题的方式是提高快门速度。
利用卷帘快门摄像头在某些光源下拍照还可能会导致成像画面中出现“水波纹”。这些光源往往是交流电驱动的，交流电传输的能量并不是恒定不变的，而是随着一个固定频率(例如50Hz或60Hz)变化，因此会导致光源光强出现周期性波动。所遵循的周期便是该固定频率(又称为工频)对应的光能量周期。例如，50Hz交流电光源的光强波动频率是100Hz，光能量周期是10毫秒，即光源1秒钟发生100次闪烁。本文中，又将这种光源称为闪烁光源。
对于卷帘快门摄像头来说，当曝光时间小于光源的光能量周期时，由于感光片上每一行像素的曝光起始、结束时间不同，因此各行像素在同样的曝光时长内累积接收到的光能量不一样，成像画面上就会出现明暗相见的条纹。
本申请各个实施例致力于解决上述部分或全部问题，下面结合附图进行说明。
图1示出了实施例一提供的运动抓拍方法。该方法可以解决运动抓拍时成像速度差于静物拍照、快门迟滞比静物拍照严重的问题。
如图1所示，该方法可包括：
S401、电子设备启动第一摄像头。
S402、电子设备在显示屏上显示预览图像，该预览图像来自第一摄像头采集的图像。
随着时间流逝，S402可以继续被执行，继续显示来自第一摄像头的预览流，以持续向用户提供预览显示。
此外，电子设备还可以提供预览帧缓存功能，以缓存最新的几帧图像(下文称为缓存帧)，这样待用户产生拍照意图时，便可以迅速从缓存帧中取出用于生成照片的图像帧。本文中，最新缓存的多帧图像可以是指，早于拍照时间但距离拍照时间最近的多帧预览图像。拍照时间可以是指电子设备检测到拍照操作的时间，例如用户按下快门键的时间。
S403、在预览时，电子设备判断预览画面中是否有运动物体。若有运动物体，则执行S405；若没有运动物体，则执行S404。
关于如何判断预览画面中是否有运动物体，后文会介绍，这里先不展开。
S404、电子设备在预览时使第一摄像头维持正常曝光。
这里，对于控制曝光时间来说，S404算不上一个额外要执行的动作，电子设备不改变第一摄像头的曝光时间(维持第一时长)便可以实现第一摄像头维持正常曝光。
S405、电子设备在预览时降低第一摄像头的曝光时间。例如，降低曝光时间至短于光能量周期。
S406、电子设备可检测到拍照操作，例如快门按键被按下。
不限于快门按键被按下，拍照操作还可以包括他用户操作，如特定内容的语音指令(如语音指令“YOYO，请拍照！”)、特定动作(如跳跃动作)、特定表情(如大笑)等。
S407、响应于拍照操作，电子设备保存照片，该照片可来自于缓存帧。
可以看出，实施例一中，电子设备在预览时检测到运动物体就降低曝光时间，进入短曝光模式，而不是在用户按下快门后才降低曝光时间。如此，缓存的预览帧便带有降曝效果，之后一旦检测到拍照操作，电子设备便可以基于缓存的预览帧获得要保存的照片，而无需等待短曝光时间生效后才能出图，因此成像速度慢和快门迟滞的问题便得到改善。
图2示例性示出降曝前后的图像质量变化。如图2所示，第1帧、第2帧为初始的正常曝光时的图像，此时曝光时间为30ms，预览画面中转动的风车出现运动拖影，图像是模糊的。第51帧、第52帧为预览时降低曝光时间后采集的图像，曝光时间降低成10ms，预览画面中转动的风车不再有运动拖影，图像是清晰的。第53帧、第54帧为用户按下快门键时的图像，由于快门按压操作导致手机明显抖动，图像出现整体模糊。第55帧位拍照操作后采集的图像，此时设备回归平稳，曝光时间为10ms，图像整体清晰。本实施例中，响 应用户点击快门的拍照操作，电子设备可以基于预览时缓存的带有降曝效果的图像帧(如第51帧、第52帧)生成照片，不仅能避免运动拖影问题，还能迅速产生照片。
实施例一中，判断预览画面中是否有运动物体(S403)的具体实现可包括：通过在相邻预览帧之间使用光流法去检测运动物体，若像素移动面积达到一定面积阈值，则认为预览画面中有运动物体。这种方式检测出的运动是摄像头和被拍摄物体之间的相对运动，具体可分为三种情况：情况一，摄像头保持静止，被拍摄物体处于运动状态；情况二，被拍摄物体保持静止，摄像头处于运动状态；情况三，被拍摄物体和摄像头都在运动，但二者的运动速度不同。这三种情况中提及的运动是指相对于地面的运动，可以称为绝对运动。
不限于通过光流法，电子设备也可以通过陀螺仪、加速度计等运动传感器检测电子设备自身(摄像头)是否处于运动状态，一旦检测到电子设备处于运动状态，也可以进而认为预览画面中有运动物体。因为，当摄像头处于运动状态时，被拍摄物体在预览画面中大概率是运动的，仅被拍摄物体和摄像头都运动且运动速度相同这种特殊情况被排除在外。
实施例一中，电子设备在降低第一摄像头的曝光时间时，还可以进一步检测预览画面中运动物体的运动速度(也是一种相对速度)，并根据运动速度控制降曝：运动速度越快，曝光时间被降低的越多，降低后的曝光时间越短。运动速度可以通过像素移动距离和帧间隔来确定，例如对于固定帧间隔的相邻两帧图像，在这两帧图像之间运动物体的像素移动距离越大，运动速度就越大。
图3示出了实施例二提供的运动抓拍方法。基于实施例一所解决的技术问题，实施例二提供的方法可以进一步改善成像质量差于静物拍照的问题。
如图3所示，该方法可包括：
预览时降低曝光时间
S501、电子设备启动第一摄像头。
S502、电子设备在显示屏上显示预览图像，该预览图像来自第一摄像头采集的图像。
S503、在预览时，电子设备判断预览画面中是否有运动物体。若有运动物体，则执行S505；若没有运动物体，则执行S504。
S504、电子设备在预览时使第一摄像头维持正常曝光。
S505、电子设备在预览时降低所述第一摄像头的曝光时间。
以上S501-S505可参考实施例一中的步骤S401-S405，这里不再赘述。
另外，预览具体可以是高动态范围(high dynamic range，HDR)预览，此时电子设备还可以在不同帧时刻应用不同EV以获得亮度不同(EV不同)的多个缓存帧。具体的，电子设备可以通过变动曝光增益、曝光时间以获得EV不同的多帧预览图像。例如，在不同帧时刻采用相同的曝光时间但不同的曝光增益以实现EV不同，或者在不同帧时刻采用不同的曝光时间且不同的曝光增益以实现EV不同，或者在不同帧时刻采用相同曝光增益但不同曝光时间以实现EV不同。
多帧融合降噪
S506、电子设备可检测到拍照操作，例如快门按键被按下。
S507、电子设备判断预览是否是HDR预览，若不是，则执行S508-S511，若是，则执行S512-S515。
S508、电子设备可以从缓存的预览帧中选择出参考帧。
参考实施例一可知，缓存的预览帧是指最新缓存的多帧图像，它们是早于拍照时间但距离拍照时间最近的多帧图像。
参考帧可以是缓存的多个预览帧中清晰度最高的，S509中选择出的多帧图像根据参考帧做对齐，消除运动对清晰度的影响，增强细节。多帧图像对齐采用的算法可以是视频编解码技术中常用的运动估计和运动补偿算法，即：整张图像会被分为一系列不重叠的块，通过计算每个块与参考帧每个块的平方误差和(sum of squared difference，SSD)或者绝对误差和(sum of absolute difference，SAD)最小值，从而估计出当前块在参考帧所对应的位置或者说运动向量，然后基于此进行补偿。
具体的，电子设备可以根据清晰度算子(如sobel算子)分析这多个预览帧的图像清晰度，还可以进一步考虑这多个预览帧时刻的设备抖动因子、快门时间等来选择参考帧。其中，设备抖动因子可以根据陀螺仪、加速度计等运动传感器采集的运动数据来确定，实质上是反映设备(摄像头)抖动程度的因子，它越大则表示设备抖动越厉害，越会导致图像模糊，不利于图像的清晰。快门时间越短，运动拖影问题越轻微，越利于图像的清晰。
电子设备可以具体运行一个算法模型来考虑清晰度算子、设备抖动因子、快门时间中一项或多项对图像清晰度的影响，该算法模型可以是神经网络模型。
S509、电子设备可以从余下的缓存帧中选择出用于多帧融合降噪的多帧图像，余下的缓存帧是指除参考帧之外的缓存帧。
S510、电子设备可以利用选择出的用于多帧融合降噪的多帧图像进行多帧融合降噪(如多帧时域降噪)，同时基于参考帧增强细节，得到融合处理后的图像帧。
多帧融合降噪的方法可以很多，例如加和求平均，本实施例对此不作限制。
S511、电子设备将融合处理后的图像帧保存为照片。
也即，实施例一中S407提及的保存的照片来自于缓存帧，在这里具体是指：保存的照片来自S508选择出的参考帧和S509选出的多帧图像，是它们融合产生的。
S512、电子设备可以从缓存的预览帧中选择出参考帧。
S513、电子设备可以从余下的缓存帧中选择出曝光值EV不同的多帧图像，余下的缓存帧是指除参考帧之外的缓存帧。
实施S512的前提是，为实行HDR融合，电子设备在预览中降低曝光时间的同时，还变动了曝光增益、曝光时间以获得EV不同的多帧预览图像。例如，在满足相对于第一时长缩短曝光时间的大前提下，在不同帧时刻采用相同的曝光时间但不同的曝光增益以实现EV不同，或者在不同帧时刻采用不同的曝光时间且不同的曝光增益以实现EV不同，或者在不同帧时刻采用相同曝光增益但不同曝光时间以实现EV不同。
S514、电子设备利用选择出的参考帧和EV不同的多帧图像进行HDR融合，得到融合处理后的图像帧。
和多帧融合降噪一样，HDR融合也包括基于参考帧对另外选择出的EV不同的多帧图像进行多帧图像对齐，然后对该多帧图像进行多帧融合降噪。二者的不同在于，HDR融合还可以基于亮度不同(EV不同)的多帧图像进行亮度融合(又称曝光融合)。
S515、电子设备将HDR融合处理后的图像帧保存为照片。
也即，实施例一中S407提及的保存的照片来自于缓存帧，在这里具体是指：保存的照片来自S512选择出的参考帧和S513选出的EV不同的多帧图像，是它们融合产生的。
可以看出，待用户产生拍照意图时，实施例二中电子设备可迅速从缓存帧中取出用于生成照片的多帧图像，并进行多帧图像融合，可以降低图像噪声和增强细节，提高了图像质量；而且，针对HDR拍摄场景，还额外进行了多个亮度帧的曝光出图，实施了多帧亮度融合。同时针对运动抓拍的情况，特殊适配HDR算法或者多帧融合的算法效果参数和ISP参数，针对高曝光增益和短曝光时间进行效果调试。其中，算法效果参数可例如降噪强度、对比度强度、锐化强度、融合比例、鬼影阈值、运动阈值、匹配阈值、融合金字塔尺寸等常见融合、降噪、动态范围类算法的可调整参数(参数会因算法原型不同而差异)。ISP调试参数可例如自动曝光、自动对焦、自动白平衡参数(又称为3A参数)、镜头阴影校正(lens shading correction，LSC)、伽马(GAMMA)、动态范围校正(dynamic range correction，DRC)、YUV降噪(YUV noise reduction，YNR)等图像调试效果参数，比如，YNR是ISP常见的YUV降噪参数，通过对运动时的高曝光增益针对性调整，来降低其带来的噪声影响。这样通过选帧、图像融合算法和参数分离来弥补曝光增益增大来的噪声等问题，同时增强图像效果。
图4示出了实施例三提供的运动抓拍方法。本实施例中，第一摄像头可以是卷帘快门摄像头，在闪烁光源下，其提供的预览图像中可能会在水波纹。基于实施例二所解决的技术问题，实施例三提供的方法可以进一步解决水波纹(banding)问题。
如图4所示，该方法可包括：
预览时降低曝光时间
S601、电子设备启动第一摄像头。
S602、电子设备在显示屏上显示预览图像，该预览图像来自第一摄像头采集的图像。
S603、在预览时，电子设备判断预览画面中是否有运动物体。若有运动物体，则执行S604；若没有运动物体，则执行S605。
S604、电子设备在预览时使第一摄像头维持正常曝光。
以上S601-S604可参考实施例二中的步骤S501-S504，这里不再赘述。
S605、电子设备在预览时降低所述第一摄像头的曝光时间。
本实施例中，电子设备在降低曝光时间的同时还可以兼顾水波纹问题，S605的实现具体可包括S6051-S6055：
S6051、判断是否进行flicker检测。
S6052、启动闪烁光源检测装置。
闪烁光源检测装置可以包括：环境光传感器(ambient light sensor，ALS)、多光谱传感器。其中，ALS运行在高频率F下，根据香农定理，当F高于光能量周期F’的两倍时， 可以通过读取ALS的数据恢复闪烁光源的能量变化情况，从而确认当前环境是否存在flicker光源。多光谱传感器可直接读出闪烁光源的能量变化信息。
也即，可以采用高采样率的环境光传感器、多光谱闪烁检测器等专用flicker检测器去检测拍摄环境中是否存在闪烁光源，配合摄像头以解决水波纹问题。但是，这类器件需要花费额外成本和设计空间。
为此，闪烁光源检测装置也可以是电子设备上的另一个卷帘快门摄像头(称为第二摄像头)。
第二摄像头并非为了闪烁光源检测专门引入的，通常它被用作景深摄像头、广角摄像头或者微距摄像头，其曝光时间可以被设置成第二时长，第二时长比光能量周期短，例如9ms，从而第二摄像头在50Hz或者60Hz闪烁光源下采集的图像必然会产生水波纹，闪烁光源能够被检测出。第二时长可进一步设定成满足多种工频下检测闪烁光源的曝光时间，即小于这多种工频的光能量周期中的最小光能量周期。以50Hz和60Hz这两种工频为例，第二时长可以设置成7ms，其值比60Hz光源的光能量周期(约8.3毫秒)还短。
另外，正常曝光所指的曝光时间可设置成光能量周期的整数倍，可避免正常曝光时用户在预览时看到水波纹。
关于如何利用第二摄像头进行闪烁光源检测，后面图8和图11实施例会详细说明。
S6053、通过闪烁光源检测装置进行闪烁光源检测，若未检测出闪烁光源，则执行S6054，否则执行S6055。
具体的，当闪烁光源检测装置是第二摄像头时，S6053的具体实现可包括：电子设备对第二摄像头采集的图像进行图像识别，判断其图像内容中是否包含水波纹特征，若包含，则认为拍摄环境中存在闪烁光源。第二摄像头采集的图像不会被送显，仅用于flicker检测，因此用户不感知。
S6054、忽略水波纹问题，按照预期降低曝光时间，例如降低该曝光时间到光能量周期以下，例如9毫秒。
这里，预期可以是指：根据运动物体的运动速度、HDR等场景下的图像质量需要去降低曝光时间，例如当物体运动速度很快时曝光时间通常要降低到很短，如5毫秒，短于光能量周期。
S6055、若检测出闪烁光源，则可以控制降低后的曝光时间是光能量周期的整数倍(如1倍、2倍等)，曝光时间最短可设置成1倍光能量周期那么长，如10ms。
如S6051所示，在启动flicker检测之前，电子设备还可以根据自动曝光控制(auto exposure control，AEC)对第一摄像头设定的曝光时间判断出环境亮度合适，如室内光照环境，进而确定出：需要进行flicker检测。因为，AEC设定的感光传感器的曝光时间能在一定程度上反映出拍摄环境的亮度。
太亮或太暗的拍摄环境下一般不需要flicker检测，因为，太亮环境下主要考虑过曝问题而需要明显降低曝光时间，难以兼顾水波纹问题；太暗环境下一般没有降低曝光时间到光能量周期以下的需求，不然容易出现曝光不足的问题。这里，太亮或者太暗可以参考电子设备100上的曝光表来定义。例如，将曝光表中高EV对应的几个最短曝光时间确定成太亮环境的曝光时间。一旦AEC曝光时间属于这几个最短曝光时间，则认为环境太亮。又 例如，将曝光表中低EV对应的几个最长曝光时间确定成太暗环境的曝光时间。一旦AEC曝光时间属于这几个最长曝光时间，则认为环境太暗。也即，太亮或太暗的参考曝光值，可以来源于曝光表。那么，若AEC曝光时间属于曝光表中余下的曝光时间则可以反映环境亮度既不是太亮也不是太暗。不限于通过AEC给第一摄像头设定的曝光时间的长短分析环境亮度，电子设备也可以通过环境光亮度传感器采集环境光从而判断是否是太亮或太暗的拍摄环境，本申请对其技术实现不作限制。
多帧融合降噪
S606、电子设备可检测到拍照操作，例如快门按键被按下。
S607、电子设备判断预览是否是HDR预览，若不是，则执行S608-S611，若是，则执行S612-S615。
S608、电子设备可以从缓存的预览帧中选择出参考帧。
S609、电子设备可以从余下的缓存帧中选择出用于多帧融合降噪的多帧图像，余下的缓存帧是指除参考帧之外的缓存帧。
S610、电子设备可以利用选择出的用于多帧融合降噪的多帧图像进行多帧融合降噪(如多帧时域降噪)，同时基于参考帧增强细节，得到融合处理后的图像帧。
S611、电子设备将融合处理后的图像帧保存为照片。
S612、电子设备可以从缓存的预览帧中选择出参考帧。
S613、电子设备可以从余下的缓存帧中选择出曝光值EV不同的多帧图像，余下的缓存帧是指除参考帧之外的缓存帧。
S614、电子设备利用选择出的参考帧和EV不同的多帧图像进行HDR融合，得到融合处理后的图像帧。
S615、电子设备将HDR融合处理后的图像帧保存为照片。
以上S606-S615可参考实施例二中的步骤S506-S515，这里不再赘述。
另外，S6051-S6055所描述的降低曝光时间的具体实现也可以结合到实施一中，作为实施例一中步骤S405的实现方式。
利用卷帘快门摄像头在某些光源下拍照可能会导致成像画面中出现“水波纹(banding)”。这些光源往往是交流电驱动的，交流电传输的能量并不是恒定不变的，而是随着一个固定频率(例如50Hz或60Hz)变化，因此会导致光源光强出现周期性波动。所遵循的周期便是该固定频率(又称为工频)对应的光能量周期。例如，50Hz交流电光源的光强波动频率是100Hz，光能量周期是10毫秒，即光源1秒钟发生100次闪烁。本文中，又将这种光源称为闪烁光源。
对于卷帘快门摄像头来说，当曝光时间小于光源的光能量周期时，由于感光片上每一行像素的曝光起始、结束时间不同，因此各行像素在同样的曝光时长内累积接收到的光能量不一样，成像画面上就会出现明暗相见的条纹。例如，如图5所示，在成像第1帧图像时，感光片上第1行像素接收到的光能量是0时刻至t2时刻积累的光能量，而感光片上第n行像素接收到的光能量是t1时刻至5毫秒积累的光能量，第n行接收到的光能量小于第1行接收到的光能量。又例如，如图5所示，在成像第N+1帧图像时，感光片上第1行像 素接收到的光能量是t3时刻至t5时刻积累的光能量，而感光片上第n行像素接收到的光能量是t4时刻至t6时刻积累的光能量，第1行接收到的光能量小于第n行接收到的光能量。一帧成像画面中，光能量大的行呈现为亮条纹，而光能量小的行呈现为暗条纹，可如图6所示。明暗相间的条纹就是水波纹，这种现象也被称为闪烁(flicker)现象。而且，在拍摄预览或者录像时，用户还可以观察到水波纹滚动。
在快门速度较快的拍摄场景下，例如运动抓拍、高动态范围(high dynamic range，HDR)等拍摄模式，成像画面中更容易出现水波纹。在检测出拍摄环境中的光源是闪烁光源之后，避免该问题的一种方式是将曝光时间设置成光能量周期的整数倍。如图7所示，光能量周期是10毫秒，可以将曝光时间也设置成10毫秒，即光能量周期的1倍。
闪烁光源的检测是解决水波纹问题的关键。一种检测方法是，通过图像识别算法识别图像内容中的水波纹特征，确认成像画面中存在水波纹之后再调整曝光时间以规避水波纹。但是，在水波纹被消除之前，用户还是会看到水波纹产生，用户体验不佳。另一种检测方法是，采用高采样率的环境光传感器、多光谱闪烁检测器等专用flicker检测器去检测拍摄环境中是否存在闪烁光源，配合摄像头以解决水波纹问题。但是，这类器件需要花费额外成本和设计空间。
本申请实施例提供了一种闪烁光源检测方法，可检测出闪烁光源，而且不需要引入额外的专用检测器件。
实现该方法的电子设备具有两个或更多卷帘快门摄像头，在一个或一些摄像头用于图像预览显示时，余下摄像头中选择出用来检测闪烁光源的摄像头。用于预览显示的摄像头称为第一摄像头，用于闪烁光源检测(又称flicker检测)的摄像头称为第二摄像头。第二摄像头并非为了闪烁光源检测专门引入的，通常它被用作景深摄像头、广角摄像头或者微距摄像头。第一摄像头、第二摄像头的快门都是卷帘快门。
图8示出了本申请实施例提供的闪烁光源检测方法的总体流程。下面展开。
S701，电子设备启动第一摄像头。
第一摄像头的启动可以是用户打开相机应用这一事件触发的。此时，相机应用的拍摄模式可以为普通拍摄模式，变焦倍率为1倍，第一摄像头可以是后置主摄像头，第二摄像头便从余下摄像头中选择。后置主摄像头可以是电子设备上像素最高的一个，其焦距通常介于广角摄像头的焦距和长焦摄像头的焦距之间。
具体的，在启动第一摄像头时，可以将第一摄像头的曝光时间设置成第一时长，第一时长大于光能量周期且为光能量周期的整数倍，例如50ms，这样在摄像头启动之初都能避免用户看到水波纹的产生。不像运动抓拍模式、HDR拍照模式那样需要降低曝光时间，普通拍摄模式下第一摄像头的曝光时间可以比较长，如50ms、100ms等光能量周期的高倍数。第一时长这个曝光时间可以是ISP通过AEC为第一摄像头设定的。光能量周期可以不限定成50Hz工频下的光能量周期，也可以是60Hz工频下的光能量周期，还可以是其他工频下的光能量周期。
不限于用户打开相机应用这一场景，启动第一摄像头的场景也可以是其他需要启动摄像头采集图像的场景，例如视频通话场景、扫码支付、身份验证等场景。例如，在扫码支付场景，检测到用户开启扫码支付时，电子设备可启动后置主摄像头以扫描支付二维码， 此时第一摄像头是后置主摄像头。又例如，在身份验证场景，电子设备可启动前置主摄像头以及前置景深摄像头以扫描获得人脸特征数据，进行用户身份验证，此时第一摄像头可以是前置主摄像头。又例如，在视频通话场景，电子设备可启动前置主摄像头以采集屏前用户的人像画面。
第一摄像头采集的图像会被送显，从而用作相机应用展示于显示屏上的预览图像。
S702，电子设备在显示屏上显示第一摄像头采集的图像。
第一摄像头可以是一个摄像头，此时相机应用展示于显示屏上的预览图像仅来自该摄像头。第一摄像头也可以是多个摄像头，此时相机应用展示于显示屏上的预览图像由这多个摄像头同时采集的图像拼接构成，以向用户提供多摄像头拍照或录像的功能。
随着时间流逝，S702可以继续被执行，继续显示来自第一摄像头的预览流，以持续向用户提供预览显示。
S703，电子设备启动第二摄像头，并将第二摄像头的曝光时间设置成第二时长，第二时长比光能量周期短，例如9ms，从而第二摄像头在50Hz或者60Hz闪烁光源下采集的图像必然会产生水波纹，闪烁光源能够被检测出。
在执行S703之前，电子设备还可以根据AEC对第一摄像头设定的曝光时间判断出环境亮度合适，如室内光照环境，进而确定出：需要进行flicker检测。因为，AEC设定的感光传感器的曝光时间能在一定程度上反映出拍摄环境的亮度。
太亮或太暗的拍摄环境下一般不需要flicker检测，因为，太亮环境下主要考虑过曝问题而需要明显降低曝光时间，难以兼顾水波纹问题；太暗环境下一般没有降低曝光时间到光能量周期以下的需求，不然容易出现曝光不足的问题。这里，太亮或者太暗可以参考电子设备100上的曝光表来定义。例如，将曝光表中高EV对应的几个最短曝光时间确定成太亮环境的曝光时间。一旦AEC曝光时间属于这几个最短曝光时间，则认为环境太亮。又例如，将曝光表中低EV对应的几个最长曝光时间确定成太暗环境的曝光时间。一旦AEC曝光时间属于这几个最长曝光时间，则认为环境太暗。也即，太亮或太暗的参考曝光值，可以来源于曝光表。那么，若AEC曝光时间属于曝光表中余下的曝光时间则可以反映环境亮度既不是太亮也不是太暗。不限于通过AEC给第一摄像头设定的曝光时间的长短分析环境亮度，电子设备也可以通过环境光亮度传感器采集环境光从而判断是否是太亮或太暗的拍摄环境，本申请对其技术实现不作限制。
S704，电子设备对第二摄像头采集的图像(不送显)进行图像识别，判断其图像内容中是否包含水波纹特征，若包含，则控制第一摄像头的曝光时长是光能量周期的整数倍。第二摄像头采集的图像不会被送显，仅用于flicker检测，因此用户不感知。flicker检测结果可包括：检测出闪烁光源、未检测出闪烁光源。若第二摄像头采集的图像内容中包含水波纹特征，则认为拍摄环境中存在闪烁光源。此后，电子设备可以根据flicker检测结果对第一摄像头进行曝光控制，控制第一摄像头的曝光时间是光能量周期的整数倍，以避免第一摄像头提供的预览图像中出现水波纹。
例如，当拍照模式从普通拍照模式切换到运动抓拍模式时，在降低曝光时间以减轻运动拖影问题时，电子设备可具体将第一摄像头的曝光时间降低成光能量周期的一倍或者两 倍等小数值整数倍，即控制降低后的曝光时间是光能量周期的整数倍，以同时避免闪烁光源下成像画面中出现水波纹的问题。
第一摄像头、第二摄像头具体是哪个摄像头不是一成不变的。
在用户打开相机应用之初，电子设备可默认开启普通拍照模式，变焦倍率为1倍，此时第一摄像头可以是主摄像头，第二摄像头便可以从余下摄像头中选择出。
但是，在拍摄模式发生变化或者变焦倍率发生变化时，第一摄像头可以从主摄像头切换到其他摄像头。
例如，在拍照预览、录像预览或视频录制时，若检测到用户提高变焦倍率至长焦摄像头的变焦范围，则电子设备可以将第一摄像头从主摄像头切换到长焦摄像头。又例如，在拍照预览、录像预览或视频录制时，若检测到用户降低变焦倍率至广角摄像头的变焦范围，则电子设备可以将第一摄像头从主摄像头切换到广角摄像头。又例如，若检测到拍照模式从普通模式切换成微距拍摄模式，则电子设备可以将第一摄像头从主摄像头切换到微距摄像头。
第二摄像头可以选择切换成原来的第一摄像头，尤其是针对电子设备只具有两个摄像头的情况，无需再去启动新的摄像头，响应速度更快。当然，实际应用中，也可以基于功耗等问题考虑将第二摄像头切换到功耗较低的摄像头，本申请实施例对此不作限制。第二摄像头也可以固定下来，例如景深摄像头，这类摄像头采集的图像数据不参与成预览图像显示，仅用作其他摄像头的辅助数据(如景深数据)，例如人脸面部的深度数据，因而不会因拍照模式改变而被占用成第一摄像头。
接下来，介绍本申请实施例提供的电子设备的软硬件架构。
电子设备可以是搭载或者其它操作系统的便携式终端设备，例如手机、平板电脑、桌面型计算机、膝上型计算机、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本，以及蜂窝电话、个人数字助理(personal digital assistant，PDA)、增强现实(augmented reality，AR)设备、虚拟现实(virtual reality，VR)设备、人工智能(artificial intelligence,AI)设备、可穿戴式设备、车载设备、智能家居设备和/或智慧城市设备，等等。
图9示例性示出了本申请实施例提供的电子设备100。电子设备100可解决运动抓拍时成像速度差于静物拍照、快门迟滞比静物拍照严重的问题。
如图9所示，电子设备100可以包括：处理器110，存储器120，摄像头130，显示屏140，以及闪烁光源检测装置(未示出)。其中：
处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。处理器110可以包括一个或多个接口，例如集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal  asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。这些接口用于处理器110与外设进行数据交互。
存储器120可包括一个或多个随机存取存储器(random access memory，RAM)和一个或多个非易失性存储器(non-volatile memory，NVM)。随机存取存储器可以由处理器110直接进行读写，可以用于存储操作系统或其他正在运行中的程序的可执行程序(例如机器指令)，还可以用于存储用户及应用程序的数据等。非易失性存储器也可以存储可执行程序和存储用户及应用程序的数据等，可以提前加载到随机存取存储器中，用于处理器110直接进行读写。处理器110中也可以设置存储单元，该存储单元可以为高速缓冲存储单元，可用于保存处理器110刚用过或循环使用的指令或数据。本申请实施例提供的闪烁光源检测方法的实现代码可存储在NVM中。在启动相机应用时，该代码可被加载到RAM中。这样，处理器110可从RAM中直接读取该程序代码，实现本申请实施例提供的闪烁光源检测方法。此外，用户使用相机应用获得的照片、视频等图像文件可被写入到NVM中保存，以供用户浏览。
摄像头130采集的图像可以被送显，从而提供预览图像。摄像头130可包括：镜头、感光传感器和柔性印刷电路板(flexible printed circuit board，FPCB)部分。FPCB负责将摄像头130的其他组件与处理器110连接起来，例如将感光传感器输出的原始数据传输到处理器110。在拍照时，摄像头130的快门被打开，光线射入并照到感光传感器上。感光传感器将光信号转换为电信号，再通过模数转换(analog digital convert，ADC)进一步将电信号转换成数字信号，以传递给ISP处理。ISP可对感光传感器的输出数据执行以下处理：自动曝光控制(auto exposure control，AEC)、自动增益控制(auto gain control，AGC)、自动白平衡(auto white balance，AWB)、色彩校正、祛除坏点等等。ISP也可以集成于摄像头130内。
显示屏140可用于显示摄像头130采集的图像。ISP处理后的图像会被送显，以在显示屏140上向用户展示摄像头采集的图像的预览。这里，送显是指，把摄像头采集的图像推送到帧缓存(frame buffer，FB)中存放。帧缓存是一段存储空间，可以位于显存，也可以位于内存，用于存储显卡芯片处理过的或者将要提取的渲染数据。帧缓存的内容对应于显示屏140上的界面显示，可以将其简单理解为显示屏140上显示内容对应的缓存，修改帧缓存中的内容，就是修改显示屏140上的内容。
闪烁光源检测装置可以用于flicker检测，具体可包括环境光传感器(ALS)、多光谱传感器。其中，ALS运行在高频率F下，根据香农定理，当F高于光能量周期F’的两倍时，可以通过读取ALS的数据恢复闪烁光源的能量变化情况，从而确认当前环境是否存在flicker光源。多光谱传感器可直接读出闪烁光源的能量变化信息。闪烁光源检测装置也可以是电子设备上的一个卷帘快门摄像头(称为第二摄像头)。第二摄像头采集的图像不会被送显，仅用于flicker检测，因此用户不感知。第二摄像头并非为了flicker检测专门引入的，通常它可以是景深摄像头、广角摄像头或者微距摄像头等等，其分辨率较低、功耗较小， 属于非主要相机，本申请实施例赋予了它新的功能。第二摄像头也可以是一个或多个摄像头。
除了第二摄像头提供的flicker检测结果，提供预览显示的摄像头(称为第一摄像头)的曝光时间还可以受自动曝光控制(AEC)这一功能控制。AEC简言之就是根据光强度自动调节曝光时间。不同环境下，光照的强度有着很大的差别，摄像头130中的感光传感器的曝光时间的长短也需要随之相适应，以确保图像的正常曝光。提供这种自适应能力的便是ISP提供的自动曝光控制(AEC)。因此，感光传感器的曝光时间能在一定程度上反映出拍摄环境的亮度。
另外，电子设备100中还可存储AEC要使用的曝光表。一个摄像头的曝光表设定了该摄像头在不同曝光值(exposure value，EV)下该采用的曝光参数(如曝光时间、曝光增益、光圈大小)，可由工程师调试记录而成。
如图9所示，电子设备100还可以包括：音频模块150，扬声器150A，受话器150B，麦克风150C，耳机接口150D。
其中，电子设备可以通过音频模块150，扬声器150A，受话器150B，麦克风150C，耳机接口150D，以及应用处理器(AP)等实现音频功能。例如音乐播放，录音等。音频模块150用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块150还可以用于对音频信号编码和解码。在一些实施例中，音频模块150可以设置于处理器110中，或将音频模块150的部分功能模块设置于处理器110中。扬声器150A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备可以通过扬声器150A收听音乐，或收听免提通话。受话器150B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备接听电话或语音信息时，可以通过将受话器150B靠近人耳接听语音。麦克风150C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风150C发声，将声音信号输入到麦克风150C。电子设备可以设置至少一个麦克风150C。在另一些实施例中，电子设备可以设置两个麦克风150C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备还可以设置三个，四个或更多麦克风150C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。耳机接口150D用于连接有线耳机。耳机接口150D可以是USB接口，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
如图9所示，电子设备100还可以包括传感器模块160，具体可以包括压力传感器160A，距离传感器160F，接近光传感器160G，触摸传感器160K，环境光传感器160L等。
图9示意的结构并不构成对电子设备的具体限定，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
图10示出了电子设备100上利用另一颗卷帘快门摄像头进行闪烁光源检测所涉及的主要软件架构。
如图10所示，该软件架构可包括如下多个功能或模块：相机应用(camera apk)、相机 服务(camera service)、相机硬件抽象层(camera HAL)、闪烁检测(CameraFlickerDetector)、camera VNDK以及Kernel部分。其中，
相机应用(camera apk)可位于应用程序层。除了相机应用，应用程序层可以包括一系列其他应用程序，例如图库应用。相机应用可主要负责人机交互，例如监听及响应选择拍照模式、调整变焦倍率、选择焦点等用户操作。
相机服务(camera service)可位于应用框架层(framework)。相机服务(camera service)可包括相机管理和相机设备等接口，起到呈上启下的作用，可通过应用编程接口(application programinterface，API)上与相机应用交互，可通过HAL接口定义语言(HAL interface definition language，HIDL)下与camera HAL交互。另外，camera service还可以与供应商本地开发套件(vendor native development kit，VNDK)交互。
相机硬件抽象层(camera HAL)可以是硬件抽象层(hardware abstract layer,HAL)中关于相机的那一部分虚拟硬件功能。硬件抽象层位于应用框架层以及驱动层之间的接口层，为操作系统提供虚拟硬件平台，上层无需关心底层硬件实现。camera HAL可通过HIDL接口上与camera service通信，可通过标准的硬件抽象层接口下发针对摄像头的控制操作。camera HAL可以包括相机启动(CamEntry)模块、自动曝光控制处理器(AECProcessor)模块，其中，CamEntry模块可用作HAL的入口，接收来自camera service的开启(open)、关闭(close)、流配置(configurestream)、请求(request)等指令，是通用的相机HAL功能入口，AECProcessor模块可提供ISP的AEC有关功能，如向CameraFlickerDetector提供第一摄像头的AEC参数(如曝光时间)，便于CameraFlickerDetector判断当前拍摄环境是否需要进行flicker检测。
闪烁检测(CameraFlickerDetector)模块可位于应用框架层(framework)或者应用程序层。CameraFlickerDetector模块可通过HIDL接口从camera HAL获知第一摄像头的AEC参数(如曝光时间)，从而决定是否要启动第二摄像头进行flicker检测。在确定要进行flicker检测时，CameraFlickerDetector模块可通过camera VNDK的接口下发启动第二摄像头的控制操作，以及设定曝光时间的控制操作。CameraFlickerDetector模块可包括CameraHelper模块、ImageReader模块，其中，CameraHelper模块可用于通过VNDK接口的调用实现对第二摄像头的camera service的开启(open)、关闭(close)、流配置(configurestream)、请求(request)等功能的调用、ImageReader模块主要用于在第二摄像头启动后，送出的图像的接收和进一步处理。
供应商本地开发套件(VNDK)是一系列库的合集，用于让供应商开发自己的HAL(vendor)。camera VNDK是第二摄像头的供应商开发的“自己的camera HAL”。camera VNDK可用于接收CameraFlickerDetector下发的启动第二摄像头的控制操作，并向camera service中转该控制操作。camera VNDK可包括ACameraManager模块、ACameraDevice模块、ACameraReader模块、ACaptureSession模块、ACaptureRequest模块，其中，ACameraManager模块可提供从vendor层访问camera service的功能，ACameraDevice模块可提供从vendor层访问摄像头设备的功能，ACameraReader模块可负责读取图像数据，例如第二摄像头捕获的数据，ACaptureSession模块可提供让vendor可以管理摄像头捕获的帧的功能，ACaptureRequest模块可负责让摄像头捕获单张图像所需要的设置和输出目标。
Kernel部分可包括各种硬件的驱动控制模块，例如ISP、感光传感器、DSP、AP等驱动控制。
图10仅示出了本申请实施例提供的闪烁光源检测方法涉及的主要软件架构，它实质仅是电子设备100的软件架构的一部分。
图10还示出了利用另一个卷帘快门摄像头进行闪烁光源检测的流程，包括：
1.第一像头启动流程
相机应用调用相机服务(camera service)，camera service对相机应用进行鉴权。经过鉴权之后，camera service再调用camera HAL，以通过camera HAL启动第一摄像头，以及控制感光传感器和ISP出图。图像帧会被送显，以实现图像预览、拍照等功能。
在这个基础上，为了进行flicker检测，同时不让用户发觉，电子设备100可以启动第二摄像头来进行检测。第二摄像头并非为了flicker检测专门引入的，通常它可以是景深摄像头、广角摄像头或者微距摄像头等等，其分辨率较低、功耗较小，属于非主要相机，本申请实施例赋予了它新的功能。
2.flicker检测流程
首先，在第一摄像头的预览开启时，camera HAL中的AECProcessor模块可通过HIDL接口向CameraFlickerDetector模块传递第一摄像头的曝光时间等参数。AEC设定的曝光时间能够反映拍摄环境的亮度。因此，CameraFlickerDetector模块可以基于曝光时间确定是否要进行flicker检测。要进行flicker检测的环境通常为室内场景。太亮或太暗的拍摄环境下一般不需要检测flicker，因为，太亮环境下主要考虑过曝问题而需要明显降低曝光时间，难以兼顾水波纹规避问题；太暗环境下一般没有降低曝光时间到光能量周期以下的需求，不然容易出现曝光不足的问题。这里，太亮或者太暗可以参考电子设备100上的曝光表来定义。例如，将曝光表中高EV对应的几个最短曝光时间确定成太亮环境的曝光时间。一旦实际曝光时间属于这几个最短曝光时间，则认为环境太亮。又例如，将曝光表中低EV对应的几个最长曝光时间确定成太暗环境的曝光时间。一旦实际曝光时间属于这几个最长曝光时间，则认为环境太暗。
然后，在确定要进行flicker检测之后，CameraFlickerDetector模块向camera VNDK传递启动第二摄像头的控制命令。
camera VNDK继而向camera service传递CameraFlickerDetector模块启动第二摄像头的控制操作。
鉴权CameraFlickerDetector模块之后，camera service再调用camera HAL，以通过camera HAL启动第二摄像头，从而可实现从camera HAL侧启动第二摄像头，不涉及应用层，因而用户不感知。同时，CameraFlickerDetector模块对第二摄像头进行曝光控制，让第二摄像头曝光时间小于光能量周期，如设定曝光时间为9ms，从而第二摄像头在50Hz或者60Hz闪烁光源下采集的图像必然会产生水波纹，闪烁光源能够被检测出。CameraFlickerDetector模块还可以调用camera HAL读取第二摄像头的出图，通过图像识别技术检测图像内容是否存在水波纹特征，并将检测结果传递到第一摄像头。第二摄像头的出图不送显，仅用于flicker检测。
第一摄像头得知flicker检测结果后，可以在预览画面中检测到运动物体后，将第一摄像头的曝光时间降低成一倍光能量周期那么长，如10ms，以兼顾解决闪烁光源下成像画面有水波纹的问题。
以上软件架构新引入了CameraFlickerDetector模块，通过VNDK、HIDL实现了CameraFlickerDetector模块和camera HAL、camera service之间的交互，不涉及相机应用和各层架构的修改，具有良好的可移植性、可扩展性和可维护性。
下面结合图11深入电子设备100的内部架构说明本申请实施例的闪烁光源检测方法。图11仅以相机应用提供的拍照或录像预览为例示例性的阐述了该方法，以便于本领域技术人员深入了解本申请实施例，其细节不应构成对本申请保护范围的限制。下面展开。
普通拍照模式
S11.检测到打开相机应用的用户操作，应用处理器(AP)可以启动后置主摄像头。
此时，后置主摄像头用作第一摄像头。拍照模式可以为普通拍照模式，后置主摄像头的曝光时间比光能量周期长，例如50ms，光能量周期的多数倍。后置主摄像头的曝光时间可以是由ISP的AEC设定的。
在S11之前，显示屏上可显示系统桌面，其中包括相机应用的桌面图标。打开相机应用的用户操作可以是指用户点击相机应用的桌面图标。
S12.后置主摄像头将采集的原始图像帧(RAW)传输至ISP。
ISP可对后置主摄像头输出RAW图执行以下处理：自动曝光控制(AEC)、自动增益控制(AGC)、自动白平衡(AWB)、色彩校正、祛除坏点等等。ISP处理后的图像可以是彩色图，例如YUV图。
S13.ISP可以将输出的后置主摄像头的YUV图像帧传输至显示屏。
这里，S13仅示意图像数据的流转路径，不表示ISP直接向显示屏发送数据，其实现包括送显这一过程。后置主摄像头的YUV图像帧可以先被缓存到帧缓存(frame buffer)中，GPU在渲染拍照预览图像时从帧缓存中读取出后置主摄像头的YUV图像帧并显示于显示屏上。
S14.显示屏显示后置主摄像头的YUV图像帧。
从此，后置主摄像头开始提供预览图像，以实现普通拍照模式下的预览功能。
普通拍照模式下，即便存在闪烁光源，由于后置主摄像头的曝光时间是长于光能量周期的整数倍，因此预览图像中也不会出现水波纹。但是，闪烁光源的尽早检测是十分必要的，因为用户随时可能切换到运动抓拍模式或者HDR拍摄模式，这些拍摄模式有降低曝光时间的需求。如何能在这些拍摄模式降低第一摄像头的曝光时间的同时又不出现水波纹，这是本申请实施例着力解决的问题。
S15.AP可以从ISP获取AEC为后置主摄像头设定的曝光时间。AEC设定的曝光时间能够反映拍摄环境的亮度。
S16.AP可以根据AEC为后置主摄像头设定的曝光时间判断是否需要进行flicker检测。
要进行flicker检测的环境通常为室内场景，太亮或太暗的拍摄环境下一般不需要检测flicker，具体说明可参考前文，这里不再赘述。
flicker检测
S17.在确定要进行flicker检测之后，应用处理器(AP)可以启动景深摄像头，用作第二摄像头，并且设定景深摄像头的曝光时间小于光能量周期，如9毫秒。
由于第二摄像头曝光时间小于光能量周期，因此第二摄像头在闪烁光源下采集的图像必然会产生水波纹，闪烁光源能够被检测出。
在启动第一摄像头之后，AP也可以直接启动第二摄像头，而不去判断是否要进行flicker检测，即可以不执行S15-S16。
S18.被启动之后，景深摄像头可将采集的原始图像帧(RAW)传输至ISP。
ISP可以对该RAW图进行一系列处理并输出YUV图像帧。作为第二摄像头，景深摄像头的YUV图像帧不送显，即不会被传输至显示屏显示。
S19.AP可以从ISP获取景深摄像头的YUV图像帧。
S20.AP可以对景深摄像头的YUV图像帧进行图像识别，检测图像内容中是否包含水波纹特征，以得到flicker检测结果。若图像内容包含水波纹特征，则认为拍摄环境中存在闪烁光源，否则，认为拍摄环境中没有闪烁光源。
其中，AP可运行图10所示软件架构，尤其是CameraFlickerDetector模块，以实现flicker检测。随着时间流逝，flicker检测可以继续被执行，尤其是在第一摄像头开启期间。
此后，如S21-S22，拍照模式可能会被切换到运动抓拍模式或HDR模式，电子设备可基于flicker检测结果设置第一摄像头的曝光时间，以规避水波纹问题。
S21.AP获知拍照模式切换到运动抓拍模式或HDR模式。
S22.AP在降低后置主摄像头的曝光时间以满足运动抓拍模式或HDR模式的需求的同时，具体可以将曝光时间降低至光能量周期的1倍那么长，以兼顾水波纹规避问题。
不限于光能量周期的1倍，曝光时间也可以降低至光能量周期的2倍、3倍等低倍数，具体可视运动抓拍模式或HDR模式的降曝需求而定。
第一摄像头不是一成不变的。后续步骤示例了几种典型场景(变焦、翻转摄像头、多路拍照或录像)下第一摄像头的切换。
变焦
变焦场景下，提高变焦倍率至长焦摄像头的变焦范围，或者降低变焦倍率至广角摄像头的变焦范围，可以改变提供预览图像的第一摄像头。
S23.应用处理器(AP)检测到变焦倍率提高至长焦摄像头的变焦范围。
S24.应用处理器(AP)可以关闭后置主摄像头。
S25.应用处理器(AP)可以启动长焦摄像头，将长焦摄像头用作第一摄像头。
此时，若处于运动抓拍或HDR这类要求低曝光时间的拍摄模式，则在启动长焦摄像头时便可以将其曝光时间设置成1倍光能量周期那么长，可参考S26。
S27.长焦摄像头将采集的原始图像帧(RAW)传输至ISP。
ISP可以对该RAW图进行一系列处理并输出YUV图像帧。
S28.ISP可以将长焦摄像头的YUV图像帧传输至显示屏。
这里，S27仅示意图像数据的流转路径，不表示ISP直接向显示屏发送数据，其实现包括送显这一过程。
S29.显示屏显示长焦摄像头的YUV图像帧。
从此，长焦摄像头开始提供拍照预览图像，用于预览显示的第一摄像头便从后置主摄像头切换成长焦摄像头。降低变焦倍率至广角摄像头的变焦范围改变第一摄像头的实现流程类似，可参考S23-S28适应性的调整，这里不再展开。
基于flicker检测结果设置长焦摄像头曝光时间也可以避免其成像画面中出现水波纹。
翻转摄像头
在用户翻转摄像头的场景下，第一摄像头可从后置主摄像头切换成前置主摄像头。
S30.应用处理器(AP)检测到翻转摄像头。
S31.应用处理器(AP)可以关闭后置主摄像头。
S32.应用处理器(AP)可以启动前置主摄像头，将前置主摄像头用作第一摄像头。
此时，若前置拍摄默认处于运动抓拍或HDR这类要求低曝光时间的拍摄模式，则在启动前置主摄像头时便可以将其曝光时间设置成1倍光能量周期那么长，可参考S33。
S34.前置主摄像头将采集的原始图像帧(RAW)传输至ISP。
ISP可以对该RAW图进行一系列处理并输出YUV图像帧。
S35.ISP可以将前置主摄像头的YUV图像帧传输至显示屏。
这里，S35仅示意图像数据的流转路径，不表示ISP直接向显示屏发送前置主摄像头的YUV图像帧，其实现包括送显这一过程。
S36.显示屏显示前置主摄像头的YUV图像帧。
从此，前置主摄像头开始提供拍照预览图像，基于flicker检测结果设置其曝光时间也可以避免其成像画面中出现水波纹。在视频通话场景等默认前置拍摄的场景下，翻转摄像头这一操作也可以触发第一摄像头从前置主摄像头切换成后置主摄像头，实现流程类似，这里不再展开。
多路拍照或录像
在拍摄模式切换成多路拍照或录像模式的场景下，启动所述主摄像头之外的一个或多个摄像头，也添加用作所述第一摄像头。
S37.应用处理器(AP)检测到拍照模式切换到前后置双路拍照模式。
前后置双路拍照模式仅是多路拍照模式的一种，多路拍照模式还可以包括：后置主摄像头，后置长焦摄像头和后置广角摄像头等多个后置摄像头同时拍照模式。多路录像类似。
S38.应用处理器(AP)可以启动前置主摄像头，将前置主摄像头添加用作第一摄像头。
此时，第一摄像头包括两个摄像头：后置主摄像头和前置主摄像头。应用处理器(AP)并不关闭后置主摄像头，后置主摄像头的YUV图像帧并不独立送显，需要与前置主摄像头的YUV图像帧拼接后一起送显。
S39.和后置主摄像头一样，前置主摄像头也可以将采集的原始图像帧(RAW)传输至ISP。
ISP也可以对该RAW图进行一系列处理并输出前置主摄像头的YUV图像帧。前置主摄像头的YUV图像帧需要和后置主摄像头的YUV图像帧一起形成拼接预览图像，以实现前后置同时拍照预览的功能。
S40.ISP可以将前、后置主摄像头的拼接图像帧传输至显示屏。
这里，S40仅示意图像数据的流转路径，不表示ISP直接向显示屏发送拼接图像帧传，其实现包括送显这一过程。
S41.显示屏显示拼接图像帧。
若第一摄像头有降低曝光时间的需求，则可以基于flicker检测结果设置第一摄像头曝光时间，可以避免其成像画面中出现水波纹。
以上通过“变焦”、“翻转摄像头”、“多路拍照或录像”说明了第一摄像头是可以更改的。不限于这些场景，电子设备还可以在其他场景下进行第一摄像头的切换，例如拍照模式切换成“微距模式”时，第一摄像头可以切换成微距摄像头。
图11实施例中，在第一摄像头发生切换时，第二摄像头始终采用景深摄像头。不限于此，第二摄像头可以发生切换，本申请实施例对其切换策略不作限制。例如，对于仅具备两个摄像头(前置摄像头和后置摄像头)的电子设备来说，当检测到翻转摄像头时，第一摄像头切换到前置摄像头，第二摄像头便可以切换成后置摄像头。
图11实施例中，各个摄像头耦合于一个ISP。不限于此，电子设备可配置有多个ISP，多个摄像头可耦合于不同ISP。
本申请的说明书和权利要求书及附图中的术语“用户界面(user interface，UI)”，是应用程序或操作系统与用户之间进行交互和信息交换的介质接口，它实现信息的内部形式与用户可以接受形式之间的转换。应用程序的用户界面是通过java、可扩展标记语言(extensible markup language，XML)等特定计算机语言编写的源代码，界面源代码在终端设备上经过解析，渲染，最终呈现为用户可以识别的内容，比如图片、文字、按钮等控件。控件(control)也称为部件(widget)，是用户界面的基本元素，典型的控件有工具栏(toolbar)、菜单栏(menu bar)、文本框(text box)、按钮(button)、滚动条(scrollbar)、图片和文本。界面中的控件的属性和内容是通过标签或者节点来定义的，比如XML通过<Textview>、<ImgView>、<VideoView>等节点来规定界面所包含的控件。一个节点对应界面中一个控件或属性，节点经过解析和渲染之后呈现为用户可视的内容。此外，很多应用程序，比如混合应用(hybrid application)的界面中通常还包含有网页。网页，也称为页面，可以理解为内嵌在应用程序界面中的一个特殊的控件，网页是通过特定计算机语言编写的源代码，例如超文本标记语言(hyper text markup language，HTML)，层叠样式表(cascading style sheets，CSS)，java脚本(JavaScript，JS)等，网页源代码可以由浏览器或与浏览器功能类似的网页显示组件加载和显示为用户可识别的内容。网页所包含的具体内容也是通过网页源代码中的标签或者节点来定义的，比如HTML通过<p>、<img>、<video>、<canvas>来定义网页的元素和属性。
用户界面常用的表现形式是图形用户界面(graphic user interface，GUI)，是指采用图形方式显示的与计算机操作相关的用户界面。它可以是在电子设备的显示屏中显示的一个图标、窗口、控件等界面元素，其中控件可以包括图标、按钮、菜单、选项卡、文本框、对话框、状态栏、导航栏、Widget等可视的界面元素。
本申请提供的上述方法实施例中的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。结合本申请实施例所公开的方法步骤可以直接体现为硬件处理器 执行完成，或者用处理器中的硬件及软件模块组合执行完成。
本申请还提供一种电子设备，该电子设备可以包括：存储器和处理器。其中，存储器可用于存储计算机程序；处理器可用于调用存储器中的计算机程序，以使得该电子设备执行上述任意一个实施例中的方法。
本申请还提供了一种芯片系统，芯片系统包括至少一个处理器，用于实现上述任意一个实施例中电子设备执行的方法中所涉及的功能。
在一种可能的设计中，芯片系统还包括存储器，存储器用于保存程序指令和数据，存储器位于处理器之内或处理器之外。
该芯片系统可以由芯片构成，也可以包含芯片和其他分立器件。
可选地，该芯片系统中的处理器可以为一个或多个。该处理器可以通过硬件实现也可以通过软件实现。当通过硬件实现时，该处理器可以是逻辑电路、集成电路等。当通过软件实现时，该处理器可以是一个通用处理器，通过读取存储器中存储的软件代码来实现。
可选地，该芯片系统中的存储器也可以为一个或多个。该存储器可以与处理器集成在一起，也可以和处理器分离设置，本申请实施例并不限定。示例性地，存储器可以是非瞬时性处理器，例如只读存储器ROM，其可以与处理器集成在同一块芯片上，也可以分别设置在不同的芯片上，本申请实施例对存储器的类型，以及存储器与处理器的设置方式不作具体限定。
示例性地，该芯片系统可以是现场可编程门阵列(field programmable gate array，FPGA)，可以是专用集成芯片(application specific integrated circuit，ASIC)，还可以是系统芯片(system on chip，SoC)，还可以是中央处理器(central processor unit，CPU)，还可以是网络处理器(network processor，NP)，还可以是数字信号处理电路(digital signal processor，DSP)，还可以是微控制器(micro controller unit，MCU)，还可以是可编程控制器(programmable logic device，PLD)或其他集成芯片。
本申请还提供一种计算机程序产品，计算机程序产品包括：计算机程序(也可以称为代码，或指令)，当计算机程序被运行时，使得计算机执行上述任一个实施例中电子设备执行的方法。
本申请还提供一种计算机可读存储介质，计算机可读存储介质存储有计算机程序(也可以称为代码，或指令)。当计算机程序被运行时，使得计算机执行上述任一个实施例中电子设备执行的方法。
计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行计算机程序指令时，全部或部分地产生按照本申请的流程或功能。计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State  Disk)等。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，该流程可以由计算机程序来指令相关的硬件完成，该程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法实施例的流程。而前述的存储介质包括：ROM或随机存储记忆体RAM、磁碟或者光盘等各种可存储程序代码的介质。
总之，以上仅为本发明技术方案的实施例而已，并非用于限定本发明的保护范围。凡根据本发明的揭露，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (19)

  1. 一种运动抓拍方法，其特征在于，包括：
    所述电子设备启动第一摄像头；
    所述电子设备在显示屏上显示预览图像，所述预览图像来自所述第一摄像头采集的图像，并缓存最新采集的多帧图像；
    在预览时，所述电子设备判断所述预览图像中是否有运动物体，若有运动物体，则在预览时降低所述第一摄像头的曝光时间；
    所述电子设备检测到拍照操作；
    响应于所述拍照操作，所述电子设备保存照片，所述照片来自于缓存的所述最新采集的多帧图像。
  2. 如权利要求1所述的方法，其特征在于，所述在预览时降低所述第一摄像头的曝光时间，具体包括：
    在预览时根据所述运动物体的运动速度降低所述第一摄像头的曝光时间，所述运动物体的运动速度越快，所述第一摄像头的曝光时间被降低的越多，降低后的所述第一摄像头的曝光时间越短。
  3. 如权利要求1或2所述的方法，其特征在于，若所述预览是高动态范围HDR预览，则所述方法还包括：在预览时，在不同帧时刻应用不同曝光值EV以缓存亮度不同的多帧图像。
  4. 如权利要求3所述的方法，其特征在于，所述在不同帧时刻应用不同曝光值EV，具体包括：在不同帧时刻采用相同的曝光时间但不同的曝光增益，或者在不同帧时刻采用不同的曝光时间且不同的曝光增益，或者在不同帧时刻采用相同曝光增益但不同曝光时间。
  5. 如权利要求1-4中任一项所述的方法，其特征在于，若所述预览不是HDR预览，则在所述电子设备保存照片之前，所述方法还包括：
    所述电子设备利用参考帧和多帧优选帧进行多帧融合降噪处理，得到多帧融合降噪处理后的图像帧；所述参考帧是预览缓存区中清晰度最高的图像帧，所述多帧优选帧是从所述参考帧之外的所述缓存的图像帧中选择出来的。
  6. 如权利要求5所述的方法，其特征在于，所述电子设备保存照片，具体包括：将所述多帧融合降噪处理后的图像帧保存为所述照片；
    所述照片来自于缓存的所述最新采集的多帧图像，具体包括：所述照片来自所述参考帧和所述多帧优选帧，是所述参考帧和所述多帧优选帧融合产生的。
  7. 如权利要求1-6中任一项所述的方法，其特征在于，若所述预览是HDR预览，则在 所述电子设备保存照片之前，所述方法还包括：
    所述电子设备利用参考帧和曝光值EV不同的多帧图像进行HDR融合处理，得到HDR融合后的图像帧；所述参考帧是预览缓存区中清晰度最高的图像帧，所述曝光值EV不同的多帧图像是从所述参考帧之外的所述缓存的图像帧中选择出来的。
  8. 如权利要求7所述的方法，其特征在于，所述电子设备保存照片，具体包括：将所述HDR融合后的图像帧保存为所述照片；
    所述照片来自于缓存的所述最新采集的多帧图像，具体包括：所述照片来自所述参考帧和所述曝光值EV不同的多帧图像，是所述参考帧和所述曝光值EV不同的多帧图像融合产生的。
  9. 如权利要求1-8中任一项所述的方法，其特征在于，还包括：在预览时，所述电子设备启动闪烁光源检测装置进行闪烁光源检测。
  10. 如权利要求9所述的方法，其特征在于，所述在预览时降低所述第一摄像头的曝光时间，具体包括：
    若在预览时检测到闪烁光源，则所述电子设备控制降低后的所述曝光时间是光能量周期的整数倍，所述曝光时间最短设置成一倍光能量周期。
  11. 如权利要求9或10所述的方法，其特征在于，所述在预览时降低所述第一摄像头的曝光时间，具体包括：
    若在预览时未检测到闪烁光源，则所述电子设备将所述曝光时间至光能量周期以下。
  12. 如权利要求9-11中任一项所述的方法，其特征在于，所述闪烁光源检测装置包括：第二摄像头，所述第二摄像头是卷帘快门摄像头，所述第二摄像头的曝光时间小于光能量周期，所述第二摄像头采集的图像不会被送显。
  13. 如权利要求12所述的方法，其特征在于，所述电子设备启动闪烁光源检测装置进行闪烁光源检测，具体包括：
    所述电子设备对所述第二摄像头采集的图像进行图像识别，判断所述图像内容中是否包含水波纹特征，若包含，则确定检测出闪烁光源；所述检测的结果用于控制所述第一摄像头的曝光时间。
  14. 如权利要求1-5中任一项所述的方法，其特征在于，在所述电子设备启动闪烁光源检测装置进行闪烁光源检测之前，还包括：所述电子设备确定环境亮度不是过亮环境，也不是过暗环境，其中，所述过亮环境的亮度高于第一高亮度，所述过暗环境的亮度低于第一低亮度。
  15. 如权利要求14所述的方法，其特征在于，所述电子设备确定出环境亮度不是过亮环境，也不是过暗环境，具体包括：根据自动曝光控制AEC设定给所述第一摄像头的曝光时间大于第三时长确定出所述环境亮度不是过亮环境，根据自动曝光控制AEC设定给所述第一摄像头的曝光时间小于第四时长确定出所述环境亮度不是过暗环境，所述第三时长小于所述第四时长。
  16. 如权利要求15所述的方法，其特征在于，所述第三时长、所述第四时长记录于所述第一摄像头的曝光表中。
  17. 一种电子设备，其特征在于，包括一个或多个处理器和一个或多个存储器；其中，所述一个或多个存储器与所述一个或多个处理器耦合，所述一个或多个存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，当所述一个或多个处理器执行所述计算机指令时，使得执行如权利要求1-16任一项所述的方法。
  18. 一种芯片系统，所述芯片系统应用于电子设备，所述芯片系统包括一个或多个处理器，其特征在于，所述处理器用于调用计算机指令以使得执行如权利要求1-16中任一项所述的方法。
  19. 一种计算机可读存储介质，包括指令，其特征在于，当所述指令在电子设备上运行时，使得执行如权利要求1-16任一项所述的方法。
PCT/CN2024/083747 2023-05-23 2024-03-26 运动抓拍方法及电子设备 Ceased WO2024239792A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202310590214.8 2023-05-23
CN202310595470.6 2023-05-23
CN202310590214.8A CN117692761B (zh) 2023-05-23 2023-05-23 运动抓拍方法及电子设备
CN202310595470.6A CN117692781B (zh) 2023-05-23 2023-05-23 闪烁光源检测方法及电子设备

Publications (1)

Publication Number Publication Date
WO2024239792A1 true WO2024239792A1 (zh) 2024-11-28

Family

ID=93588893

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/083747 Ceased WO2024239792A1 (zh) 2023-05-23 2024-03-26 运动抓拍方法及电子设备

Country Status (1)

Country Link
WO (1) WO2024239792A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109286758A (zh) * 2018-10-15 2019-01-29 Oppo广东移动通信有限公司 一种高动态范围图像的生成方法、移动终端及存储介质
WO2019071613A1 (zh) * 2017-10-13 2019-04-18 华为技术有限公司 一种图像处理方法及装置
CN115529409A (zh) * 2021-06-24 2022-12-27 荣耀终端有限公司 一种拍摄方法及相关装置
CN116074634A (zh) * 2022-05-27 2023-05-05 荣耀终端有限公司 一种曝光参数确定方法和装置
CN117692781A (zh) * 2023-05-23 2024-03-12 荣耀终端有限公司 闪烁光源检测方法及电子设备
CN117692761A (zh) * 2023-05-23 2024-03-12 荣耀终端有限公司 运动抓拍方法及电子设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019071613A1 (zh) * 2017-10-13 2019-04-18 华为技术有限公司 一种图像处理方法及装置
CN109286758A (zh) * 2018-10-15 2019-01-29 Oppo广东移动通信有限公司 一种高动态范围图像的生成方法、移动终端及存储介质
CN115529409A (zh) * 2021-06-24 2022-12-27 荣耀终端有限公司 一种拍摄方法及相关装置
CN116074634A (zh) * 2022-05-27 2023-05-05 荣耀终端有限公司 一种曝光参数确定方法和装置
CN117692781A (zh) * 2023-05-23 2024-03-12 荣耀终端有限公司 闪烁光源检测方法及电子设备
CN117692761A (zh) * 2023-05-23 2024-03-12 荣耀终端有限公司 运动抓拍方法及电子设备

Similar Documents

Publication Publication Date Title
CN113382169B (zh) 一种拍照方法及电子设备
CN117692761B (zh) 运动抓拍方法及电子设备
CN111212235A (zh) 一种长焦拍摄的方法及电子设备
EP3135028A1 (en) Realtime capture exposure adjust gestures
CN107222680A (zh) 一种全景图像的拍摄方法和移动终端
CN115689963B (zh) 一种图像处理方法及电子设备
CN117119285B (zh) 一种拍摄方法
CN116055855B (zh) 图像处理方法及其相关设备
WO2024179100A1 (zh) 一种拍摄方法
CN115883957B (zh) 一种拍摄模式推荐方法
WO2023231595A9 (zh) 一种拍摄方法和电子设备
WO2023160230A9 (zh) 一种拍摄方法及相关设备
EP4543031A1 (en) Photographing method and electronic device
CN116723382A (zh) 一种拍摄方法及相关设备
WO2024239779A1 (zh) 一种视频拍摄方法及电子设备
WO2024179109A1 (zh) 一种拍照方法及电子设备
CN117692781B (zh) 闪烁光源检测方法及电子设备
WO2024088074A1 (zh) 拍摄月亮的方法和电子设备
CN115883958A (zh) 一种人像拍摄方法
WO2024109213A1 (zh) 拍摄模式切换方法及相关装置
CN119255113B (zh) 一种闪烁光源检测的方法和电子设备
CN117692753B (zh) 一种拍照方法及电子设备
WO2025146120A1 (zh) 一种拍摄方法、电子设备及存储介质
CN119254904A (zh) 图像处理方法、芯片系统及电子设备
WO2024093518A1 (zh) 一种出图模式切换方法及相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24810040

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE