WO2025147237A1

WO2025147237A1 - Real-time generative video on resource-constrained devices

Info

Publication number: WO2025147237A1
Application number: PCT/US2024/010160
Authority: WO
Inventors: Ian Chen; Shang Chih Chuang; Chihchi CHENG; Bang Sian LIU; Yu-Chuan Huang
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2024-01-03
Filing date: 2024-01-03
Publication date: 2025-07-10
Anticipated expiration: 2026-07-03
Also published as: TW202533162A

Abstract

This disclosure describes systems and techniques for real-time generative video on a resource-constrained device. A mobile electronic device (104) initiates a video stream (402, 502) capture. The mobile electronic device (104) pre-processes (404) the video stream (402, 502) and generates associated keyframes (510). Two or more keyframes (510) are selected for processing, and interpolation images (522) are generated based on the selected keyframe images (510) and/or associated thumbnail images (506) and features (508). The selected keyframes (510) represent a subset of the total video stream (402, 502) images, thus requiring fewer resources for processing than were the entire video stream (402, 502) used.

Description

REAL-TIME GENERATIVE VIDEO ON RESOURCE-CONSTRAINED DEVICES

BACKGROUND

[0001] Modem video-editing techniques, especially those employing generative artificial-intelligence-type algorithms, are limited to the computation power of the device on which they are performed. This is particularly true in mobile electronic devices such as smart phones and tablets, among other resource-constrained devices. The user experience with resource- constrained devices employing video editing software may suffer from low frame rates, low- resolution, lagging feature generation due to processing being done completely offline or remotely, etc. One of the adverse effects of a resource-constrained device producing generative video is the lack of real-time capabilities. The lack of real-time capability not only provides an end user with a subpar editing experience, but may have other deleterious effects such as hindering development pace for content creators, slowing adoption of new editing features, etc.

[0002] Existing approaches to this problem generally employ strictly off-device techniques, such as fully remote processing. In such techniques, the generating speed is negatively affected by the speed of the network, the throughput capabilities of the network, access to real-time network resources, etc. This approach also does not solve the problem of having an on-device, real-time capability to enable generative video editing.

SUMMARY

[0003] This disclosure describes systems and techniques for real-time generative video on a resource-constrained device. A user equipment (e.g., a mobile phone) is used to capture a video stream. The video stream may be comprised of a plurality of images and may include an audio component. There may be elements of the video stream that the user would like to edit, including techniques such as upscaling, increasing frame rate, removal of unwanted objects, addition of desired object(s), or similar techniques. In aspects, this may involve an interpolation, herein defined as an addition of content. For instance, removal of an object may be accomplished by an addition of content to fill in the area of the image where the object previously was. Interpolation of a video stream, such as an interpolation involving generative artificial intelligence (GenAI), often requires a high level of processing pow er as each frame or image in a video stream must be processed.

[0004] The systems and techniques described herein allow for interpolation processes from a video stream, such as GenAI for a video stream, to be employed on a resource-constrained device by using less than the full image set within a captured video stream for processing interpolated images to use in an output video stream. This allows for fewer resources to be required to process a certain amount of the video stream and generate the output video stream.

[0005] In some aspects, a method is described for real-time generative video on a resource-constrained device. The method includes driving, by one or more processors, one or more image sensors of an electronic device to capture an initial video stream, the initial video stream comprising a plurality of images in sequence, generating, by the one or more processors and based on the initial video stream, one or more features for each of the plurality of images, selecting, by the one or more processors and based on the one or more generated features, two or more images from the plurality of images to act as keyframes, generating, by the one or more processors and based on the keyframes and the two or more features associated with the keyframes, one or more interpolation images, and generating, by the one or more processors and based on the one or more interpolation images, an output video stream.

[0006] The method may further, for example, include the generation of the one or more features being done using a machine-learned model stored in a memory of the electronic device. The method may also, by way of example, have the machine-learned model be a convolutional neural network (CNN). The selection of the two or more keyframes may also be done using a machine-learned model stored in a memory of the electronic device. In aspects, the method may also include generating, by the one or more processors, one or more second interpolation images based on the one or more interpolation images and wherein the output video stream comprises the interpolation images and the second interpolation images. According to some examples, the one or more interpolation images may include a first image, a second image, and a third image, the one or more second interpolation images may include a primary image and a secondary image, and the primary image may be based on the first image and the second image, and the secondary image may be based on the second and the third image. Further, the generation of the one or more interpolation images may, in some examples, be done by a machine-learned model.

[0007] In aspects, the method may further include comparing the one or more features with one or more parameters, and wherein the selection of the keyframes is based on the comparison of the one or more features with the one or more parameters. The one or more parameters may, in some examples, include one or more of a motion detection threshold, an object detection classification, a hardware threshold, a composition threshold, and a time threshold. According to some examples, the output video stream may include an upscaling of the initial video stream, the upscaling including an increase in the initial video stream resolution, frame rate, or both. Further, in some examples the output video stream may include the initial video stream with one or more elements omitted from and/or one or more elements added to the initial video stream and/or an enhancement of the initial video stream, the enhancement including one or more of a change in contrast, a change in color, an application of a filter effect, a change in white balance, or a reduction of an artifact. The generated one or more features may, in aspects, include at least a second plurality of images, the second plurality of images being a lower resolution set of the plurality of images.

[0008] This document also describes computer-readable media having instructions for performing the above-summarized method and other methods set forth herein, as well as systems and means for performing these methods. Such a system may, for example comprise an electronic device comprising one or more image sensors, one or more processors, and a memory containing instructions that, when accessed by the one or more processors, cause the one or more processors to perform any one of the methods set forth herein.

[0009] This summary is provided to introduce simplified concepts for real-time generative video on a resource-constrained device, which is further described below in the Detailed Description and Drawings. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The details of one or more aspects of real-time generative video on a resource-constrained device are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:

[0011] FIG. 1 illustrates an example environment, in which techniques for real-time generative video on a resource-constrained device can be implemented;

[0012] FIGs. 2A-C illustrate an example video stream processed using real-time generative video on a resource-constrained device;

[0013] FIG. 3 illustrates an example of a mobile device for implementing real-time generative video on a resource-constrained device;

[0014] FIG. 4 illustrates details of a processing algorithm used for generative video on a resource-constrained device;

[0015] FIG. 5 illustrates a more comprehensive algorithm than that of FIG. 4;

[0016] FIG. 6 illustrates a generic machine-learning (ML) setup, as described in this disclosure;

[0017] FIG. 7 illustrates an example of how one of the ML models of FIG. 6 may be trained in the context of this disclosure; [0018] FIG. 8 illustrates an example method for generative video on a resource-constrained device;

[0019] FIG. 9 illustrates another example method for generative video on a resource-constrained device;

[0020] FIG. 10 illustrates another example method for generative video on a resource-constrained device.

DETAILED DESCRIPTION

[0021] This disclosure describes techniques and systems for real-time generative video on a resource-constrained device. A resource-constrained device may, for example, be a mobile electronic device (e.g. s smart phone or tablets, mobile electronic device), which does not have access to unlimited power and/or processing capabilities. Such a resource-constrained device may, for example, be battery powered. Real-time generative video may use generative artificial intelligence (Gen Al), increase a frame-rate, upscale a video resolution, exclude elements from a video (e g., people in a crowd, unwanted cars around a landmark, etc.), include elements into a video (e.g., a person who was not there at the time of video taking, a clear sky on a cloudy day, etc.), or other elements, which require generative video. Processing of a video stream to produce an output video stream may be completely done on a user device (e.g., smart phone, augmented- reality (AR) goggles, smart glasses, etc.), which ty pically has fewer available resources than a traditional, stand-alone, or array-based computation device (e.g., desktop computer, cloud computing array, server, etc.).

[0022] The process of producing the output video stream using minimal resources leverages picking keyframes to represent the video stream or portions of the video stream. Byway of example, a portion of the video stream containing 5,000 images may have little change over these images, and thus the entire video stream portion may be represented by only two frames, at the beginning and end of the video stream portion. Other amounts of keyframes maybe selected, the two keyframes may represent more or less total images, or any range of other ratios of keyframe to video stream portion images may be used; the example of two keyframes to 5,000 images is meant to be illustrative and not limiting. In this example, the user device may use the two keyframes to produce 5,000 upscales frames or images based on these two key-frames, which may be used, in aspects, as an output video stream representing an upscaled version of the original video stream portion. In this way, only two of the original frames from the video stream portion were used to generate the total output video stream, thus minimizing resource usage on the user device. [0023] In aspects, intermediate steps may be taken in addition to the above outlined steps. For example, the user device may generate one or more features based on each image of a plurality of images from the video stream portion. The one or more features may be low-resolution thumbnails representing each image, one or more parameters (e.g., contrast values, object identification, motion detection, etc.), compressed images, etc. The picking of the keyframes may be based on these one or more features. The generation of the one or more features may be done based on a preset algorithm, a machine-learned (ML) model, or any other technique know n to a person of ordinary skill in the art.

[0024] According to some examples, an enhanced or interpolated image may be generated from a keyframe. The interpolated image may be a final image or a low-res, intermediate image from which the final image may be generated. The generation of the interpolated image may be done by a ML model or another processing algorithm. The interpolated image may be, for example, a reproduction of the keyframe image with one or more enhancements from the processing, such as the addition of an element or an increase in the native resolution of the keyframe. One or more interpolated images, based on the one or more keyframes for the video stream portion, may be used to, for example, generate additional interpolation images, which may be in sequence between a first keyframe image and a second keyframe image. The additional interpolation images may each be based on both the first and the second keyframe image, or the interpolation images associated with the first and second keyframe images.

Example Environment

[0025] FIG. 1 illustrates an example environment 100, in which techniques for real-time generative video on a resource-constrained device can be implemented. A user 102 may use a mobile electronic device 104, such as a smart phone, to capture a video of the environment 100. The environment 100 may contain a first person 106, a bicyclist 108, and a second person 110. Each of these elements 106. 108, and 110 may be present in the video captured by the mobile electronic device 104.

[0026] FIG. 2A illustrates an example of a video stream 200A based on the captured video. The video stream 200A may be, in aspects, on the mobile electronic device 104. The video stream may be displayed to the user, such as on the screen of a smart phone or the display in an augmented-reality (AR) headset. Here, it is shown as being displayed on the screen of a smart phone (e.g., the mobile electronic device 104 of FIG. 1). This is illustrative and not meant to be limiting; a person of ordinary' skill in the art will readily understand that other display embodiments and devices are compatible with this description. [0027] The video stream 200A includes a first person 202A, a bicyclist 204 A, and a second person 206A. The mobile electronic device 104 may, in some examples, use an identification algorithm to identify and tag the elements 202A, 204A, and 206A in the video stream 200A. For instance, a convolutional neural network (CNN) may be employed to extract features such as identifying the objects/elements 202A, 204A. and 206 A, identifying motion, such as the motion of the bicyclist 204A, identifying weather conditions, etc. This type of identification may also be performed, in some examples, without the use of a CNN.

[0028] The user may decide that elements of the video stream are not wanted. Additionally or alternately, the mobile electronic device 104 may automatically or otherwise autonomously select elements of the video stream for removal. FIG. 2B illustrates an example where a determination has been made that elements 204A and 206 A of FIG. 2A are not desired in the video stream. Such a determination may be an automatic action from the mobile electronic device 104, a user-initiated determination, or a combination of these or other methods of determination. The mobile electronic device 104 may remove these elements, as in FIG. 2B. The video stream 200B of FIG. 2B, in this way, represents a post-processing video or a video output using the techniques for real-time generative video on a resource-constrained device. Additionally or alternately, FIG. 2B may represent a video stream 200B that is a post-processing video but not yet the full output video stream, such as a lower resolution rendering.

[0029] In FIG. 2B, element 202B has been retained and elements 204A (the bicyclist) and 206A (the second person) have been removed from the video stream 200A of FIG. 2A to create the video stream 200B. In order to provide a realistic video, the video stream 200B may include the environment in places formerly occupied by elements 204A and 206A. This may, in aspects, be accomplished by generation of the background environment by the mobile electronic device 104 in the areas of the video stream 200B formerly occupied by the elements 204 A and 206A.

[0030] In addition or as an alternative to generating the missing background in video stream 200B, it may be necessary' or desirable to add one or more additional elements to the video stream 200B. FIG. 2C illustrates an example video stream 200C where novel elements aside from background filling have been added to the original video stream 200 A. The video stream 200C again, as in the video stream 200B, retains element 202C, but further contains an element 204C, a flowering plant, and an element 206C, a dog. The elements 204C and 206C may not have been present when the initial video stream 200A was captured but were added in the processing stage by the mobile electronic device 104.

[0031] Additional elements, such as 204C and 206C, may' be, in some examples, user-selected elements for generation. For example, a user such as user 102 may want to have the first person 202C in the video stream 200A with, by way of example, first person 202C’s dog 206C, but the dog 206C is not present during video capture. The user 102 may also desire to only have the elements 202C and 206C in the video stream 200 A. The user 102, in this example, may select the dog 206C from a video rendering menu, or similar interface. The dog 206C, or other elements such as the flowering plant 204C, may be added to video stream 200C.

[0032] In an alternate example, the mobile electronic device 104 may automatically generate one or more new elements, such as the elements 204C and 206C. For example, the flowering plant 204C may not have been selected for addition by the user 102, but may have been determined to be advantageous to add to the video stream 200C due to one or more automatic determination factors. The automatic determination factors may be. for example, based on one or more rendering consideration, one or more ambiance consideration, one or more user preference considerations, etc.

[0033] In aspects, a rendering consideration may be a determination by the mobile electronic device 104 that there could be an artifact in the video stream 200C if there is not an element, for example the element 204C, added to the video stream 200C in that position. Examples of the artifact may be, but are not limited to, motion blurring, pixelation, blocking, noise, or other artifact types common to digital video processing.

[0034] In aspects, an ambiance consideration may be a prominence or lack of a certain feature, such as flowering plant 204C, a type of scene for the environmental background, other factors, or a combination of factors. According to some examples, the ambience consideration may be based on one or more parameters of the scene, such as, but not limited to, a color composition, a contrast composition, a determination of one or more weather conditions, or similar aspects of the scene or environment.

[0035] In aspects, a user preference consideration may be a history of user additions or subtractions, a prediction of user preferred additions or subtractions, a matching of the user to a profile or profile type, or other or combination factors relating to the user. The prediction of user preferred additions or subtractions may be based on the user’s history, a ML model output taking one or more parameters derived from the environment as inputs, prior user behavior or actions, a general preference inference based on one or more persons apart from the user, or other or a combination of these things.

Example Device

[0036] FIG. 3 illustrates an example of a mobile device 300 for implementing real-time generative video on a resource-constrained device. The mobile device 300 may be, for example, a smart phone 300-2, a tablet device 300-4, smart glasses 300-6. AR goggles or an AR headset 300-8, a smart helmet 300-10. a smart watch 300-12, or any other device known to a person of ordinary skill in the art where such a device is capable of capturing and processing video. The list of example devices given is meant to be illustrative and not limiting. The mobile device 300 may be a battery powered device.

[0037] The mobile device 300 includes one or more processors 302 and one or more computer-readable media (memory) 304. The one or more computer-readable media 304 may include instructions 306, such as those for generating real-time generative video on a resource-constrained device, parameters 308, and a GenAI module 310. The mobile device 300 may also, in some examples, include one or more cameras 312, such as video cameras, infrared cameras, multiple cameras at different aspect ratios, etc. The one or more cameras 312 may be configured to capture video data. The captured video data, may, for example comprise a plurality of images, each comprising a two-dimensional data array comprising a plurality of data items, each data item comprising a digital representation of an intensity⁷ of light at a respective of image location (e.g. pixels). Each data item may comprise a plurality of digital intensity values (e.g. RGB values). The methods described herein may operate on such captured video data.

[0038] The mobile device 300 may also include a communications module 314. The communications module 314 may include data network interfaces that provide connection and/or communication links between the mobile device and other data networks (e.g., a mesh network, external network, etc.), devices, or remote computing systems (e.g., servers). The communications module 314 may, in aspects, couple the mobile device 300 to a variety of different types of components, peripherals, or accessory⁷ devices. Data input ports of the communications module 314 may receive data, including image data, user inputs, communication data, audio data, video data, and the like. The communications module 314 may enable wired or wireless communicating of device data between the mobile device 300 and other devices, computing systems, and networks. Transceivers of the communications module 314 may enable cellular phone communication and other types of network data communication. While connections or communication links may be provided to other data networks, it may still be preferred to perform certain processing tasks on the mobile device 300, for example to minimize latency, or stability⁷ issues.

[0039] The mobile device 300 may also include one or more sensors 316, such as, but not limited to, one or more infrared sensors, one or more capacitive sensors, one or more radar sensors, one or more magnetic sensors, or any other sensors known to a person of ordinary skill in the art.

[0040] The processors 302 and the memory 304, which includes memory⁷ media and storage media, are the main processing complex of the mobile device 300. The instructions 306, the parameters 308, and the GenAI module 310 may, in aspects, be implemented as computer-readable instructions on the memory 304, which may be executed by the one or more processors 302 to provide functionalities described herein, such as some or all of the functions of the GenAI module 310 (shown within the memory 304, though this is not required).

[0041] The one or more processors 302 may include any combination of one or more controllers, microcontrollers, processors, microprocessors, hardware processors, hardware processing units, digital-signal-processors, graphics processors, graphics processing units, and the like. The processors 302 may be an integrated processor and memory subsystem (e.g., implemented as a “system-on-chip”), which processes computer-executable instructions to control operations of the mobile device 300.

[0042] The memory 304 may be, in aspects, configured as persistent and non-persistent storage of executable instructions (e.g., firmware, recovery' firmware, software, applications, modules, programs, functions, and the like) and data (e.g., user data, operational data) to support execution of the executable instructions. Examples of the memory 304 include volatile memory' and non-volatile memory’, fixed and removable media devices, and any suitable memory device or electronic data storage that maintains executable instructions and supporting data. The memory 304 may include various implementations of random-access memory (RAM), read-only memory' (ROM), flash memory, and other types of storage memory' in various memory' device configurations. The memory' 304 may exclude propagating signals. The memory 304 may be a solid-state drive (SSD) or a hard disk drive (HDD).

[0043] The one or more sensors 316 generally obtain contextual information indicative of operating conditions (virtual or physical) of the mobile device 300 or the surroundings of the mobile device 300. The mobile device 300 monitors the operating conditions based in part on sensor data generated by the sensors 316. Additional examples of the sensors 316 include movement sensors, temperature sensors, position sensors, proximity sensors, light sensors, infrared sensors, moisture sensors, pressure sensors, and the like.

[0044] The interface module 318 may, in aspects, act as an output and input component for obtaining user input and providing a user interface. As an output component, the interface module 316 may, in some examples, be a display, a speaker or audio system, a haptic-feedback system, or another system for outputting information to the user 102. When configured as an input component, the interface module 316 can include a touchscreen, a camera, a microphone, a physical button or switch, a radar input system, or another system for receiving input from the user 102. Other examples of the interface module 316 include a mouse, a keyboard, a fingerprint sensor, an optical, an infrared, a pressure-sensitive, a presence-sensitive, or a radar-based gesture detection system. The interface module 316 often includes a presence-sensitive input component operatively coupled to (or integrated within) a display. [0045] When configured as a presence-sensitive screen, the interface module 316 detects when the user 102 provides two-dimensional or three-dimensional gestures at or near the locations of a presence-sensitive feature. In response to the gestures, the interface module 316 may output information to other components of the mobile device 300 to indicate relative locations (e.g., X, Y, Z coordinates) of the gestures, and to enable the other components to interpret the gestures. The interface module 316 may output data based on the information generated by an output component or an input component which, for example, may be used as input for the GenAI module 310.

Example Algorithms

[0046] FIG. 4 illustrates details of a processing algorithm 400 used for generative video on a resource-constrained device. A raw input 402 is received, which may be, in aspects, the raw video stream from the user device (e.g. comprising image data). The raw input 402 may then be subject to pre-processing 404, which may, in aspects, result in an image 406 and one or more features 408. By way of example, the image 406 may be a thumbnail style image, meaning a lower resolution image than the raw image 402 it is based on. The one or more features 408 may be additional low-res imagery, masks, image parameters (e.g., contrast values, color values, etc.), or other parameters pertaining to the raw input 402 or the image 406.

[0047] The one or more features 408 may, in aspects, be subject to a processing 410. The processing 410 may proceed via an ML model, such as a GenAI or other ML model. The processing 410 may produce a generated output (generation 412). The generation 412 may be, by way of example, subtraction of one or more elements present in the raw input 402, such as in the video stream 200B, addition of one or more elements not present in the raw input 402, such as in the video stream 200C, an upscaling of the raw input 402, additional frames for the raw input 402, or other enhancements.

[0048] The generation 412 and the image 406 may be subject to a post-processing 414. The post-processing 414 may, in aspects, combine elements from the generation 412 and the image 406. For example, if the goal is upscaling, the post-processing 414 may take the image 406, which may be a lower resolution thumbnail sty le image, and combine it with the generation 412, which may be interpolated information that allows for additional information to be added to the raw input 402. In the example of upscaling, the post-processing 414 gives an output 416, which would be an upscaled, or higher resolution, version of the raw input 402.

[0049] According to some examples, the processing 410 and post-processing 414 steps may be combined. In such examples, the generation 412 and the output 416 would be the same element. Other combinations of the elements of the processing algorithm 400 used for generative video on a resource-constrained device may be easily and directly inferred from the above disclosed algorithmic flow.

[0050] FIG. 5 illustrates a more comprehensive algorithm 500 than the algorithm 400 of FIG. 4. The algorithm 500 may be described using rows 502 through 522 and columns 1 through 9. For example, a raw input 502- 1 lies on the intersection of row 502 and column 1. The row 502 represents raw inputs, with elements 502-1 through 502-9. More or less elements may be in a video stream used for processing; the number of elements here is meant to be illustrative. Raw inputs 502 may be raw video footage from a video stream produced by a resource-limited device, such as the mobile device 300.

[0051] Raw inputs 502 may be used to produce one or more raw' images, as in row 504, from which thumbnail images, as in row 506, and/or features, as in row' 508, may be produced. Each of the elements 504-1 through 504-9 may comprise a frame of raw⁷ image data. Thumbnail images 506 and features 508 may contain less data overall than raw images 504. Additionally or alternately, thumbnail images 506 and features 508 may, in aspects, include one or more characterizations, which may be wholly found in features 508, such as contrast values, color values, object identification, etc., or any combination of these things.

[0052] One or more keyframes, as in row 510, may be produced. By way of example, the keyframe 510-1 may be produced based on the thumbnail image 506-1 and the one or more features 508-1. The keyframe 510-1 may be a data representation, an image representation, or any other representation useful for image processing. The keyframe 510-1 may be generated based on a set algorithm or may be generated from a dynamic algorithm, both of which may include, in aspects. ML models.

[0053] Rows 512-518 do not have members for every column. This is because, for example, two or more keyframes from row⁷510 may be selected for processing. The exact number of keyframes 510 selected during a video processing session need not be set but may, in aspects, be based on features 508, the exact ty pe of enhancement being done, thumbnails 506, etc. For example, suppose that keyframes 510-1 and 510-4 are generated based on features 508-1 and 508-4 and thumbnails 506-1 and 506-4, and that features 508-1 and 508-4 indicate that there are identified elements (e.g., 204A, 206A, etc.) that have associated identified motions. Further consider, in this example, that the identified motions do not change over a threshold motion value from the image 504-1 to the image 504-4, where such a determination is made based on keyframes 510-1 and 510-4. The algorithm 500 may then select the keyframes 510-1 and 510-4. Other criteria and/or combination criteria may be used to select keyframes for processing.

[0054] Once the keyframes 510-1 and 510-4 are selected, they may be processed in row⁷ 512 to generate novel image data. For example, the processing element 512-1 may be a GenAI model, which takes the thumbnail image 506-1 and the one or more features 508-1 as inputs and produces an image 514-1. The produced image 514-1 may, in aspects, be produced by a GenAI algorithm, such as an ML algorithm capable of adding novel data to a video stream. In some examples, the produced image 514-1 may be a low-resolution image. In such examples, the produced image 514-1 may be up-processed (such as by upscaling) 516-1 to a full generated image 518-1.

[0055] Similarly, in aspects, keyframe 510-4 being selected may result in a processing element 512-4 producing an image 514-4 and, optionally, upscaling 516-4 the produced image 514-4 to create a full generated image 518-4.

[0056] In aspects, once full generated images 518-1 and 518-4 are created, they may be used as inputs to generate interpolated images 522 via interpolation 520. For example, interpolations 520-2 and 520-3 may take both 518-1 and 518-4 as inputs. Interpolated images 522-2 and 522-3 may be produced from interpolations 520-2 and 520-3, respectively. The interpolation row 520 may take into account the sequencing of all elements 520-1 through 520-9, thus there may also, in aspects, be an interpolation 520-1 and an interpolation 520-4 to create interpolation images 522-1 and 522-4 from full generated images 518-1 and 518-4.

[0057] In some examples, another keyframe 510-9 may be selected. In this example, there are now three keyframes selected: 510-1. 510-4. and 510-9. The algorithm for producing interpolation images 522-1 through 522-3 does not change, but the added interpolation images, in this example, 522-5 through 522-9 are also produced in a similar fashion using full generated images 518-4 and 518-9 as inputs. Further, in aspects, the interpolation 520-4 used to produce the interpolation image 522-4 may take into account both interpolations before, such as the interpolation 520-3, as well as interpolations after, such as the interpolation 520-5. In another example, interpolation images 522-1, 522-4, and 522-9 may be solely based on full generated images 518-1, 518-4, and 518-9, respectively.

[0058] The resulting interpolation image row 522, in aspects, serves as an output video stream 522. The output video stream 522 may. for example, contain elements not present in the raw video stream 502, remove elements present in the raw video stream 502, upscale the raw video stream 502, increase the frame rate of the raw video stream 502, or any combination of these or other, similar video processing enhancements. Other video processing enhancements provided in the output video stream 522 may include one or more of a change in contrast, a change in color, an application of a filter effect, a change in white balance, or a reduction of an artifact. As the output video stream 522 was generated based on the limited keyframes 510-1, 510-4, and 510-9, the processing requirements to generate the output video stream 522 may be significantly reduced from a generative video processing procedure where all raw images 504 (and/or thumbnails 506, features 508, and keyframes 510) are used for the processing. This may allow a resource- constrained device, such as the examples outlined for mobile device 300, to produce real-time generative video enhancements.

[0059] It should be noted that, while processing rows 512 and 520 (and generation of elements such as keyframes 510) are shown here together, they need not be present on the same device. For example, a smart phone may be connected over a communication network and offload the processing 512 or the interpolation 520 to a cloud computing device. In another example, all steps may be achieved on the smart phone (or other mobile electronic device) itself. Other combinations of processing devices should be apparent to a person of ordinary skill in the art.

Machine-Learning

[0060] An ML model, as discussed in this disclosure, refers to a computer model that has been trained using one or more machine-learning techniques. In general, this training may be done by providing training inputs to one or more training models, which in turn may provide an output. The output may be in the form of a prediction, a confidence score, or other probability -based metrics.

[0061] FIG. 6 illustrates a generic ML model training setup. An ML model generator 600 may comprise training elements in the form of inputs 602, training models 604, and outputs 606 and may be used to generate an ML model 608. The inputs 602 may be training data. The training data 602 may be processed by the training models 604. Examples of the training models 604 include multi-layer perceptron (MLP) models, convolutional neural networks (CNN), long short-term memory (LSTM) algorithms, generative adversarial networks (GAN), K-means clustering, Gaussian mixture models (GMM), or any other of a number of machine-learning training techniques known to a person of ordinary skill in the art. The training models 604 may comprise a single type of model or multiple types, as well as various combinations of types including combinations of single types and of multiple types.

[0062] The training models 604 may use supervised learning methods, where the training data may be labeled and the outputs 606 may be graded based on their fidelity to a ‘'truth” output. The training models 604 may use an unsupervised learning method, where there may not be labels on the training data and the training models 604 may classify correlations without reference to a “truth” value. The training models 604 may combine supervised and unsupervised techniques.

[0063] The input 602 may be data from one or more sensors 316. For example, the input 602 may be raw inputs 502, raw images, 504, thumbnails 506, features 508, keyframes 510, produced images 518, or other or a combination of these input ty pes. The input 602 may also be from training data used to generate the ML model 608. The training data may comprise data of any of the input types sorts described above, created for the purpose of training the ML model 608. The ML model 608 may be generated once the training of the ML model generator 600 is complete. The ML model 608 may be trained on a same device where the ML model 608 is stored or on at least one other device.

[0064] The ML model 608 may be updated with additional training after it has been initially trained. In aspects, the ML model 608 may have the same structure before and after training, but may have one or more different values, such as starting vs. final values for weights and biases. In some examples, the ML model 608 may start with a different architecture prior to training, and in this way the generated ML model 608 may have architecture that is a product of the training done in the ML model generator 600.

[0065] FIG. 7 illustrates an example of how the ML model 608 may be trained in the context of this disclosure. FIG. 7 is meant not to be limiting but to show one possible example out of many, which a person skilled in the art may be able to employ. A trainer 700 may have inputs 702. The inputs 702 may be raw inputs 502, raw images, 504, thumbnails 506, features 508, key frames 510, produced images 518, or other or a combination of these input types. In this example, the inputs 702 are first processed by a CNN 704. The CNN 704 may extract features from the inputs 702. For example, extracted features may be the features 508, and may contain elements such as motion detection, object recognition, person recognition, or other elements and features common to video and image processing and identification.

[0066] The CNN 704 may be a disconnected neural network, meaning all members of the neural network, such as perceptrons, may not be in a causal relationship with one another. It should be mentioned that other feature extraction techniques besides CNNs may be equally employed in lieu of the CNN 704. An MLP neural network 706 may receive the extracted feature outputs from the CNN 704 as inputs. The MLP 706 may be a fully connected network, which may comprise all perceptrons in a causal relationship with one another. The MLP 706 may have as an output a detection probability prediction 708. The prediction 708 may be, for example, the keyframes selected (e.g.. 510-1, 510-4, and 510-9), the produced images 514, the fully generated images 518, the interpolation images 522, etc.

[0067] An ML model trained by a trainer such as the ML trainer 700 may exhibit similar architecture. For example, suppose the ML trainer 700 uses a matrix containing thumbnails 506, features 508, and/or keyframes 510 as the inputs 702 and provides a prediction of which keyframes 510 are selected for processing (e.g., key frames 510-1, 510-4, and 510-9) as numbers between 0 and 1 as the prediction 708. An ML model trained by the ML trainer 700 may take a matrix of input data as described in the example above as its input and output a probability⁷ vector with elements between 0 and 1 indicating the desirability of the associated key frames 510 for selection. Such an ML model may have one or more feature extraction layers, such as the CNN 704, and one or more fully connected layers, such as the MLP 706.

[0068] In aspects, feature extraction, such as that done by the feature extraction layers of the CNN 704, may be accomplished by correlating areas of the inputs 702. Such correlations may, for example, be established using a disconnected network, the disconnected network not having all neurons causally connected by necessity. The feature extraction layers may, in aspects, utilize windows of analysis within the inputs 702, the windows of analysis in some examples overlapping one another. The feature extraction layers may, in aspects, be a product of unsupervised learning. For example, the thumbnail images 506. the features 508, or both may be outputs of or part of the composition of the feature extraction layers.

[0069] By way of example, an ML model trained by the ML trainer 700 may be used to generate features from the inputs 702. Such a model, in aspects, may be referred to as a generation ML model or a feature generation ML model. In aspects, a feature generation ML model may take the inputs 702, such as the raw inputs 502, and extract and/or generate one or more features from them, such as the thumbnail images 506 and the features 508. Such a feature generation ML model may be separate from an ML model or other analysis and/or generation tool. In this example, the feature generation ML model may have one or more features or feature maps, such as the feature maps shown in FIG. 7 for the CNN 704, as the final outputs of the feature generation model. In aspects, such an output may be at least a second plurality of images distinct from the raw inputs 502, the second plurality of images being a lower resolution set of the plurality of images. For example, the second plurality of images may be the thumbnail images 506. The extracted and/or generated features may be, in aspects, the thumbnail images 506, the features 508, or a combination of both of these or other extracted and/or generated features.

[0070] Though FIG. 7 depicts the ML trainer 700 as incorporating both feature extraction, such as from the CNN portion 704, and selection and/or interpolation, such as from the MLP portion 706, it should be understood that these ML models may be distinct. For example, there may be a trained generation model corresponding to the CNN 704, a trained selection model corresponding to at least a portion of the MLP 706, and a trained interpolation model corresponding to at least a portion of the MLP 706. Some or all of these ML models may be combined, or other ML models, including types not shown in FIG. 7.

[0071] It will be understood by a person of ordinary skill in the art that the aforementioned are merely examples of the types of machine-learning training that may be done to produce an ML model for any aspect of real-time generative video on a resource-constrained device. Many other combinations and techniques may be employed, which do not alter the scope of the inventive concept of real-time generative video on a resource-constrained device. Examples such as employing a CNN, use of raw images 504 as inputs, having a fully connected portion of the machine-learning training, etc. are not meant to limit the scope of the invention, but rather to serve as examples of how an ML model for generating, in real-time, interpolated video on a resource- constrained device may be accomplished.

[0072] By way of example, training for the ML model may equivalently be accomplished by incorporating, as either a part of or the complete machine-learning trainer 700, an LSTM. The LSTM algorithm is a ty pe of recurrent neural network (RNN), which may process data in a time-indexed fashion. The LSTM may solve a so-called “vanishing gradient problem,” in which gradients used in fitting may tend to zero and, thus, may not yield useful fit parameters for a given model (e.g., weights and biases). During the training phase, the LSTM may allow for persistent gradients used to fit when the gradients may otherwise go to zero (e.g., in a traditional RNN). For the purpose of the present disclosure, an example of implementing the LSTM in the machine-learning trainer 700 may be to replace the MLP 706 with the LSTM or to include the LSTM before or after the MLP 706. This may be preferred, for example, in the interpolation step 520 as time-indexing may be desired to ensure the proper order of interpolation images 522 for the final output video stream. A person of ordinary' skill in the art will understand that other configurations using the LSTM may also be equivalently used, and that the example given here is meant to be illustrative and not limiting.

Example Methods

[0073] Example methods 800-1000 are described with reference to FIGs. 8 through 10 in accordance with one or more aspects for real-time generative video on a resource-constrained device. The order in which the method blocks are descnbed is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement a method or an alternate method. Generally, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods may be described in the general context of executable instructions stored on computer-readable storage memory that is local and/or remote to a computer processing system, and implementations can include software applications, programs, functions, and the like. Alternatively or in addition, any of the functionalities described herein can be performed, at least in part, by one or more hardware logic components, such as, and without limitation, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SoCs), complex programmable logic devices (CPLDs). and the like. [0074] FIG. 8 illustrates an example method 800 for generative video on a resource-constrained device. Operation 802 includes driving one or more image sensors to capture an initial video stream. Such driving may be, by way of example, done by one or more processors. The initial video stream may, in aspects, comprise a plurality of images in sequence. The initial video stream may be materially similar to the video streams 200A, 200B, 200C, 402, 502, or 504.

[0075] Operation 804 includes generating one or more features. The generating may be, by way of example, done by one or more processors. The generating may be based on the initial video stream. There may be one or more features for each one of the images in the plurality of images in sequence. The feature generation may be an output of a ML model, taking the plurality of images in sequence as inputs. The ML model may be referred to as a generation machine- learned model.

[0076] Operation 806 includes selecting two or more images to act as keyframes. The selecting may be, for example, done by one or more processors. The selecting may be based on the one or more generated features. In aspects, selecting two or more images to act as keyframes may further include generating a keyframe image for each of the selected two or more images, with the generation of the ke frame image being based on the one or more generated features, a generated thumbnail image, or other features. Each of the plurality of images may have associated features, which may be used in the selection of the two or more keyframes, including generation of keyframes for each of the plurality of images.

[0077] The selection of two or more key frames may be accomplished with an ML model. The ML model may be referred to as a selection machine-learned model. Additionally or alternately, the selection may be accomplished by a different computational algorithm. In the example where an ML model is used for selection, the ML model may take the generated features and/or the initial video stream as inputs. The ML model may be a static model, or it may be a dynamic model able to be further fine-tuned based on inputs used for processing and selection.

[0078] Operation 808 includes generating one or more interpolation images. The generation of the one or more interpolation images may be done by one or more processors. The generation of the one or more interpolation images may be, for example, based on the images from the plurality of images associated with the two or more keyframes, the features associated with the two or more key frames, the two or more key frames, or a combination of these elements.

[0079] The generation of the one or more interpolation images may be accomplished with an ML model. The ML model may be referred to as an interpolation machine-learned model. The ML model may make inferences or prediction in order to generate the one or more interpolation images. The ML model may be a supervised model, and unsupervised model, or any other model or combination of models as described in this disclosure or as known to a person of skill in the art.

[0080] Operation 810 includes generating an output video stream. The output video stream may be generated by one or more processors. The output video stream may be a series of sequential images. The output video stream may be an enhancement of the initial video stream, such as an upscaled version, an increased frame-rate version, a version with the removal or addition of various elements, or another version or a combination of these versions. The output video stream may include, in aspects, the one or more interpolation images.

[0081] The generation of the output video stream may be done by an ML model, as described in this disclosure. The method 800 may, in aspects, be described as a GenAI method for video processing and enhancement.

[0082] FIG. 9 illustrates another example method 900 for generative video on a resource-constrained device. As with the method 800, in aspects the method 900 may be described as a GenAI method for video processing and enhancement.

[0083] Operation 902 includes driving one or more image sensors to capture an initial video stream. Such driving may be, by way of example, done by one or more processors. The initial video stream may, in aspects, comprise a plurality of images in sequence. The initial video stream may be materially similar to the video streams 200A, 200B, 200C, 402, 502, or 504.

[0084] Operation 904 includes generating one or more features. The generating may be, by way of example, done by one or more processors. The generating may be based on the initial video stream. There may be one or more features for each one of the images in the plurality of images in sequence. The feature generation may be an output of a ML model, taking the plurality of images in sequence as inputs. The ML model may be referred to as a generation machine- learned model.

[0085] Operation 906 includes selecting two or more images to act as keyframes. The selecting may be, for example, done by one or more processors. The selecting may be based on the one or more generated features. In aspects, selecting two or more images to act as keyframes may further include generating a keyframe image for each of the selected two or more images, with the generation of the keyframe image being based on the one or more generated features, a generated thumbnail image, or other features. Each of the plurality of images may have associated features, which may be used in the selection of the two or more keyframes, including generation of keyframes for each of the plurality of images.

[0086] The selection of two or more keyframes may be accomplished with an ML model. The ML model may be referred to as a selection machine-learned model. Additionally or alternately, the selection may be accomplished by a different computational algorithm. In the example where an ML model is used for selection, the ML model may take the generated features and/or the initial video stream as inputs. The ML model may be a static model, or it may be a dynamic model able to be further fine-tuned based on inputs used for processing and selection.

[0087] Operation 908 includes generating one or more interpolation images. The generation of the one or more interpolation images may be done by one or more processors. The generation of the one or more interpolation images may be, for example, based on the images from the plurality of images associated with the two or more keyframes, the features associated with the two or more key frames, the two or more key frames, or a combination of these elements.

[0088] The generation of the one or more interpolation images may be accomplished with an ML model. The ML model may be referred to as an interpolation machine-learned model. The ML model may make inferences or prediction in order to generate the one or more interpolation images. The ML model may be a supervised model, and unsupervised model, or any other model or combination of models as described in this disclosure or as known to a person of skill in the art.

[0089] Operation 910 includes generating one or more second interpolation images. The generation of the one or more second interpolation images may be done by one or more processors. The generation of the one or more second interpolation images may be, for example, based on the one or more interpolation images.

[0090] The generation of the one or more second interpolation images may be accomplished with an ML model. The ML model may be referred to as an interpolation machine- learned model. The ML model may be the same or different from the ML model described above with reference to operation 908. The ML model may make inferences or prediction in order to generate the one or more second interpolation images. The ML model may be a supervised model, and unsupervised model, or any other model or combination of models as described in this disclosure or as known to a person of skill in the art.

[0091] Operation 912 includes generating an output video stream. The output video stream may be generated by one or more processors. The output video stream may be a series of sequential images. The output video stream may be an enhancement of the initial video stream, such as an upscaled version, an increased frame-rate version, a version with the removal or addition of various elements, or another version or a combination of these versions. The output video stream may be based on the one or more interpolation images and the one or more second interpolation images. In aspects, the output video stream may include the one or more interpolation images and the one or more second interpolation images. The one or more interpolation images may include a first image, a second image, and a third image. The one or more second interpolation images may include a primary image and a secondary image, and the primary image may be based on the first image and the second image, and the secondary image may be based on the second and the third image. The primary and secondary images may be inserted between the first and second images, and second and third images respectively. In this way, a frame rate may be increased.

[0092] FIG. 10 illustrates another example method 1000 for generative video on a resource-constrained device. As with the methods 800 and 900, in aspects, the method 1000 may be described as a GenAI method for video processing and enhancement.

[0093] Operation 1002 includes driving one or more image sensors to capture an initial video stream. Such driving may be, by way of example, done by one or more processors. The initial video stream may, in aspects, comprise a pl ural i ty of images in sequence. The initial video stream may be materially similar to the video streams 200A, 200B, 200C, 402, 502, or 504.

[0094] Operation 1004 includes generating one or more features. The generating may be, by way of example, done by one or more processors. The generating may be based on the initial video stream. There may be one or more features for each one of the images in the plurality of images in sequence. The feature generation may be an output of an ML model, taking the plurality of images in sequence as inputs. The ML model may be referred to as a generation machine- learned model.

[0095] Operation 1006 includes comparing the one or more features with one or more parameters. The one or more parameters may be static parameters stored in a memory, dynamic parameters computed in real-time, or some combination of static and dynamic parameters. In aspects, the parameters may represent threshold values. Examples of threshold values include, but are not limited to, a motion score or motion detection threshold, an object identification confidence level or object detection classification, a person identification confidence level, a contrast level, a color profile, and other values common to video processing. Further examples of threshold values may include a hardware threshold, a composition threshold, and a time threshold,

[0096] For example, a motion score may be a value computed for the severity of a motion detected. In aspects, a running dog may have a higher motion score than a walking person. In aspects, an identification confidence level may be a number between 0 and 1 indicating the level of confidence a machine model, such as an ML model, may have in the classification of an object, individual, etc. The one or more parameters may be based on ML models, static values, etc.

[0097] Operation 1008 includes selecting two or more images to act as keyframes. The selecting may be, for example, done by one or more processors. The selecting may be based on the one or more generated features and/or the comparison of the one or more features with the one or more parameters. In aspects, selecting two or more images to act as keyframes may further include generating a keyframe image for each of the selected two or more images, with the generation of the keyframe image being based on the one or more generated features, a generated thumbnail image, or other features. Each of the plurality of images may have associated features, which may be used in the selection of the two or more keyframes, including generation of key frames for each of the plurality of images.

[0098] The selection of two or more key frames may be accomplished with an ML model. The ML model may be referred to as a selection machine-learned model. Additionally or alternately, the selection may be accomplished by a different computational algorithm. In the example where an ML model is used for selection, the ML model may take the generated features, the initial video stream, and/or the comparison of the one or more features with the one or more parameters as inputs. The ML model may be a static model, or it may be a dynamic model able to be further fine-tuned based on inputs used for processing and selection.

[0099] Operation 1010 includes generating one or more interpolation images. The generation of the one or more interpolation images may be done by one or more processors. The generation of the one or more interpolation images may be, for example, based on the images from the plurality of images associated with the two or more keyframes, the features associated with the two or more key frames, the two or more key frames, the comparison of the one or more features with the one or more parameters, or a combination of these elements.

[00100] The generation of the one or more interpolation images may be accomplished with an ML model. The ML model may be referred to as an interpolation machine-learned model. The ML model may make inferences or prediction in order to generate the one or more interpolation images. The ML model may be a supervised model, and unsupervised model, or any other model or combination of models as described in this disclosure or as known to a person of skill in the art.

[00101] Operation 1012 includes generating an output video stream. The output video stream may be generated by one or more processors. The output video stream may be a series of sequential images. The output video stream may be an enhancement of the initial video stream, such as an upscaled version, an increased frame-rate version, a version with the removal or addition of various elements, or another version or a combination of these versions. The output video stream may be based on the one or more interpolation images. In aspects, the output video stream may include the one or more interpolation images.

Additional Examples

[00102] The following are additional examples of the described devices and methods for real-time generative video on a resource-constrained device. [00103] Example 1. A method for generating interpolated video, the method comprising driving, by one or more processors, one or more image sensors of an electronic device to capture an initial video stream, the initial video stream comprising a plurality of images in sequence; generating, by the one or more processors and based on the initial video stream, one or more features for each of the plurality of images; selecting, by the one or more processors and based on the one or more generated features, two or more images from the plurality of images to act as keyframes; generating, by the one or more processors and based on the keyframes and the two or more features associated with the keyframes, one or more interpolation images; and generating, by the one or more processors and based on the one or more interpolation images, an output video stream.

[00104] Example 2. The method of example 1, wherein the generation of the one or more features is done using a generation machine-learned model stored in a memory' of the electronic device.

[00105] Example 3. The method of example 2, wherein the machine-learned model is a convolutional neural network (CNN).

[00106] Example 4. The method of any one of the previous examples, wherein the selection of the two or more keyframes is done using a selection machine-learned model stored in a memory of the electronic device.

[00107] Example 5. The method of any one of the previous examples, further comprising generating, by the one or more processors, one or more second interpolation images based on the one or more interpolation images and wherein the output video stream comprises the interpolation images and the second interpolation images.

[00108] Example 6. The method of example 5, wherein the one or more interpolation images comprise a first image, a second image, and a third image; the one or more second interpolation images comprise a primary' image and a secondary' image; and the primary' image is based on the first image and the second image and the secondary' image is based on the second image and the third image.

[00109] Example 7. The method of any one of the previous examples, wherein the generation of the one or more interpolation images is done by an interpolation machine-learned model.

[00110] Example 8. The method of any one of the previous examples, further comprising comparing the one or more features with one or more parameters, and wherein the selection of the keyframes is based on the comparison of the one or more features with the one or more parameters. [00111] Example 9. The method of example 8, wherein the one or more parameters include one or more of a motion detection threshold, an object detection classification, a hardware threshold, a composition threshold, or a time threshold.

[00112] Example 10. The method of any one of the previous examples, wherein the output video stream comprises an upscaling of the initial video stream, the upscaling comprising an increase in the initial video stream resolution, frame rate, or both.

[00113] Example 11. The method of any one of the previous examples, wherein the output video stream comprises the initial video stream with one or more elements omitted from and/or one or more elements added to the initial video stream.

[00114] Example 12. The method of any one of the previous examples, wherein the output video stream comprises an enhancement of the initial video stream, the enhancement comprising one or more of a change in contrast, a change in color, an application of a fdter effect, a change in white balance, or a reduction of an artifact.

[00115] Example 13. The method of any one of the previous examples, wherein the generating of the output video stream comprises generating higher resolution versions of the one or more interpolation images.

[00116] Example 14. The method of any one of the previous examples, wherein the generated one or more features comprise at least a second plurality of images, the second plurality of images being a lower resolution set of the plurality of images.

[00117] Example 15. A mobile electronic device comprising one or more image sensors; one or more processors; and a memory' containing instructions that, when accessed by the one or more processors, cause the one or more processors to perform any one of the methods of examples 1-14.

[00118] Example 16. A non-transitory, computer-readable storage medium comprising instructions that, when executed, configure at least one processor of an electronic device to perform any one of the methods of examples 1-14.

Conclusion

[00119] While the present subject matter has been described in detail with respect to various specific example implementations thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, vanations of, and equivalents to such implementations. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations, and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one implementation can be used with another implementation to yield a still further implementation. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

[00120] While various embodiments of the disclosure are described in the foregoing description and shown in the drawings, it is to be distinctly understood that this disclosure is not limited thereto but may be variously embodied to practice within the scope of the following claims. From the foregoing description, it will be apparent that various changes may be made without departing from the spirit and scope of the disclosure as defined by the following claims.

Claims

CLAIMS What is claimed is:

1. A method for generating interpolated video, the method comprising: driving, by one or more processors, one or more image sensors of an electronic device to capture an initial video stream, the initial video stream comprising a plurality of images in sequence; generating, by the one or more processors and based on the initial video stream, one or more features for each of the plurality of images: selecting, by the one or more processors and based on the one or more generated features, two or more images from the plurality of images to act as keyframes; generating, by the one or more processors and based on the ke frames and the two or more features associated with the key frames, one or more interpolation images; and generating, by the one or more processors and based on the one or more interpolation images, an output video stream.

2. The method of claim 1. wherein the generation of the one or more features is done using a generation machine-learned model stored in a memory of the electronic device.

3. The method of claim 2, wherein the machine-learned model is a convolutional neural network (CNN).

4. The method of any one of the previous claims, wherein the selection of the two or more keyframes is done using a selection machine-learned model stored in a memory of the electronic device.

5. The method of any one of the previous claims, further comprising generating, by the one or more processors, one or more second interpolation images based on the one or more interpolation images and wherein the output video stream comprises the interpolation images and the second interpolation images.

6. The method of claim 5, wherein: the one or more interpolation images comprise a first image, a second image, and a third image; the one or more second interpolation images comprise a primary image and a secondary image; and the primary image is based on the first image and the second image and the secondary' image is based on the second image and the third image.

7. The method of any one of the previous claims, wherein the generation of the one or more interpolation images is done by an interpolation machine-learned model.

8. The method of any one of the previous claims, further comprising comparing the one or more features with one or more parameters, and wherein the selection of the keyframes is based on the comparison of the one or more features with the one or more parameters.

9. The method of claim 8, w herein the one or more parameters include one or more of a motion detection threshold, an object detection classification, a hardware threshold, a composition threshold, or a time threshold.

10. The method of any one of the previous claims, wherein the output video stream comprises an upscaling of the initial video stream, the upscaling comprising an increase in the initial video stream resolution, frame rate, or both.

11. The method of any' one of the previous claims, wherein the output video stream comprises the initial video stream with one or more elements omitted from and/or one or more elements added to the initial video stream.

12. The method of any one of the previous claims, wherein the output video stream comprises an enhancement of the initial video stream, the enhancement comprising one or more of a change in contrast, a change in color, an application of a filter effect, a change in white balance, or a reduction of an artifact.

13. The method of any one of the previous claims, wherein the generated one or more features comprise at least a second plurality' of images, the second plurality' of images being a lower resolution set of the plurality of images.

14. A mobile electronic device comprising: one or more image sensors; one or more processors; and a memory containing instructions that, when accessed by the one or more processors, cause the one or more processors to perform any one of the methods of claims 1-13.

15. A non-transitory, computer-readable storage medium comprising instructions that, when executed, configure at least one processor of an electronic device to perform any one of the methods of claims 1-13.