TW202334902A - Systems and methods for image reprojection - Google Patents

Abstract

An imaging system receives depth data (corresponding to an environment) from a depth sensor and first image data (a depiction of the environment) from an image sensor. The imaging system generates, based on the depth data, first motion vectors corresponding to a change in perspective of the depiction of the environment in the first image data. The imaging system generates, using grid inversion based on the first motion vectors, second motion vectors that indicate respective distances moved by respective pixels of the depiction of the environment in the first image data for the change in perspective. The imaging system generates second image data by modifying the first image data according to the second motion vectors. The second image data includes a second depiction of the environment from a different perspective than the first image data. Some image reprojection applications (e.g., frame interpolation) can be performed without the depth data.

Description

Systems and methods for image reprojection

This case is about image processing. More specifically, the present case relates to systems and methods that reproject a first image captured from a first perspective using, for example, grid inversion to produce a second image that appears to be captured from a second perspective.

A camera is a device that uses an image sensor to receive light and capture an image frame, such as a still image or video frame. The camera captures an image of the illustrated environment from a perspective corresponding to the camera's field of view.

Extended reality (XR) devices are devices that display the environment to the user, such as through a head-mounted display (HMD) or a mobile handheld terminal. The environment is at least partially different from the user's real-world environment. Users can generally interactively change their view of the environment, such as by tilting or moving the HMD or other device. Virtual reality (VR), augmented reality (AR), and mixed reality (MR) are examples of XR. XR devices can include sensors that capture information from the environment.

In some examples, systems and techniques for image processing are described. In some examples, the imaging system receives depth information (corresponding to the environment). The imaging system receives first image data (including a representation of the environment) captured by the image sensor. The imaging system generates a first motion vector corresponding to a change in perspective of the representation of the environment in the first image data based on the depth information. The imaging system uses grid inversion based on the first plurality of motion vectors to generate a second motion vector indicating a corresponding distance that a corresponding primitive of the representation of the environment in the first image material moves in response to a change in viewing angle. The imaging system generates second image data by modifying the first image data based on the first motion vector and/or the second motion vector. The second image material includes a second illustration of the environment from a different perspective than the first image material. The imaging system outputs the second image data. Some image reprojection applications (e.g., frame interpolation) can be performed without depth information.

In one example, an image processing device is provided. The apparatus includes memory and one or more processors coupled to the memory (eg, implemented in circuitry). The one or more processors are configured to and may: receive depth data including depth information corresponding to the environment; receive first image data captured by the image sensor, the first image data including an illustration of the environment; based on at least the depth data, generating a first plurality of motion vectors corresponding to changes in perspective of the illustration of the environment in the first image data; based on the first plurality of motion vectors using Grid inversion generates a second plurality of motion vectors indicating corresponding distances that corresponding primitives of the representation of the environment in the first image material move in response to the change in viewing angle; at least in part Second image data is generated by modifying the first image data based on the second plurality of motion vectors, wherein the second image data includes a second view of the environment from a different perspective than the first image data. icon; and output the second image data.

In another example, an image processing method is provided. The method includes: receiving depth data including depth information corresponding to the environment; receiving first image data captured by an image sensor, the first image data including an illustration of the environment; based on at least the Depth data, generating a first plurality of motion vectors corresponding to changes in perspective of the illustration of the environment in the first image data; using grid inversion to generate a second plurality of motion vectors based on the first plurality of motion vectors Motion vectors, the second plurality of motion vectors indicating corresponding distances in which corresponding primitives of the representation of the environment in the first image data move in response to the change in viewing angle; at least in part by moving according to the second plurality of motions vector modifying the first image data to generate second image data, wherein the second image data includes a second illustration of the environment from a different perspective than the first image data; and outputting the second image Like data.

In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive information including communicating with an environment. Depth data corresponding to depth information; receiving first image data captured by an image sensor, the first image data including an illustration of the environment; based on at least the depth data, generating the first image data A first plurality of motion vectors corresponding to changes in the viewing angle of the illustration of the environment in the image data; using grid inversion to generate a second plurality of motion vectors based on the first plurality of motion vectors, the second plurality of motion vectors A vector indicating a corresponding distance that a corresponding primitive of the representation of the environment in the first image data moves in response to the change in perspective; at least in part by modifying the first image data based on the second plurality of motion vectors. generating second image data, wherein the second image data includes a second representation of the environment from a different perspective than the first image data; and outputting the second image data.

In another example, an image processing device is provided. The device includes: means for receiving first image data captured by an image sensor, the first image data including an illustration of the environment; and means for generating an image corresponding to the first image data based on at least the depth data. means for a first plurality of motion vectors corresponding to a change in viewing angle of the illustration of the environment in the image data; means for generating a second plurality of motion vectors based on the first plurality of motion vectors using grid inversion , the second plurality of motion vectors indicates the corresponding distance that the corresponding graphic element of the illustration of the environment in the first image data moves in response to the change of viewing angle; for at least partially by moving according to the second plurality of motions means for vector modifying the first image data to generate second image data, wherein the second image data includes a second representation of the environment from a different perspective than the first image data; and for outputting The component of the second image data.

In some aspects, the second image material includes an interpolated image configured to illustrate the environment at a second time between the first time and a third time, wherein the first image material includes an illustration At least one image of the environment at at least one of the first time or the third time.

In some aspects, the first image data includes a plurality of video data frames including parallax motion, wherein the second image data includes a stable variant of the plurality of video data frames that reduces the parallax motion.

In some aspects, the first image data includes the person viewing the image sensor from a first angle, wherein the second image data includes the person viewing the image from a second angle different from the first angle. Like a sensor.

In some aspects, the viewing angle changes include viewing angle rotations based on angles and about an axis. In some aspects, the viewing angle changes include viewing angle translation based on direction and distance. In some aspects, perspective changes include transitions. In some aspects, the change in perspective includes movement along an axis between an original perspective of the illustration of the environment in the first image material and a position of an object in the environment, wherein at least a portion of the object This is illustrated in the first image material.

In some aspects, one or more of the methods, apparatus, and computer-readable media described above further includes identifying the second image based on one or more gaps in the second plurality of motion vectors. one or more gaps in the second image data; and before outputting the second image data, at least partially modifying the second image by filling one or more gaps in the second image data using interpolation material.

In some aspects, one or more of the methods, apparatus, and computer-readable media described above further includes identifying the second image based on one or more gaps in the second plurality of motion vectors. one or more occluded areas in the image data; and before outputting the second image data, modify the second image at least in part by filling one or more gaps in the second image data using inpainting material.

In some aspects, one or more of the methods, apparatus, and computer-readable media described above further includes identifying the second image based on one or more gaps in the second plurality of motion vectors. one or more occluded areas in the image data; and before outputting the second image data, at least partially filling in one or more occluded areas in the second image data by using inpainting with one or more trained machine learning models. or multiple gaps to modify the second image data.

In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include: based on one or more of the second plurality of motion vectors from the first image data. conflict values to identify one or more conflicts in the second image data; and selecting the one or more conflict values from the first image data based on motion data associated with the second plurality of motion vectors one of them.

In some aspects, the depth information includes a three-dimensional representation of the environment from a first-person perspective. In some aspects, the depth data is received from at least one depth sensor, wherein the at least one depth sensor includes at least one time-of-flight sensor.

In some aspects, outputting the second image data includes using at least a display to display the second image data. In some aspects, outputting the second image data includes causing the second image data to be sent to at least one recipient device using at least one communication interface.

In some aspects, the illustration of the environment in the first image material illustrates the environment from a first perspective, wherein the change in perspective is a combination of the first perspective and a third perspective of the environment in the second image material. The two diagrams show the corresponding changes between different viewing angles.

In some aspects, the change in perspective includes at least one of a parallax shift of the perspective or a rotation of the perspective about the axis, the method further comprising receiving, via the user interface, one of: an indication of a distance of the parallax shift of the perspective, or an indication of the angle of rotation or axis of the viewing angle.

In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include identifying, based on one or more gaps in corresponding endpoints of the first plurality of motion vectors, One or more gaps in the second plurality of motion vectors create one or more gaps in the second image data; and before outputting the second image data, filling the second plurality of motion vectors at least partially by using interpolation. one or more gaps in the second image data to modify the second image data.

In some aspects, the device is a wearable device, an extended reality device (eg, a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device , wireless communication devices, mobile devices (such as mobile phones and/or mobile handheld terminals and/or so-called "smartphones" or other mobile devices), cameras, personal computers, laptops, server computers, vehicles or An element of a computing device or vehicle, part of another device or combination thereof and/or includes them. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and/or other displayable information. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and/or other sensors).

This Summary is neither intended to identify key or essential features of the claimed subject matter, nor is it intended to be used solely to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the appropriate portions of the complete specification of this patent, any or all of the drawings, and each claim.

The foregoing, together with other features and embodiments, will become more apparent by reference to the following specification, claims and drawings.

Some aspects and examples of this case are provided below. Some of these aspects and embodiments may be used independently and some of them may be used in combination, as will be apparent to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the present embodiments. It may be apparent, however, that various embodiments may be practiced without these specific details. The drawings and descriptions are not intended to be limiting.

The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present invention. Rather, the following description of the exemplary embodiments will provide those skilled in the art with a feasible description for implementing the exemplary embodiments. It should be understood that various changes could be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims.

A camera is a device that uses an image sensor to receive light and capture an image frame, such as a still image or video frame. The terms "image", "image frame" and "frame" are used interchangeably in this article. Cameras can be configured with various image capture and image processing settings. Different settings cause images to have different appearances. Some camera settings (such as ISO, exposure time, aperture size, aperture range, shutter speed, focus, and gain) are determined and applied before or during the acquisition of one or more image frames. For example, settings or parameters may be applied to an image sensor to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, levels, curves, or color. For example, settings or parameters may be applied to a processor (eg, an image signal processor or ISP) to process one or more image frames captured by an image sensor.

A depth sensor is a sensor that measures depth, range, or distance from one or more parts of the environment in which the depth sensor is located. Examples of depth sensors include light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) ) sensor, time-of-flight (ToF) sensor, structured light sensor or a combination thereof. The depth data captured by the depth sensor may include point clouds, 3D models and/or depth images.

Extended reality (XR) systems or devices can provide users with virtual content and/or a view of the real world that can combine physical environments (scenes) and virtual environments (including virtual content). The XR system facilitates user interaction with such combined XR environments. Real-world views may include real-world objects (also referred to as physical objects), such as people, vehicles, buildings, tables, chairs, and/or other real-world or physical objects. XR systems or devices can facilitate interaction with different types of XR environments (e.g., users can use XR systems or devices to interact with XR environments). XR systems may include virtual reality (VR) systems that facilitate interaction with VR environments, augmented reality (AR) systems that facilitate interaction with AR environments, mixed reality (MR) systems that facilitate interaction with MR environments, and/or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and more. In some cases, the XR device can track a portion of the user (eg, the user's hands and/or fingertips) to allow the user to interact with virtual content items.

The imaging system may include a depth sensor and an image sensor of the camera. The depth sensor captures depth data including depth information corresponding to the environment, such as a point cloud, a 3D model, a depth image, a set of disparity values, and/or a 3D representation of the environment. The image sensor captures first image data including a 2D representation of the environment.

The imaging system uses the depth data to generate a first set of motion vectors. The first set of motion vectors corresponds to a change in the perspective of the illustration of the environment in the first image material from a first perspective to a second perspective.

The imaging system applies grid inversion to the first set of motion vectors to produce a second set of motion vectors. The second set of motion vectors is indicative of the corresponding distance that corresponding primitives of the representation of the environment in the first image material move in response to a change in perspective from the first perspective to the second perspective. In some cases, to apply grid inversion, the imaging system works by prioritizing larger motions over smaller motions and/or by prioritizing the motion of closer objects in the environment over the motion of farther objects in the environment. Resolve conflicts with grid inversion. In some cases, to apply grid inversion, imaging systems use interpolation to fill in missing areas.

The imaging system generates second image data by modifying the image data based on the second set of motion vectors. For example, the imaging system may modify the image based on the second set of motion vectors by moving corresponding primitives of primitive data representing the environment in the first image data by corresponding distances indicated by the second set of motion vectors. material. The second image material includes a second illustration of the environment from a different perspective than the first image material. The imaging system outputs the second image data, for example, by displaying the second image data or sending the second image data to the recipient device.

Various useful applications of perspective changes are achieved by generating second image data by modifying the first image data based on a second set of motion vectors based on grid inversion. For example, perspective changes can be used for 3D stabilization of video material, such as to reduce or eliminate parallax movement that may be caused by an unstable hand holding the camera and/or by the user's footsteps. Viewing angle changes can be used for frame interpolation to increase the effective frame rate of a video by creating an intermediate frame between two existing frames. Perspective changes can be used for a "3D zoom" effect, which zooms the foreground of an environment more quickly than the background of the environment to look more like real movement forward into the environment rather than zooming in. Viewing angle changes can be used to accommodate offsets between two sensors (e.g., two cameras, a camera and a depth sensor, etc.). Perspective changes can be used for head pose correction, for example, making the camera appear level with the person's head when the camera is actually below or above the person, which is common in video conferencing. Perspective changes can be used in XR to quickly simulate different perspectives of an environment even if the different perspectives have not yet finished rendering. Perspective changes can be used for a variety of special effects, such as simulating rotation around objects in the scene.

In some examples, systems and techniques for image processing are described. In some examples, the imaging system receives depth data (corresponding to the environment) captured by the depth sensor, and the imaging system receives first image data (a representation of the environment) captured by the image sensor. The imaging system generates a first motion vector corresponding to a change in perspective of the representation of the environment in the first image data based on the depth information. The imaging system uses grid inversion based on the first motion vector to generate a second motion vector indicating a corresponding distance that a corresponding primitive of the representation of the environment in the first image material moves in response to a change in viewing angle. The imaging system generates second image data by modifying the first image data based on the first motion vector and/or the second motion vector. The second image material includes a second illustration of the environment from a different perspective than the first image material. The imaging system outputs the second image data.

The imaging systems and techniques described herein provide several technical improvements over previous image processing systems. For example, the image processing systems and techniques described herein can provide reprojection to different viewing angles for any viewing angle translation and/or rotational movement. The image processing systems and techniques described in this article make it possible to use this reprojection and the grid inversion techniques that support it for a variety of applications, including using optical flow to improve video frame quality and aligning depth and image data to overcome these two problems. Offset distance between sensors, 3D depth-based video stabilization, 3D depth-based zoom (also called movie zoom), aligning image data from two different cameras to overcome the two senses Offset distance between monitors, head pose correction, post-production reprojection for extended reality (XR), special effects, or a combination thereof. Using grid inversion can increase efficiency, reduce computational load, reduce power usage, reduce heat generation, and reduce the need for cooling components.

Each aspect of this case will be described with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating the architecture of an image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (eg, images of scene 110). The image capture and processing system 100 can capture individual images (or photos) and/or can capture video that includes multiple images (or video frames) in a specific sequence. Lens 115 of system 100 faces scene 110 and receives light from scene 110 . Lens 115 deflects light toward image sensor 130 . Light received by lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by image sensor 130 . In some examples, scene 110 is a scene in an environment. In some examples, scene 110 is the scene of at least a portion of the user. For example, scene 110 may be a scene of one or both eyes of the user and/or at least a portion of the user's face.

One or more control mechanisms 120 may control exposure, focus, and/or zoom based on information from the image sensor 130 and/or based on information from the image processor 150 . One or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and/or one or more zoom control mechanisms 125C . One or more control mechanisms 120 may also include additional control mechanisms in addition to those shown, such as controls that control analog gain, flash, HDR, depth of field, and/or other image capture attributes.

The focus setting can be obtained by the focus control mechanism 125B of the control mechanism 120 . In some examples, focus control mechanism 125B stores focus settings in a memory register. Based on the focus setting, focus control mechanism 125B may adjust the position of lens 115 relative to the position of image sensor 130 . For example, based on the focus setting, the focus control mechanism 125B can adjust the focus by actuating a motor or servo system to move the lens 115 closer to or away from the image sensor 130 . In some cases, additional lenses may be included in system 100 , such as one or more microlenses above each photodiode of image sensor 130 , each of which responds when light received from lens 115 reaches the corresponding photodiode. The polar body deflects light toward the photodiode. Focus setting can be determined via contrast-detection autofocus (CDAF), phase-detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and/or the image processor 150. Focus settings may be called image capture settings and/or image processing settings.

The exposure setting can be obtained by the exposure control mechanism 125A of the control mechanism 120 . In some cases, exposure control mechanism 125A stores exposure settings in a storage register. Based on the exposure setting, the exposure control mechanism 125A can control the size of the aperture (eg, aperture size or aperture range), the duration of the aperture opening (eg, exposure time or shutter speed), the sensitivity of the image sensor 130 (eg, ISO speed or film speed), analog gain applied by image sensor 130, or any combination thereof. Exposure settings may be referred to as image capture settings and/or image processing settings.

The zoom setting can be obtained by the zoom control mechanism 125C of the control mechanism 120 . In some examples, zoom control mechanism 125C stores zoom settings in a memory register. Based on the zoom setting, zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses relative to each other. The zoom settings may be called image capture settings and/or image processing settings. In some examples, the lens assembly may include a hyperfocal zoom lens or a zoom zoom lens. In some examples, the lens assembly may include a focusing lens (which may be lens 115 in some cases) that first receives light from scene 110 and then passes the light through the focusing lens (eg, Afocal zoom system between lens 115) and image sensor 130). In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses of equal or similar (e.g., within a threshold difference) focal length, with a negative (e.g., divergent, concave) lens between them . In some cases, zoom control mechanism 125C moves one or more lenses in an afocal zoom system, such as a negative lens and one or two positive lenses.

Image sensor 130 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific primitive in the image produced by image sensor 130 . In some cases, different photodiodes may be covered by different color filters so that light matching the color of the color filter covering the photodiodes can be measured. For example, Bayer filters include red filters, blue filters, and green filters, where each primitive of the image is based on red light from at least one photodiode covered in the red filter data, blue light data from at least one photodiode covered in a blue color filter, and green light data from at least one photodiode covered in a green color filter. Other types of color filters may use yellow, magenta, and/or cyan (also called "emerald") color filters instead of or in addition to red, blue, and/or green color filters. Some image sensors may have no color filters at all and may instead use different photodiodes (in some cases stacked vertically) throughout the array of primitives. Different photodiodes throughout the array of primitives can have different spectral sensitivity curves and therefore respond to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore lack color depth.

In some cases, image sensor 130 may alternatively or additionally include opaque and/or reflective masks that block light from reaching certain photodiodes or photodiodes at certain times and/or from certain angles. Part of the polar body, this can be used for phase detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier to amplify the analog signal output by the photodiode and/or include an analog-to-digital converter (ADC) to output the analog signal of the photodiode (and/or by the analog signal). A gain amplifier amplifies) and converts it into a digital signal. In some cases, certain elements or functionality discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130 . The image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active picture element sensor (APS), a complementary metal oxide semiconductor (CMOS), or an N-type metal Oxide semiconductor (NMOS), hybrid CCD/CMOS sensor (e.g., sCMOS), or some other combination thereof.

Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and/or Computing system 4100 may be one or more of the other types of processors 4110 discussed. Host processor 152 may be a digital signal processor (DSP) and/or other types of processors. In some implementations, image processor 150 is a single integrated circuit or die (eg, referred to as a system on a chip or SoC) that includes host processor 152 and ISP 154 . In some cases, the chip may also include one or more input/output ports (eg, input/output (I/O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem ( For example, 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth®, Global Positioning System (GPS), etc.), any combination thereof, and/or other components. I/O port 156 may include any suitable input/output port or interface in accordance with one or more protocols or specifications, such as an inter-integrated circuit 2 (I2C) interface, an inter-integrated circuit 3 (I3C) interface, a serial peripheral Interface (SPI) interface, Serial General Purpose Input/Output (GPIO) interface, Mobile Industry Processor Interface (MIPI) (such as MIPI CSI-2 Physical (PHY) layer port or interface), Advanced High Performance Bus (AHB) bus, any combination thereof, and/or other input/output ports. In one illustrative example, host processor 152 may communicate with image sensor 130 using an I2C port, and ISP 154 may communicate with image sensor 130 using a MIPI port.

The image processor 150 can perform many tasks, such as demosaicing, color space conversion, image frame downsampling, primitive interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance , merging image frames to form HDR images, image recognition, object recognition, feature recognition, input reception, managing output, managing memory, or some combination thereof. The image processor 150 may store the image frame and/or the processed image in the random access memory (RAM) 140 and/or 4120, the read only memory (ROM) 145 and/or 4125, or the cache memory. body, a memory unit, another storage device, or some combination thereof.

Various input/output (I/O) devices 160 may be connected to image processor 150 . I/O device 160 may include a display screen, a keyboard, a keypad, a touch screen, a trackpad, a touch-sensitive surface, a printer, any other output device 4135, any other input device 4145, or some combination thereof. In some cases, subtitles may be input into image processing device 105B via a physical keyboard or keypad of I/O device 160 or via a virtual keyboard or keypad of a touch screen of I/O device 160 . I/O 160 may include one or more ports, jacks, or other connectors that enable wired connections between system 100 and one or more peripheral devices through which system 100 may Other connectors receive data from and/or send data to one or more peripheral devices. I/O 160 may include a wireless transceiver that enables wireless connection between system 100 and one or more peripheral devices, through which system 100 may receive data from one or more peripheral devices and/or transmit data to one or more peripheral devices. Send data to multiple peripheral devices. Peripheral devices may include any of the previously discussed types of I/O devices 160 and may themselves be considered I/O devices once they are coupled to a port, jack, wireless transceiver, or other wired and/or wireless connector 160.

In some cases, image capture and processing system 100 may be a single device. In some cases, image capture and processing system 100 may be two or more separate devices, including image capture device 105A (eg, a camera) and image processing device 105B (eg, a camera coupled to the camera). computing equipment). In some embodiments, image capture device 105A and image processing device 105B may be coupled together wirelessly, such as via one or more wires, cables, or other electrical connectors and/or via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

As shown in Figure 1, the vertical dotted line divides the image acquisition and processing system 100 of Figure 1 into two parts, which respectively represent the image acquisition device 105A and the image processing device 105B. The image capturing device 105A includes a lens 115, a control mechanism 120 and an image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145 and I/O 160. In some cases, certain components shown in image capture device 105A, such as ISP 154 and/or host processor 152, may be included in image capture device 105A.

Image capture and processing system 100 may include electronic devices such as mobile or landline handheld terminals (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, on-board box, television, camera, display device, digital media player, video game console, video streaming device, Internet Protocol (IP) camera or any other suitable electronic device. In some examples, image capture and processing system 100 may include one or more devices for wireless communications, such as cellular network communications, 802.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof. wireless transceiver. In some embodiments, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device and the image processing device 105B may include a computing device, such as a mobile handheld terminal, a desktop computer, or other computing device.

Although the image capture and processing system 100 is shown as including certain elements, those skilled in the art will understand that the image capture and processing system 100 may include more elements than those shown in FIG. 1 . The components of image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include and/or may be implemented using electronic circuits or other electronic hardware. May include one or more programmable electronic circuits (e.g., microprocessor, GPU, DSP, CPU and/or other suitable electronic circuits); and/or may include computer software, firmware or any combination thereof and/or may Implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. Software and/or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 100 .

2 is a block diagram illustrating an exemplary architecture of an imaging system 200 for performing reprojection operations for various applications. In some examples, imaging system 200 includes at least one image capture and processing system 100, image capture device 105A, image processing device 105B, or a combination thereof. In some examples, imaging system 200 includes at least one computing system 4100. In some examples, imaging system 200 includes at least one neural network 3900.

In some examples, imaging system 200 includes one or more sensors 205 . Sensor 205 captures sensor data that measures and/or tracks information about various aspects of the environment in which imaging system 200 and/or a user of imaging system 200 is. In some examples, sensors 205 may capture sensor data that measures and/or tracks information about the user's body and/or the user's behavior. In some examples, sensors 205 include one or more cameras facing at least a portion of the environment and/or the user. One or more cameras may include one or more image sensors that capture images of at least a portion of the environment and/or the user. In some examples, sensors 205 include one or more depth sensors facing at least a portion of the environment and/or the user. One or more depth sensors may capture depth data of at least a portion of the environment and/or the user (e.g., depth images, point clouds, 3D models, ranges between the depth sensor and portions of the environment, depth the depth between the sensor and parts of the environment and/or the distance between the depth sensor and parts of the environment). In some examples, stereoscopic depth sensing may also be used to determine depth data (such as any of the types of depth data listed above) using image data from a stereo camera. In some examples, image data from a stereo camera can be used to determine depth data by inputting the image data into a trained machine learning model trained based on the training data. The training data includes other images captured by a stereo camera (or other cameras in a similar stereo arrangement) along with corresponding depth data. In some examples, sensors 205 include one or more other types of sensors, such as microphones, accelerometers, gyroscopes, positioning receivers, inertial measurement units (IMUs), biosensors, or combinations thereof. In Figure 2, one or more sensors 205 are shown as a camera diagram and a microphone diagram.

Sensors 205 may include one or more cameras, image sensors, microphones, heart rate monitors, oximeters, biosensors, positioning transceivers, inertial measurement units (IMUs), accelerometers, gyroscopes, Gyroscope, barometer, thermometer, altimeter, depth sensor, other sensors discussed in this article, or combinations thereof. Examples of depth sensors include light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) ) sensor, time-of-flight (ToF) sensor, structured light sensor or a combination thereof. Examples of positioning receivers include Global Navigation Satellite System (GNSS) receivers, Global Positioning System (GPS) receivers, cellular signal transceivers, Wi-Fi transceivers, Wireless Local Area Network (WLAN) transceivers, Bluetooth transceivers , beacon transceiver, near field communication (NFC) transceiver, personal area network (PAN) transceiver, radio frequency identification (RFID) transceiver, communication interface 4140 or a combination thereof. In some examples, one or more sensors 205 include at least one image capture and processing system 100, image capture device 105A, image processing device 105B, or a combination thereof. In some examples, one or more sensors 205 includes at least one input device 4145 of computing system 4100 . In some implementations, one or more of sensors 205 may supplement or improve sensor readings from other sensors 205 . For example, application engine 210 and/or image reprojection engine 215 may use sensor data from positioning receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, and/or other sensors to improve and/or or supplement image data and/or depth data. For example, the application engine 210 and/or the image reprojection engine 215 may use such sensors during the acquisition of image data and/or depth data and/or in the context of image stabilization and/or motion compensation. Data is used to help determine the posture (eg, 3D position coordinates and/or orientation (eg, pitch, yaw, and/or roll)) of the imaging system 200 in the environment.

In some examples, imaging system 200 includes virtual content generator 207 that generates virtual content. Virtual content may include two-dimensional (2D) shapes, three-dimensional (3D) shapes, 2D objects, 3D objects, 2D models, 3D models, 2D animations, 3D animations, 2D images, 3D images, textures, portions of other images , characters, strings, or combinations thereof. In some examples, imaging system 200 may combine virtual content generated by virtual content generator 207 with sensor data from sensor 205 to form media data 285 . In some examples, imaging system 200 may combine virtual content generated by virtual content generator 207 with media material 285 . In FIG. 2, the virtual content generated by the virtual content generator 207 is shown as a tetrahedron. In some examples, virtual content generator 207 includes running on one or more processors of imaging system 200 , such as processor 4110 of computing system 4100 , image processor 150 , host processor 152 , ISP 154 , or a combination thereof. One or more software elements, such as one or more sets of instructions corresponding to one or more programs. In some examples, virtual content generator 207 includes one or more hardware components. For example, virtual content generator 207 may include a processor such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or a combination thereof. In some examples, virtual content generator 207 includes a combination of one or more software elements and one or more hardware elements.

Imaging system 200 includes a set of application engines 210 . Application engine 210 receives media data 285 from sensor 205 . Media data 285 is captured by sensor 205. Media material 285 may include image material, such as one or more images or portions thereof. Image data may include video data, such as video frames including video. Media data 285 may include depth data, including, for example, depth images, point clouds, 3D models, ranges between a depth sensor and portions of the environment, depth between a depth sensor and portions of the environment, and/or a sense of depth. distance between the detector and parts of the environment, or a combination thereof. Media data 285 may include audio data, such as audio recorded by one or more microphones of sensor 205 . In some cases, the audio data may include an audio track corresponding to the video of the image data. In some cases, the audio data may be multi-channel audio from multiple microphones of sensor 205, such as allowing separate tracks corresponding to the audio to reach sensor 205 from different directions in the environment. Media data 285 may include attitude data, including, for example, the position of imaging system 200 in the environment (e.g., latitude, longitude, and/or altitude), the orientation of imaging system 200 (e.g., pitch, yaw, and/or roll), imaging The speed of movement of system 200, the acceleration of imaging system 200, the velocity of imaging system 200, the momentum of imaging system 200, the rotation of imaging system 200, or a combination thereof. In some examples, attitude data may be acquired using a positioning receiver, an inertial measurement unit (IMU), an accelerometer, and/or a gyroscope of the imaging system 200 . In some examples, imaging system 200 may infer aspects of pose data, and/or may improve pose data based on pose decisions based on other types of media data 285 , such as image data, depth data, and/or audio data. .

Application engine 210 includes an image reprojection engine 215 having a motion vector engine 220 and a mesh inversion engine 225. Motion vector engine 220 of image reprojection engine 215 may determine and/or generate a first set of motion vectors corresponding to movement from a first view of the environment to a second view of the environment. In some examples, motion vector engine 220 may identify or generate images based on depth data captured by a depth sensor of sensor 205 and/or image data captured by an image sensor of sensor 205 . 3D representation of the environment. Motion vector engine 220 may rotate, translate, and/or transform the 3D representation of the environment from a first perspective representing the environment to a second perspective representing the environment. Motion vector engine 220 may determine the first set of motion vectors based on this change of perspective from the first perspective to the second perspective.

The motion vectors output by the motion vector engine 220 of the image reprojection engine 215 may be output to the grid inversion engine 225 . Grid inversion engine 225 of image reprojection engine 215 may perform grid inversion on the motion vectors to generate a second set of motion vectors. Image reprojection engine 215 may modify at least a subset of media material 285 using the second set of motions to produce modified media material 290 . For example, image reprojection engine 215 may receive an image of media material 285 from a third perspective graphical environment and may apply a second set of motion vectors to the image to produce a modified version of media material 290 . images. The modified image can illustrate the environment from a fourth perspective. The change from the third perspective to the fourth perspective may match the change from the first perspective to the second perspective, such as applying the same amount, distance, and/or angle of rotation, translation, and/or transformation. For example, in some examples, the change from the first perspective to the second perspective includes a perspective rotation according to the angle, and the change from the third perspective to the fourth perspective includes a perspective rotation according to the angle. In some examples, the change from the first perspective to the second perspective includes a perspective translation according to the direction and the distance, and the change from the third perspective to the fourth perspective includes the perspective translation according to the direction and the distance. . In some examples, the change from the first perspective to the second perspective includes a transformation, and the change from the third perspective to the fourth perspective includes a perspective translation according to the transformation.

In some examples, image reprojection engine 215 is included on one or more processors of imaging system 200 (such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or combinations thereof) One or more software elements that run, such as one or more sets of instructions corresponding to one or more programs. In some examples, image reprojection engine 215 includes one or more hardware components. For example, image reprojection engine 215 may include a processor such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or a combination thereof. In some examples, image reprojection engine 215 includes a combination of one or more software elements and one or more hardware elements.

In some examples, image reprojection engine 215 includes an ML system and/or a trained ML model that receives media material 285 as input from sensor 205 and/or virtual content generator 207 . The ML system and/or the trained ML model outputs modified media material 290 based on the media material 285 and/or the virtual content. In some cases, the ML system and/or trained ML model may modify media material 285 and/or virtual content such that modified media material 290 includes illustrations and/or representations of the environment from media material 285 Illustrations and/or representations of the environment from different perspectives. In some examples, the ML system and/or the trained ML model of image reprojection engine 215 may include one or more neural networks (NN) (eg, neural network 3900 ), one or more convolutional neural networks network (CNN), one or more trained time delay neural networks (TDNN), one or more deep networks, one or more autoencoders, one or more deep belief networks (DBN), a or multiple Recurrent Neural Networks (RNN), one or more Generative Adversarial Networks (GAN), one or more other types of neural networks, one or more trained Support Vector Machines (SVM), One or more trained random forests (RF), one or more computer vision systems, one or more deep learning systems, or a combination thereof.

Application engine 210 includes multiple engines that apply image reprojection of image reprojection engine 215 (eg, including motion vector engine 220 and/or mesh inversion engine 225) in various ways for various applications. These engines of application engine 210 include time warp engine 230, depth sensor support engine 235, 3D stabilization engine 240, 3D zoom engine 245, reprojection SAT engine 250, head pose correction engine 255, extended reality (XR) Post-production reprojection engine 260 and special effects engine 265. "SAT" in reprojection SAT engine 250 may represent sensor alignment, spatial alignment transformation, or both. Reprojection SAT engine 250 may use sensor alignment, spatial alignment transformation, or both. The engines of application engine 210 modify at least a subset of media data 285 to produce modified media data 290 , such as using image reprojection engine 215 (eg, including motion vector engine 220 and/or mesh inversion engine 225 ) to do so by reprojecting the image.

In some examples, at least one of the application engines 210 includes an ML system and/or a trained ML model that receives media data 285 as input from the sensor 205 and/or the virtual content generator 207 . The ML system and/or the trained ML model outputs modified media material 290 based on the media material 285 and/or the virtual content. In some cases, the ML system and/or trained ML model may modify media material 285 and/or virtual content such that modified media material 290 includes illustrations and/or representations of the environment from media material 285 Illustrations and/or representations of the environment from different perspectives. In some examples, the ML system and/or the trained ML model of at least one of the application engines 210 may include one or more NNs, one or more CNNs, one or more TDNNs, one or more deep network, one or more autoencoders, one or more DBNs, one or more RNNs, one or more GANs, one or more trained SVMs, one or more trained RFs, one or more A computer vision system, one or more deep learning systems, or a combination thereof.

In some examples, application engine 210 , including image reprojection engine 215 , may analyze (eg, to determine motion vectors), process, and/or modify media material 285 into which virtual content generated by virtual content generator 207 is incorporated. Media information 285. In some examples, application engine 210 including image reprojection engine 215 may analyze (eg, to determine motion vectors), process, and/or modify media material 285 in which virtual content generated by virtual content generator 207 is not Incorporated into media material 285. In some examples, such as if virtual content generated by virtual content generator 207 is incorporated into media material 285 (which is input into application engine 210), then by the application engine including image reprojection engine 215 The modified media material 290 output by 210 may already include the virtual content. In some examples, such as if the virtual content generated by the virtual content generator 207 is not incorporated into the media data 285 (the media data is input into the application engine 210), then by the application including the image reprojection engine 215 The modified media material 290 output by the engine 210 may lack this virtual content. In such examples, the virtual content generated by virtual content generator 207 may be generated after application engine 210 outputs modified media data 290 but before outputting modified media data 290 using output device 270 and/or transceiver 275 . Content added to modified media material 290.

In some examples, at least one of application engines 210 includes one or more processors in imaging system 200 (such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or other One or more software elements that run on a combination), such as one or more sets of instructions corresponding to one or more programs. In some examples, at least one of application engines 210 includes one or more hardware elements. For example, at least one of application engines 210 may include a processor such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or combinations thereof. In some examples, at least one of application engines 210 includes a combination of one or more software elements and one or more hardware elements.

In some examples, imaging system 200 includes one or more output devices 270 that are configured to and can output modified media material 290 . In some examples, output device 270 includes a display that is configured to and can display visual media, such as images and/or video. In some examples, output device 270 includes an audio output device, such as a speaker or a headset, or a connector configured to couple imaging system 200 to a speaker or headset. The audio output device is configured to and can play audio media such as music, sound effects, a soundtrack corresponding to video, audio recordings recorded by a microphone (eg, of sensor 205), or a combination thereof. The output device 270 may output an output including an environment representation (eg, media material 285 captured by the sensor 205 ), virtual content (eg, generated by the virtual content generator 207 ), a combination of the environment representation and the virtual content, a combination of the environment representation and the virtual content. or modification of virtual content and/or combination (eg, modification using application engine 210 and/or image reprojection engine 215), or combination thereof. In some examples, output device 270 may be directed to a user of imaging system 200 . For example, a display of output device 270 may face a user of imaging system 200 , and/or visual media may be displayed toward (eg, toward) a user of imaging system 200 . Similarly, the audio output device of output device 270 may be oriented toward a user of imaging system 200 and/or may play audio media toward (eg, toward) a user of imaging system 200 . In some examples, output device 270 includes output device 4135. In some examples, output device 4135 may include output device 270. In Figure 2, output device 270 is shown as a display that displays visual media material and corresponding speakers that play audio media material.

Imaging system 200 also includes one or more transceivers 275 that imaging system 200 may use (e.g., by sending media to a recipient device) to output data generated by application engine 210 (e.g., including images). Modified media material 290 generated by the reprojection engine 215). The recipient device may output the media using its own output device, such as by displaying the media's visual media data using a display of the output device and/or by playing the audio media data of the media using an audio output device of the output device. Transceiver 275 may include a wired or wireless transceiver, communication interface, antenna, connector, coupler, coupling system, or combinations thereof. In some examples, transceiver 275 may include communication interface 4140 of computing system 4100 . In some examples, communication interface 4140 of computing system 4100 may include transceiver 275 . In Figure 2, transceiver 275 is shown as a wireless transceiver 275 transmitting media material.

In some examples, imaging system 200 includes feedback engine 280 . Feedback engine 280 may detect feedback received from a user via the user interface of the imaging system. Feedback engine 280 may detect feedback regarding one engine of imaging system 200 received from another engine of imaging system 200, such as whether one engine decides to use data from another engine. The feedback may be about any of the application engines 210 (such as image reprojection engine 215, motion vector engine 220, mesh inversion engine 225, time warp engine 230, depth sensor support engine 235, 3D stabilization transformation engine 240, 3D zoom engine 245, re-projection SAT engine 250, head posture correction engine 255, XR post-production re-projection engine 260, special effects engine 265 or a combination thereof). The feedback received by the feedback engine 280 may be a positive feedback or a negative feedback. For example, if one engine of imaging system 200 uses data from another engine of imaging system 200, feedback engine 280 may interpret this as a positive feedback. If one engine of imaging system 200 rejects data from another engine of imaging system 200, feedback engine 280 may interpret this situation as a negative feedback. Positive feedback may also be based on sensor data from sensor 205 and/or attributes of input from the user interface, such as the user smiling, laughing, nodding, pressing a button associated with positive feedback, making a Positive feedback associated with gestures (e.g., thumbs up), uttering affirmative statements (e.g., "yes," "confirm," "ok," "next"), or otherwise affirming the media reaction. Negative feedback may also be based on sensor data from sensor 205 and/or attributes of input from the user interface, such as the user frowning, crying, shaking his head (e.g., for a "no" action), pressing, and negative feedback. associated buttons, making gestures associated with negative feedback (e.g., thumbs down), saying negative statements (e.g., "no," "no," "no," "not this"), or Otherwise negative reactions to virtual content.

In some examples, feedback engine 280 provides feedback as training material to one or more ML systems of imaging system 200 to update one or more ML systems of imaging system 200 . For example, feedback engine 280 may provide feedback as training data to a trained ML model of any of the ML systems and/or application engines 210, such as image reprojection engine 215, motion vector engine 220, mesh reflection engine 215, etc. Performance engine 225, time warp engine 230, depth sensor support engine 235, 3D stabilization engine 240, 3D zoom engine 245, re-projection SAT engine 250, head posture correction engine 255, XR post-production re-projection engine 260, special effects engine 265 or combination thereof. Positive feedback can be used to strengthen and/or enhance the weights associated with the output of the ML system and/or the trained ML model. Negative feedback can be used to weaken and/or remove weights associated with the output of the ML system and/or the trained ML model.

In some examples, feedback engine 280 includes software elements, such as a program corresponding to a program, running on a processor such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or combinations thereof. Group instructions. In some examples, feedback engine 280 includes one or more hardware components. For example, feedback engine 280 may include a processor such as processor 4110 of computing system 4100, image processor 150, host processor 152, ISP 154, or combinations thereof. In some examples, feedback engine 280 includes a combination of one or more software elements and one or more hardware elements.

3A is a perspective view 300 showing a head mounted display (HMD) 310 used as an extended reality (XR) system 200. HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof . HMD 310 may be an example of imaging system 200 . The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310 . The first camera 330A and the second camera 330B may be examples of the sensor 205 of the imaging system 200 . When the user's eyes face the display 340, the HMD 310 includes a third camera 330C and a fourth camera 330D facing the user's eyes. The third camera 330C and the fourth camera 330D may be examples of the sensor 205 of the imaging system 200 . In some examples, HMD 310 may only have a single camera with a single image sensor. In some examples, HMD 310 may include one or more additional cameras in addition to first camera 330A, second camera 330B, third camera 330C, and fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, which may also include other types of sensing. 205 and/or the sensor 205 of the imaging system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C and/or the fourth camera 330D may be the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or the like. Example of combination.

HMD 310 may include one or more displays 340 that are visible to user 320 who wears HMD 310 on the user's 320 head. One or more displays 340 of HMD 310 may be an example of one or more displays of output device 270 of imaging system 200 . In some examples, HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320 . The left viewfinder may be oriented so that the left eye of user 320 sees the left side of the display. The right viewfinder may be oriented so that the left eye of user 320 looks to the right side of the display. In some examples, HMD 310 may include two displays 340 including a left display that displays content to the left eye of user 320 and a right display that displays content to the right eye of user 320 . One or more displays 340 of HMD 310 may be digital "pass-through" displays or optical "see-through" displays.

HMD 310 may include one or more earbuds 335 , which may function as speakers and/or headphones that output audio to one or more ears of the user of HMD 310 . One earbud 335 is illustrated in Figures 3A and 3B, but it should be understood that the HMD 310 may include two earbuds, one for each ear (left and right) of the user. In some examples, HMD 310 may also include one or more microphones (not shown). One or more microphones may be examples of sensors 205 of imaging system 200 . One or more earbuds may be examples of output devices 270 of imaging system 200 . In some examples, audio output by HMD 310 to the user via one or more earbuds 335 may include or be based on audio recorded using one or more microphones.

FIG. 3B is a perspective view 350 showing the head mounted display (HMD) of FIG. 3A worn by user 320 . The user 320 wears the HMD 310 on the head of the user 320 above the eyes of the user 320 . The HMD 310 may capture images using the first camera 330A and the second camera 330B. In some examples, HMD 310 uses display 340 to display one or more output images to the eyes of user 320 . In some examples, the output image may include virtual content generated by virtual content generator 207, synthesized using a compositor, and/or displayed by the display of output device 270. The output image may be based on images captured by the first camera 330A and the second camera 330B, for example with superimposed virtual content. The output image may provide a stereoscopic view of the environment, in some cases with overlaid virtual content and/or other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on the image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on the image captured by the second camera 330B. For example, the HMD 310 may provide superimposed virtual content in the display image over the images captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the user's eyes before, during, and/or after viewing the display image displayed by the display 340 . In this way, the sensor data from the third camera 330C and/or the fourth camera 330D can capture the reaction of the user's eyes (and/or other parts of the user) to the virtual content. Earbud 335 of HMD 310 is shown in the ear of user 320 . HMD 310 may output audio to user 320 via earbud 335 and/or via another earbud (not shown) of HMD 310 in the other ear (not shown) of user 320 .

4A is a perspective view 400 showing the front surface of a mobile handheld terminal 410 that includes a front-facing camera and may be used as an extended reality (XR) system 200. Mobile handheld terminal 410 may be an example of imaging system 200 . Mobile handheld terminal 410 may be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop computer, a mobile device, or any other device discussed herein. type of computing device or computing system, or a combination thereof.

Front surface 420 of mobile handheld terminal 410 includes display 440 . The front surface 420 of the mobile handheld terminal 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B may be examples of the sensor 205 of the imaging system 200 . The first camera 430A and the second camera 430B may face the user, including the user's eyes, while content (eg, modified media output by the media modification engine 235) is displayed on the display 440. Display 440 may be an example of a display of output device 270 of imaging system 200 .

The first camera 430A and the second camera 430B are shown in a bezel surrounding the display 440 on the front surface 420 of the mobile handheld terminal 410 . In some examples, the first camera 430A and the second camera 430B may be positioned in notches or cuts cut out from the display 440 on the front surface 420 of the mobile handheld terminal 410 . In some examples, the first camera 430A and the second camera 430B may be under-screen cameras positioned between the display 440 and the remainder of the mobile handheld terminal 410 such that light passes through before reaching the first camera 430A and the second camera 430B. part of display 440. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face a direction perpendicular to the plane of the front surface 420 of the mobile handheld terminal 410 . The first camera 430A and the second camera 430B may be two cameras of one or more cameras of the mobile handheld terminal 410 . The first camera 430A and the second camera 430B may be a first image sensor and a second image sensor, respectively. In some examples, the front surface 420 of the mobile handheld terminal 410 may have only a single camera.

In some examples, in addition to the first camera 430A and the second camera 430B, the front surface 420 of the mobile handheld terminal 410 may include one or more additional cameras. One or more additional cameras may also be examples of sensors 205 of imaging system 200 . In some examples, in addition to the first camera 430A and the second camera 430B, the front surface 420 of the mobile handheld terminal 410 may include one or more additional sensors. One or more additional sensors may also be examples of sensor 205 of imaging system 200 . In some cases, front surface 420 of mobile handheld terminal 410 includes more than one display 440. One or more displays 440 of the front surface 420 of the mobile handheld terminal 410 may be examples of displays of the output device 270 of the imaging system 200 . For example, one or more displays 440 may include one or more touch screen displays.

Mobile handheld terminal 410 may include one or more speakers 435A and/or other audio output devices (e.g., earphones or headphones or connectors thereof) that may output audio to a user of mobile handheld terminal 410 One or more ears. One speaker 435A is illustrated in Figure 4A, but it should be understood that the mobile handheld terminal 410 may include more than one speaker and/or other audio devices. In some examples, mobile handheld terminal 410 may also include one or more microphones (not shown). The one or more microphones may be the sensor 205 of the imaging system 200 and/or an example of the one or more sensors 205 . In some examples, mobile handheld terminal 410 may include one or more microphones along and/or adjacent front surface 420 of mobile handheld terminal 410 , which microphones are examples of sensors 205 of imaging system 200 . In some examples, audio output by mobile handset 410 to the user via one or more speakers 435A and/or other audio output devices may include or be based on audio recorded using one or more microphones.

FIG. 4B is a perspective view 450 illustrating a rear surface 460 of a mobile handheld terminal that includes a rear camera and may be used as an extended reality (XR) system 200 . The mobile handheld terminal 410 includes third and fourth cameras 430C and 430D on the rear surface 460 of the mobile handheld terminal 410 . The third camera 430C and the fourth camera 430D of the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D may be examples of the sensor 205 of the imaging system 200 of FIG. 2 . The third camera 430C and the fourth camera 430D face a direction perpendicular to the plane of the rear surface 460 of the mobile handheld terminal 410 .

The third camera 430C and the fourth camera 430D may be two cameras of one or more cameras of the mobile handheld terminal 410 . In some examples, the rear surface 460 of the mobile handheld terminal 410 may have only a single camera. In some examples, in addition to the third camera 430C and the fourth camera 430D, the rear surface 460 of the mobile handheld terminal 410 may include one or more additional cameras. One or more additional cameras may also be examples of sensors 205 of imaging system 200 . In some examples, in addition to the third camera 430C and the fourth camera 430D, the rear surface 460 of the mobile handheld terminal 410 may include one or more additional sensors. One or more additional sensors may also be examples of sensor 205 of imaging system 200 . In some examples, the first camera 430A, the second camera 430B, the third camera 430C and/or the fourth camera 430D may be the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or the like. Example of combination.

Mobile handheld terminal 410 may include one or more speakers 435B and/or other audio output devices (e.g., earphones or headphones or connectors thereof) that may output audio to a user of mobile handheld terminal 410 One or more ears. One or more speakers 435B may be an example of output device 270 of imaging system 200 . One speaker 435B is illustrated in Figure 4B, but it should be understood that the mobile handheld terminal 410 may include more than one speaker and/or other audio devices. In some examples, mobile handheld terminal 410 may also include one or more microphones (not shown). The one or more microphones may be the sensor 205 of the imaging system 200 and/or an example of the one or more sensors 205 . In some examples, mobile handheld terminal 410 may include one or more microphones along and/or adjacent rear surface 460 of mobile handheld terminal 410 , which microphones are examples of sensors 205 of imaging system 200 . In some examples, audio output by mobile handset 410 to the user via one or more speakers 435B and/or other audio output devices may include or be based on audio recorded using one or more microphones.

Mobile handheld terminal 410 may use display 440 on front surface 420 as a pass-through display. For example, display 440 may display an output image. The output image may be based on an image captured by the third camera 430C and/or the fourth camera 430D, for example, with overlaid virtual content and/or modifications applied by the media modification engine 235 . The first camera 430A and/or the second camera 430B may capture images of the user's eyes (and/or other parts of the user) before, during, and/or after displaying the output image with virtual content on the display 440 . In this way, the sensor data from the first camera 430A and/or the second camera 430B can capture the reaction of the user's eyes (and/or other parts of the user) to the virtual content.

FIG. 5 is a conceptual diagram showing an example of grid inversion. The input to the grid inversion includes a first set of motion vectors, represented as a motion vector (MV) grid using the solid black arrows from the first image Img1 510 to the second image Img2 515 in Figure 5 . For each primitive (or group of primitives), the motion vector grid uses the motion vector in the motion vector (MV) grid 505 to indicate that the primitive (or group of primitives) will be in the first image of the environment Img1 510 ( The distance moved between, for example, vision or depth) and the second image Img2 515 of the environment (e.g., vision or depth). The motion vector grid 505 may be referred to as the motion vector map of the image. The motion vectors of motion vector grid 505 may be determined using motion vector engine 220 (eg, using optical flow).

The grid inversion engine 225 may perform a grid inversion that changes the characteristics (eg, direction, origin, position, length, and/or size) of the motion vectors in the first set of motion vectors (the motion vector grid 505 ) to A second set of motion vectors is generated (inversion MV grid 520). The motion vectors in the second set of motion vectors (inversion MV grid 520 ) show how each primitive from Img2 515 can move back to graph 1510 , rather than how each primitive from Img1 510 moves to Img2 515 (as in MV grid 505). The motion vectors in the second set of motion vectors (inversion MV grid 520) are shown in Figure 5 using the black dashed arrow from the second image Img2 515 to the first image Img1 510.

The various black illustrations in Figure 5 represent various elements in the environment illustrated by the two images Img1 510 and Img2 515. For example, elements include houses, birds, people, cars, and trees. According to MV grid 505, the house and tree did not move from Img1 510 to Img2 515, indicated by the zeros in MV grid 505. Likewise, in the inversion MV grid 520, the house and tree do not move from Img2 515 to Img1 510. The house is represented by zeros in the MV grid 505 and the inverted MV grid 520, which are located in cell 0 where the house is located. The tree can be represented by zeros in the inversion MV grid at cell 8 where the tree resides, but as discussed below conflicts with the car, represented by a black circle. The bird moves 1 grid cell to the right from Img1 510 to Img2 515 (moving from cell 1 to cell 2), represented by the 1 at cell 1 in MV grid 505. The bird moves 1 grid cell left from Img2 515 to Img1 510 (moving from cell 2 to cell 1), represented by the -1 at cell 2 in the inverted MV grid 520. Not only are the values inverted (multiplied by -1) from the MV grid 505 to the inverted MV grid 520, but they are also moved from the cell corresponding to the element's old position in Img1 510 to the element's new position in Img2 515 The unit corresponding to the location. The black star in bin 1 (where the bird is in Img1 510 but is missing in Img2 515) indicates that in the inversion MV grid 520, the area of the image corresponding to bin 1 is missing and may need to be filled ( For example, using interpolation and/or repair filling). The person moves 2 grid cells left from Img1 510 to Img2 515 (moving from cell 6 to cell 4), represented by the -2 at cell 6 in MV grid 505. The person moves 2 grid cells to the right from Img2 515 to Img1 510 (moving from cell 4 to cell 6), represented by the 2 at cell 4 in the inverted MV grid 520. The black star in cell 6 (where the person is in Img1 510 but is missing in Img2 515) indicates that in the inversion MV grid 520, the area of the image corresponding to cell 1 is missing and may need to be filled ( For example, using interpolation and/or repair filling). The car moves 1 grid cell to the right from Img1 510 to Img2 515 (from cell 7 to cell 8), represented by the 1 in MV grid 505. The car will move 1 grid cell left from Img2 515 to Img1 510 (moving from cell 8 to cell 7), which can be represented by a -1 in the inverted MV grid 520. However, the car and the tree are in the same grid cell (cell 8) in Img2 515, so the red circle indicates the conflicting value of that cell in the inversion MV grid 520 (e.g. 0 for tree, -1 for car ).

Figure 6 is a conceptual diagram 600 illustrating an example of depth-based reprojection. Depth-based reprojection is performed by image reprojection engine 215. The example shows a camera image 610 of an environment (called world scene 605) that has a table with a toolbox on the table and some chairs around the table. Image reprojection engine 215 reprojects camera image 610 using depth information 620 of the environment (eg, world scene 605 ) to produce reprojected image 615 . Reprojected image 615 illustrates the same environment (eg, world scene 605 ) as camera image 610 , but is reprojected as if the environment were from a different perspective than camera image 610 in reprojected image 615 or Viewpoint captured. In the example shown in FIG. 6 , reprojected image 615 appears to be captured from a perspective or viewpoint of the environment that is translated to that of the environment illustrated in camera image 610 left side. In some examples, image reprojection engine 215 may perform image reprojection using an inverted MV grid (eg, inverted MV grid 520 ) generated by grid inversion engine 225 , such as based on depth profile 620 .

FIG. 7 is a conceptual diagram 700 illustrating an example of time warping 705 performed by time warping engine 230. On the left, a large or dense motion vector map 720 is shown as a solid black arrow, illustrating how primitives move between image frame n and image frame n-4. Image frames n and n-4 are shown as tall vertical lines. Time warp 705 uses grid inversion (using grid inversion engine 225) on a large or dense motion vector map 720 to build a smaller motion vector map, shown e.g. from image frame n to image Frame n-1, from image frame n-1 to image frame n-2, from image frame n-2 to image frame n-3 and from image frame n-3 to image Shorter vertical arrow like frame n-4.

To create smaller vector maps, the time warp engine 230 uses resampling. For example, to generate a reduced vector map, the time warp engine 230 multiplies a value in the motion vector map (representing the distance an element moves between frame n and frame n-4) by ¼, for example. smaller. Additionally, similar to the value movement in the grid inversion of Figure 5, the time warp engine 230 moves the values to each element's new position in the corresponding plot frame.

For example, if optical flow is only performed every k frames, time warping 705 can be used to interpolate motion vector maps between existing motion vector maps. Optical flow is a computationally expensive operation that can use a lot of power to perform, whereas the time warp 705 demonstrated here is a less expensive and lower power operation. Thus, optical flow can be used sparingly to reduce computational cost and power usage, and time warping 705 can still allow imaging system 200 to obtain information for use between any two adjacent frames (and in some cases, any two frames). motion vector for each frame transition between).

In some examples, the smaller motion vector maps produced by time warping 705 can be used to interpolate additional frames between existing frames of the video, such as increasing the frame rate of the video from a first frame rate to a higher The second frame rate of the first frame rate.

In some examples, the smaller motion vector maps generated by time warping 705 can be used to improve the quality of certain frames of the video. For example, if a particular video frame is blurry, includes extensive compression artifacts, including compression artifacts that make it difficult to clearly see the scene illustrated in the image, or otherwise suffers from low quality, time warping 705 can improve The quality of this video frame. Time warping 705 can be used to determine a motion vector map from one or more adjacent or nearby frames of the video, and the image data from those frames can be used to generate a modified image to replace the specific frame in question. frame to improve the image quality of the specific frame in question. Conceptual diagram 700 illustrates two examples of images of a boy - a first image 710 on the left with no time warp 705 applied, and a second image 715 on the right with time warp 705 applied, thus comparing to the first image 710 The clarity of the illustration of the boy in the second image 715 has been improved. The right image 715 improved using time warp 705 appears sharper and clearer than the left image 710, especially at and near various edges in the illustration of the boy, as shown by the use of solid lines As indicated by the various lines and edges representing the illustration of the boy in image 715 . Additionally, in some examples, patterns such as hair patterns, fabric patterns, another pattern, text, logos, and/or other designs may appear in the image to which time warp 705 is applied (eg, as in image 715 on the right). Images without time warp 705 (eg, image 710 on the left) appear sharper than in images without time warp 705 applied.

Additional examples of time warping 705 and image improvements using time warping 705 are illustrated in FIGS. 23 and 29 .

8 is a conceptual diagram 800 illustrating an example of depth sensor support 805 performed by the depth sensor support engine 235. The cluster of sensors 205 on the imaging system 200 is shown to include a set of image sensors 810 and a set of depth sensors 815, which may include time-of-flight (ToF) sensors. In some cases, it may be useful to use image data from image sensor 810 and depth data from depth sensor 815 together in image processing, for example to generate bokeh, simulate depth blur, object recognition wait. However, the image sensor 810 and the depth sensor 815 are not co-located. Instead, image sensor 810 and depth sensor 815 are offset from each other by offset 820. Therefore, using image data from image sensor 810 and depth data from depth sensor 815 may create parallax issues due to a slight mismatch in viewing angles caused by offset 820 . Therefore, the depth in the depth data may not match the object depicted in the image data. This mismatch can be especially pronounced for objects in the environment that are close to the sensor, which can appear in significantly different locations in the image data than in the depth data. Objects that are further away may appear more similar in image data and depth data.

To correct for this mismatch, in some examples, image reprojection engine 215 may reproject depth data from depth sensor 815 to represent the perspective from image sensor 810 . In some examples, image reprojection engine 215 may reproject image data from image sensor 810 to represent the perspective from depth sensor 815 . Because image reprojection engine 215 may require depth information to perform reprojection, image reprojection engine 215 may rely on external calibration between image sensor 810 and depth sensor 815 to obtain appropriate depth information.

FIG. 9 is a conceptual diagram 900 illustrating an example of 3D stabilization 905 performed by the 3D stabilization engine 240. Traditional stabilization techniques can compensate for rotational motion, but generally cannot compensate for real-world translational (e.g., parallax) motion. Image reprojection of environment-based depth data using an image reprojection engine 215 can provide true 3D stabilization 905 that corrects for parallax movement, including translational movement, rotational movement, or both. For each video frame of the video captured using the sensor 205 , including the four video frames labeled original (“orig”) in FIG. 9 , reprojection is performed using the image reprojection engine 215 to generate the original video. A stable variant of the frame ("stable"). The resulting reprojected video frames have their corresponding viewing angles all adapted to a line representing a virtual stable movement path, without any parallax movement perpendicular to the line or around the axis corresponding to the line (or any other axis ) any rotation. The line can be curved to represent a curved movement path, but does not have any jagged edges corresponding to such parallax movement or rotation.

For the illustrated 3D stabilization 905, the input video represented by the video frame swings in different directions - translation up, translation down, translation left, translation right, translation forward, translation back and/or rotation ( For example, pitch, yaw, and/or roll). Because the image reprojection engine 215 reprojects the image to change the perspective of the environment, all of these movements in the swing are stabilized by reprojection using the image reprojection engine 215.

In some cases, white space may appear in the stable frame, such as at the edges of the frame and/or around the person in the frame (e.g., to the right of the woman in the fourth stable frame in the lower right corner of Figure 9 ). These can represent occluded areas that have no corresponding material in the original image. These occluded areas may be filled by the image reprojection engine 215, such as using interpolation and/or inpainting (eg, deep learning based inpainting). An additional example 3205 of 3D stabilization 905 is shown in Figure 30. In some examples, these empty areas may appear black. In some examples, these empty areas may appear white. In Figure 9, these blank areas are shown in white.

In some examples, for 3D stabilization, and for certain other applications of image reprojection engine 215, distant primitives are treated as if they were at an infinite distance such that the location of such primitives is under reprojection. Immutability may be useful. In some examples, image reprojection engine 215 may use translation attenuation to smoothly transition translation values to values representing infinity to treat distant primitives as if they were at infinite distance.

FIG. 10 is a conceptual diagram 1000 illustrating an example of 3D zoom 1005 (also referred to as movie zoom) performed by the 3D zoom engine 245. 3D zoom 1005 performed by 3D zoom engine 245 may include enlarging the image (e.g., making certain portions of the image larger while removing other portions of the image), moving the virtual camera in different directions (e.g., panning, rotating etc.) and/or other types of zoom. In some cases, to perform digital zoom on an image, the entire image is traditionally enlarged and cropped, as shown in the sequence of four images labeled Digital Zoom ("dig.zm.") in Figure 10. These images illustrate a skateboarder in front of a house. Performing a digital zoom (or in some examples even an optical zoom using an optical zoom lens or switching between cameras and/or lenses) loses a large portion of the view of the house. However, if you move the camera closer to the skateboarder, the view of the house is not lost as much as it would be if you used digital zoom. This is because the skateboarder is closer to the camera than the house. In other words, the skateboarder is in the foreground and the house is in the background.

3D zoom 1005 or depth-based zoom or movie zoom uses image reprojection based on depth information 1020 for the environment using an image reprojection engine 215 to simulate forward movement of the camera in the environment, in this case more Move closer to the skateboarder. As shown in the sequence of four images labeled Depth-based Zoom ("depth.zm.") in Figure 10, the skateboarder's size increases as much as digital zoom, but the house's depth of field is less lost. For example, in the last of the four images in the sequence, the span of the house's four windows is at least partially within the frame under digital zoom, while the span of the house's six windows is in the frame based on 3D depth. The zoom down is at least partially in the frame (although one of those windows is entirely behind the skateboarder). Therefore, 3D depth-based zoom (or movie zoom) minimizes the loss of field of view, especially the loss of background elements. An additional example of 3D zoom 1005 (or depth-based zoom or movie zoom) is shown in Figure 31.

FIG. 11 is a conceptual diagram 1100 illustrating an example of reprojection 1105 performed by the reprojection SAT engine 250. The cluster of sensors 205 of imaging system 200 is shown in Figure 11, with telephoto sensor 1110, wide angle sensor 1115, and another sensor 1125. In some cases, imaging system 200 may switch between telephoto sensor 1110 and wide-angle sensor 1115, for example, to provide different levels of zoom in images of the environment. However, similar to the image sensor 810 and depth sensor 815 scene in FIG. 8 , the telephoto sensor 1110 and the wide-angle sensor 1115 are not co-located. In contrast, there is an offset 1120 between the telephoto sensor 1110 and the wide-angle sensor 1115 . Therefore, switching between the telephoto sensor 1110 and the wide-angle sensor 1115 creates a parallax effect. For example, the illustration shows a telephoto image 1130 (labeled "tele") captured using the telephoto sensor 1110, while the illustration is captured using the wide-angle sensor 1115 and cropped to match the telephoto field of view (i.e., Digitally zooming before transitioning to the telephoto sensor) wide-angle image 1135 (labeled "wide"). Both images illustrate a man in front of a distant background. In the telephoto image 1130 , the man appears slightly to the right of the position of the man in the wide-angle image 1135 .

Similar to the depth sensor support 805 of FIG. 8 , the reprojection SAT engine 250 can perform reprojection 1105 to correct the offset 1120 based on the depth data 1160 . For example, the reprojection SAT engine 250 may perform reprojection 1105 to modify the telephoto image to modify the viewing angle such that the modified telephoto image 1140 (labeled "modif.tele") appears as the viewing angle from the wide-angle sensor 1115 captured (eg, as in wide-angle image 1135) rather than from the perspective of telephoto sensor 1110 (eg, as in telephoto image 1130). In the modified telephoto image 1140, the man appears slightly to the left of the man's position in the unmodified telephoto image 1130. In the modified telephoto image 1140, the man appears similarly positioned as the man in the wide-angle image 1135. A black shadow appears to the right of the man in the modified telephoto image 1140, which is caused by the parallax movement of the image material of the illustrated man relative to the background. Black shading represents "holes" that can be filled with image data (eg using interpolation and/or inpainting as discussed further).

In some examples, the reprojection SAT engine 250 may instead perform reprojection 1105 based on the depth information 1160 to modify the wide-angle image to modify the viewing angle such that the modified wide-angle image (not shown) appears to be from telephoto sensing It is captured from the viewing angle of the wide-angle sensor 1110, rather than from the viewing angle of the wide-angle sensor 1115. Unlike sensor-to-sensor transformation, where a set of digitally zoomed images from one sensor is warped based on image estimates to match the second sensor before switching, the reprojection SAT engine 250 can be depth-based. The data corrects the offset, thereby reducing parallax problems (e.g., parallax errors), especially for closer objects (e.g., objects in the foreground and/or objects with depths less than a threshold). An additional example of reprojection 1105 is shown in Figure 32.

FIG. 12 is a conceptual diagram 1200 illustrating an example of head posture correction 1205 performed by the head posture correction engine 255. In some cases, images of the user may be captured from suboptimal and/or unflattering angles (eg, angles other than vertical to the user's face). For example, when a user captures a selfie image of himself or points the camera at himself during a video conference, the angle of the captured image is usually not aligned with the user's head posture, causing the user to look downward. , look up, left and/or right. In some cases, a user's hands may become tired and/or uncomfortable from holding their phone or other imaging system 200 for extended periods of time, which may be exacerbated when the user's hands are dropped or resting against a nearby surface. problem.

Head pose correction 1205 performed by head pose correction engine 255 may use image reprojection engine 215 to perform reprojection to reproject real sensors to match virtual sensor positions for more optimization and/or or a more flattering viewing angle, such as from a vertical angle perpendicular to the user's face.

For example, the original head pose of the woman in input image 1210 is captured from an unflattering angle slightly below the level of the woman's head, thereby emphasizing the woman's neck and chin area. Head pose correction 1205 uses an image reprojection engine 215 based on the input image 1210 and depth data 1220 to generate a reprojected image 1215 from a perspective perpendicular to the vertical angle of the user's face. The reprojected image 1215 appears to view the woman's face from a more flattering vertical angle, thereby emphasizing the woman's facial features rather than the woman's neck and chin in the input image 1210 . Additional examples of head pose correction 1205 are shown in Figure 33.

FIG. 13 is a conceptual diagram 1300 illustrating an example of XR post-reprojection 1305 performed by the XR post-reprojection engine 260. Some XR devices (eg, HMD 1320) or other mobile devices use their sensors 205 to capture sensor data (eg, images, videos, depth images, and/or point clouds) at a low frame rate to save Battery power. Interpolation can be used to generate additional frames between frames of low frame rate sensor data to increase frame rate. A high frame rate may be important for XR applications, as low frame rate XR may cause the user to feel nauseated and/or may cause XR to appear to be interferencey and unrealistic.

Interpolation techniques are not always able to faithfully represent all viewing angle changes of an XR device (e.g., HMD 1320). For example, interpolation may use digital zoom to simulate a user moving closer or further away from an object, which may result in a mismatch in the field of view similar to that discussed with respect to 3D zoom 1005 of FIG. 10 . Interpolation techniques may also have difficulty handling parallax movements caused, for example, by translational movement of the XR device (e.g., HMD 1320). Interpolation techniques may also have difficulty handling rotational movements caused, for example, by changes in orientation (eg, pitch, roll, and/or yaw) of the XR device (eg, HMD 1320).

XR post-reprojection 1305 performed by XR post-reprojection engine 260 may perform image reprojection using image reprojection engine 215 to reproject an image of the environment based on changes to the position of the XR device. Position changes of the XR device (e.g., HMD 1320) may be determined based on sensor data from an attitude sensor of the XR device (e.g., HMD 1320), which may use more sensors than image sensors or depth sensors. less bandwidth and/or power. Inferences may be made based on image data, depth data, and/or audio data from the image sensor, depth sensor, and/or microphone of the sensor 205 of the XR device (e.g., HMD 1320) HMD 1320) position changes.

For example, an input image 1310 is illustrated based on which the XR post-reprojection engine 260 uses the XR post-reprojection 1305 to generate a reprojected image 1315 based on the illustrated orientation changes of the HMD 1320 (which is an example of an XR device).

FIG. 14 is a conceptual diagram 1400 illustrating an example of special effects 1405 performed by special effects engine 265. The special effects 1405 performed by the special effects engine 265 may perform image reprojection using the image reprojection engine 215 to reproject the input image 1410 to rotate around the object, translate next to the object, rotate the perspective around an axis, move the perspective along a path Move, or some combination thereof. In the example shown in Figure 14, an input image 1410 of the environment is reprojected from different perspectives of the environment to form a reprojected image 1415. The perspective with respect to the environment in the reprojected image 1415 is to the left of the perspective with respect to the environment in the input image 1410 , such that the toolbox appears in the reprojected image 1415 rotated to the right relative to the input image 1410 and /or tilt.

Figure 15 is a conceptual diagram 1500 illustrating an image reprojection transformation based on matrix operations. Conceptual diagram 1500 illustrates how image reprojection engine 215 can reproject a captured image 1510 of the environment to produce a reprojected image 1515 of the environment from a different perspective than the captured image 1510 . Image reprojection engine 215 receives captured image 1510 from sensor 205, specifically from a camera. The captured image illustrates the environment from a first-person perspective ("first persp."). An example of a captured image 1510 is shown in Figure 15. For example, using the pinhole camera paradigm and focal length (f) and depth, the imaging system can determine the position of objects in the environment relative to the camera. The image reprojection engine 215 may use an intrinsic matrix illustrating a first camera (also known as an original camera, a source camera, or a first view), a illustrating second camera, or a virtual camera in the 3D world (also known as a target camera, or second perspective), and a 3D transformation matrix for movement or reprojection from the first camera to the second camera. In some examples, the image reprojection engine may also perform depth reprojection to establish a second depth map illustrating the environment from a second perspective based on the same principles as image reprojection described herein. Additionally, various transformation paradigms may be used for image and/or depth reprojection, such as those that account for lens distortion (eg, radial distortion).

The image reprojection engine 215 receives a depth map ("depth in the image domain", for example, from a depth sensor and/or based on depth determination using a camera (eg, stereo depth sensing, ToF sensor, and/or structured light) ) (e.g., depth profile 620). Based on the depth map, the image reprojection engine 215 may determine that any given object in the captured image 1510 (such as any of the chairs or tables or tool boxes illustrated in the captured image 1510) is The exact location in 3D coordinates (for example, X, Y, and Z). For example, Figure 15 identifies the coordinates for the object in the captured image 1510 based on the object's depth, the camera's intrinsic matrix (intrinsic _cam ) and A set of equations to determine the X, Y, and Z coordinates of an object. The equation is as follows:

The camera's intrinsic matrix (intrinsic _cam ) can be used to transform 3D camera coordinates into 2D image coordinates, and can be based on focal length (f _x and/or f _y ) and/or principal point offset (c _x and/or _cy ), as indicated below:

The 3D transformation can be based on intrinsic matrices at the source camera position and the target camera position corresponding to the reprojection, for example as indicated below:

Image reprojection engine 215 receives and/or determines a reprojection matrix that indicates how the perspective moves in the reprojected environment (eg, simulated movement of the camera). The values in the reprojection matrix illustrated in Figure 15 are labeled R11, R12, R13, Tx, R21, R22, R23, Ty, R31, R32, R33 and Tz. In another example, the image reprojection engine can obtain this transformation directly as a 3DTransform matrix (eg, without performing at least some of the calculations indicated above). Once the image reprojection engine 215 knows how the perspective moves in the environment in the form of a reprojection matrix, the image reprojection engine 215 can determine the position of the object in the environment after the camera moves by determining X _out , Y _out and Z _out as follows. new 3D position in (e.g., in reprojected image 1515):

The image reprojection engine 215 may use the object's new position in the environment (defined by the coordinates X _out , Y _out , and Z _out ) to determine the object's new coordinates in the reprojected image 1515 , respectively represented as and . The new coordinates of the object in the reprojected image 1515 ( and ) is determined by the image reprojection engine 215 as follows:

The image reprojection engine 215 may use the coordinates of the object in the captured image 1510 ( and ) and the new coordinates of the object in the reprojected image 1515 ( and ) to determine the motion vector of the object from the captured image 1510 to the reprojected image 1515. Image reprojection engine 215 may determine the horizontal value of the motion vector as MV _x and the vertical value of the motion vector as MV _y as follows:

Image reprojection engine 215 may use motion vectors MV _x and MV _y to learn for any primitive of any object in captured image 1510 where that primitive should fall in reprojected image 1515 . In the illustrative example, portions of the chair may be moved 4 primitives to the right from captured image 1510 to reprojected image 1515 . At the same time, the toolbox portion can be moved 10 primitives to the right from the captured image 1510 to the reprojected image 1515 because the toolbox is closer to the camera than the chair. Therefore, for each object, the image reprojection engine 215 can calculate the position where the object should be moved into the reprojected image 1515 relative to the captured image 1510 .

The motion vector may represent a primitive displacement of each primitive in the first image data to a position of the primitive in the second image data, where the displacement will depend on the relative viewing viewpoints and depths of the first and second perspectives. The countdown. As discussed above, motion vectors can be determined based on depth data (eg, "depth" in the equation above). For example, in some examples, motion vectors may be determined based on the location of the object in the environment, such as 3D coordinates (eg, X, Y, Z) that may be determined based on depth data from captured image data. In some examples, the output of a transformation (eg, a 3DTransformation) of the object's position in the environment (eg, X _out , Y _out , Z _out )) to determine the motion vector.

In some examples, the camera's focal length f may also affect some of the above equations. For example, the determination of the X and Y coordinates of the object in the environment may be based on the focal length f and the coordinates of the object in the reprojected image 1515 ( and ), for example as indicated below:

Figure 16 is a block diagram 1600 illustrating mesh inversion transformation and 3D transformation based on depth data. The mesh inversion transformation obtains the 3D transformation 1605 (e.g., in the form of a reprojection matrix) and the depth map 1610 and uses MV computation 1615 to produce motion vectors (MVs) 1620 that indicate the object's position in the environment from the captured image 1510 Movement to the reprojected image 1515, as shown in Figure 15. In some examples, the initial motion vector may be referred to as an existing motion vector. Grid Inversion Transformation Grid inversion 1625 is performed on the existing MV 1620 to become inverted motion vectors 1630 . In some examples, the inverted motion vector may be referred to as the desired motion vector.

Figure 17 is a block diagram 1700 illustrating a motion vector based image reprojection transformation. Illustration warping engine 1705, which may be part of image reprojection engine 215. The warp engine 1705 uses the inverted motion vector 1730 (eg, the inverted MV of Figures 15-16) instead of the originally determined motion vector (the MV of Figures 15-16). This is because the inverted motion vector 1730 is an outside-in motion vector, while the initially determined motion vector (MV) is an inside-out motion vector. The outside-in motion vector transformation is less computationally expensive than the inside-out motion vector transformation. Specifically, if the warp engine 1705 generates the reprojected image 1715 using an outside-in motion vector, such as the inverted motion vector 1730, the warp engine 1705 may produce the reprojected image 1715 in raster order (or the reverse raster order or whatever) of the reprojected image. preferred order) to generate a reprojected image 1715 primitive by primitive. For each primitive in the reprojected image 1715 , the outside-in inversion motion vector 1730 instructs the warp engine 1705 to extract primitive data from a specific location in the captured image 1710 and use The primitive information of 1710 populates the primitive of the reprojected image 1715 . For example, for a certain primitive in the reprojected image 1715, the warp engine 1705 can read the outside-in inversion motion vector 1730 to determine that the value of the primitive should be taken from the left 4 in the captured image 1710. primitives, and so on.

An inside-out motion vector may represent a motion vector indicating the motion of a primitive from an initial image of the scene (from the initial perspective) to a target image of the scene (from the target perspective). The initially decided motion vector (eg, MV of FIGS. 15 and 16 ) may be an example of an inside-out motion vector. An outside-in motion vector may represent a motion vector indicating the motion of a primitive from a target image of the scene (from the target perspective) to an initial image of the scene (from the initial perspective). Inversion MV 1730 may be an example of an outside-in motion vector.

When the warp engine 1705 performs a warp (e.g., from captured image 1710 to reprojected image 1715), an outside-to-in motion vector (e.g., inversion motion vector 1730) is used to warp relative to an inside-to-outer motion vector. The use of motion vectors (eg, MVs of Figures 15-16) for warping can provide a reduction in computational resource consumption. The inside-out motion vectors (eg, MVs of FIGS. 15-16 ) are organized based on the captured image 1710 rather than the reprojected image 1715 . On the other hand, outside-in motion vectors (eg, inverted motion vectors 1730 ) are instead organized based on the reprojected image 1715 . When the warp engine 1705 performs the warp to produce the reprojected image 1715 , the reprojected image 1715 is produced according to a primitive order based on the reprojected image 1715 (eg, in a raster order based on the reprojected image 1715 ). It is optimal to generate the reprojected image 1715 rather than in primitive order based on the captured image 1710 (eg, in raster order based on the captured image 1710). Using outside-in motion vectors (eg, inverted motion vectors 1730 ) for warping may allow the warp engine 1705 to be in a primitive order based on the reprojected image 1715 (eg, in a raster order based on the reprojected image 1715 ) The resulting reprojected image 1715. For example, using the inverted motion vectors 1730, the warping engine 1705 may produce each primitive of the reprojected image 1715 in which any conflicts or missing areas have been resolved as discussed with respect to FIG. 5. On the other hand, in order for the warp engine 1705 to use the inside-out motion vectors to generate the reprojected image 1715 in the raster order of the primitives in the reprojected image 1715 , the warp engine 1705 will search the capture by primitive by primitive. Take the inside-out motion vector for each specific primitive of the reprojected image 1710 and the reprojected image 1715 and iteratively search among the motion vectors to find what should be in that specific primitive of the reprojected image 1715 Ending information. Repeatedly searching through the captured image 1710 and the inside-out motion vectors is computationally expensive and uses a lot of power. In some cases, the warp engine 1705 may also need to resolve conflicts or fill in missing areas, and if these searches produce motion vectors in the wrong order (e.g., mistakenly prioritizing farther objects over closer ones instead of closer ones) objects over more distant objects), conflicts may be resolved incorrectly or missing areas may be filled incorrectly. Therefore, even though generating an outside-in motion vector (eg, inverting motion vector 1730) from an inside-out motion vector (eg, the motion vectors of Figures 15-16) requires some computational cost, using an outside-in motion vector (eg, the motion vector of Figures 15-16) , inverse motion vector 1730), the final result of the distortion still saves computational resources and improves accuracy.

In some examples, the inside-out MV (existing MV) is determined at a low resolution, such as ¼ of the resolution of the captured image, since determining the inside-out MV can be expensive. Generating the outside-in MV (required MV) by applying grid inversion to the inside-out MV is computationally inexpensive. Furthermore, reprojection using outside-in MVs (required MVs) is computationally inexpensive. The computationally inexpensive nature of these operations allows grid inversion and/or reprojection using outside-in MVs (required MVs) to be performed efficiently even at higher resolutions such as the full range of captured images. resolution). Therefore, although the inside-out MV (existing MV) is determined at a lower resolution, the warp engine 1705 can generate the reprojected image as a complete reprojection of the captured image. This allows further savings in computing resources and power.

The grid inversion engine 225 includes several mechanisms to handle missing data and/or conflicts in the inverted MV grid. As explained previously, the grid inversion engine changes the position of the MV to correlate the primitive's position in the target image (eg, reprojected image 1715). In some cases, there are primitives that do not have MVs pointing to them in the input mesh, so inversion alone will not place MVs at these locations. The grid inversion engine fills these cells in the inverted MV grid by interpolation during its process. Referring again to Figure 5, inversion MV grid 520 is generated via grid inversion and includes missing cells marked with an asterisk. For example, cell 1 in the inversion MV grid 520 does not have a corresponding motion vector from the MV grid 505, but is filled using repair. One interpolation option is to interpolate the value of cell 1 using the values in its neighbor cells 0 and 2. For example, the weight of the interpolation can be by distance, so based on the value 0 in cell 0 and the value -1 in cell 2, the interpolated value of cell 1 can be -1/2. A similar type of interpolation can be performed on cells 3, 5, 6, and 7.

The grid inversion engine 225 also includes mechanisms for handling conflicts in the inverted MV grid. In some cases, multiple MVs in the MV mesh 505 may point to the same primitive in the second image (eg, the second image Img2 515, the reprojected image 1715), so in the inverted MV mesh An MV conflict occurs in grid 520 , requiring the grid inversion engine to select one of the conflict values for a given cell in inversion MV grid 525 . An example of such a conflict is shown in cell 8 of the inversion MV grid 520. According to the motion vectors expanded from cells 7 and 8 in the MV grid 505, the car in cell 7 of the first image Img1 510 and the tree in cell 8 of the first image Img1 510 end up in the same position as the first image Img1 510. The second image ends in the same primitive corresponding to cell 8 in image Img2 515. As a result, it may be unclear which value the grid inversion engine should select to place into cell 8 of the inversion MV grid 520 .

To resolve conflicts, grid inversion engine 225 may select one value or the other. In some examples, a weighted average of conflicting values may be used. If grid inversion engine 225 has depth information corresponding to the two objects (eg, from depth data 620 ), grid inversion engine 225 may select the value corresponding to the object closer to sensor 205 . This is because in many cases, closer objects cover, block, or obscure the view of farther objects. If the mesh inversion engine 225 lacks depth information corresponding to these two objects, the mesh inversion engine 225 can select the value based on other heuristics or techniques, such as selecting a value that corresponds to larger motion or representation. for larger objects. Regardless of object size, objects that experience greater motion are more likely to be closer to the sensor 205 because the movement of a closer object covers a larger field of view of the sensor 205 than the movement of a farther object, even if the movement is the same. in this way. In some examples, objects that appear larger may also be closer to the sensor 205 .

In some examples, referring to Figure 5, the car moving from cell 7 of the first image Img1 510 to cell 8 of the second image Img2 515 is closer to the sensor 205 than the tree, in which case the network Lattice inversion engine 225 may choose to invert the value in bin 8 of MV grid 520 as -1 (as the reciprocal of the corresponding value of 1 in bin 7 of MV grid 505). In some examples, in FIG. 5 , the tree is closer to the sensor 205 than the car, in which case the grid inversion engine 225 may choose to invert a value of 0 in cell 8 of the MV grid 520 ( Based on the corresponding value in cell 8 of MV grid 505 being 0). In some examples, grid inversion engine 225 may lack information regarding the relative depth of the car relative to the tree. In this case, the value in cell 8 of the inverted MV grid 520 is selected because the car is experiencing a large motion (its value is 1 in the MV grid 505 and the tree's value is 0). is -1 because the car may be closer to the sensor 205 than the tree. In some examples, if the car appears larger than the tree in the image, the value in cell 8 of the inversion MV grid 520 is selected to be -1 because the car may be closer to the sensor 205 than the tree. In some examples, the value in cell 8 of the inverted MV grid 520 is chosen to be -1/2 as the average of the reciprocal of the values in cells 7 and 8 of the MV grid 505 .

Different types of interpolation can be performed, in one example the interpolation can weight the value based on distance from a neighborhood cell. In another example, interpolation can weight values based on the depth of the neighborhood. Other methods can be applied. For example, for larger gaps, such as in cells 5, 6, and 7 of inversion MV grid 520, interpolation can weight information from closer cells higher than information from farther cells. For example, the value in cell 6 of the inverted MV grid 520 may be between the value (2) in the cell 4 of the inverted MV grid 520 and the value in cell 8 of the inverted MV grid 520 average value. Inverting the values in cell 8 of MV grid 520 may depend on how conflicts in cell 8 are resolved, as discussed above. Assuming that the value in cell 8 of the inversion MV grid 520 is -1, the value in cell 6 of the inversion MV grid 520 may be ½. In its interpolation, the value in cell 5 of the inverted MV grid can 520 cause the value (2) in cell 4 of the inverted MV grid 520 to be weighted higher than the value in the cell of the inverted MV grid 520 The value in 8 is, for example, the average value of the value in cell 4 of the inverted MV grid 520 and the interpolated value in the cell 6 of the inverted MV grid 520 . Similarly, in its interpolation, the value in bin 7 of the inverted MV grid 520 may weight the value (2) in bin 4 of the inverted MV grid 520 less than the value in the inverted MV grid 520 The value in cell 8 is, for example, the average value of the value in cell 8 of the inverted MV grid 520 and the interpolated value in the cell 6 of the inverted MV grid 520 . For example, assuming the value in cell 8 of the inverted MV grid is -1, the value in cell 5 of the inverted MV grid could be set to 1.25, while the value in cell 7 of the inverted MV grid could be set to 1.25. Value can be set to -0.25.

Figure 18 is a conceptual diagram 1800 illustrating an example of repair to address occlusions. Some regions in some reprojected images may not have appropriate material from the input image and may therefore represent gaps or occlusions in such reprojected images. In the reprojected image 1805, the occluded areas appear as black areas. For example, occluded areas are visible on the left side of each chair (especially the leftmost chair), the left side of the tool box, and the left side of the table. These occlusion areas may occur when an object close to the sensor 205 moves from side to side. Occlusion map 1810 of reprojected image 1805 shows occluded areas in white and all unoccluded areas in black. Imaging system 200 modifies the reprojected image 1805 to fill the occluded areas using inpainting to produce an inpainted image 1815 . In some examples, deep learning-based inpainting is used, which can provide high-quality inpainting based on the training of a deep learning model for deep learning-based inpainting, which may have been based on including the original copy of the image. Training is performed with training material on a second copy of the image with occlusions added, similar to the occlusions shown in the reprojected image 1805 and occlusion map 1810 . An example of deep learning based inpainting is illustrated in inpainted image 1815.

In some examples, a computationally less expensive repair, such as interpolation or online or nearest value repair, may be used based on the available computational bandwidth and/or power margin of the imaging system 200 for the repair operation. An example of interpolation based inpainting is illustrated at the bottom of Figure 18 using a 3D depth based zoom example, eg using interpolation and/or line or nearest value inpainting. A 3D depth-based zoom image 1825 is illustrated in Figure 18, with an occlusion area 1835 visible between the skateboarder's legs at the previous position of the skateboard. The repaired image 1830 is shown repairing the occluded region 1835 using interpolation based repair, such as using interpolation or line or nearest value repair.

Figure 19 is a block diagram 1900 illustrating the architecture of a reprojection and grid inversion system 1905. The reprojection and grid inversion system 1905 can read data in raster order. In some examples, the reprojection and grid inversion system 1905 reads the MV grid 1910 in raster order, and/or reads the depth data (eg, first option 1915 ) in raster order (eg, from a depth sensor). ), and obtain a 3D matrix. For each primitive in the input, and for each motion vector and/or depth value in the input, the reprojection and mesh inversion system 1905 places the primitive in the output at the location in the output. Each tile number represents a set of primitives in the output. In raster order, the primitive indicated by arrow 1930 will go into tile 1, while the primitive indicated by arrow 1935 will go into tile 2. Primitives that are not close to each other in the input mesh can be closer together in the output mesh. For this reason, it may be useful to save the tiles in cache in the event that the reprojection and grid inversion system 1905 needs to write more data to the tiles. For example, if the reprojection and grid inversion system 1905 starts with tile 1 and then moves to tile 2, the reprojection and grid inversion system 1905 may need tile 1 again later. Keep the tiles in cache (as long as the reprojection and grid inversion system 1905 can be based on a least recently used (LRU) cache system) allowing the reprojection and grid inversion system 1905 to quickly modify the tiles again instead of reading it from DRAM.

In some cases, using depth-based reprojection, closer objects move more than farther objects. Therefore, objects from different regions in the input image can appear in the same region in the reprojected image. Primitives/Arrows 1930 and Primitives/Arrows 1940 are an example of a situation where they originate from different locations in the input (e.g. MV grid 1910) but fall in the same area in the output, e.g. in a tile 1 in. Reprojection and mesh inversion system 1905 can therefore save Tile 1 in memory so that it can modify Tile 1 (eg, overwrite Tile 1 with the primitive values indicated by arrow 1940). Keeping the entire output buffer in memory hardware may be excessive, so the reprojection and mesh inversion system 1905 may include a cache mechanism to keep the tiles in memory hardware.

If the reprojection and grid inversion system 1905 starts at the beginning of the raster sequence and this is the first time the reprojection and grid inversion system 1905 wants to write to a tile (e.g., the primitive value indicated by arrow 1930 is written to the tile block 1), then the reprojection and grid inversion system 1905 simply resets tile 1 and writes the values in question to tile 1 without first reading the tile from DRAM. In some examples, the value from Tile 1 may be moved from cache to DRAM. The reprojection and grid inversion system 1905 uses cache memory so that it does not need to perform many read/modify/write operations, but the reprojection and grid inversion system 1905 does have the ability to read/write when necessary. Ability to modify/write operations. As long as the tiles are in cache, the reprojection and mesh inversion system 1905 can immediately access them. At some point in time, the cache may become full, and the reprojection and grid inversion system 1905 may send a tile from the cache to DRAM to make room for another tile (based on LRU). At some other point in time, the reprojection and grid inversion system 1905 again needs the tile sent from cache to DRAM, and then the reprojection and grid inversion system 1905 can read the tile back from DRAM to cache. The memory is fetched in order to modify it, and at some other point in time the tile can be written to DRAM.

Additionally, the reprojection and grid inversion system 1905 has a prefetch mechanism that allows the reprojection and grid inversion system 1905 to generate required tiles in advance and before processing to avoid latency issues in reading tiles from DRAM. The reprojection and grid inversion system 1905 works in an ordered manner, and the prefetch mechanism ensures that the reprojection and grid inversion system 1905 always has what it needs in cache. The reprojection and grid inversion system 1905 can switch between prefetching and processing in a lock-step manner rather than randomly to ensure that the reprojection and grid inversion system 1905 can process all data in an orderly manner and will be required Everything processed is placed in cache.

In a first option 1915, the reprojection and grid inversion system 1905 can receive depth data and 3D matrices. In some examples, the reprojection and mesh inversion system 1905 can generate the MV mesh 1910 from the depth data and the 3D matrix. In a second option 1920, the reprojection and mesh inversion system 1905 may receive the MV mesh with depth data and a 2D matrix. In some examples, the reprojection and mesh inversion system 1905 can generate the MV mesh 1910 from the MV mesh with depth data and a 2D matrix. Reproject if reprojection and grid inversion system 1905 receives depth and 3D matrices (first option 1915) or if reprojection and grid inversion system 1905 receives MV grids and/or 2D matrices (second option 1920) And the grid inversion system 1905 uses its coordinate calculation system to calculate the output coordinates (outCoord) and output data (outData). In some examples, the output data may include an output motion vector (outMV) and an output depth (outDepth). The reprojection and grid inversion system 1905 may also output additional output data (as part of outData), such as confidence (outConf) and/or occlusion (outOcc), to determine the location of the occlusion region. Output from the reprojection and grid inversion system 1905 may be output as output data to one or more buffers, caches, or other memories. In one illustrative example, the output buffer (or cache or other memory) shown on the right side of Figure 19 includes an output buffer (or cache or other memory) for depth, an output buffer (or cache or other memory) for MV An output buffer (or cache or other memory) for the mesh (eg, with depth and/or confidence), and an output buffer (or cache or other memory) for occlusion. These output buffers (or cache or other memory) can be output as multiple output images. The prefetch and cache mechanisms can handle three buffers simultaneously. Because each output buffer can store a different number of bits per tile, the prefetch and cache mechanisms can handle all the different levels of bits and tiles of different sizes at each stage. Synchronize.

In some examples, the reprojection and mesh inversion system 1905 uses specialized hardware specifically designed to be efficient in motion vector manipulation, coordinate calculations, caching, prefetching, and generating output buffers. In some aspects, certain operations may be performed using a processor such as a CPU or GPU.

In some examples, the output confidence (outConf) is not generated specifically for reprojection, but is a by-product of the depth measurement from the depth sensor. In some examples, the acquired depth may suffer from measurement inaccuracies and/or other issues that may be represented by a confidence map. Improving depth based on confidence maps and/or visual (RGB) images may be beneficial. The reprojection and grid inversion system 1905 can reproject the depth and confidence to match the visual (RGB) image and allow the confidence to be used for the correct domain in the reprojected image. Once the depth matches the RGB image, the reprojection and grid inversion system 1905 can use confidence to improve the depth.

In some examples, the imaging system may use a "triangle walking" operation to determine that a given primitive from the input image (eg, first image Img1 510, captured image 1710) should be moved to the reprojected image The position in the image (eg, second image Img2, 515, reprojected image 1715).

FIG. 20 is a conceptual diagram 2000 showing an example of a triangular walking operation. In some examples, different primitives from the input image can be moved to different locations in the reprojected image. The system can process X inputs at a time, where X equals any integer value (e.g., 3, 4, 5, 6, 10, etc.). The system can produce Y output triangles (e.g., for each set of inputs), where Y equals any integer value (e.g., 6, 7, 8, 9, 10, 15, etc.). The primitives in the input include primitive a, primitive b, primitive c, etc. In some examples, primitive data from primitive a in the input image can be moved to the first position in the reprojected image, and primitive data from primitive b in the input image can be moved to the first position in the reprojected image. to the second position in the image, and the primitive data from primitive c in the input image can be moved to the third position in the reprojected image, and so on. Through a map (eg, MV grid 505 or inverted MV grid 520), the system finds where each primitive of the input image should be in the reprojected image. Thus, in the illustrative example, primitive a of the input image ends in primitive primitive 2010 of output, primitive primitive b of the input ends in primitive primitive 2015 of output, and primitive primitive 1 of the input ends in primitive primitive 2015 of output 2020, and so on. For each input primitive, the imaging system calculates the position at which the input primitive's value is configured to end in the output. For the area between specific primitives in the output (for example, the shaded triangle area between primitives 2010, 2015, and 2020), the imaging system uses interpolation to fill the area. To perform interpolation, the imaging system can have a processor (eg, a GPU or other processor) iterate through each triangle individually and perform interpolation for each output primitive individually, one after the other.

However, to improve efficiency, the imaging system can put the triangles together to form a large polygon, that is, a polygon composed of the combination of all triangles on the output side of Figure 20 (including triangles with primitives 2010, 2015, and 2020). The imaging system may have a dedicated hardware processor specifically designed for efficient interpolation, or have other processors (eg, a GPU or other processor) perform the interpolation. Imaging systems using a processor (e.g., GPU) to iterate through each triangle individually and interpolate each output primitive separately can be inefficient because many of these triangles include image material that is close together and similar. To increase efficiency, the imaging system can combine triangles into polygons and can have the processor (e.g., GPU) perform interpolation on the entire polygon's primitives by traversing the entire polygon at once.

The imaging system includes a main walking engine 2025, N triangle control engines 2030 (where N can be equal to any integer value, such as 6, 8, 10 or other values), and M primitive interpolation engines 2035 (where M can be equal to any integer value). A numerical value such as 6, 8, 10 or other values, and in some embodiments may be equal to N). The main walking engine 2025 (shown as a dashed box with white shading) traverses the entire polygon at a time. N triangle control engines 2030, two of which are shown as boxes with dashed lines and light shading, and each triangle control engine is responsible for one of the triangles. The main walking engine 2025 traverses the entire polygon, effectively pre-scanning the output locations and/or areas to be used by the imaging system for image reprojection, allowing the imaging system to early prefetch and/or retrieve data from DRAM (e.g., image blocks) to cache memory data, thereby reducing or eliminating delays that might otherwise be caused by retrieving data from DRAM (e.g., delays in padding, interpolation, or other image processing operations).

Figure 21 is a conceptual diagram 2100 illustrating an example of occlusion masking. Occlusion areas are areas of the reprojected image where image reprojection engine 215 has no image material available. As mentioned previously, the image reprojection engine 215 performs interpolation on regions in the original captured image that do not have specific values. This interpolation is performed even for occluded areas, eg to avoid filling those areas with unreliable data (eg any data occurring in DRAM). Image 2110 may be an example of filling using such unreliable material. In order to perform reprojection, some objects (such as toolboxes) may be stretched slightly in some directions (e.g., horizontally), but this stretching is generally not enough to have a negative impact, and in some cases can enhance the new perspective in Appearance in the reprojected image. However, in some areas, holes or gaps exceed a threshold size, beyond which interpolation may be unreliable, and the image reprojection engine 215 may determine the area to be an occlusion area.

In some examples, image reprojection engine 215 may determine whether an occlusion region exists based on corner depth. For example, if the difference between the depths at the corners of a region exceeds a threshold difference, the image reprojection engine 215 may determine that there is an occlusion region (eg, a triangle or other shape of Figure 20) in the region. The threshold difference can be changed based on the depth minimum.

Once the image reprojection engine 215 defines that an occlusion region exists (e.g., based on the difference between the depths at the corners of the region exceeding a threshold difference), the image reprojection engine 215 can perform repair to fill in the reprojection with the image material. occluded area of the image. The "unreliable residue" in image 2110 may represent a form of repair using portions of the toolbox image data in occluded areas. In some cases this type of repair may work well, even if it looks unusual in image 2110. In some examples, occlusion can be performed using deep learning (e.g., using one or more trained ML models).

Figure 22 is a conceptual diagram 2200 showing an example of hole filling. Hole filling refers to interpolating gaps where motion vector data does not exist. Flow 2220 illustrates that with hole filling turned off, the reprojected image has many visual artifacts, such as black and white dots in the pattern of visual artifacts that are particularly noticeable on toolboxes and other objects near the camera. When hole filling is turned on, holes in the reprojected image are filled using interpolation, and the image looks clean and free of such visual artifacts or patterns of visual artifacts. In some examples, hole filling may use inpainting (such as deep learning-based inpainting) as an alternative to or in addition to interpolation.

23 is a conceptual diagram 2300 illustrating additional examples of time warping 705 performed by time warping engine 230. The time warp engine 230 computes dense optical flow, here between frame n+1 and frame n and between frame n and frame n-1 respectively. The input frame rate (in frames per second (FPS)) is equal to Fin, which can be 30 FPS, 60 FPS, 120 FPS, 240 FPS, or other frame rates. The output frame rate is equal to Fout, which can be 60 FPS, 120 FPS, 240 FPS, 480 FPS or other frame rates. These dense optical flows are computed with high quality, but may be computationally expensive and/or use large amounts of power. Similar to time warp 705 of Figure 7, time warp engine 230 divides the dense optical flow to produce smaller pixels between other frames (eg, between frames n-1 and n or between frames n and n+1). partial optical flow. For example, the time warp engine 230 divides the dense optical flow to produce smaller partial optical flows for frames n + ¾, n + ½, n + ¼, n – ¼, n – ½, and n – ¾. These partial optical flows can be used as surrogates for optical flow, as if each partial optical flow was calculated directly using optical flow calculations. These partial optical flows can be broken down into quarters as in this example, or other similar fractions. These partial optical flows can be used to improve existing frames when present at frames n+¾, n+½, n+¼, n–¼, n–½, and n–¾. These partial optical flows can be used to generate new interpolated frames at frames n+¾, n+½, n+¼, n–¼, n–½, and n–¾. In some examples, time warping 705 may be used by first dividing the computed dense optical flow into optical flow for intermediate frames at a lower frame rate (eg, 30 or 60 fps) and using time warping 705 High frame rates (e.g., 90, 120, 240, 480, or 960 fps) produce dense optical flow for video.

In some examples, time warp engine 230 may take the motion vectors of the optical flow, combine the motion vectors with the global matrix, and after combination divide the result into partial optical flow or motion vectors as in time warp 705.

Additional examples of image sharpening benefits for images without time warp 705 and with time warp 705 are illustrated. Use Time Warp 705 to restore detail, as indicated by the areas pointed by arrows (such as the boy's hair, ears, and T-shirt in the middle image, and the area at the mark in the right image). Specifically, edges and/or areas that appear blurry are represented by dashed lines, while edges and/or areas that appear sharp are represented by solid lines.

24 is a block diagram 2400 illustrating an exemplary architecture for the time warp engine 230 in some examples of the reprojection engine 24341. Optical flow engine 2420 receives frame n and frame n-M from camera 2405 having image sensor 2410 and dynamic random access memory (DRAM) 2415. The optical flow engine 2420 generates motion information. In some examples, the motion information includes two types of motion information, including global motion and local motion. For example, a matrix (eg, a global matrix) can represent global motion in some cases. Optical flow engines can produce dense grids of motion vectors to indicate local motion and 3D motion. In other examples, a dense motion vector grid may also indicate global motion and/or local motion, a combination of 3D motion and global motion.

The grid inversion engine 2425 receives motion information from the optical flow engine 2420 (eg, a dense grid of motion vectors and, in some cases, a matrix representing global motion). The grid inversion engine 2425 is run multiple (M) times, each run dividing the motion vector and outputting a different portion of the motion vector. The grid inversion engine 2425 outputs M motion vectors. In some cases, the motion vector can be multiplied by a factor. The warp engine 2430 can be used to shrink motion vectors to provide different resolutions. Warp engine 2430 may receive motion vectors from the dense mesh and perform some warping, scaling, and/or other operations on the dense motion mesh. In some examples, the warp engine 2430 also obtains the transformation matrix and warps the dense mesh based on it. In other examples, the warp engine 2430 can obtain the transformation matrix and combine it with the dense mesh. The inverted motion vectors output by the mesh inversion engine 2425 and/or the warping engine 2430 are output to the image processing engine 2440 to generate a reprojected image based on the inverted motion vectors.

25 is a block diagram 2500 illustrating an exemplary architecture for a time warp engine 230 with temporal blur in some examples of a reprojection engine 2535 with temporal blur. The architecture in Figure 25 is similar to the architecture in Figure 24, but the system's temporal deblurring engine 2505 (eg, based on motion detection and/or image analysis) determines which M frames are blurred and uses grid inversion Partial motion vectors generated by engine 2425 are used to deblur and/or sharpen blurred frames. In some examples, the temporal deep learning algorithm of the reprojection engine 2535 analyzes the attitude sensor data and looks at how much distance was moved (and therefore how much blur was present) during the capture of each frame. In some examples, the original motion vectors are provided from the optical flow engine 2420 to the image processing engine 2440, in some cases after further transformation 2520 (eg, shrinking).

26 is a block diagram 2600 illustrating an exemplary architecture of the depth sensor support engine 235. A time-of-flight (ToF) sensor is an example of a depth sensor, but the depth sensor support engine 235 may use different types of depth sensors as described herein in some examples. Post-processing (e.g., by filtering out outliers and/or normalizing noise) can be applied to clean up the depth values from the depth sensor to provide higher quality depth values. In some cases, the post-processing can also receive a confidence map and depth, and the post-processing can then also clean the confidence map, and/or use the confidence map to assist in depth processing. The depth and in some cases the confidence are sent to a reprojection engine, which can reproject the depth image and confidence map based on the 3D transformation, for example to align with the image sensor (eg wide angle or telephoto). The reprojection engine can produce reprojected depth and confidence values, which can be run again with depth post-processing to clean up the depth and confidence values. The depth post-processing can also accept images from the wide-angle and telephoto sensors, and/or secondary depth sensor data from the secondary depth sensor (e.g., DFS depth), and the depth post-processing can adjust the depth to further Improve it and correct inaccuracies from the original depth. The 3D transformation can be based on 3D calibration between the image sensor and the depth sensor. If the depth sensor and image sensor move relative to each other (eg, focus change, zoom, OIS, and/or other), the 3D calibration can take this into account and update the 3D transform. It should be understood that the auxiliary depth flow (i.e., DFS with wide-angle and telephoto images) at the bottom of Figure 26 is an illustrative example. In other examples, the auxiliary depth can come from another depth sensor, a deep learning depth engine, and/or any other depth source. In some examples, the depth postprocessing will have no auxiliary depth. In some examples, depth post-processing can have more than two depth sources.

27 is a conceptual diagram 2700 illustrating additional examples of depth sensor support 805 performed by depth sensor support engine 235. In these additional examples, the main image sensor (eg, RGB3) and depth sensor (eg, TOF system) are shown on the circuit board. Illustrated depth maps and images. In the example on the left (projection alignment 2705), some elements are aligned, but other objects at different distances from the camera (such as a teddy bear or a character's head) are misaligned between the image data and the depth data. For example, the bear's depth data (e.g., shown using a dashed line) is on the right (parallax shift) compared to the bear's image data. Similarly, the depth profile of a person (e.g. shown using dashed lines) is on the right (parallax shift) compared to the image profile of the person. On the other hand, in the example on the right (depth-based alignment 2710), the disparity is fixed and the depth data and image data of each object are aligned.

28 is a block diagram 2800 illustrating an exemplary architecture of an imaging system including image reprojection engine 215 and/or 3D stabilization engine 240. The imaging system accepts the input and reprojects the perspective to a new location in the environment. In the case of 3D stabilization, this reprojection can be done to reduce or eliminate camera wobble, and/or simulate the situation of camera stabilization and/or stabilization such that any movement has no (or very little) wobble or Signal interference. For example, the imaging system's 3D stabilization engine 240 can establish a virtual path as if the video was captured along a virtual path that includes little or no signal interference and/or wobble. The imaging system may also be used for at least some other applications of image reprojection described herein, such as time warping, head pose correction, sensor support, and the like. The imaging system receives image data and/or depth data as input, stabilizes or otherwise corrects any distortion in the data, and then provides the data to the reprojection engine. For 3D stabilization, the imaging system's 3D stabilization engine 240 may build a matrix indicating a stabilized smooth virtual path. Imaging systems can create 3D transformations to change the perspective of an image. For example, for 3D stabilization, the 3D transformation can change the corresponding viewing angles of a series of images such that the corresponding viewing angles of the images have origins along a virtual path (eg, a stabilized smooth virtual path). 3D transformations and in some cases virtual paths can be fed to the reprojection engine. The reprojection engine can generate motion vectors (MVGrid) to warp the image to the identified perspective (e.g., so that the captured perspective follows a virtual path). In some examples, the imaging system may perform lens distortion correction (LDC) and/or rolling shutter correction (RSC) on the image using another motion vector grid to reduce any distortion from the lens and/or rolling shutter. In other examples, motion vectors and/or matrices may also be used to correct other distortions and/or transformation errors. As shown in Figure 30, in some examples, 3D stabilization and meshes for LDC and RSC are combined by combining motion vectors from both, and warped together. The new set of MVs can perform both 3D stabilization and LDC and RSC. In some examples, the LDC and RSCMV meshes can be sparser than the 3D stabilized MV mesh, in which case the LDC and RSCMV meshes can be amplified before combining. In some examples, the 3D stabilized MV mesh can be sparser than the LDC and RSCMV meshes, in which case the 3D stabilized MV mesh can be enlarged before combining. The combined MV mesh can be sent to the warp engine which performs the warp. The figure shows the resulting image, to which 3D stabilization (via reprojection), LDC and RSC were applied.

Since reprojection is used for 3D stabilization, occluded areas may still remain in the resulting image. Depth reprojection, occlusion map, low-resolution copy of the image (e.g., with full field of view (FoV)) and/or Q high-resolution patches from the image (e.g., 500 patches of size 64x64, or with Any other number of patches of suitable size) can be sent to the deep learning engine (NSP) to perform repair. For example, the 3D stabilization engine 240 may take patches from one region without reading another region. Thanks to the occlusion map, the 3D stabilization engine 240 knows which areas to focus with high-resolution patches. In some examples, the patches and occlusion maps are small (e.g., the occlusion map is binary or can include a small number of bits, such as 3 bits, 4 bits, 6 bits, etc.), making the patch useful for performing repairs. Cheap input for deep learning engines (NSP). Depth reprojection can help ensure that the correct type of material is used for the restoration. For example, the deep learning engine (NSP) will not use nearby objects like toolboxes to repair background areas - the only data that should be used to repair background areas is the depth of the image data from the background area. This smart fix is very efficient and uses less power.

In some examples, restoration can use temporal filtering, such as using previous images in a video to generate image content for a specific region. For example, if the previous image has clear image content in the scene area illustrated in the occlusion area in the current image frame, the image material from the previous image can be used for inpainting and/or for 3D stabilization to calm any wobbles. Patches can be aligned with compressed tiles such that the inpainted patches output by the deep learning engine (NSP) can be moved into memory (e.g., directly into DRAM) for the relevant portions of the resulting image.

29 is a conceptual diagram 2900 illustrating an additional example of time warping 705 performed using time warping engine 230 compared to an image processed without time warping engine 230. Especially in and around the edges and corners of the image, the sample with the time warp engine 230 looks sharper than the image without the time warp engine 230. For example, edges that look blurry are reproduced using dashed lines in Figure 29, while edges that look shared and clear are reproduced using solid lines in Figure 29.

30 is a conceptual diagram 3000 illustrating an additional example 3005 of 3D stabilization 905 performed by the 3D stabilization engine 240. Additional example 3005 includes four video frames of a video, shown in original (unsteady) and stabilized forms. As discussed previously, reprojection is used to eliminate wobble and/or parallax motion.

31 is a conceptual diagram 3100 illustrating an additional example of 3D zoom 1005 performed by 3D zoom engine 245. Digital zoom 3105 crops and enlarges, as shown by the dotted box and dotted line on the left side of the figure. A depth image of a skateboarder is shown with 3D depth-based zoom. 3D depth-based zoom uses depth-based image reprojection to simulate moving the camera closer to the skateboarder, as shown in illustration 3110 of moving the phone closer to the man.

32 is a conceptual diagram 3200 illustrating additional examples of reprojection 1105 performed by the reprojection SAT engine 250. Reprojection 1105 shifts the perspective by an offset using reprojection from one sensor's perspective to a different sensor's perspective.

33 is a conceptual diagram 3300 illustrating an additional example of head posture correction 1205 performed by head posture correction engine 255. Illustration of depth image 3515 of a woman's head as a basis for reprojection. Also illustrated is an occlusion map 3320 for the reprojected image 1215. An illustration illustrating the relative position of the person relative to the camera below the input image 1210 indicates that the camera is taking the photo from slightly below the user's face and slightly upward. Below the reprojected image 1215 is an illustration illustrating the simulated relative position of the person relative to the camera, indicating that the simulated camera position is retrieving the input image from an elevation or height that matches the elevation or height of the user's face, from The position of the image 1210 is separated by an offset distance 3305 and the photo is taken from an angle separated by an offset angle 3310 from the angle at which the input image 1210 was captured. The captured angle of the reprojected image 1215 is perpendicular to the person's face, body and/or perpendicular to gravity.

Figure 34 is a conceptual diagram 3400 illustrating additional examples of grid inversion. The original MV grid and the inverted MV grid are shown for a target image with sun and clouds. An example of using asterisks (via interpolation and/or repair) to fill in missing content, such as where part of the sun is blocked by clouds in the input image but is not in the reprojected image. Use circles to show examples of conflicting values, such as where both clouds and the sun have data, and the cloud data ultimately wins because the clouds are in front of the sun.

Figure 35 is a conceptual diagram 3500 illustrating an example of the use of deep learning-based repair. A plurality of sets of images are illustrated, each of the plurality of images including an occluded region 3505 in one of the set of images. Occluded areas are shown as blank before being filled in using a trained deep learning inpainting engine such as Neural Network 3900.

Figure 36 is a conceptual diagram 3600 illustrating an example of the use of inpainting without deep learning. Groups of images are shown arranged in columns. The first column includes the image output by the Grid Inversion Engine (RGE), which includes the occluded area 3605 shown as blank. The second column includes images output by the Grid Inversion Engine (RGE) where inpaintings were issued to fill occluded areas 3605. For example, the repair of Figure 36 may use interpolation and/or online or nearest value repair. As shown, patches can be selected for remediation based on similarity and/or priority. The third column includes the image 3605 without occluded areas output by the Grid Inversion Engine (RGE). The images in the third column include blur or visual "tailing" around some edges of the occlusion area 3605 in the first column images, which may look similar to motion blur and may be caused by other locations and/or the use of mesh. The lattice inversion engine (RGE) transforms the representation of objects in the original captured image.

Figure 37 is a conceptual diagram 3700 illustrating an example of the use of edge filters and depth filters on edges. In some examples, edge filters can be used to smooth block edges in depth data and/or image data, which can reduce visual artifacts in image reprojection. Although the filter is shown as being 3x3 in size, the filter can be larger in some cases (eg, 4x4, 6x6, etc.). Edge filters detect edges in depth maps. Depth filters on edges reduce interpolated depth values that do not belong to any object.

Figure 38 is a conceptual diagram 3800 illustrating an example of reprojection. Sensor 205 includes camera cam1 that captures images of the 3D scene and depth data (cam1 depth). Inter-camera 3D translation is used to reproject the 3D scene illustrated in the image in 3D space using the perspective of camera cam2. The forward mapping (eg, motion vector grid) is shown using dashed lines. Backward mapping (eg, inverted motion vector grid) is shown using a solid arrow from cam2 back to cam1.

Figure 39 is a block diagram illustrating an example of a neural network (NN) 3900 that may be used for media processing operations. Neural network 3900 may include any type of deep network, such as convolutional neural network (CNN), autoencoder, deep belief network (DBN), recurrent neural network (RNN), generative adversarial network (GAN) ) and/or other types of neural networks. Neural network 3900 may be an example of one of one or more trained neural networks of imaging system 200 , such as a neural network of any of application engines 210 , such as image reprojection engine 215 , Motion vector engine 220, grid inversion engine 225, time warp engine 230, depth sensor support engine 235, 3D stabilization engine 240, 3D zoom engine 245, reprojection SAT engine 250, head pose correction engine 255, XR Post-reprojection engine 260, special effects engine 265, or a combination thereof.

The input layer 3910 of the neural network 3900 includes input data. Input data to input layer 3910 may include data representing primitives of one or more input image frames, such as media data 285, sensor data from sensor 205, virtual content from virtual content generator 207, or combination thereof. Input data to input layer 3910 may include depth data from a depth sensor. The input data of the input layer 3910 may include motion vectors and/or optical flow. The input data of the input layer 3910 may include matrices. Input data to input layer 3910 may include occlusion maps.

Images may include image data from an image sensor that includes raw primitive data (including a single color per primitive based on, for example, a Bayer filter) or processed primitive values (e.g., RGB for an RGB image) Graph element). Neural network 3900 includes multiple hidden layers 3912A, 3912B through 3912N. Hidden layers 3912A, 3912B through 3912N include "N" hidden layers, where "N" is an integer greater than or equal to one. The number of hidden layers can include as many layers as necessary for a given application. Neural network 3900 also includes an output layer 3914 that provides output resulting from processing performed by hidden layers 3912A, 3912B through 3912N.

In some examples, output layer 3914 may provide an output image or a portion thereof, such as modified media material 290 , any reprojected image discussed herein, any reprojected depth material discussed herein, any motion discussed herein vector or optical flow, any inpainted image material discussed in this article, or a combination thereof.

Neural Network 3900 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent the input data. The information associated with the filter is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 3900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back to itself. In some cases, network 3900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read in.

In some cases, information can be exchanged between layers via inter-node interconnections between layers. In some cases, a network may include a convolutional neural network, which may not connect every node in one layer to every other node in the next layer. In a network that exchanges information between multiple layers, a node in the input layer 3910 can activate a group of nodes in the first hidden layer 3912A. For example, as shown, each input node of input layer 3910 may be connected to each node of first hidden layer 3912A. Hidden layer nodes can transform the information of each input node by applying an activation function (e.g., a filter) to that information. The information derived from the transformation can then be passed to and enable the nodes of the next hidden layer 3912B, which can perform their own designated functions. Exemplary functions include convolution, reduction, expansion, data transformation, and/or any other suitable function. The output of hidden layer 3912B can then initiate the nodes of the next hidden layer, and so on. The output of the last hidden layer 3912N may activate one or more nodes of the output layer 3914, thereby providing a processed output image. In some cases, although a node in neural network 3900 (eg, node 3916) is shown as having multiple output lines, the node has a single output and all lines shown as output from the node represent the same output value.

In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 3900. For example, interconnections between nodes can represent a piece of information learned about the interconnected nodes. The interconnections may have tunable numerical weights that can be tuned (eg, based on a training data set), allowing the neural network 3900 to adapt to the input and learn as it processes more and more data.

Neural network 3900 is pre-trained to process features from the data in input layer 3910 using different hidden layers 3912A, 3912B through 3912N to provide an output via output layer 3914.

FIG. 40 is a flowchart showing a procedure for media processing operations. Process 4000 may be executed by a media processing system. In some examples, the media processing system may include, for example, image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, imaging system 200, HMD 310. Mobile handheld terminal 410, reprojection and grid inversion system 2490, system of Figure 25, system of Figure 26, system of Figure 27, system of Figure 28, neural network 3900, computing system 4100, processor 4110 or its combination.

At operation 4005, the media processing system is configured to and can receive depth data including depth information corresponding to the environment. In some examples, the depth information may include depth measurements from a first-view representation of the environment. In some examples, the depth information includes point clouds corresponding to the environment. In some examples, depth data may be captured using one or more depth sensors, such as one or more light detection and ranging (LIDAR) sensors, radio detection and ranging (LIDAR) sensors, radio detection and ranging (LIDAR) sensors, range (RADAR) sensor, sound detection and ranging (SODAR) sensor, sound navigation and ranging (SONAR) sensor, time of flight (ToF) sensor, structured light sensor or a combination thereof. In some examples, depth data may be captured using one or more cameras and/or image sensors, such as based on stereoscopic depth sensing using a stereoscopic camera arrangement. In some examples, image capture and processing system 100, sensor 205, cameras 330A-330B, cameras 430A-430D, image sensor 810, depth sensor 815, telephoto sensor 1110 may be used , wide-angle sensor 1115, sensor 1125, image sensor 2610, cam1 in Figure 38, cam2 in Figure 38, any other sensor described herein, or a combination thereof to capture depth data. Examples of depth data include media data 285, depth data 620, depth data 1020, depth data 1160, depth data 1220, depth data of Figure 15, depth map 1610, depth data associated with first option 1915, depth input 2402, Depth of Figure 26, depth information of Figure 27, depth information of Figure 28, depth information 3315, depth image 3410, depth map of Figure 37, Cam1 depth of Figure 38, any other depth information described herein, or combinations thereof.

At operation 4010, the media processing system is configured and can receive first image data captured by the image sensor, the first image data including a representation of the environment. In some examples, image capture and processing system 100, sensor 205, cameras 330A-330B, cameras 430A-430D, image sensor 810, depth sensor 815, telephoto sensor 1110 may be used , wide-angle sensor 1115, sensor 1125, image sensor 2610, cam1 in Figure 38, cam2 in Figure 38, any other sensor described herein, or a combination thereof to capture the first image data . Examples of first image data include media data 285, first image Img1 510, camera image 610, image 710, the "orig" image in Figure 9, the original non-zoomed image of Figure 10 (before zooming) , telephoto image 1130, input image 1210, input image 1310, input image 1410, captured image 1510, captured image 1710, input image Image1 of process 2310, input image of process 2320 , the input image 705 without time distortion in Figure 25, the frames n and n-M in Figures 24 to 25, the m blurred frames in Figure 25, the wide-angle and telephoto images in Figure 26, the input image in Figure 27 Image, "orig" image in Figure 30, unzoomed input image in Figure 31, input image in Figure 34, input image in Figure 35, input image in Figure 36, original primitive in Figure 38, provided An image, other image material described herein, or a combination thereof is given to input layer 3910.

At operation 4015, the media processing system is configured to and can generate a first plurality of motion vectors corresponding to changes in perspective of the illustration of the environment in the first image material based on at least the depth material. Examples of the first plurality of motion vectors include the motion vectors in MV grid 505, the motion vectors of Figure 15 (eg, _MVin , _MVx , _MVy ), MV 1620, the dense MV of Figure 23, and the optical flow engine 2420 The associated motion vectors, the MV grid of Figure 28, the original MV and MV grid of Figure 34, the forward map of Figure 38, other motion vectors described herein, or combinations thereof.

At operation 4020, the media processing system is configured to and may use grid inversion to generate a second plurality of motion vectors based on the first plurality of motion vectors, the second plurality of motion vectors indicative of the first plurality of motion vectors in the first image material. The corresponding distance that the corresponding primitives of the representation of the environment move in response to the change in perspective. Examples of the second plurality of motion vectors include motion vectors in inversion MV grid 520, inversion MV 1630, inversion MV 1730, inversion motion vectors associated with grid inversion engine 2425, the MV net of Figure 28 lattice, the inverted MV and MV grid of Figure 24, the backward mapping of Figure 38, other inverted motion vectors described herein, or combinations thereof.

At operation 4025, the media processing system is configured to and can generate second image data by, at least in part, modifying the first image data based on the second plurality of motion vectors, wherein the second image data includes from A second illustration of the environment from a different perspective than the first image material. Examples of second image data include modified media data 290, second image Img2 515, reprojected image 615, image 715, the "steady" image of Figure 9, the 3D zoomed image of Figure 10, Modified Telephoto Image 1140, Reprojected Image 1215, Input Image 1315, Reprojected Image 1415, Reprojected Image 1515, Reprojected Image 1715, Reprojected Image 1805, Repair image 1815, reprojected image 2110, reprojected image 2115, reprojected image of process 2210, reprojected image of process 2220, reprojected image 705 with time warping in Figure 23 , using the image output by the image processing engine 2440, the depth-based alignment 2710 image of Figure 27, the time warp image of Figure 29, the "stable" image of Figure 30, the depth-based 3D zoom image of Figure 31 , the output image of Figure 34, the output image of Figure 35, the output image of Figure 36, the reprojected image in Figure 38, the image output using the output layer 3914, other image materials described herein, or their combination.

In some examples, the second image material includes an interpolated image configured to illustrate the environment at a second time between the first time and the third time. In such examples, the first image material includes at least one image illustrating an environment at least one of a first time or a third time. Examples of such image interpolation may be performed using time warping 705 as in Figure 7 and/or Figure 23. In some examples, the imaging system can produce interpolated images without using depth information.

In some examples, the first image data includes a plurality of video data frames that include parallax motion, and the second image data includes a stable variant of the plurality of video data frames that reduces the parallax motion. For example, 3D stabilization 905 may stabilize, reduce, and/or eliminate parallax movement, rotation, or a combination thereof, as shown in FIG. 9 and/or FIG. 30 .

In some examples, the first image material includes the person viewing the image sensor from a first angle, and the second image material includes the person viewing the image from a second angle different from the first angle. sensor. Examples of this include head pose correction 1205, as shown in Figure 12 and/or Figure 33.

In some examples, the viewing angle changes include viewing angle rotations based on angles and about an axis. In some examples, the perspective changes include perspective translation as a function of direction and distance. In some examples, the change in perspective includes a shift. In some examples, the change in perspective includes movement along an axis between an original perspective of the illustration of the environment in the first image material and a position of an object in the environment, where at least a portion of the object is in This first image material is illustrated. In some examples, rotations, translations, transformations, and/or movements may be identified based on what is required to perform any type of reprojection and/or distortion described herein, such as in any of the examples of FIGS. 7-14. In some examples, user interface identification may be used to identify rotations, translations, transformations, and/or movements. For example, in some examples, the change in perspective includes at least one of a parallax shift of the perspective or a rotation of the perspective about the axis, the method further comprising receiving, via the user interface, one of: an indication of a distance of the parallax shift of the perspective , or an indication of the angle of rotation or axis of the viewing angle.

At operation 4030, the media processing system is configured and can output the second image material (eg, using output device 270). For example, the media processing system may display the second image data, output the second image data for further processing, store the second image data, any combination thereof, and/or otherwise output the second image data.

In some examples, outputting the second image material includes using at least a display to display the second image material. In some examples, outputting the second image data includes using at least one communication interface to cause the second image data to be sent to at least one recipient device.

In some examples, the media processing system is configured to and can identify one or more gaps in the second image material based on the one or more gaps in the second plurality of motion vectors, and upon outputting the second Before image data, the second image data is at least partially modified by filling one or more gaps in the second image data using interpolation. In some examples, the media processing system is configured to and can identify one or more gaps in the second plurality of motion vectors based on the one or more gaps in corresponding endpoints of the first plurality of motion vectors. One or more gaps are generated in the second image data, and before outputting the second image data, the first image data is modified, at least in part, by filling the one or more gaps in the second image data using interpolation. 2. Image data. Examples of gaps include gaps in the inverted MV grid 520 (and/or in the second image Img2 515) indicated by an asterisk in Figure 5.

In some examples, the media processing system is configured to and can identify one or more occlusion regions in the second image material based on one or more gaps in the second plurality of motion vectors, and upon outputting the first The second image data is modified at least in part by filling one or more gaps in the second image data using inpainting. Repair can use interpolation, machine learning, neural networks, or a combination thereof. Examples of repairs are illustrated in Figures 18, 21, 22, 28, 33, 34, 35, 36 and/or 37.

In some examples, the media processing system is configured to and can identify one or more occlusion regions in the second image material based on one or more gaps in the second plurality of motion vectors, and upon outputting the first The second image data is modified at least in part by filling one or more gaps in the second image data using one or more trained machine learning models using inpainting. Repair can use interpolation, machine learning, neural networks, or a combination thereof. Examples of repairs are illustrated in Figures 18, 21, 22, 28, 33, 34, 35, 36 and/or 37.

In some examples, the media processing system is configured to and can identify one or more of the second image material based on one or more conflicting values in the second plurality of motion vectors from the first image material. conflict, and selecting one of the one or more conflict values from the first image data based on motion data associated with the second plurality of motion vectors. An example of one or more conflicts includes a conflict at cell 8 of the inversion MV grid 520 .

In some examples, the illustration of the environment in the first image material illustrates the environment from a first perspective, and the change in perspective is a combination of the first perspective and a second perspective of the environment in the second image material. The diagram shows the corresponding changes between different viewing angles. In some examples, a first plurality of motion vectors point from a first viewing angle to different viewing angles, and a second plurality of motion vectors point from different viewing angles to the first viewing angle.

In some examples, the procedures described herein (eg, procedure 4000 and/or other procedures described herein) may be executed by a computing device or apparatus. In some examples, the procedures described herein may be performed by image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, imaging system 200, HMD 310. Mobile handheld terminal 410, reprojection and grid inversion system 2490, system of Figure 23, system of Figure 24, system of Figure 25, system of Figure 26, system of Figure 28, system of Figure 29, neural network 3900, computing system 4100, processor 4110 or a combination thereof.

Computing devices may include any suitable device, such as mobile devices (eg, cell phones), desktop computing devices, tablet computing devices, wearable devices (eg, VR headsets, AR headsets, AR glasses, connected watch or smart watch or other wearable device), server computer, autonomous vehicle or autonomous vehicle computing device, robotic device, television, and/or any other computing device with resource capabilities for executing the procedures described herein. In some cases, a computing device or apparatus may include various elements, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers , one or more cameras, one or more sensors, and/or other elements configured to perform the steps of the procedures described herein. In some examples, a computing device may include a display, a network interface configured to transmit and/or receive data, any combination thereof, and/or other elements. The network interface may be configured to transmit and/or receive Internet Protocol (IP)-based data or other types of data.

Elements of a computing device may be implemented in circuits. For example, the components may include and/or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU) and/or other suitable electronic circuits); and/or may include computer software, firmware or any combination thereof and /Or implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

The programs described herein are illustrated as logic flow diagrams, block diagrams, or conceptual diagrams, the actions of which represent sequences of operations that may be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, these operations mean computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations . Generally speaking, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and the procedures may be implemented by combining any number of the described operations in any order and/or in parallel.

Additionally, the programs described herein may execute under the control of one or more computer systems configured with executable instructions and may be implemented for execution jointly on one or more processors, by hardware, or a combination thereof Code (e.g., executable instructions, one or more computer programs, or one or more applications). As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example in the form of a computer program including a plurality of instructions executable by one or more processors. Computer-readable or machine-readable storage media may be non-transitory.

41 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, FIG. 41 illustrates an example of a computing system 4100 , which may be, for example, any computing device that constitutes an internal computing system, a remote computing system, a camera, or any element thereof, where the elements of the system are connected to each other using connection 4105 Communication. Connection 4105 may be a physical connection using a bus, or a direct connection into processor 4110 such as in a chipset architecture. Connection 4105 may also be a virtual connection, a network connection, or a logical connection.

In some embodiments, computing system 4100 is a distributed system in which the functionality described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, and the like. In some embodiments, one or more of the system elements described represent a number of such elements, each of which performs some or all of the functions described for that element. In some embodiments, the element may be a physical device or a virtual device.

The exemplary system 4100 includes at least one processing unit (CPU or processor) 4110 and a connection 4105 that will include system memory 4115 such as read only memory (ROM) 4120 and random access memory (RAM) 4125 Various system elements are coupled to processor 4110. Computing system 4100 may include a high-speed memory cache 4112 directly connected to, in close proximity to, or integrated as part of the processor 4110 .

Processor 4110 may include any general purpose processor as well as hardware services or software services (such as services 4132, 4134, and 4136 stored in storage 4130) configured to control processor 4110 and in which software instructions are incorporated into the actual processing Specialized processors in device designs. Processor 4110 may essentially be a completely independent computing system that includes multiple cores or processors, a bus, a memory controller, cache, etc. Multicore processors can be symmetric or asymmetric.

To enable user interaction, computing system 4100 includes an input device 4145, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, Voice, etc. Computing system 4100 may also include an output device 4135, which may be one or more of a number of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input/output to communicate with computing system 4100 . Computing system 4100 may include a communication interface 4140 that may generally control and manage user input and system output. Communications interfaces may use wired and/or wireless transceivers to perform or facilitate the receipt and/or sending of wired or wireless communications, including wired or wireless communications that take advantage of: audio jack/plug, microphone jack/plug, universal Serial bus (USB) port/plug, Apple® Lightning® port/plug, Ethernet port/plug, fiber optic port/plug, dedicated wired port/plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low power consumption ( BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short-range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless Area Network (WLAN) signal transmission, Visible Light Communications (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Infrared (IR) communications wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G/4G/5G/LTE cellular data network wireless signal transmission, self-organizing network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, Wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 4140 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for performing communication based on one or more signals received from one or more satellites associated with one or more GNSS systems. signals to determine the location of computing system 4100. GNSS systems include, but are not limited to, the American Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese Beidou Navigation Satellite System (BDS), and the European Galileo Global GNSS. There are no restrictions on operating on any specific hardware device, so the basic functionality here can easily be replaced with improved hardware or firmware arrangements as they are developed.

Storage device 4130 may be a non-volatile and/or non-transitory and/or computer-readable storage device and may be a hard drive or other type of computer-readable medium that may store data that may be accessed by a computer that may be read by the computer. Retrieve media such as cassette tapes, flash memory cards, solid state memory devices, digital versatile disks, tape cartridges, floppy disks, floppy disks, hard disks, tapes, magnetic strips/disks, any other magnetic storage media, fast Flash memory, memristor memory, any other solid-state memory, CD-ROM, CD-ROM, DVD, BDD , Holographic Disc, Another Optical Media, Secure Digital (SD) Card, Micro Secure Digital (microSD) Card, Memory Stick® Card, Smart Card Chip, EMV Chip, Subscriber Identity Module (SIM) Card, Mini/Micro/Nay Mi/Pico SIM card, another integrated circuit (IC) chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read only memory (ROM), programmable Read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory EPROM (FLASHEPROM), cache memory (L1/L2/L3/L4/L5/L#), resistive random access memory (RRAM/ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or tape cartridge, and/or combinations thereof.

Storage 4130 may include software services, servers, services, etc., that when code defining such software is executed by processor 4110, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include a combination of software components stored in computer-readable media and necessary hardware components (such as processor 4110, connection 4105, output device 4135, etc.) to perform Function.

The term "computer readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain or carry instructions and/or data. Computer-readable media may include non-transitory media in which data can be stored and does not include carrier waves and/or transient electronic signals that propagate wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. A computer-readable medium on which code and/or machine-executable instructions may be stored, which may represent a program, function, subroutine, routine, routine, subroutine, module, package, class, or instructions, data Any combination of structural or program statements. A code fragment may be coupled to another code fragment or hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded or sent using any suitable method, including memory sharing, message passing, token passing, network sending, etc.

In some embodiments, computer-readable storage devices, media, and memory may include wired or wireless signals including bit streams and the like. However, references to non-transitory computer-readable storage media specifically exclude media such as energy, carrier signals, electromagnetic waves and signals themselves.

Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, one of ordinary skill in the art will understand that these embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be represented as including individual functional blocks that include equipment, equipment elements, steps or routines in a method embodied in software or a combination of hardware and software. Functional blocks. Additional components other than those shown in the figures and/or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, procedures, algorithms, structures and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Various embodiments may be described above as procedures or methods illustrated as flowcharts, flowcharts, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart can describe operations as a sequential program, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. A process terminates when its operations are complete, but may have additional steps not included in the figure. Procedures can correspond to methods, functions, programs, subroutines, subroutines, etc. When a procedure corresponds to a function, its termination can correspond to the function's return to the calling function or to the main function.

Programs and methods according to the above examples may be implemented using computer-executable instructions stored in or obtainable from a computer-readable medium. Such instructions may include, for example, instructions and material that cause or otherwise configure a general purpose computer, special purpose computer or processing device to perform a particular function or group of functions. Various parts of the computer resources used can be accessed via the network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to the described examples include magnetic disks, optical disks, flash memory, and non-volatile memory. USB devices, networked storage devices, etc.

Devices implementing programs and methods according to these disclosures may include hardware, software, firmware, intermediary software, microcode, hardware description languages, or any combination thereof, and may adopt any of a variety of form factors. When implemented in software, firmware, intermediary software, or microcode, the code or code fragments used to perform the necessary tasks (e.g., a computer program product) may be stored on a computer-readable or machine-readable medium. The processor can perform necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, and the like. The functionality described in this article can also be implemented in peripheral devices or expansion cards. By further example, such functionality may also be implemented on a circuit board between different wafers or different processes executed within a single wafer.

Instructions, the media used to communicate such instructions, the computing resources used to execute them, and other structures used to support such computing resources are exemplary components for providing the functionality described herein.

In the foregoing description, various aspects of the present case are described with reference to specific embodiments of the present case, but those familiar with the art will realize that the present case is not limited thereto. Thus, while illustrative embodiments of the present invention have been described in detail herein, it is to be understood that the inventive concepts may be otherwise practiced and employed differently, and the appended claims are intended to be construed as including variations in addition to limitations subject to the prior art of this variation. The various features and aspects of the applications described above can be used individually or in combination. Furthermore, the embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. For illustrative purposes, the methods are described in a specific order. It will be understood that, in alternative embodiments, the methods may be performed in an order different than that described.

Those skilled in the art will understand that, without departing from the scope of this specification, the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≦"), respectively. ") and greater than or equal to ("≧") symbols.

Where an element is described as being "configured to" perform certain operations, such configuration may, for example, be by designing electronic circuitry or other hardware to perform the operations, by programming programmable electronic circuitry (e.g., a microprocessor) , or other suitable electronic circuit) programmed to perform the operations or any combination thereof.

The phrase "coupled to" refers to any element that is directly or indirectly physically connected to, and/or is in direct or indirect communication with, another element (e.g., via a wired or wireless connection and/or other suitable communications interface to another component).

Request language that states "at least one" of a set and/or "one or more" of a set or other language indicates that a member of the set or multiple members of the set (in any combination) satisfy the claim . For example, claim language stating "at least one of A and B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. "At least one" of a collection and/or "one or more" of a collection does not limit the collection to the items listed in the collection. For example, claim language stating "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been briefly described above as to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handheld terminals, or integrated circuit devices having a variety of uses, including in wireless communication device handheld terminals and other devices. Application. Any features described as modules or elements may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the technology may be implemented, at least in part, by a computer-readable data storage medium including program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. Computer readable media may include memory or data storage media such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory Memory (NVRAM), electronically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, technology may be implemented, at least in part, by a computer-readable communication medium, such as a propagated signal or wave, that carries or transfers program code in the form of instructions or data structures, and may be implemented by a computer or other processors to access, read and/or execute.

The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable Logic array (FPGA) or other equivalent integrated or discrete logic circuit system. Such processors may be configured to perform any of the techniques described in this case. A general purpose processor may be a microprocessor; but alternatively, the processor may be any conventional processor, controller, microcontroller or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term "processor" as used herein may represent any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided in dedicated software modules or hardware modules configured for encoding and decoding or incorporated into a combined video transcoder-decoder (CODEC) within.

Illustrative aspects of this case include:

Aspect 1A. An image processing device, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: e.

Aspect 2A. The apparatus of aspect 1A, wherein the second image material includes an interpolated image configured to illustrate the environment at a second time between the first time and a third time, wherein the The first image material includes at least one image illustrating the environment at at least one of the first time or the third time.

Aspect 3A. The device of any one of Aspects 1A to 2A, wherein the first image data includes a plurality of video data frames including parallax motion, and wherein the second image data includes the plurality of video data frames Stable variant of the data frame that reduces parallax movement.

Aspect 4A. The device of any one of Aspects 1A to 3A, wherein the first image data includes a person viewing the image sensor from a first angle, and wherein the second image data includes the person The image sensor is viewed from a second angle different from the first angle.

Aspect 5A. The device of any one of aspects 1A to 4A, wherein the change in viewing angle includes rotation of viewing angle as a function of angle and about an axis.

Aspect 6A. The device of any one of aspects 1A to 5A, wherein the viewing angle change includes viewing angle translation as a function of direction and distance.

Aspect 7A. The device of any one of aspects 1A to 6A, wherein the change in viewing angle includes a transformation.

Aspect 8A. The device of any one of Aspects 1A to 7A, wherein the change in perspective includes an original perspective along an axis of the illustration of the environment in the first image data and objects in the environment. Movement between locations in which at least a portion of the object is illustrated in the first image data.

Aspect 9A. The apparatus of any one of aspects 1A-8A, wherein the at least one processor is configured to identify the second image based on one or more gaps in the second plurality of motion vectors. one or more gaps in the second image data; and before outputting the second image data, modify the second image at least in part by filling one or more gaps in the second image data using interpolation material.

Aspect 10A. The apparatus of any one of aspects 1A-9A, wherein the at least one processor is configured to identify the second image based on one or more gaps in the second plurality of motion vectors. one or more occluded areas in the image data; and before outputting the second image data, modify the second image at least in part by filling one or more gaps in the second image data using inpainting material.

Aspect 11A. The apparatus of any one of aspects 1A-10A, wherein the at least one processor is configured to identify the second image based on one or more gaps in the second plurality of motion vectors. one or more occluded areas in the image data; and before outputting the second image data, at least partially filling in one or more occluded areas in the second image data by using inpainting using one or more trained machine learning models. or multiple gaps to modify the second image data.

Aspect 12A. The apparatus of any one of aspects 1A to 11A, wherein the at least one processor is configured to: based on one or more of the second plurality of motion vectors from the first image data conflict values to identify one or more conflicts in the second image data; and selecting the one or more conflict values from the first image data based on motion data associated with the second plurality of motion vectors one of them.

Aspect 13A. The device of any one of aspects 1A to 12A, wherein the depth information includes a three-dimensional representation of the environment from the first perspective.

Aspect 14A. The device of any one of aspects 1A to 13A, wherein the depth data is received from at least one depth sensor.

Aspect 15A. The apparatus of any one of aspects 1A to 14A, further comprising: a display, wherein to output the second image data, the at least one processor is configured to display the second image data using at least the display. Second image data.

Aspect 16A. The device of any one of aspects 1A to 15A, further comprising: a communication interface, wherein in order to output the second image data, the at least one processor is configured to use at least the communication interface to At least the second image material is sent to at least the recipient device.

Aspect 17A. The device of any one of Aspects 1A to 16A, wherein the device includes at least one of a head-mounted display (HMD), a mobile handheld terminal, or a wireless communication device.

Aspect 18A. The device of any one of aspects 1A to 17A, wherein the illustration of the environment in the first image data illustrates the environment from a first perspective, wherein the change in perspective is the first perspective Changes between different viewing angles corresponding to the second representation of the environment in the second image material.

Aspect 19A. The device of any one of aspects 1A to 18A, wherein the change in perspective includes at least one of a parallax shift of perspective or a rotation of perspective about an axis, wherein the at least one processor is configured to: One of the following is received via the user interface: an indication of a distance of disparity movement of the viewing angle, or an indication of a rotation angle or axis of the viewing angle.

Aspect 20A. The apparatus of any one of aspects 1A to 19, wherein the at least one processor is configured to identify, based on one or more gaps in corresponding endpoints of the first plurality of motion vectors, One or more gaps in the second plurality of motion vectors create one or more gaps in the second image data; and before outputting the second image data, filling the second plurality of motion vectors at least partially by using interpolation. one or more gaps in the second image data to modify the second image data.

Aspect 21A. An image processing method, the method comprising: receiving depth data including depth information corresponding to the environment; receiving first image data captured by an image sensor, the first image data including an illustration of the environment; generating, based on at least the depth data, a first plurality of motion vectors corresponding to changes in perspective of the illustration of the environment in the first image data; using the first plurality of motion vectors based on the first plurality of motion vectors Grid inversion generates a second plurality of motion vectors indicating corresponding distances that corresponding primitives of the representation of the environment in the first image material move in response to the change in viewing angle; at least in part Second image data is generated by modifying the first image data based on the second plurality of motion vectors, wherein the second image data includes a second view of the environment from a different perspective than the first image data. icon; and output the second image data.

Aspect 22A. The method of aspect 21A, wherein the second image material includes an interpolated image configured to illustrate the environment at a second time between the first time and a third time, wherein the The first image material includes at least one image illustrating the environment at at least one of the first time or the third time.

Aspect 23A. The method of any one of aspects 21A to 22A, wherein the first image data includes a plurality of video data frames including parallax motion, and wherein the second image data includes the plurality of video data frames Stable variant of the data frame that reduces parallax movement.

Aspect 24A. The method of any one of aspects 21A to 23A, wherein the first image data includes a person viewing the image sensor from a first angle, and wherein the second image data includes the person The image sensor is viewed from a second angle different from the first angle.

Aspect 25A. The method of any one of aspects 21A to 24A, wherein the viewing angle change includes a viewing angle rotation according to an angle and about an axis.

Aspect 26A. The method of any one of aspects 21A to 25A, wherein the viewing angle change includes viewing angle translation as a function of direction and distance.

Aspect 27A. The method of any one of aspects 21A to 26A, wherein the change in viewing angle includes a transformation.

Aspect 28A. The method of any one of aspects 21A to 27A, wherein the change in perspective includes an original perspective along an axis of the illustration of the environment in the first image data and objects in the environment. Movement between locations in which at least a portion of the object is illustrated in the first image data.

Aspect 29A. The method of any one of aspects 21A to 28A, further comprising identifying one or more of the second image data based on one or more gaps in the second plurality of motion vectors. a plurality of gaps; and before outputting the second image data, at least partially modifying the second image data by filling one or more gaps in the second image data using interpolation.

Aspect 30A. The method of any one of aspects 21A-29A, further comprising identifying one or more of the second image data based on one or more gaps in the second plurality of motion vectors. a plurality of occluded regions; and before outputting the second image data, modifying the second image data at least in part by filling one or more gaps in the second image data using inpainting.

Aspect 31A. The method of any one of aspects 21A to 30A, further comprising identifying one or more of the second image data based on one or more gaps in the second plurality of motion vectors. a plurality of occluded regions; and modifying, at least in part, by using inpainting to fill one or more gaps in the second image data using one or more trained machine learning models before outputting the second image data. the second image data.

Aspect 32A. The method of any one of aspects 21A-31A, further comprising: identifying the third motion vector based on one or more conflict values from the first image data in the second plurality of motion vectors. one or more conflicts in two image data; and selecting one of the one or more conflict values from the first image data based on motion data associated with the second plurality of motion vectors.

Aspect 33A. The method of any one of aspects 21A to 32A, wherein the depth information includes a three-dimensional representation of the environment from the first perspective.

Aspect 34A. The method of any one of aspects 21A to 33A, wherein the depth data is received from at least one depth sensor.

Aspect 35A. The method of any one of aspects 21A to 34A, wherein outputting the second image material includes using at least a display to display the second image material.

Aspect 36A. The method of any one of aspects 21A to 35A, wherein outputting the second image data includes causing the second image data to be sent to at least one recipient device using at least one communication interface.

Aspect 37A. The method of any one of aspects 21A to 36A, wherein the method is performed using a device including at least one of a head mounted display (HMD), a mobile handheld terminal, or a wireless communication device .

Aspect 38A. The method of any one of Aspects 21A to 37A, wherein the illustration of the environment in the first image data illustrates the environment from a first perspective, wherein the change in perspective is the first perspective Changes between different viewing angles corresponding to the second representation of the environment in the second image material.

Aspect 39A. The method of any one of aspects 21A to 38A, wherein the change in viewing angle includes at least one of a parallax shift of the viewing angle or a rotation of the viewing angle about an axis, the method further comprising: receiving, via a user interface Either: an indication of the distance by which the parallax of the viewing angle has moved, or an indication of the angle or axis of rotation of the viewing angle.

Aspect 40A. The method of any one of aspects 21A-39A, further comprising identifying the second plurality of motions based on one or more gaps in corresponding endpoints of the first plurality of motion vectors. One or more gaps in the vector create one or more gaps in the second image data; and before outputting the second image data, at least partially filling in the second image data by using interpolation one or more gaps to modify the second image data.

Aspect 41A: A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive depth information including depth corresponding to the environment Depth data of information; receiving first image data captured by an image sensor, the first image data including an illustration of the environment; based on at least the depth data, generating information related to the first image data A first plurality of motion vectors corresponding to changes in the viewing angle of the illustration of the environment; using grid inversion based on the first plurality of motion vectors to generate a second plurality of motion vectors, the second plurality of motion vectors indicating the A corresponding distance that a corresponding primitive of the representation of the environment in the first image data moves in response to the change in viewing angle; generating the second image data at least in part by modifying the first image data based on the second plurality of motion vectors. Image data, wherein the second image data includes a second representation of the environment from a different perspective than the first image data; and outputting the second image data.

Aspect 42A: The non-transitory computer-readable medium of aspect 41A, further comprising the operations of any one of aspects 2A to 20A and/or any one of aspects 22A to 40A.

Aspect 43A: An image processing device, the device comprising: means for receiving first image data captured by an image sensor, the first image data including an illustration of the environment; Means for at least the depth data generating a first plurality of motion vectors corresponding to changes in perspective of the representation of the environment in the first image data; for using grid inversion based on the first plurality of motion vectors means for generating a second plurality of motion vectors indicative of respective distances that corresponding primitives of the representation of the environment in the first image material move in response to the change in viewing angle; for at least partially Means for generating second image data by modifying the first image data based on the second plurality of motion vectors, wherein the second image data includes a view of the environment from a different perspective than the first image data. a second illustration; and a component for outputting the second image data.

Aspect 44A: The apparatus of aspect 43A, further comprising means for performing the operations of any one of aspects 2A-20A and/or any one of aspects 22A-40A.

Aspect 1B. An image processing device, the device comprising: at least one memory; and one or more processors, the one or more processors coupled to the at least one memory, the one or more processors configured for: receiving depth data captured by a depth sensor, the depth data including a three-dimensional representation of the environment as viewed from a first perspective; based on at least the depth data, determining to correspond to a change from the first perspective to a second perspective a first plurality of motion vectors; receiving first image data captured by an image sensor, the first image data illustrating the environment from a third perspective; using a grid based on the first plurality of motion vectors Inverting to determine a second plurality of motion vectors corresponding to the change from the third viewing angle to the fourth viewing angle; generating a second plurality of motion vectors at least in part by modifying the first image data based on the second plurality of motion vectors. image data, the second image data illustrating the environment from the fourth perspective; and outputting the second image data.

Aspect 2B. The apparatus of aspect 1B, wherein the second image material includes an interpolated image configured to illustrate the environment at a second time between the first time and a third time, wherein the The first image material includes a first image illustrating the environment at the first time and a second image illustrating the environment at the third time.

Aspect 3B. The apparatus of any one of Aspects 1B to 2B, wherein the first image data includes video data including parallax movement, and wherein the second image data includes the video data without the parallax movement. stable variant.

Aspect 4B. The device of any one of aspects 1B to 3B, wherein the first image data includes an illustration of a person viewing the image sensor from a first angle, and wherein the second image data includes The figure shows the person viewing the image sensor from a second angle different from the first angle.

Aspect 5B. The device of any one of aspects 1B to 4B, wherein the fourth viewing angle is the first viewing angle.

Aspect 6B. The device of any one of aspects 1B to 5B, wherein the fourth viewing angle is the second viewing angle.

Aspect 7B. The device of any one of aspects 1B to 6B, wherein the change from the first viewing angle to the second viewing angle includes a viewing angle rotation according to an angle, wherein from the third viewing angle to the fourth viewing angle The changes include rotation of the perspective based on that angle.

Aspect 8B. The device of any one of aspects 1B to 7B, wherein the change from the first viewing angle to the second viewing angle includes a viewing angle translation as a function of direction and distance, wherein the change from the third viewing angle to the third viewing angle The change of the four viewing angles includes a translation of the viewing angle according to the direction and the distance.

Aspect 9B. The device of any one of aspects 1B to 8B, wherein the change from the first viewing angle to the second viewing angle includes a transformation, wherein the change from the third viewing angle to the fourth viewing angle includes the Transform.

Aspect 10B. The method of any one of aspects 1B-9B, wherein the one or more processors are configured to identify the first plurality of motion vectors based on one or more gaps. one or more gaps in the second image data; and before outputting the second image data, modify the second image data at least in part by filling one or more gaps in the second image data using interpolation. Image data.

Aspect 11B. The method of any one of aspects IB-10B, wherein the one or more processors are configured to identify the first plurality of motion vectors based on one or more gaps. one or more occluded areas in the second image data; and before outputting the second image data, modify the second image data at least in part by filling one or more gaps in the second image data using inpainting. Image data.

Aspect 12B. An image processing method, the method comprising: receiving depth data captured by a depth sensor, the depth data including a three-dimensional representation of the environment as viewed from a first perspective; and based on at least the depth data, determining and selecting from the depth data. A first plurality of motion vectors corresponding to the change from the first perspective to the second perspective; receiving first image data captured by the image sensor, the first image data illustrating the environment from a third perspective; using grid inversion based on the first plurality of motion vectors to determine a second plurality of motion vectors corresponding to the change from the third viewing angle to the fourth viewing angle; at least in part by determining based on the second plurality of motion vectors modifying the first image data to generate second image data illustrating the environment from the fourth perspective; and outputting the second image data.

Aspect 13B. The method of aspect 12B, wherein the second image material includes an interpolated image configured to illustrate the environment at a second time between the first time and a third time, wherein the The first image material includes a first image illustrating the environment at the first time and a second image illustrating the environment at the third time.

Aspect 14B. The method of any one of aspects 12B to 13B, wherein the first image data includes video data including parallax motion, and wherein the second image data includes the video data without the parallax motion. stable variant.

Aspect 15B. The method of any one of aspects 12B to 14B, wherein the first image data includes an illustration of a person viewing the image sensor from a first angle, and wherein the second image data includes The figure shows the person viewing the image sensor from a second angle different from the first angle.

Aspect 16B. The method of any one of aspects 12B to 15B, wherein the fourth viewing angle is the first viewing angle.

Aspect 17B. The method of any one of aspects 12B to 16B, wherein the fourth viewing angle is the second viewing angle.

Aspect 18B. The method of any one of aspects 12B to 17B, wherein the change from the first viewing angle to the second viewing angle includes a viewing angle rotation according to an angle, wherein from the third viewing angle to the fourth viewing angle The changes include rotation of the perspective based on that angle.

Aspect 19B. The method of any one of aspects 12B to 18B, wherein the change from the first viewing angle to the second viewing angle includes a viewing angle translation as a function of direction and distance, wherein the change from the third viewing angle to the third viewing angle The change of the four viewing angles includes a translation of the viewing angle according to the direction and the distance.

Aspect 20B. The method of any one of aspects 12B to 19B, wherein the change from the first perspective to the second perspective includes a transformation, wherein the change from the third perspective to the fourth perspective includes the Transform.

Aspect 21B. The method of any one of aspects 12B to 20B, further comprising: identifying one or more of the second image data based on one or more gaps in the second plurality of motion vectors. a plurality of gaps; and modifying the second image data at least in part by filling one or more gaps in the second image data using interpolation before outputting the second image data.

Aspect 22B. The method of any one of aspects 12B to 21B, further comprising: identifying one or more of the second image data based on one or more gaps in the second plurality of motion vectors. a plurality of occluded regions; and before outputting the second image data, modifying the second image data at least in part by filling one or more gaps in the second image data using inpainting.

Aspect 23B. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform any of Aspects 1B through 22B. one of the operations described.

Aspect 24B. An image processing device comprising one or more means for performing the operation according to any one of aspects 1B to 22B.

100:Image capture and processing system 105A:Image capture device 105B:Image processing equipment 110: Scene 115: Lens 120:Control mechanism 125A: Exposure control mechanism 125B: Focus on control mechanism 125C:Zoom control mechanism 130:Image sensor 140: Random access memory (RAM) 145: Read-only memory (ROM) 150:Image processor 152: Host processor 154:ISP 156: Input/output (I/O) port 160: Input/output (I/O) device 200:Imaging system 205: Sensor 207:Virtual content generator 210:Application Engine 215:Image reprojection engine 220: Motion vector engine 225:Grid inversion engine 230:Time Warp Engine 235: Depth sensor support engine 240:3D stabilization engine 245:3D zoom engine 250:Reprojection SAT engine 255:Head posture correction engine 260: Extended Reality (XR) post-reprojection engine 265:Special effects engine 270:Output device 275:Transceiver 280:Feedback engine 285:Media materials 290: Modified media information 300:Perspective 310:HMD 320:User 330A:The first camera 330B: Second camera 330C: Third camera 330D: fourth camera 335:Earplugs 340:Display 350:Perspective 400:Perspective 410: Mobile handheld terminal 420: Front surface 430A:First camera 430B: Second camera 430C: Third camera 430D: The fourth camera 435A: Speaker 435B: Speaker 440:Display 450:Perspective 460:Rear surface 505: Motion Vector (MV) Grid 510: First image Img1 515: Second image Img2 520:Inversion MV grid 600:Concept map 605: World scene 610:Camera image 615:Reproject image 620:In-depth information 700:Concept map 705: Time Warp 710: First image 715: Second image 720: Motion vector diagram 800:Concept map 810: A set of image sensors 815: A set of depth sensors 820:Offset 900:Concept map 905:3D stabilization 1000:Concept map 1005:3D zoom 1020:In-depth information 1100:Concept map 1105:Reprojection 1110: Telephoto sensor 1115:Wide angle sensor 1120:Offset 1125: Sensor 1130: Telephoto image 1135:Wide angle image 1140:Telephoto image 1160:In-depth information 1200:Concept map 1205: Head posture correction 1210:Input image 1215:Reproject image 1220:In-depth information 1300:Concept map 1305:XR post-reprojection 1310:Input image 1315:Input image 1320:HMD 1400:Concept map 1405:Special effects 1410:Input image 1415:Image 1500:Concept map 1510: Captured image 1515:Reprojected image 1600:Block diagram 1605:3D transformation 1610:Block diagram 1615:MV calculation 1620: Motion vector (MV) 1625:Grid inversion 1630:Invert motion vector 1700:Block diagram 1705:Warp Engine 1710: Captured image 1715:Reprojected image 1730:Reverse MV 1800:Concept map 1805:Reprojected image 1810: Occlusion map 1815: Repaired image 1825:Zoom image 1830: Repaired image 1835: Occluded area 1900:Block diagram 1905: Reprojection and grid inversion system 1910:MV Grid 1915:First option 1920:Second option 1930:arrow 1935:Arrow 1940: Graph/Arrow 2000:Concept map 2010:Graphics 2015:Graphics 2020:Graphics 2025: Main travel engine 2030: Triangle Control Engine 2035: Graph element interpolation engine 2100:Concept map 2110:Image 2115:Reprojected image 2200:Concept map 2220:Process 2300:Concept map 2400:Block diagram 2405:Camera 2410:Image sensor 2415: Dynamic Random Access Memory (DRAM) 2420: Optical flow engine 2425:Grid inversion engine 2430:Warp Engine 2440:Image processing engine 2500:Block diagram 2505: Temporal deblurring engine 2520: Further transformation 2535:Reprojection engine 2600:Block diagram 2700:Concept map 2705: Projection alignment 2710:Alignment 2800:Block diagram 2900:Concept map 3000:Concept map 3005: Additional examples 3100:Concept map 3105:Digital zoom 3110: Illustration 3200:Concept map 3300:Concept map 3305:Offset distance 3310:Offset angle 3315:In-depth information 3320: Occlusion map 3400:Concept map 3500:Concept map 3505: Occluded area 3605: Occluded area 3700:Concept map 3800:Concept map 3900:Neural Network 3910:Input layer 3912A:Hidden layer 3912B:Hidden layer 3912N:Hidden layer 3914:Output layer 3916:node 4000:Program 4005: Operation 4010: Operation 4015:Operation 4020: Operation 4025: Operation 4030: Operation 4100:Computing Systems 4105:Connect 4110: Processor 4112: cache memory 4115:System memory 4120: Read-only memory (ROM) 4125: Random access memory (RAM) 4130:Storage device 4132:Service 4134:Service 4135:Output device 4136:Service 4140: Communication interface 4145:Other input devices RGB:Visual RGB3: Main image sensor

Illustrative embodiments of the present case are described in detail below with reference to the following figures:

1 is a block diagram illustrating an exemplary architecture of an image capture and processing system according to some examples;

2 is a block diagram illustrating an exemplary architecture of an imaging system for performing reprojection operations for various applications, according to some examples;

3A is a perspective view illustrating a head-mounted display (HMD) used as an extended reality (XR) system, according to some examples;

3B is a perspective view illustrating the head mounted display (HMD) of FIG. 3A worn by a user, according to some examples;

4A is a perspective view illustrating the front surface of a mobile handheld terminal that includes a front-facing camera and may be used as an extended reality (XR) system, according to some examples;

4B is a perspective view illustrating the rear surface of a mobile handheld terminal that includes a rear camera and may be used as an extended reality (XR) system, according to some examples;

Figure 5 is a block diagram illustrating an example of grid inversion according to some examples;

6 is a conceptual diagram illustrating an example of depth-based reprojection according to some examples;

7 is a conceptual diagram illustrating an example of time warping performed by a time warping engine, according to some examples;

8 is a conceptual diagram illustrating an example of depth sensor support performed by a depth sensor support engine, according to some examples;

9 is a conceptual diagram illustrating an example of 3D stabilization performed by a 3D stabilization engine, according to some examples;

10 is a conceptual diagram illustrating an example of 3D zoom (or movie zoom) performed by a 3D zoom engine, according to some examples;

11 is a conceptual diagram illustrating an example of reprojection performed by a reprojection SAT engine, according to some examples;

12 is a conceptual diagram illustrating an example of head posture correction performed by a head posture correction engine, according to some examples;

13 is a conceptual diagram illustrating an example of XR post-reprojection performed by an XR post-reprojection engine, according to some examples;

14 is a conceptual diagram illustrating an example of special effects performed by a special effects engine, according to some examples;

Figure 15 is a conceptual diagram illustrating an image reprojection transformation based on matrix operations according to some examples;

Figure 16 is a block diagram illustrating depth data-based mesh inversion transformation and 3D transformation according to some examples;

17 is a block diagram illustrating a motion vector-based image reprojection transformation according to some examples;

Figure 18 is a conceptual diagram illustrating an example of repair to address occlusions according to some examples;

Figure 19 is a block diagram illustrating the architecture of a reprojection and grid inversion system according to some examples;

20 is a conceptual diagram illustrating an example of a triangular walking operation according to some examples;

Figure 21 is a conceptual diagram illustrating an example of occlusion masking according to some examples;

22 is a conceptual diagram illustrating an example of hole filling according to some examples;

23 is a conceptual diagram illustrating additional examples of time warping performed by a time warping engine, according to some examples;

24 is a block diagram illustrating an exemplary architecture for a time warp engine in some examples, according to some examples;

25 is a block diagram illustrating an exemplary architecture for a time warp engine with temporal deblurring in some examples, according to some examples;

26 is a block diagram illustrating an exemplary architecture of a depth sensor support engine for a time-of-flight (ToF) sensor, according to some examples;

27 is a conceptual diagram illustrating additional examples of depth sensor support performed by a depth sensor support engine, according to some examples;

28 is a block diagram illustrating an exemplary architecture of an imaging system including an image reprojection engine and/or a 3D stabilization engine, according to some examples;

29 is a conceptual diagram illustrating additional examples of time warping performed using a time warping engine compared to images processed without a time warping engine, according to some examples;

30 is a conceptual diagram illustrating additional examples of 3D stabilization performed by a 3D stabilization engine, according to some examples;

31 is a conceptual diagram illustrating additional examples of 3D zoom (or movie zoom) performed by a 3D zoom engine, according to some examples;

32 is a conceptual diagram illustrating additional examples of reprojection performed by a reprojection SAT engine, according to some examples;

33 is a conceptual diagram illustrating additional examples of head pose correction performed by a head pose correction engine, according to some examples;

34 is a conceptual diagram illustrating additional examples of grid inversion according to some examples;

35 is a conceptual diagram illustrating an example of the use of deep learning-based repair according to some examples;

36 is a conceptual diagram illustrating an example of the use of inpainting without deep learning, according to some examples;

37 is a conceptual diagram illustrating an example of the use of edge filters and depth filters on edges, according to some examples;

38 is a conceptual diagram illustrating an example of reprojection according to some examples;

39 is a block diagram illustrating an example of a neural network that may be used for media processing operations, according to some examples;

Figure 40 is a flowchart illustrating a media processing routine according to some examples; and

41 is a diagram illustrating an example of a computing system for implementing certain aspects described herein.

Domestic storage information (please note in order of storage institution, date and number) without Overseas storage information (please note in order of storage country, institution, date, and number) without

100:Image capture and processing system

105A:Image capture device

105B:Image processing equipment

110: Scene

115: Lens

120:Control mechanism

125A: Exposure control mechanism

125B: Focus on control mechanism

125C:Zoom control mechanism

130:Image sensor

140: Random Access Memory (RAM)

145: Read-only memory (ROM)

150:Image processor

152: Host processor

154:ISP

160: Input/output (I/O) device

Claims (30)

An image processing device, the device includes: at least one memory; and At least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: receiving depth data including depth information corresponding to an environment; receiving first image data captured by an image sensor, the first image data including an illustration of the environment; Generating a first plurality of motion vectors corresponding to a change in viewing angle of the representation of the environment in the first image data based on at least the depth data; Grid inversion is used to generate a second plurality of motion vectors based on the first plurality of motion vectors, the second plurality of motion vectors indicating that corresponding primitives of the representation of the environment in the first image data are for the The corresponding distance moved when the perspective changes; Second image data is generated at least in part by modifying the first image data based on the second plurality of motion vectors, wherein the second image data includes an image of the first image data from a different perspective than the first image data. a second representation of the environment; and Output the second image data. The apparatus of claim 1, wherein the second image data includes an interpolated image configured to illustrate the environment at a second time between a first time and a third time, wherein the first The image material includes at least one image illustrating the environment at at least one of the first time or the third time. The device according to claim 1, wherein the first image data includes a plurality of video data frames including a parallax movement, and wherein the second image data includes a plurality of video data frames that reduces the parallax movement. Stable variant. The device according to claim 1, wherein the first image data includes a person viewing the image sensor from a first angle, and wherein the second image data includes the person viewing the image sensor from a second angle different from the first angle. View the image sensor at an angle. The device of claim 1, wherein a change in viewing angle includes a rotation of viewing angle according to an angle and about an axis. The device according to claim 1, wherein a viewing angle change includes a viewing angle translation according to a direction and a distance. The device of claim 1, wherein a change in viewing angle includes a transformation. The apparatus of claim 1, wherein the change in perspective includes a change along an axis between an original perspective of the representation of the environment in the first image data and a position of an object in the environment. Movement, wherein at least a portion of the object is illustrated in the first image material. The device according to claim 1, wherein the at least one processor is configured to: identifying one or more gaps in the second image data based on one or more gaps in the second plurality of motion vectors; and Before outputting the second image data, the second image data is modified at least in part by filling one or more gaps in the second image data using interpolation. The device according to claim 1, wherein the at least one processor is configured to: identifying one or more occlusion regions in the second image data based on one or more gaps in the second plurality of motion vectors; and Before outputting the second image data, the second image data is modified at least in part by filling one or more gaps in the second image data using repair. The device according to claim 1, wherein the at least one processor is configured to: identifying one or more occlusion regions in the second image data based on one or more gaps in the second plurality of motion vectors; and Modifying the second image data at least in part by filling one or more gaps in the second image data using one or more trained machine learning models before outputting the second image data. . The device according to claim 1, wherein the at least one processor is configured to: identifying one or more conflicts in the second image data based on one or more conflict values in the second plurality of motion vectors from the first image data; and One of the one or more conflict values is selected from the first image data based on motion data associated with the second plurality of motion vectors. The device of claim 1, wherein the depth information includes a three-dimensional representation of an environment from a first perspective. The device of claim 1, wherein the depth data is received from at least one depth sensor. The device according to claim 1, further comprising: A display, wherein to output the second image data, the at least one processor is configured to display the second image data using at least the display. The device according to claim 1, further comprising: A communication interface, wherein in order to output the second image data, the at least one processor is configured to send at least the second image data to at least one recipient device using at least the communication interface. The device according to claim 1, wherein the device includes at least one of a head-mounted display (HMD), a mobile handheld terminal or a wireless communication device. An image processing method, the method includes: receiving depth data including depth information corresponding to an environment; receiving first image data captured by an image sensor, the first image data including an illustration of the environment; generating a first plurality of motion vectors corresponding to a change in viewing angle of the representation of the environment in the first image data based on at least the depth data; Grid inversion is used to generate a second plurality of motion vectors based on the first plurality of motion vectors, the second plurality of motion vectors indicating the corresponding primitives of the representation of the environment in the first image data for The corresponding distance moved by the change in perspective; Second image data is generated at least in part by modifying the first image data based on the second plurality of motion vectors, wherein the second image data includes an image of the first image data from a different perspective than the first image data. a second representation of the environment; and Output the second image data. The method of claim 18, wherein the second image data includes an interpolated image configured to illustrate the environment at a second time between a first time and a third time, wherein the first The image material includes at least one image illustrating the environment at at least one of the first time or the third time. The method of claim 18, wherein the first image data includes a plurality of video data frames including a parallax movement, and wherein the second image data includes a stabilization of the plurality of video data frames that reduces the parallax movement. Variants. The method of claim 18, wherein the first image data includes a person viewing the image sensor from a first angle, and wherein the second image data includes the person viewing the image sensor from a second angle different from the first angle. View the image sensor at an angle. The method of claim 18, wherein a viewing angle change includes a viewing angle rotation according to an angle and about an axis. The method of claim 18, wherein a viewing angle change includes a viewing angle translation according to a direction and a distance. According to the method of claim 18, wherein a change of perspective includes a transformation. The method of claim 18, wherein the change of perspective includes movement along an axis between an original perspective of the representation of the environment in the first image data and a position of an object in the environment , wherein at least a portion of the object is illustrated in the first image data. According to the method of claim 18, it further includes: identifying one or more gaps in the second image data based on one or more gaps in the second plurality of motion vectors; and Before outputting the second image data, the second image data is modified at least in part by filling one or more gaps in the second image data using interpolation. According to the method of claim 18, it further includes: identifying one or more occlusion regions in the second image data based on one or more gaps in the second plurality of motion vectors; and Before outputting the second image data, the second image data is modified at least in part by filling one or more gaps in the second image data using repair. According to the method of claim 18, it further includes: identifying one or more occlusion regions in the second image data based on one or more gaps in the second plurality of motion vectors; and Modifying the second image data at least in part by filling one or more gaps in the second image data using one or more trained machine learning models before outputting the second image data. . According to the method of claim 18, it further includes: identifying one or more conflicts in the second image data based on one or more conflict values in the second plurality of motion vectors from the first image data; and One of the one or more conflict values is selected from the first image data based on motion data associated with the second plurality of motion vectors. The method of claim 18, wherein outputting the second image data includes using at least one display to display the second image data.