WO2019019172A1

WO2019019172A1 - Adaptive Image Processing in a Robotic Vehicle

Info

Publication number: WO2019019172A1
Application number: PCT/CN2017/094965
Authority: WO
Inventors: Yin Huang; Xiaoyi Zhu; Leslie Chiming Chan; Ruowei WANG
Original assignee: Qualcomm Incorporated
Priority date: 2017-07-28
Filing date: 2017-07-28
Publication date: 2019-01-31

Abstract

Embodiments include devices and methods for adaptive image processing in a robotic vehicle. A robotic vehicle processor may capture an image by an image sensor of the robotic vehicle. The processor may perform lens distortion correction on center pixels of each line of the captured image to produce estimated movement information. The processor may generate a pixel map, wherein the pixel map comprises a mapping of input pixel positions to output pixel positions based, at least in part, on the estimated movement information. The processor may perform lens distortion correction on the pixels based, at least in part, on the estimated movement information to produce pixel movement information. The processor may perform rolling shutter corrections on the pixels based, at least in part, on the pixel movement information to produce corrected pixels. The processor may generate a corrected image using the corrected pixels and the generated pixel map.

Description

Adaptive Image Processing in a Robotic Vehicle

BACKGROUND

Robotic vehicles, such as unmanned autonomous vehicles (UAVs) , are being developed for a wide range of applications. UAVs are typically equipped with one or more sensors, such as cameras capable of capturing an image, a sequence of images, or video. However, motions of the robotic vehicle may cause an unacceptably distorted or wobbly image or video.

Image stabilization (IS) refers to the process of detecting and correcting spurious motion introduced due to camera shake during the capture of an image or video. In the most general sense, spurious global motion may include any deviation from and intended camera path and jitter introduced due to unintended camera movement.

A variety of mechanical image stabilization mechanisms and techniques are available. However, such mechanisms are typically too heavy and too expensive for incorporation into and use with most UAVs.

SUMMARY

Various embodiments include methods, as well as robotic vehicles and processing devices executing such methods, for adaptive processing of an image captured by an image sensor of a robotic vehicle. Various embodiments may include capturing an image by an image sensor of the robotic vehicle, performing lens distortion correction on center pixels of each line of the captured image to produce estimated movement information, generating a pixel map in which the pixel map may include a mapping of input pixel positions to output pixel positions based, at least in part, on the estimated movement information, performing lens distortion correction on the pixels based, at least in part, on the estimated movement information to produce pixel movement information, performing rolling shutter corrections on the pixels based, at least in part, on the pixel movement information to produce corrected pixels, and generating a corrected image using the corrected pixels and the generated pixel map.

In some embodiments, performing lens distortion correction on the pixels based, at least in apart, on the estimated movement information may include determining an image sensor pose matrix, and performing a single step correction and stabilization of the pixels using the image sensor pose matrix and the estimated movement information.

In some embodiments, performing rolling shutter corrections on the pixels to produce a transformation matrix based, at least in part, on the pixel movement information may include determining an estimated yaw, estimated pitch, and estimated roll of a center pixel, determining an estimated yaw, estimated pitch, and estimated roll of all the pixels other than the center pixel, determining a yaw difference, a pitch difference, and a roll difference based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels, and modifying each pixel based, at least in part, on the yaw difference, the pitch difference, and the roll difference. In such embodiments, performing lens distortion correction and performing rolling shutter corrections may be performed by a first processing unit of the processing device, and modifying each pixel may be performed by a second processing unit of the processing device.

In some embodiments, performing rolling shutter corrections on the pixels based, at least in part, on the pixel movement information may include determining an estimated yaw, estimated pitch, and estimated roll of a center pixel of a tile of the captured image, determining an estimated yaw, estimated pitch, and estimated roll of all the pixels of the tile other than the center pixel, determining a yaw difference, a pitch difference, and a roll difference based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels of the tile, and modifying each tile based, at least in part, on the yaw difference, the pitch difference, and the roll difference. In such embodiments, performing lens distortion correction and performing rolling shutter corrections may be performed by a first processing unit of the processing device, and modifying each tile may be performed by a second processing unit of the processing device.

In some embodiments, performing lens distortion correction on the pixels and performing rolling shutter corrections may be performed in parallel.

In some embodiments, generating a corrected image using the corrected pixels and the generated pixel map may include executing interpolation on the corrected pixels, and generating the corrected image using interpolated corrected pixels and the generated pixel map.

Further embodiments may include a robotic vehicle having an image sensor and a processor configured to perform operations of the methods summarized above. Further embodiments may include a processing device for use in a robotic vehicle having an image sensor in which the processing device is configured to perform operations of the methods summarized above. Further embodiments may include a robotic vehicle having means for performing functions of the methods summarized above.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate example embodiments, and together with the general description given above and the detailed description given below, serve to explain the features of various embodiments.

FIGS. 1A and 1B are system block diagrams of robotic vehicles operating within communication system according to various embodiments.

FIG. 2 is a component block diagram illustrating components of a robotic vehicle according to various embodiments.

FIG. 3 is a component block diagram illustrating a processing device suitable for use in robotic vehicles implementing various embodiments.

FIG. 4 is a component block diagram illustrating components of an image capture and processing system of a robotic vehicle suitable for use with various embodiments.

FIG. 5A illustrates a distorted image according to various embodiments.

FIG. 5B illustrates a corrected image according to various embodiments.

FIGS. 6A and 6B illustrate image distortion in an image captured by an image sensor on a moving platform according to various embodiments.

FIGS. 7A and 7B illustrate coordinate systems of robotic vehicles according to various embodiments.

FIG. 8 is a control flow diagram illustrating embodiment methods for correcting images captured with an image sensor of a robotic vehicle according to various embodiments.

FIG. 9 is a process flow diagram illustrating embodiment methods for processing of images captured by an image senor of a robotic vehicle according to various embodiments.

FIG. 10 is a process flow diagram illustrating embodiment methods for image processing by an image sensor of a robotic vehicle according to various embodiments.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and embodiments are for illustrative purposes, and are not intended to limit the scope of the claims.

Various embodiments include methods that may be implemented on a processing device of a robotic vehicle for processing an image captured by an image sensor of the robotic vehicle to adaptively track regions of interest within images for captured by an image sensor of a robotic vehicle without the need for a physical gimbal. Various embodiments improve the efficiency and accuracy of image processing of such images captured using a rolling shutter type image sensor in a robotic vehicle subject to pitch, yaw, and roll. Various embodiments further improve efficiency and accuracy of correcting for rolling shutter distortion and lens distortion by images captured by a robotic vehicle in motion, particularly an aerial robotic vehicle.

As used herein, the term “robotic vehicle” refers to one of various types of vehicles including an onboard processing device configured to provide some autonomous or semi-autonomous capabilities. Examples of robotic vehicles include but are not limited to: aerial vehicles, such as an unmanned aerial vehicle (UAV) ； ground vehicles (e.g., an autonomous or semi-autonomous car, a vacuum robot, etc. ) ； water-based vehicles (i.e., vehicles configured for operation on the surface of the water or under water) ； space-based vehicles (e.g., a spacecraft or space probe) ； and/or some combination thereof. In some embodiments, the robotic vehicle may be manned. In other embodiments, the robotic vehicle may be unmanned. In embodiments in which the robotic vehicle is autonomous, the robotic vehicle may include an onboard computing device configured to maneuver and/or navigate the robotic vehicle without remote operating instructions (i.e., autonomously) , such as from a human operator (e.g., via a remote computing device) . In embodiments in which the robotic vehicle is semi-autonomous, the robotic vehicle may include an onboard computing device configured to receive some information or instructions, such as from a human operator (e.g., via a remote computing device) , and autonomously maneuver and/or navigate the robotic vehicle consistent with the received information or instructions. In some implementations, the robotic vehicle may be an aerial vehicle (unmanned or manned) , which may be a rotorcraft or winged aircraft. For example, a rotorcraft (also referred to as a multirotor or multicopter) may include a plurality of propulsion units (e.g., rotors/propellers) that provide propulsion and/or lifting forces for the robotic vehicle. Specific non-limiting examples of rotorcraft include tricopters (three rotors) , quadcopters (four rotors) , hexacopters (six rotors) , and octocopters (eight rotors) . However, a rotorcraft may include any number of rotors. A robotic vehicle may include a variety of components and/or payloads that may perform a variety of functions.

Robotic vehicles are increasingly equipped with image sensor devices for capturing images and video. Robotic vehicles equipped to image the ground suffer from the problem that pitch, yaw, and roll of the vehicle leads to images that are not aligned with the horizon. Further, spurious motions of the robotic vehicle may cause jitter or other distortions in images and video. The fisheye lens type cameras used to capture a larger portion of the environment also lead to distortion of captured images. While a variety of mechanical image stabilization mechanisms are available (e.g., mechanical gimbals and optical image stabilization (OIS) ) , such mechanisms are typically too heavy and too expensive for incorporation into and use with many robotic vehicles, particularly aerial robotic vehicles (e.g., drones) .

Digital image stabilization (DIS) and electronic image stabilization (EIS) techniques may reduce or eliminate the need for mechanical image stabilization mechanisms, such as gimbals. A processing device employing a DIS technique may estimate spurious motion of the robotic vehicle based on image data, such as changes from image to image, or frame to frame. For example, the processing device may determine one or more image statistics from the image data. A processing device may, for example, analyze consecutive frames to calculate a transform that when applied to an image or frame reduces the effects of motion with respect to the previous image or frame. However, image statistics cannot be used to easily distinguish motion of an image sensor from motion of a subject in an image sensor’s field of view. Also, use of image statistics in image stabilization may result in additional jitter or shake in a video in particular when moving subjects are present in the image sensor’s field of view. Additionally, DIS performance may be impaired in conditions of low light or changing illumination.

To enable EIS, a processing device of a robotic vehicle may analyze sensor data from a sensor of the robotic vehicle to determine spurious motion of the robotic vehicle. For example, a processing device of the robotic vehicle may detect an orientation (e.g., pitch and roll) of the robotic vehicle, motion of the robotic vehicle (e.g., in three dimensions plus motion about the pitch, roll and yaw axes) , accelerations (e.g., vibrations and jitter) , and/or other information that may be available from one or more sensors (e.g., gyroscopes and accelerometers) of the robotic vehicle. Using the estimated orientation and motions of the robotic vehicle, the processing device of the robotic vehicle may process an image or video to correct the image of distortions caused by the orientation and motions. In some embodiments, such processing may be performed in real time or in post-processing of the image or video. For example, a processing device of the robotic vehicle may use sensor data to determine a rotation and translation to be applied to the output of the image sensor between two consecutive images or frames using, e.g., a gyroscope and accelerometer.

In an EIS system, the processing device the robotic vehicle may process the image or video based on a coordinate system of the robotic vehicle, and information about the mounting of an image sensor on the robotic vehicle, as well as information about an orientation of the output of the image sensor.

For example, robotic vehicle s may include a wide variety of body frames, and manufacturers of such body frames may utilize different coordinate systems, For example, in a flight controller or another processor of the robotic vehicle. One example of a body frame coordinate system is North-East-Down (NED) , in which positive values along the x-axis indicates north, positive values along the y-axis indicates east, and positive values along the x-axis indicates down (i.e., toward gravity) . Another example of a body frame coordinate system is North-West-Up (NWU) , in which positive values along the x-axis indicates north, positive values along the y-axis indicates west, and positive values along the x-axis indicates up (i.e., away from gravity) . Different robotic vehicle manufacturers and suppliers may use different coordinate systems.

Various embodiments provide adaptive image correction for digital gimbal in a robotic vehicle. Various embodiments include an image sensor (e.g., a line-read image sensor) of the robotic vehicle capturing an image. A processing device of the robotic vehicle may determine the movement of each pixel of each line of the captured image using information derived from an inertial measurement unit (IMU) of the robotic vehicle. Various embodiments may include the processing device performing lens distortion correction on the pixels based, at least in part, on the determined movement of each pixel. Output of the lens distortion correction process may be passed as input to an operation of performing rolling shutter corrections on the pixels to produce a transformation matrix based, at least in part, on the determined movement of each pixel. The processing device may then apply the transformation matrix to the captured image to produce a corrected image.

Various embodiments may enable a processing device within a robotic vehicle to perform adaptive two-dimensional image correction (i.e., rolling shutter corrections and lens distortion correction) . Various embodiments include processing device devices and methods that bi-linearly calculate the correction for both rolling shutter distortion and lens distortion in an image captured by an image sensor of a robotic vehicle. Processing of the image correction calculations may be performed across processors of the robotic vehicle, such as a central processing unit (CPU) and a graphics processing unit (GPU) , which may be in the same processing device. Performing bi-linear correction calculation and application may reduce the complexity of such calculations and the amount of time and processing resources required to accurately correct captured images.

In various embodiments, a processing device of a robotic vehicle may utilize tracking information obtained from various sensors on a robotic vehicle (e.g., inertial measurement units) to estimate the relative motion of individual pixels of image sensor output. The exposure time for each pixel may be determined or calculated by the processing device based on this estimated motion. The exposure time, position, and motion of pixels may be used by the processing device to calculate a correction matrix, which may be applied to the image sensor output to produce a corrected image.

Various embodiments may be implemented within a robotic vehicle operating within a variety of communication systems 100, an example of which is illustrated in FIG. 1. With reference to FIG. 1, the communication system 100 may include a robotic vehicle 102, a base station 104, an access point 106, a communication network 108, and a network element 110. In some embodiments, the robotic vehicle 120 may be equipped with an image sensor 102a. In some embodiments, the image sensor 102a may include a monocular image sensor.

The base station 104 and the access point 106 may provide wireless communications to access the communication network 108 over a wired and/or

wireless communication backhaul

116 and 118, respectively. The base station 104 may include base stations configured to provide wireless communications over a wide area (e.g., macro cells) , as well as small cells, which may include a micro cell, a femto cell, a pico cell, and other similar network access points. The access point 106 may include access points configured to provide wireless communications over a relatively smaller area. Other examples of base stations and access points are also possible.

The robotic vehicle 102 may communicate with the base station 104 over a wireless communication link 112, and with the access point 106 over a wireless communication link 114. The

wireless communication links

112 and 114 may include a plurality of carrier signals, frequencies, or frequency bands, each of which may include a plurality of logical channels. The

wireless communication links

112 and 114 may utilize one or more radio access technologies (RATs) . Examples of RATs that may be used in a wireless communication link include 3GPP Long Term Evolution (LTE) , 3G, 4G, 5G, Global System for Mobility (GSM) , Code Division Multiple Access (CDMA) , Wideband Code Division Multiple Access (WCDMA) , Worldwide Interoperability for Microwave Access (WiMAX) , Time Division Multiple Access (TDMA) , and other mobile telephony communication technologies cellular RATs. Further examples of RATs that may be used in one or more of the various wireless communication links within the communication system 100 include medium range protocols such as Wi-Fi, LTE-U, LTE-Direct, LAA, MuLTEfire, and relatively short range RATs such as ZigBee, Bluetooth, and Bluetooth Low Energy (LE) .

The network element 110 may include a network server or another similar network element. The network element 110 may communicate with the communication network 108 over a communication link 122. The robotic vehicle 102 and the network element 110 may communicate via the communication network 108. The network element 110 may provide the robotic vehicle 102 with a variety of information, such as navigation information, weather information, information about local air, ground, and/or sea traffic, movement control instructions, and other information, instructions, or commands relevant to operations of the robotic vehicle 102.

In various embodiments, the robotic vehicle 102 may move in an environment 120. In some embodiments, the robotic vehicle may use the image sensor 102a to capture one or more images of a target image 125 in the environment 120. In some embodiments, the target image 125 may include a test image, which may include known characteristics, such as a height, a width, a texture, and/or other characteristics.

Robotic vehicles may include winged or rotorcraft varieties. FIG. 2 illustrates an example robotic vehicle 200 of a ground vehicle design that utilizes one or more wheels 202 driven by corresponding motors to provide locomotion to the robotic vehicle 200. The robotic vehicle 200 is illustrated as an example of a robotic vehicle that may utilize various embodiments, but is not intended to imply or require that various embodiments are limited to ground robotic vehicles. For example, various embodiments may be used with rotorcraft or winged robotic vehicles, water-borne robotic vehicles, and space-based robotic vehicles.

With reference to FIGS. 1 and 2, the robotic vehicle 200 may be similar to the robotic vehicle 102. The robotic vehicle 200 may include a number of wheels 202, a body 204, and an image sensor 206. The frame 204 may provide structural support for the motors and their associated wheels 202 as well as for the image sensor 206. For ease of description and illustration, some detailed aspects of the robotic vehicle 200 are omitted such as wiring, frame structure interconnects, or other features that would be known to one of skill in the art. While the illustrated robotic vehicle 200 has wheels 202, this is merely exemplary and various embodiments may include any variety of components to provide propulsion and maneuvering capabilities, such as treads, paddles, skids, or any combination thereof or of other components.

The robotic vehicle 200 may further include a control unit 210 that may house various circuits and devices used to power and control the operation of the robotic vehicle 200. The control unit 210 may include a processing device 220, a power module 230, sensors 240, one or more payload securing units 244, one or more image sensors 245, an output module 250, an input module 260, and a radio module 270.

The processing device 220 may be configured with processor-executable instructions to control travel and other operations of the robotic vehicle 200, including operations of various embodiments. The processing device 220 may include or be coupled to a navigation unit 222, a memory 224, a gyro/accelerometer unit 226, and a maneuvering data module 228. The processing device 220 and/or the navigation unit 222 may be configured to communicate with a server through a wireless connection (e.g., a cellular data network) to receive data useful in navigation, provide real-time position reports, and assess data.

The maneuvering data module 228 may be coupled to the processing device 220 and/or the navigation unit 222, and may be configured to provide travel control-related information such as orientation, attitude, speed, heading, and similar information that the navigation unit 222 may use for navigation purposes, such as dead reckoning between Global Navigation Satellite System (GNSS) position updates. The gyro/accelerometer unit 226 may include an accelerometer, a gyroscope, an inertial sensor, an inertial measurement unit (IMU) , or other similar sensors. The maneuvering data module 228 may include or receive data from the gyro/accelerometer unit 226 that provides data regarding the orientation and accelerations of the robotic vehicle 200 that may be used in navigation and positioning calculations, as well as providing data used in various embodiments for processing images.

The processing device 220 may further receive additional information from one or more image sensors 245 (e.g., a camera, which may be a monocular camera) and/or other sensors 240. In some embodiments, the image sensor (s) 245 may include an optical sensor capable of infrared, ultraviolet, and/or other wavelengths of light. The sensors 240 may also include a wheel sensor, a radio frequency (RF) sensor, a barometer, a sonar emitter/detector, a radar emitter/detector, a microphone or another acoustic sensor, or another sensor that may provide information usable by the processing device 220 for movement operations as well as navigation and positioning calculations. The sensors 240 may include contact or pressure sensors that may provide a signal that indicates when the robotic vehicle 200 has made contact with a surface. The payload-securing units 244 may include an actuator motor that drives a gripping and release mechanism and related controls that are responsive to the control unit 210 to grip and release a payload in response to commands from the control unit 210.

The power module 230 may include one or more batteries that may provide power to various components, including the processing device 220, the sensors 240, the payload-securing unit (s) 244, the image sensor (s) 245, the output module 250, the input module 260, and the radio module 270. In addition, the power module 230 may include energy storage components, such as rechargeable batteries. The processing device 220 may be configured with processor-executable instructions to control the charging of the power module 230 (i.e., the storage of harvested energy) , such as by executing a charging control algorithm using a charge control circuit. Alternatively or additionally, the power module 230 may be configured to manage its own charging. The processing device 220 may be coupled to the output module 250, which may output control signals for managing the motors that drive the rotors 202 and other components.

The robotic vehicle 200 may be controlled through control of the individual motors of the rotors 202 as the robotic vehicle 200 progresses toward a destination. The processing device 220 may receive data from the navigation unit 222 and use such data in order to determine the present position and orientation of the robotic vehicle 200, as well as the appropriate course towards the destination or intermediate sites. In various embodiments, the navigation unit 222 may include a GNSS receiver system (e.g., one or more global positioning system (GPS) receivers) enabling the robotic vehicle 200 to navigate using GNSS signals. Alternatively or in addition, the navigation unit 222 may be equipped with radio navigation receivers for receiving navigation beacons or other signals from radio nodes, such as navigation beacons (e.g., very high frequency (VHF) omni-directional range (VOR) beacons) , Wi-Fi access points, cellular network sites, radio station, remote computing devices, other robotic vehicles, etc.

The radio module 270 may be configured to receive navigation signals, such as signals from aviation navigation facilities, etc., and provide such signals to the processing device 220 and/or the navigation unit 222 to assist in robotic vehicle navigation. In various embodiments, the navigation unit 222 may use signals received from recognizable RF emitters (e.g., AM/FM radio stations, Wi-Fi access points, and cellular network base stations) on the ground.

The radio module 270 may include a modem 274 and a transmit/receive antenna 272. The radio module 270 may be configured to conduct wireless communications with a variety of wireless communication devices (e.g., a wireless communication device (WCD) 290) , examples of which include a wireless telephony base station or cell tower (e.g., the base station 104) , a network access point (e.g., the access point 106) , a beacon, a smartphone, a tablet, or another computing device with which the robotic vehicle 200 may communicate (such as the network element 110) . The processing device 220 may establish a bi-directional wireless communication link 294 via the modem 274 and the antenna 272 of the radio module 270 and the wireless communication device 290 via a transmit/receive antenna 292. In some embodiments, the radio module 270 may be configured to support multiple connections with different wireless communication devices using different radio access technologies.

In various embodiments, the wireless communication device 290 may be connected to a server through intermediate access points. In an example, the wireless communication device 290 may be a server of a robotic vehicle operator, a third party service (e.g., package delivery, billing, etc. ) , or a site communication access point. The robotic vehicle 200 may communicate with a server through one or more intermediate communication links, such as a wireless telephony network that is coupled to a wide area network (e.g., the Internet) or other communication devices. In some embodiments, the robotic vehicle 200 may include and employ other forms of radio communication, such as mesh connections with other robotic vehicles or connections to other information sources (e.g., balloons or other stations for collecting and/or distributing weather or other data harvesting information) .

In various embodiments, the control unit 210 may be equipped with an input module 260, which may be used for a variety of applications. For example, the input module 260 may receive images or data from an onboard camera or sensor, or may receive electronic signals from other components (e.g., a payload) .

While various components of the control unit 210 are illustrated in FIG. 2 as separate components, some or all of the components (e.g., the processing device 220, the output module 250, the radio module 270, and other units) may be integrated together in a single processing device 310, an example of which is illustrated in FIG. 3.

With reference to FIGS. 1–3, the processing device 310 may be configured to be used in a robotic vehicle and may be configured as or including a system-on-chip (SoC) 312. The SoC 312 may include (but is not limited to) a processing device 314, a memory 316, a communication interface 318, and a storage memory interface 320. The processing device 310 or the SoC 312 may further include a communication component 322, such as a wired or wireless modem, a storage memory 324, an antenna 326 for establishing a wireless communication link, and/or the like. The processing device 310 or the SoC 312 may further include a hardware interface 328 configured to enable the processing device 314 to communicate with and control various components of a robotic vehicle. The processing device 314 may include any of a variety of processing devices, for example any number of processor cores.

The term “system-on-chip” (SoC) is used herein to refer to a set of interconnected electronic circuits typically, but not exclusively, including one or more processing devices (e.g., 314) , a memory (e.g., 316) , and a communication interface (e.g., 318) . The SoC 312 may include a variety of different types of processors 314 and processor cores, such as a general purpose processor, a central processing unit (CPU) , a digital signal processor (DSP) , a graphics processing unit (GPU) , an accelerated processing unit (APU) , a subsystem processor of specific components of the processing device, such as an image processor for a camera subsystem or a display processor for a display, an auxiliary processor, a single-core processor, and a multicore processor. The SoC 312 may further embody other hardware and hardware combinations, such as a field programmable gate array (FPGA) , an application-specific integrated circuit (ASIC) , other programmable logic device, discrete gate logic, transistor logic, performance monitoring hardware, watchdog hardware, and time references. Integrated circuits may be configured such that the components of the integrated circuit reside on a single piece of semiconductor material, such as silicon.

The SoC 312 may include one or more processing devices 314. The processing device 310 may include more than one SoC 312, thereby increasing the number of processing devices 314 and processor cores. The processing device 310 may also include processing devices 314 that are not associated with an SoC 312 (i.e., external to the SoC 312) . Individual processing devices 314 may be multicore processors. The processing devices 314 may each be configured for specific purposes that may be the same as or different from other processing devices 314 of the processing device 310 or SoC 312. One or more of the processing devices 314 and processor cores of the same or different configurations may be grouped together. A group of processing devices 314 or processor cores may be referred to as a multi-processor cluster.

The memory 316 of the SoC 312 may be a volatile or non-volatile memory configured for storing data and processor-executable instructions for access by the processing device 314. The processing device 310 and/or SoC 312 may include one or more memories 316 configured for various purposes. One or more memories 316 may include volatile memories such as random access memory (RAM) or main memory, or cache memory.

Some or all of the components of the processing device 310 and the SoC 312 may be arranged differently and/or combined while still serving the functions of the various aspects. The processing device 310 and the SoC 312 may not be limited to one of each of the components, and multiple instances of each component may be included in various configurations of the processing device 310.

FIG. 4 illustrates an image capture and processing system 400 of a robotic vehicle (e.g., 102, 200 in FIGS. 1 and 2) suitable for use with various embodiments. With reference to FIGS. 1–4, the image capture and processing system 400 may be implemented in hardware components and/or software components of the robotic vehicle, the operation of which may be controlled by one or more processing devices (e.g., the processing device 220, the processing device 310, the SoC 312, and/or the like) of the robotic vehicle.

An image sensor 406 may capture light of an image 402 that enters through a lens 404. The lens 404 may include a fish eye lens or another similar lens that may be configured to provide a wide image capture angle. The image sensor 406 may provide image data to an image signal processing (ISP) unit 408. A region of interest (ROI) selection unit 410 may provide data to the ISP 408 data for the selection of a region of interest within the image data. In some embodiments, the image sensor 406 may be similar to the

image sensor

102a, 245.

The ISP 408 may provide image information and ROI selection information to a rolling-shutter correction, image warp, and crop unit 412. An image rectification unit 414 may provide information and/or processing functions to the rolling-shutter correction, image warp, and crop unit 426. In some embodiments, the image rectification unit 414 may provide information and/or processing functions to correct for image distortion caused by the lens 404, an image distortion effect caused by the image sensor 406 (e.g., distortion such as wobble, skew, smear, and the like) , or other image distortion.

A movement parameters unit 416 may determine inertial measurement data and robotic vehicle position and orientation data. For example, the movement parameters unit 416 may obtain or receive the inertial measurement data and robotic vehicle position and orientation data from one or more sensors of the robotic vehicle (e.g., the sensors 240) The movement parameters unit 416 may provide the inertial measurement data and robotic vehicle position and orientation data to a pose estimation unit 418. ( “Pose” is a portmanteau of “position” and “orientation. ” )

The pose estimation unit 418 may determine a position and orientation of the robotic vehicle based on the inertial measure data and the position and orientation data. In some embodiments, the pose estimation unit 418 may determine the position and orientation (e.g., pitch, roll, and yaw) of the robotic vehicle based on a coordinate system of the robotic vehicle (e.g., NED or NWU) . The pose estimate unit 418 may provide the determined position and orientation of the robotic vehicle to a motion filter unit 420. Additionally, a pan and tilt control unit 410 may provide data about the pan and/or tilt of the image sensor to the motion filter unit 420.

The motion filter unit 420 may determine physical and/or virtual pose changes of an image sensor of the robotic vehicle (e.g., a sensor 240) based on the position and orientation information from the pose estimation unit 418 and the pan and/or tilt information from the pan and tilt control unit 410. In some embodiments, the motion filter unit 420 may determine the physical or virtual pose changes of the image sensor over time. In some embodiments, the motion filter unit 420 may determine the physical or virtual pose changes based on one or more changes between a first image and a second subsequent image. In some embodiments, the motion filter unit 420 may determine the physical or virtual pose changes of the image sensor on a frame-by-frame basis. The motion filter unit may provide the determined physical and/or virtual pose changes of an image sensor to a per-line camera rotation calculation unit 422.

The 1^st pass perspective filtering unit 422 may determine a per image frame transformation matrix/map to determine a coarse mapping of each output pixel’s to the input pixel. Based on this coarse mapping, unit 422 may provide information about the determined rotation to a transform matrix calculation unit 424.

The transform matrix calculation unit 424 may determine pixel by pixel or pixel tile by pixel tile transformation matrices/map for use in processing an image. The transform matrix calculation unit 424 may provide the transformation matrices to the rolling-shutter correction and warp unit 426.

The rolling-shutter correction and warp unit 426 may crop the image information, correct for distortions in the image caused by the lens 404, and may apply the transformation matrices/map to the image information. The rolling-shutter correction and warp unit 426 may provide as output a corrected image 428 based on the cropping, distortion correction, and/or application of the transformation matrix. In some embodiments, the corrected image may include an image having a corrected horizontal orientation or horizontal rotation. In some embodiments, the corrected image may include a stabilized video output.

FIG. 5A illustrates a distorted image 500 according to various embodiments. With reference to FIGS. 1–5A, the distorted image 500 may include one or more distortions, for example a curvature of a straight object 552, or the distortions indicated by

distortion markers

554 and 556, and by the test image 558.

FIG. 5B illustrates a corrected image 528 according to various embodiments. With reference to FIGS. 1–5B, the illustrated corrected image 528 has been rotated 90 degrees counterclockwise, and includes corrections to, for example, the straight object 552 and the test image 558.

FIGS. 6A and 6B illustrate image distortion in an image captured by an image sensor on a moving platform according to various embodiments. With reference to FIGS. 1–6B, a processing device of a robotic vehicle (e.g., the processing device 220 and/or the like) and hardware components and/or software components of the robotic vehicle may capture and process an image or video using an image sensor of the robotic vehicle (e.g., the sensor 240) .

FIG. 6A illustrates an image 602 captured by a moving image sensor, which includes a skewed object 604. For example, rolling shutter distortion may occur in images, and particularly video, captured by certain image sensors (e.g., complementary metal-oxide-semiconductor (CMOS) image sensors) , which record every frame line-by-line from top to bottom of the image, rather than as a single snapshot at a point in time. Because parts of the image are captured at different times, image sensor motion may cause image distortion referred to as a “jelly-effect” or “Jell-O wobble. ” The distortion illustrated in the image 602 may be caused by an object moving quickly through the image sensor’s field of view, or by camera translation (e.g., horizontal or rotational motion of the camera) . In addition, fast-moving objects may be distorted with diagonal skews, as illustrated by a skewed object 604 in the image 602. A processing device may determine motion of the images sensor during the time taken to traverse from the first to the last line of the frame, and the processing device may correct for sensor-motion induced rolling shutter distortion.

FIG. 6B illustrates rolling shutter distortion that may be cause by a pitch and a yaw of a motion sensor. Image sensor rotation (e.g., caused by pitch and yaw of a platform of the image sensor, e.g., a robotic vehicle) may cause two distinct effects due to the rolling shutter. For example, changes in yaw during exposure of a frame may cause vertical lines to develop a diagonal skew 606. In addition, changes in pitch during exposure of a frame may change a separation 608 between horizontal lines and may lead to a perception of residual motion along a Y-axis (e.g., horizontal axis) of the image. In some embodiments, a processing device may correct rolling shutter distortion and lens distortion by modeling a motion of pixels within the image or frame.

FIGS. 7A and 7B illustrate body frame coordinate systems of robotic vehicles according to various embodiments. With reference to FIGS. 1–7B, a processing device of a robotic vehicle (e.g., the processing device 220 and/or the like) and hardware components and/or software components of the robotic vehicle may capture and process an image or video based on a coordinate system of the robotic vehicle (in addition to information about the mounting or orientation of an image sensor on the robotic vehicle, and information about an orientation of the output of the image sensor) . Robotic vehicles may include a wide variety of body frames, and manufacturers of such body frames may utilize different coordinate systems, for example, in a flight controller or another processing device of the robotic vehicle.

FIG. 7A illustrates an example of a North-East-Down (NED) body frame coordinate system, in which positive values along the x-axis indicates north, positive values along the y-axis indicates east, and positive values along the x-axis indicates down (i.e., toward gravity) .

FIG. 7B illustrates an example of a North-West-Up (NWU) body frame coordinate system in which positive values along the x-axis indicates north, positive values along the y-axis indicates west, and positive values along the x-axis indicates up (i.e., away from gravity) .

FIG. 8 illustrate lens distortion correction and rolling shutter correction by processing devices of a robotic vehicles according to various embodiments. With reference to FIGS. 1–8, a processing device of a robotic vehicle (e.g., the processing device 220 and/or the like) and hardware components and/or software components of the robotic vehicle may capture and process an image or video based on a coordinate system of the robotic vehicle, in addition to information about the mounting or orientation of an image sensor on the robotic vehicle, and information about an orientation of the output of the image sensor.

In various embodiments a processing device, such as CPU 220 may analyze a captured image 802 to determine the movement of individual pixels or groups of pixels, tiles, and may use the determined movement of the pixels to generate a two-dimensional map associating input pixel positions, in x and y coordinates, to estimated output positions, based on the movement information. This movement information may be estimated during a first pass of lens distortion correction, which may be performed on the center pixels of lines of a captured image.

Various embodiments may perform a second pass of lens distortion correction in parallel with rolling shutter correction. These techniques may act directly on individual pixels or on tiles of pixels, thereby negating the need for a homography matrix to adjust the captured image. Once bi-linear correction is complete, the processing device such as a graphical processing unit (GPU) 802, may use the generated pixel map to look up the appropriate output position for each pixel or pixel tile. More specifically, the processing device may determine the input position of the pixel that was modified into the corrected pixel. The output position associated with the identified input position may be determined and the pixel placed in that output position. The combined impact of correcting and positioning all pixels may be a corrected image 806.

Robotic vehicles, particularly relatively small robotic vehicles such as aerial robotic vehicles, may experience rotor spinning with high revolutions per minute (RPM) (e.g., 10x thousands RPM) that may cause the robotic vehicle to shake or wobble. As such, a rolling shutter image sensor may capture images subject to significant distortion. In order to correct rolling shutter distortion in captured images, such as captured image 804, the processing device may track the attitude and movement of individual pixels of an image. Determining the attitude of individual pixels of an image may include performing either an interpolation based attitude calculation per pixel, per line of the image sensor output. Alternatively, various embodiments may include using a direct integration of the raw (i.e., distorted) pixels per line of the image sensor output to reduce complexity and improve performance.

Various embodiments using integration techniques may use per frame attitude estimates for reference. Because each pixel may have a different exposure time and thus a different degree of rolling shutter distortion, the robotic vehicle processing device may be able to determine the relative movement of each pixel if the per pixel, per line exposure time Δt is known.

The processing device may perform short term integration over an angular speed of the robotic vehicle to calculate per line attitudes. Such techniques may introduce only minimal integration error into the calculation of per-line attitudes. The estimated pitch θ, yaw ψ, and roll

of a pixel may be calculated from the raw/distorted IMU output information. For each line of the image sensor output, the yaw, pitch, and roll may be calculated by:

θ_n ＝ Δt ∑_n g_y (i) [Equation 2]ψ_n ＝ Δt ∑_n g_z (i) [Equation 3]

In various embodiments, the estimation of pixel attitude may be performed twice in order to increase accuracy. For example, equations 1-3 may be performed by a processing device for a pixel located in the center of a line of the image sensor output to obtain an overall estimate of attitude for pixels in the line. During the first execution of equations 1-3, the processing device may not have an accurate estimate of the exposure time for each individual pixel； however, after the time necessary to obtain the center pixel estimate has elapsed, the processing device may be able to accurately determine the exposure time for each unique pixel. A second round of equations 1-3 may be executed on each pixel of the line in order to determine an attitude for each pixel. The difference between yaw, pitch, and roll of the center pixel and each of the pixels of the line is then calculated.

The calculated differences in yaw, pitch, and roll may be used as input for a local rotation matrix. The local rotation matrix may be represented by the following matrix:

In Equation 5, I is the identity matrix and the local rotation matrix may be a 3x1 vector that may be applied to individual pixels of a line “a” of the image sensor output to correct for rolling shutter distortion.

The processing device may also perform a linear distortion or warp correction in combination with, prior to, or subsequent to the rolling shutter correction process. Performing an image warp process may require a processing device to perform a series of pixel-by-pixel read/write operations resulting in an unnecessary intermediate mid-processing image. Such techniques may be processor-intensive and may require both high processing throughput and high bandwidth throughput. To reduce complexity and avoid unnecessary read/write operations, various embodiments enable the processing device to use a one step warp correction process. Performing the composite or one step operation reduces processing demands on the processing device and other processing resources, as well as reducing consumption of a battery or other power resources.

In various embodiments, performing warp corrections may require a determination of the motion of each pixel of each line of image senor output. The processing device may compare the IMU to the image sensor calibration information (e.g., initial position and orientation information for image sensor) to obtain motion information. Each pixel of the image may be represented by a 3x1 matrix (i.e., a vector) . In various embodiments, a corrected pixel may be calculated by the processing device using the equation:

In equation 6,

represents a distorted pixel, prior to any digital gimbal processing (e.g., rolling shutter correction) , and

represents a digitally gimbaled, but not originally distorted pixel. The rotation matrix R_c may be the image sensor pose rotation matrix. The distortion function D () may vary according to image sensors and lens configuration.

The image sensor capture may be mapped to the matrix K. A point (X, Y, Z) in 3D space may be mapped to an image plane (x, y) based on pin-hole. The image sensor capture may be represented as:

In equation 9, F is a focal length in unit of pixels which relates to the image resolution, lens focal length and camera sensor size.

Further, w_dg may be described in terms of the image corrected pixel elements u and v and the elements of the transformation matrix:

w_dg ＝ tf₃₁u_dg+tf₃₂v_dg+tf₃₃ [Equation 8]

The resulting pixel movement information may be passed to a rolling shutter correction process as described. The application of the rolling shutter process to the output of the lens distortion correction/warp correction process may result in the local rotation matrix. These local rotation matrices may enable the correction of warp and rolling shutter distortion in individual pixels of image sensor output.

In various embodiments, the lens distortion correction and the rolling shutter corrections may be applied in parallel with image correction. For example, an applications processor or central processing unit (CPU) of the robotic vehicle may calculate an appropriate transformation matrix and continuously update the transformation matrix as new lines are read in by the image sensor, at the same time that the GPU 802 may apply the transformation matrix to distorted image pixels to produce a corrected image 806. Corrected image pixels may be passed as inputs to the applications processor or CPU for use in generating updated transformation matrices.

FIG. 9 illustrates a method 900 of correcting lens distortion and rolling shutter distortion in images captured by an image sensor of a robotic vehicle according to various embodiments. With reference to FIGS. 1–9, the method 800 may be implemented by a processing device (e.g., the processing device 220 and/or the like) or SoC 300 of the robotic vehicle (e.g., 102, 200) .

In block 902, an image sensor of the robotic vehicle may capture an image. In various embodiments, an image sensor mounted on or integrated into a robotic vehicle may capture an image or frame of a video.

In block 904, the processing device may perform lens distortion correction on center pixels of each line of the captured image to produce estimated movement information. An IMU of the robotic vehicle may provide movement and orientation information to the processing device. The processing device may use this information to determine the movement and attitude of individual pixels of the captured image. The movement of all pixels may be estimated using the calculated movement of the center pixels, thereby providing estimated movement information. Such estimated movement information may be used in a second iteration of the lens distortion correction to obtain more accurate pixel movement information.

In block 906, the processing device may generate a pixel mp, in which the pixel map may include a mapping of input pixel positions to output pixel positions based, at least in part on the estimated movement information. In various embodiments, the processing device may create a 2-dimensional image map accounting for each pixel and its associated movement. More specifically, the x and y coordinates of each pixel of the raw/unprocessed captured image may be mapped to an estimate output pixel x and y coordinate position. The output position may be estimated using the estimated movement information such that the estimated movement information indicates a difference between the input pixel position and the estimated output pixel position.

In block 908, the processing device may perform lens distortion correction on the pixels based, at least in part, on the estimated movement information to produce pixel movement information. In various embodiments, the processing device may use the determined movement information in combination with an image sensor matrix R_c representing the rotation needed in order to correct for the pose of the image sensor, a captured image matrix K, and pixel movement information to correct each pixel for warping and to stabilize the pixels. In various embodiments, the correction and stabilization may be performed in a single step, eliminating the need for an intermediate image and associated read/write operations.

In block 910, the processing device may perform rolling shutter corrections on the pixels based, at least in part, on the pixel movement information to produce corrected pixels. The processing device may use the pixel movement information to determine the attitude of a center pixel of a line of the captured image. The processing device may repeat the attitude calculation process for the remaining pixels of the line of the captured image. Movement estimates may include an estimated yaw, estimated pitch, and estimated roll of a pixel. The various estimates for each pixel may be subtracted from the corresponding estimates for the center pixel. The yaw difference, pitch difference, and roll difference for each pixel may be used to modify or correct each pixel or row.

In block 912, the processing device may generate a corrected image using the corrected pixels and the generated pixel map. In various embodiments, the processing device may use the generated pixel map to lookup the output location for a corrected pixel based on the position of the corresponding input pixel. For example, the processor may use the generated pixel map to look up the position of the raw/unprocessed input pixel associated with the corrected pixel and may place the corrected pixel in an appropriate output position.

FIG. 10 illustrates a method 1000 of advanced image processing in an image captured in a robotic vehicle according to various embodiments. With reference to FIGS. 1–10, the method 1000 may be implemented by a processing device (e.g., the processing device 220 and/or the like) or SoC 300 of the robotic vehicle (e.g., 102, 200) .

In block 1002, the processing device may determine an image sensor pose matrix. Image sensor calibration or initialization information may be used to populate a matrix representing the rotation of the image sensor pose with reference to a body frame coordinate. Specifically, the image sensor pose matrix may represent the rotation to be applied to portions of a captured image to compensate for the mounting angle and orientation of the image sensor.

In block 1004, the processing device may perform a single step correction and stabilization of the pixels using the image sensor pose matrix and the determined movement of each pixel. As discussed in greater detail with reference to FIG. 8, the processing device may reduce complexity and processing time by combining the correction and stabilization of pixels of the image sensor output into a single step.

In various embodiments, the operations of

blocks

1002 and 1004 may be sub-operations of either of

blocks

904 and 906 of the method 900.

In block 1006, the processing device may determine an estimated yaw, estimated pitch, and estimated roll of a center pixel. For example, the processing device may use equations 1-3 to estimate the yaw pitch, and roll of the center pixel of a line of the captured image.

In block 1008, the processing device may determine an estimated yaw, estimated pitch, and estimated roll of all the pixels other than the center pixel. As in block 1006, the processing device may use equations 1-3 to calculate an estimated yaw, pitch, and roll for all remaining pixels of the line of the captured image.

In block 1010, the processing device may determine a yaw difference, pitch difference, and roll difference, based on the estimated yaw, estimate, pitch, and estimated roll of the center pixels and other pixels. Equation 4 may be used by the processing device to calculate a difference between the yaw, pitch, and roll of the center pixel with respect to each other pixel in the line of the captured image. If the robotic vehicle is moving slowly or is not moving, then the yaw difference, pitch, difference, and roll difference may be slight.

In block 1012, the processing device may modify pixels or tiles of the captured image based, at least in part, on the yaw difference, the pitch difference, and the roll difference. In various embodiments, the processing device may modify or correct individual pixels of the captured image to remove rolling shutter distortion and produce corrected pixels. In various embodiments, the processing device may modify or correct tiles, blocks of pixels, with the same correction.

In various embodiments, the processing device may further execute interpolation techniques as described in greater detail with reference to FIG. 8. To increase the accuracy of the modification of the pixels and the resulting corrected pixels.

In various embodiments, the operations of blocks 1006-1012 may correspond to sub-operations of block 910 of the method 800. After performing these operations, the processing device may generate a corrected image in block 912.

Various embodiments enable the processing device of the robotic vehicle to improve image capture and processing by the robotic vehicle. Various embodiments also improve the efficiency of image capture and processing by the robotic vehicle. Various embodiments further improve the accuracy of image processing during image capture by the image sensor. Various embodiments enable improved image capture and processing by the robotic vehicle for a variety of body frame coordinate systems.

Various embodiments illustrated and described are provided merely as examples to illustrate various features of the claims. However, features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment and may be used or combined with other embodiments that are shown and described. Further, the claims are not intended to be limited by any one example embodiment. For example, one or more of the operations of the

methods

800 and 900 may be substituted for or combined with one or more operations of the

methods

800 and 900, and vice versa.

The foregoing method descriptions and the process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the operations of various embodiments must be performed in the order presented. As will be appreciated by one of skill in the art the order of operations in the foregoing embodiments may be performed in any order. Words such as “thereafter, ” “then, ” “next, ” etc. are not intended to limit the order of the operations； these words are used to guide the reader through the description of the methods. Further, any reference to claim elements in the singular, for example, using the articles “a, ” “an, ” or “the” is not to be construed as limiting the element to the singular.

Various illustrative logical blocks, modules, circuits, and algorithm operations described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and operations have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such embodiment decisions should not be interpreted as causing a departure from the scope of the claims.

The hardware used to implement various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP) , an application specific integrated circuit (ASIC) , a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but, in the alternative, the processing device may be any conventional processor, controller, microcontroller, or state machine. A processing device may also be implemented as a combination of receiver smart objects, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by circuitry that is specific to a given function.

In one or more aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable storage medium or non-transitory processor-readable storage medium. The operations of a method or algorithm disclosed herein may be embodied in a processor-executable software module or processor-executable instructions, which may reside on a non-transitory computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable storage media may be any storage media that may be accessed by a computer or a processing device. By way of example but not limitation, such non-transitory computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage smart objects, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD) , laser disc, optical disc, digital versatile disc (DVD) , floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable storage medium and/or computer-readable storage medium, which may be incorporated into a computer program product.

The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the claims. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

A method of processing an image in a robotic vehicle, comprising:

capturing an image by an image sensor of the robotic vehicle；

performing, by a processing device of the robotic vehicle, lens distortion correction on center pixels of each line of the captured image to produce estimated movement information；

generating, by the processing device, a pixel map, wherein the pixel map comprises a mapping of input pixel positions to output pixel positions based at least in part on, the estimated movement information；

performing, by the processing device, lens distortion correction on the pixels based at least in part on the estimated movement information to produce pixel movement information；

performing, by the processing device, rolling shutter corrections on the pixels based at least in part on the pixel movement information to produce corrected pixels； and

generating, by the processing device, a corrected image using the corrected pixels and the generated pixel map.
The method of claim 1, wherein performing lens distortion correction on the pixels based at least in apart on the estimated movement information comprises:

determining, by the processing device, an image sensor pose matrix； and

performing, by the processing device, a single step correction and stabilization of the pixels using the image sensor pose matrix and the estimated movement information.
The method of claim 1, wherein performing rolling shutter corrections on the pixels to produce a transformation matrix based at least in part on the pixel movement information comprises:

determining, by the processing device, an estimated yaw, estimated pitch, and estimated roll of a center pixel；

determining, by the processing device, an estimated yaw, estimated pitch, and estimated roll of all the pixels other than the center pixel；

determining, by the processing device, a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels； and

modifying, by the processing device, each pixel based at least in part on the yaw difference, the pitch difference, and the roll difference.
The method of claim 3, wherein:

performing lens distortion correction and performing rolling shutter corrections is performed by a first processing unit of the processing device, and

modifying each pixel is performed by a second processing unit of the processing device.
The method of claim 1, wherein performing rolling shutter corrections on the pixels based, at least in part, on the pixel movement information comprises:

determining, by the processing device, an estimated yaw, estimated pitch, and estimated roll of a center pixel of a tile of the captured image；

determining, by the processing device, an estimated yaw, estimated pitch, and estimated roll of all the pixels of the tile other than the center pixel；

determining, by the processing device, a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels of the tile； and

modifying, by the processing device, each tile based at least in part on the yaw difference, the pitch difference, and the roll difference.
The method of claim 5, wherein:

performing lens distortion correction and performing rolling shutter corrections is performed by a first processing unit of the processing device, and

modifying each tile is performed by a second processing unit of the processing device.
The method of claim 1, wherein performing lens distortion correction on the pixels and performing rolling shutter corrections are performed in parallel.
The method of claim 1, wherein generating, by the processing device, a corrected image using the corrected pixels and the generated pixel map comprises:

executing, by the processing device, interpolation on the corrected pixels； and

generating, by the processing device, the corrected image using interpolated corrected pixels and the generated pixel map.
A robotic vehicle, comprising:

an image sensor；

a processor coupled to the image sensor and configured with processor-executable instructions to:

capture an image by the image sensor；

perform lens distortion correction on center pixels of each line of the captured image to produce estimated movement information；

generate a pixel map, wherein the pixel map comprises a mapping of input pixel positions to output pixel positions based, at least in part on, the estimated movement information；

perform lens distortion correction on the pixels based, at least in part on the estimated movement information to produce pixel movement information；

perform rolling shutter corrections on the pixels based, at least in part, on the pixel movement information to produce corrected pixels； and

generate a corrected image using the corrected pixels and the generated pixel map.
The robotic vehicle of claim 9, wherein the processor is further configured with processor-executable instructions to:

determine an image sensor pose matrix； and

perform a single step correction and stabilization of the pixels using the image sensor pose matrix and the estimated movement information.
The robotic vehicle of claim 9, wherein the processor is further configured with processor-executable instructions to:

determine an estimated yaw, estimated pitch, and estimated roll of a center pixel；

determine an estimated yaw, estimated pitch, and estimated roll of all the pixels other than the center pixel；

determine a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels； and

modify each pixel based, at least in part, on the yaw difference, the pitch difference, and the roll difference.
The robotic vehicle of claim 11, wherein the processor is further configured with processor-executable instructions to:

perform lens distortion correction and performing rolling shutter corrections by a first processing unit, and

modify each tile by a second processing unit.
The robotic vehicle of claim 9, wherein the processor is further configured with processor-executable instructions to:

determine an estimated yaw, estimated pitch, and estimated roll of a center pixel of a tile of the captured image；

determine an estimated yaw, estimated pitch, and estimated roll of all the pixels of the tile other than the center pixel；

determine a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels of the tile； and

modify each tile based, at least in part, on the yaw difference, the pitch difference, and the roll difference.
The robotic vehicle of claim 13, wherein the processor is further configured with processor-executable instructions to:

perform lens distortion correction and performing rolling shutter corrections by a first processing unit, and

modify each tile by a second processing unit.
The robotic vehicle of claim 9, wherein the processor is further configured with processor-executable instructions to perform lens distortion correction on the pixels and perform rolling shutter corrections in parallel.
The robotic vehicle of claim 9, wherein the processor is further configured with processor-executable instructions to:

execute interpolation on the corrected pixels； and

generate the corrected image using interpolated corrected pixels and the generated pixel map.
A processing device for use in a robotic vehicle configured to:

capture an image by an image sensor of the robotic vehicle；

perform lens distortion correction on center pixels of each line of the captured image to produce estimated movement information；

generate a pixel map, wherein the pixel map comprises a mapping of input pixel positions to output pixel positions based, at least in part, on the estimated movement information；

perform lens distortion correction on the pixels based, at least in part, on the estimated movement information to produce pixel movement information；

perform rolling shutter corrections on the pixels based, at least in part, on the pixel movement information to produce corrected pixels； and

generate a corrected image using the corrected pixels and the generated pixel map.
The processing device of claim 17, further configured to:

determine an image sensor pose matrix； and

perform a single step correction and stabilization of the pixels using the image sensor pose matrix and the estimated movement information.
The processing device of claim 17, further configured to:

determine an estimated yaw, estimated pitch, and estimated roll of a center pixel；

determine an estimated yaw, estimated pitch, and estimated roll of all the pixels other than the center pixel；

determine a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels； and

modify each pixel based, at least in part, on the yaw difference, the pitch difference, and the roll difference.
The processing device of claim 19, further configured to:

perform lens distortion correction and performing rolling shutter corrections by a first processing unit of the processing device, and

modify each tile by a second processing unit of the processing device.
The processing device of claim 17, further configured to:

determine an estimated yaw, estimated pitch, and estimated roll of a center pixel of a tile of the captured image；

determine an estimated yaw, estimated pitch, and estimated roll of all the pixels of the tile other than the center pixel；

determine a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels of the tile； and

modify each tile based, at least in part, on the yaw difference, the pitch difference, and the roll difference.
The processing device of claim 21, further configured to:

perform lens distortion correction and performing rolling shutter corrections by a first processing unit of the processing device, and

modify each tile by a second processing unit of the processing device.
The processing device of claim 17, further configured to perform lens distortion correction on the pixels and perform rolling shutter corrections in parallel.
The processing device of claim 17, further configured to:

execute interpolation on the corrected pixels； and

generate the corrected image using interpolated corrected pixels and the generated pixel map.
A robotic vehicle, comprising:

an image sensor；

means for capturing an image by the image sensor of the robotic vehicle；

means for performing lens distortion correction on center pixels of each line of the captured image to produce estimated movement information；

means for generating a pixel map, wherein the pixel map comprises a mapping of input pixel positions to output pixel positions based, at least in part, on the estimated movement information；

means for performing lens distortion correction on the pixels based, at least in part, on the estimated movement information to produce pixel movement information；

means for performing rolling shutter corrections on the pixels based, at least in part, on the pixel movement information to produce corrected pixels； and

means for generating a corrected image using the corrected pixels and the generated pixel map.
The robotic vehicle of claim 25, wherein means for performing lens distortion correction on the pixels based at least in apart on the estimated movement information comprises:

means for determining an image sensor pose matrix； and

means for performing a single step correction and stabilization of the pixels using the image sensor pose matrix and the estimated movement information.
The robotic vehicle of claim 25, wherein means for performing rolling shutter corrections on the pixels to produce a transformation matrix based, at least in part, on the pixel movement information comprises:

means for determining an estimated yaw, estimated pitch, and estimated roll of a center pixel；

means for determining an estimated yaw, estimated pitch, and estimated roll of all the pixels other than the center pixel；

means for determining a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels； and

means for modifying each pixel based, at least in part, on the yaw difference, the pitch difference, and the roll difference.
The robotic vehicle of claim 25, wherein means for performing rolling shutter corrections on the pixels based, at least in part, on the pixel movement information comprises:

means for determining an estimated yaw, estimated pitch, and estimated roll of a center pixel of a tile of the captured image；

means for determining an estimated yaw, estimated pitch, and estimated roll of all the pixels of the tile other than the center pixel；

means for determining a yaw difference, a pitch difference, and a roll difference, based on the estimated yaw, estimated pitch, and estimated roll of the center pixel and other pixels of the tile； and

means for modifying each tile based, at least in part, on the yaw difference, the pitch difference, and the roll difference.
The robotic vehicle of claim 25, wherein means for performing lens distortion correction on the pixels and means for performing rolling shutter corrections operate in parallel.
The robotic vehicle of claim 25, wherein means for generating a corrected image using the corrected pixels and the generated pixel map comprises:

means for executing interpolation on the corrected pixels； and

means for generating the corrected image using interpolated corrected pixels and the generated pixel map.