WO2018090258A1

WO2018090258A1 - Image processing method, device, and system

Info

Publication number: WO2018090258A1
Application number: PCT/CN2016/106152
Authority: WO
Inventors: 翁松伟
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2016-11-16
Filing date: 2016-11-16
Publication date: 2018-05-24
Also published as: CN107113406A

Abstract

Disclosed in embodiments of the present invention are an image processing device, system, and method. The device comprises: a receiving module used to receive first image data acquired by a camera; a camera processing module used to process the first image data to obtain second image data; and a memory module used to store the second image data in a memory. The image processing device of the present invention is arranged within a main body of an unmanned aerial vehicle to implement an integrated aerial photography system. In addition, the image processing device facilitates installation, and a camera mount provides a superior stabilization effect to a camera.

Description

Image processing method, device and system

Technical field

The present invention relates to the field of images, and in particular, to an image processing method, apparatus, and system. Background technique

Unmanned Aerial Vehicle (UAV), referred to as the UAV, is a non-manned aircraft operated by radio remote control equipment and self-contained program control devices. The existing professional-grade aerial photography system uses a multi-axis platform equipped with a professional movie machine, but this combination is not only complicated to install, but also cannot quickly and easily set camera parameters through the wireless remote ground end, and the stability of the gimbal is poor. Summary of the invention

The embodiment of the invention provides an image processing method and device for implementing an integrated aerial photography system, which is convenient to install and has a good stability effect of the cloud platform. A first aspect of the present invention provides an image processing apparatus, including: a receiving module, configured to receive first image data collected by a camera; and a camera processing module, configured to process the first image data to obtain second image data a storage module, configured to store the second image processing data to a memory. Optionally, the first image data includes: a color filter array CFA image that is induced by the camera sensor and converted. Optionally, the camera processing module includes: an image processing ISP unit, a ProRes encoder;

The ISP unit is configured to restore, render, and convert the first image data into a YUV image, and transmit the YUV image to the ProRes encoder;

The ProRes encoder, configured to receive a YUV image from the ISP unit, and encode the YU V image in a ProRes standard and encapsulate the second image data; wherein the second image data includes a MOV Formatted video;

The storage module is configured to store the video in the MOV format to a memory through a PCIE interface.

Optionally, the memory comprises a solid state hard disk SSD.

Optionally, the image processing device is disposed in the body of the drone.

A second aspect of the present invention provides an image processing system, comprising the image processing apparatus according to the first aspect, further comprising: a pan-tilt camera, a ground PC end.

Optionally, the pan/tilt camera is configured to sense light by using a camera sensor, convert the obtained color filter array CFA image, and send the CFA image to the image processing device.

Optionally, the ground PC is configured to read a MOV format video in the memory through a card reader.

A third aspect of the present invention provides an image processing method for use in a drone, the method comprising:

Receiving first image data collected by the camera;

Processing the first image data to obtain second image data; The second image data is stored to a memory.

Optionally, the first image data includes: a color filter array CFA image that is induced by the camera sensor and converted.

Optionally, the processing the first image data to obtain the second image data includes:

Restoring, rendering, and converting the first image data into a γυν image;

The YUV image is encoded in a ProRes standard and encapsulated as the second image data; wherein the second image data includes a video in a MOV format.

Optionally, the memory comprises a solid state hard disk SSD.

An embodiment of the present invention provides an image processing apparatus, system, and method, the apparatus comprising: a grazing module for receiving first image data collected by a camera; and a camera processing module, configured to perform the first image data Processing, obtaining second image data; and a storage module, configured to store the second image processing data to the memory. The image processing device is deployed inside the drone body to realize an integrated aerial photography system, which is convenient to install. At the same time, the PTZ does not need to be equipped with an image processing module, which makes the PTZ more stable to the camera, and the image processing module is placed inside the fuselage. The UAV body has larger space than the camera, and the heat dissipation effect is better.

I figure description

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention, Those skilled in the art can also obtain other drawings based on these drawings without paying for labor. FIG. 1 is a schematic diagram of a movable object according to an embodiment of the present invention; FIG. 2A is a schematic diagram of an image processing system according to an embodiment of the present invention;

2B is a schematic diagram of a sky end according to an embodiment of the present invention;

3 is a flowchart of recording a ProRes video according to an embodiment of the present invention; FIG. 4 is a flowchart of image processing according to an embodiment of the present invention;

5A-5B are schematic diagrams of an image processing apparatus provided by an embodiment of the present invention. detailed description

Embodiments of the present invention provide an image processing apparatus, system, and method, which realize an integrated aerial photography system, are convenient to install, and have a better stability effect of the gimbal.

The following description of the invention uses a drone as an example of a movable object. It will be apparent to those skilled in the art that other types of movable animals can be made without limitation.

The technical solutions in the embodiments of the present invention will be clearly described in conjunction with the drawings in the embodiments of the present invention. Some embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without the inventive work should be protected by the present invention.

The detailed description will be respectively made below through specific embodiments. Movable object

FIG. 1 is a schematic diagram of a movable object 100 according to an embodiment of the present invention. This can The moving object 100 includes a carrier 102 and a load 104. Although the movable object 100 is described as an aircraft, this description is not limiting, and any type of movable object is applicable. Those skilled in the art will appreciate that any of the embodiments described herein with respect to an aircraft system are applicable to any movable object (such as an unmanned aerial vehicle). In some embodiments, the load 104 can be located directly on the movable object 100 without the need for a carrier: 102. The movable object 100 can include a power unit 106, a sensing system 108, and a communication system 110.

The power mechanism 106 can include one or more rotating bodies, propellers, blades, engines, motors, wheels, bearings, magnets, nozzles. For example, the rotating body of the power mechanism may be a self-tighting rotating body, a rotating body assembly, or other rotating body power unit. A movable object can have one or more power mechanisms. All power mechanisms can be of the same type. Alternatively, one or more of the power mechanisms can be of different types. The power mechanism 106 can be mounted to the movable object by suitable means, such as by a support member such as a drive shaft. The power mechanism 106 can be mounted at any suitable position on the movable object 100, such as the top end, the lower end, the front end, the rear end, the side, or any of the ports therein.

In some embodiments, the power mechanism 106 can cause the movable object to take off vertically from the surface, or land vertically on the surface without requiring any horizontal movement of the movable object 100 (eg, without taxiing on the runway). Alternatively, the power mechanism 106 can allow the movable object 100 to hover in a preset position and/or direction in the air. One or more of the power mechanisms 106 can be independent of other power mechanisms when controlled. Alternatively, one or more of the power units 106 can be simultaneously controlled. For example, the movable object 100 can have a plurality of horizontally rotating bodies to track the lifting and/or pushing of the target. Horizontal direction The rotating body can be actuated to provide the ability of the movable object 100 to take off vertically, vertically, and spiral. In some embodiments, one or more of the horizontally rotating bodies may be rotated in a clockwise direction, while the other one or more of the horizontally rotating bodies may be rotated in a counterclockwise direction. For example, the number of rotating bodies rotating clockwise is the same as the number of rotating bodies rotating counterclockwise. The rotation rate of each horizontally rotating body can be independently varied to achieve lifting and/or pushing motion caused by each rotating body, thereby adjusting the spatial orientation, velocity and/or acceleration of the movable object 100 (eg, relative to many Up to three degrees of freedom of rotation and translation).

Sensing system 108 can include one or more sensors to sense the spatial orientation, velocity, and/or acceleration of the movable object (e.g., relative to rotation and translation of up to three degrees of freedom). The one or more sensors include any of the sensors described above, including GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors. Sensing data provided by sensing system 108 can be used to track the spatial orientation, velocity, and/or acceleration of target 100 (as appropriate, using suitable processing units and/or control units). Alternatively, sensing system 108 can be used to acquire data for the environment of the movable object, such as climatic conditions, potential obstacles to be approached, location of geographic features, location of man-made structures, and the like.

Communication system 110 is capable of communicating with terminal 112 having communication system 114 via wireless signal 116. Communication systems 110, 114 may include any number of transmitters, receivers, and/or transceivers for wireless communication. The communication can be one-way communication so that data can be sent from one direction. For example, one-way communication may include that only the movable object 100 transmits data to the terminal 112, or vice versa. One of the communication systems 110 One or more transmitters can transmit data to one or more receivers of communication system 112, and vice versa. Alternatively, the communication may be two-way communication, such that data may be transmitted between the movable object 100 and the terminal 112 in two directions. To the communication includes - one or more transmitters of the communication system 110 can send the digital saw to one or more receivers of the communication system 114, and vice versa.

In some implementations, terminal 112 can provide control data to one or more of movable object 100, carrier 102, and load 104, and from one or more of movable object 100, carrier 102, and load 104. Receive information (such as the position and/or motion information of the movable object, carrier or load, load-sensing data, such as image data captured by the camera). In some embodiments, the control data for the terminal may include instructions regarding position, motion, actuation, or control of the movable object, carrier, and/or load. For example, control data can result in changes in the position and/or orientation of the movable object (e.g., by controlling the power unit 抅 106) or cause movement of the carrier relative to the movable object.

(eg by control of the carrier 102). Terminal control data can lead to load control, such as controlling camera or other image capture device operations (capturing still or moving images, zooming, turning on or off, switching imaging modes, changing image resolution, changing focus, changing depth of field, changing Exposure time, change the viewing angle or field of view). In some embodiments, the communication of the movable object, the carrier, and/or the load may include one or more sensors

Information sent (such as sensing system 108 or load 104). The communication may include sensing information transmitted from one or more different types of sensors, such as GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors. The sensing information is about a position (such as direction, position) of the movable object, the carrier, and/or the load, Movement, or acceleration. The sensing information transmitted from the load includes the data captured by the load or the state of the load. The control data transmitted by terminal 112 can be used to track the status of one or more of movable object 100, carrier 102. or load 104. Alternatively or simultaneously, the carrier 102 and the load 104 may each include a communication module for communicating with the terminal U2 so that the terminal can individually communicate or track the movable object i00, the carrier 102 and the load 104.

In some embodiments, the movable object 100 can communicate with other remote devices other than the terminal 112, and the terminal 12 can also communicate with other remote devices than the movable object 100. For example, the movable object and/or terminal 112 can communicate with a carrier or load of another movable object or another movable object. The additional remote device can be a second terminal or other computing device (e.g., a computer, desktop, tablet, smartphone, or other mobile device) when needed. The remote device can transmit data to the movable object 100, receive data from the movable object 100, transmit data to the terminal 112, and/or receive data from the terminal 112. Alternatively, the remote device can be connected to the Internet or other telecommunications network to upload data received from the removable object 100 and/or terminal 112 to a website or server.

In some embodiments, the motion of the movable object, the motion of the carrier, and the movement of the load relative to a fixed reference (such as an external environment), and/or motion between each other, can be controlled by the terminal. The terminal may be a remote control terminal located remotely from the movable object, the carrier and/or the load. The terminal can be located or affixed to the support platform. Optionally, the terminal may be handheld or wearable. For example, the terminal may include a smartphone, a tablet, a desktop computer, a computer, glasses, gloves, a helmet, Microphone or any combination of them. The terminal can include a user interface such as a keyboard, mouse, a squat, a touch screen, or a display. Any suitable user input can interact with the terminal, such as manual input commands, sound control, gesture control, or position control (eg, by motion, position, or tilt of the terminal). Image processing system

The hardware of the camera processing module is placed in the fuselage to more effectively ensure the heat dissipation, stabilization and recording stability; and the function adds support for the video recording of the mov format encoded by ProRes, combined with stable flight, Yuntai Integrated support for stabilization, remote setup and on-site quick export viewing ensures high image quality and better meets the shooting needs of professional filmmakers while reducing hard disk space usage.

2A is a schematic diagram of an image processing system provided by an embodiment of the present invention. The image processing system includes a sky end 201, a ground moving end 202, and a personal PC end 203.

The sky end 201 can be a drone, which mainly achieves stable flight, the pan/tilt is stabilized, and the video is recorded and stored in the PreRes encoded MOV format.

The ground mobile terminal 202 can be a remote controller and a mobile terminal. Specifically, the mobile terminal can be a smart phone or a tablet computer. The ground mobile terminal 202 mainly realizes remote control of the drone, remotely sets the camera parameters and transmits images in real time.

The sky end 201 and the ground moving end 202 pass the radio frequency signal to realize remote wireless communication, and the remote control of the ground mobile end 202 can remotely disperse the drone to fly, and the ground mobile application APP (such as DJI GO) can remotely set the camera. Parameters, and view real The image is transmitted at the time.

The ground PC end 203 may include a solid state drive (SSD) card reader and a PC device, and mainly implements fast data reading, previewing, and exporting of the SSD.

The SSD of the sky end 201 is taken out, and the ground PC end is connected through the SSD card reader, and the data in the SSD can be quickly read, previewed and clipped.

FIG. 2B is a schematic diagram of a sky end according to an embodiment of the present invention. The camera processing module 212 is disposed in the stable flightable body, and receives image information sensed by the sensor of the pan/tilt camera module 21 1 having the pan/tilt stabilizing effect, and then rendered and encoded in the ProRes format, and packaged into the MOV format. Video, save video to SSD (pluggable) module via PCI-E interface.

Embodiments of the present invention provide an image processing apparatus for deploying an image processing apparatus inside a human-free body to implement an integrated aerial photography system, which is convenient to install. At the same time, the PTZ does not need to be equipped with an image processing module, which makes the PTZ more stable to the camera, and the image processing module is placed inside the fuselage. The UAV body has larger space than the camera, and the heat dissipation effect is better. FIG. 3 is a flowchart of recording a ProRes video according to an embodiment of the present invention. When recording video in MOV format encoded by ProRes, the system mainly works with the sky end and the ground moving end; and when the recording is finished, the SSD connection ground PC end of the sky side can be quickly previewed and exported. The process of recording ProRes video is supported by the sky side, the ground moving end, and the ground PC side.

Record the ProRes process:

S 301, a mobile terminal APP connected to a data line for a remote controller (for example, DJI GO On the top, select the required ProRes coded video settings, turn on the video, and the mobile terminal APP transmits the corresponding command to the remote control processing module through the USB port for encoding.

5302. The remote control processing module transmits the encoded data to a radio frequency module of the sky end (for example, a drone) via a radio frequency module of a ground mobile terminal (for example, a remote controller).

5303. The encoding module of the sky end receives and decodes the data transmitted to the image transmission module, and sends a corresponding instruction to the corresponding module via the graphic transmission module.

5304. The flight control module adjusts the flight status of the drone according to the flight settings of the user to achieve flight stability.

The 5305 and PTZ camera modules adjust the state of the pan/tilt three axes according to the user's pan/tilt parameter settings to achieve the stabilization of the gimbal.

5306. The camera processing module adjusts the parameters in the camera processing module according to the user's camera parameter settings, and calls the camera sensor to start recording the ProRes encoded MOV format video with the pan/tilt stabilization effect for the aerial scene.

5307. During recording, the optical data of the aerial scene is sensed by the camera sensor and converted into a Color Filter Array (CFA) data to the camera processing module;

5308. The camera processing module performs image restoration rendering by the Image Signal Processing (ISP) module and converts it into YUV data, transmits it to the DDR register in real time, pushes it to the ProRes encoder, encodes and encapsulates it into the MOV format by ProRes. The video is then passed back to the DDR register and transferred to the SSD (pluggable) module for storage via the PCIE interface. YUV is a color space, where Y is the brightness, and U and V are the chromaticities, that is, υ is the difference between the luminance and blue components (Y-B), and V is the difference between the luminance and red components: Y-R). S309. At the end of the recording, the MOV format video encoded by ProRes corresponding to the aerial scene is completely stored in the SSD module of the drone.

After the aerial photography is finished, the user only needs to take out the SSD in the SSD pluggable module in the sky, and connect the SSD to the card reader of the PC1E to the USB interface and connect to the USB interface of the MAC.

310. The ground PC side recognizes the FAT32 public file system in the SSD, and the user can view, preview, export and edit by opening the recorded ProRes encoded MOV format file. FIG. 4 is a flowchart of image processing according to an embodiment of the present invention, including:

5401. Receive first image data collected by a camera.

Optionally, the first image data comprises: a color filter array CFA image that is induced by the camera sensor and converted.

5402. Process the first image data to obtain second image data.

Restoring, rendering, and converting the first image data into a YUV image;

5403. Store the second image data to a memory.

In some implementations, the memory includes a solid state drive SSD. FIG. 5A is a schematic diagram of an image processing apparatus according to an embodiment of the present invention, including: a receiving module 501, configured to receive first image data collected by a camera.

In some embodiments, the first image data comprises: sensing light by a camera sensor, And convert the resulting color filter array CFA image.

The camera processing module 502 is configured to process the first image data to obtain second image data.

Specifically, as shown in FIG. 5B, the camera processing module 502 includes an image processing ISP unit 521 and a ProRes encoder 522.

The ISP unit 521 is configured to restore, render, and convert the first image data into a YUV image, and transmit the YUV image to the ProRes encoder;

The ProRes encoder 522 is configured to receive a YUV image from the LSP unit, and encode the YUV image in a ProRes standard and package the second image data; wherein the second image data includes a MOV-style video.

The storage module 503 is configured to store the second image processing data to the memory. In some embodiments, the storage module 503 is configured to store the video of the MOV format obtained by the ProRes encoder 522 to the memory through the PCIE interface. In some embodiments, the memory can be an E1 state hard disk SSD.

In some embodiments, image processing device 500 is deployed within the drone body. The various embodiments in the specification are described in a progressive manner, and each embodiment is focused on differences from the other embodiments, and the same or similar parts may be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the method part.

A person skilled in the art can also appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functions in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein can be implemented directly in hardware, a software module executed by a processor, or a combination of both. The software module can be placed in random access memory (RAM), memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or technical field. Any other form of storage medium known.

The image processing apparatus, system and method provided by the present invention have been described in detail above. The principles and embodiments of the present invention have been described herein with reference to specific examples, and the description of the above embodiments is only to assist in understanding the method of the present invention and its core idea. It is to be understood that those skilled in the art can make various modifications and changes to the present invention without departing from the spirit and scope of the invention.

Claims

^S3⁄4, ^ 4 ^ i . An image processing apparatus, characterized in that the apparatus comprises:

a receiving module, configured to receive first image data collected by the camera;

a camera processing module, configured to process the first image data to obtain second image data;

a storage module, configured to store the second image processing data to a memory.

The device according to claim 1, wherein

The first image data includes: a color filter array CFA image that is induced by the camera sensor and converted.

The apparatus according to claim 1 or 2, wherein the camera processing module comprises an image processing ISP unit, a ProRes encoder;

4. A device according to any one of claims 1 to 3, wherein the memory comprises a solid state drive SSD.

The apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is disposed in a body of the drone.

An image processing system, comprising the image processing apparatus according to any one of claims 1 to 5, further comprising: a pan-tilt camera, a ground PC end.

7. The image processing system according to claim 6, wherein:

The pan/tilt camera is configured to sense light by using a camera sensor, and convert the obtained color filter array CFA image, and send the CFA image to the image processing device.

8. The image processing system according to claim 6 or 7, wherein the ground PC is configured to read a video of the MOV format in the memory through the card reader.

9. An image processing method for use in a drone, characterized in that the method comprises:

Receiving first image data collected by the camera;

Processing the first image data to obtain second image data;

The second image data is stored to a memory.

10. The method of claim 9 wherein:

The method according to claim 9 or 10, wherein the processing the first image data to obtain the second image data comprises:

Restoring, rendering, and converting the first image data into a YUV image;

The method according to any one of claims 9 to 11, wherein the memory comprises a solid state hard disk SSD.