US20200358979A1

US20200358979A1 - System and method for supporting selective backtracking data recording

Info

Publication number: US20200358979A1
Application number: US16/939,653
Authority: US
Inventors: Sheldon Schwartz; Tao Wang; Mingyu Wang; Zisheng Cao
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd
Priority date: 2014-12-14
Filing date: 2020-07-27
Publication date: 2020-11-12
Also published as: EP3167604A1; US11095847B2; US20200186749A1; US10284808B2; EP3123644A4; US10771734B2; CN107005624B; US9973728B2; EP3123644A1; US20170256288A1; EP3846454A1; CN107223316A; DK3123644T3; EP3167604B1; WO2016095367A1; US20190174090A1; US10567700B2; EP3123644B1; CN107223316B; CN111432152A

Abstract

Systems and methods can support a data processing apparatus. The data processing apparatus can include a data processor that is associated with a data capturing device on a stationary object and/or a movable object. The data processor can receive data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence. Then, the data processor can receive a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time. Furthermore, the data processor can determine a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of International Application No. PCT/CN2015/076015, entitled “SYSTEM AND METHOD FOR SUPPORTING SELECTIVE BACKTRACKING DATA RECORDING,” filed on Apr. 7, 2015, which claims priority to International Application No. PCT/CN2014/093785, entitled “VIDEO PROCESSING METHOD, VIDEO PROCESSING DEVICE AND PLAYING DEVICE,” filed on Dec. 14, 2014, International Application No. PCT/CN2014/093786, entitled “VIDEO PROCESSING METHOD, VIDEO PROCESSING DEVICE AND DISPLAY DEVICE,” filed on Dec. 14, 2014, and International Application No. PCT/CN2015/075458, entitled “METHODS AND SYSTEMS OF VIDEO PROCESSING,” filed on Mar. 31, 2015. The contents of all four International Applications are hereby incorporated by reference in their entirety.

BACKGROUND OF THE INVENTION

The disclosed embodiments relate generally to data processing and more particularly, but not exclusively, to multimedia data recording.
Data recording, such as video recording, is traditionally an action in one direction along the timeline, where the past has already been settled and the future is yet to come. Typically, the data can be stored as soon as the recording starts. However, critical data, such as memorable moments, may still be missed since the user may not be able to start the recording in time.
On the other hand, there are physical limits that prevent a user from recording everything without qualification. For example, an unmanned aerial vehicle (UAV) may be used for a wide range of applications including surveillance, search and rescue operations, exploration, and other fields. As the image technology develops, the amount of image data that needs to be recorded can grow very fast. Thus, excessive bandwidth may be consumed, which prohibits the downloading of the captured video from an UAV.
This is the general area that embodiments of the invention are intended to address.

BRIEF SUMMARY OF THE INVENTION

Described herein are systems and methods that can support a data processing apparatus. The data processing apparatus can include a data processor that is associated with a data capturing device on a stationary object and/or a movable object. The data processor can receive data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence. Then, the data processor can receive a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time. Furthermore, the data processor can determine a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.
Also described herein are systems and methods that can support a data processing apparatus. The data processing apparatus can include a data processor that is associated with a data capturing device on a stationary object and/or a movable object. The data processor can store, in a memory, data received from one or more data sources in a time sequence. Then, the data processor can remove a portion of the data stored in the memory after receiving a synchronization signal. Furthermore, the data processor can forward the removed data to a storage, when a switch is turned on, for a time period in the time sequence.
Also described herein are systems and methods that can support a data processing apparatus. The data processing apparatus can include a data processor that is associated with a data capturing device on a stationary object and/or a movable object. The data processor can associate one or more tags with a data flow received from one or more data sources, wherein said one or more tags are associated with one or more timestamps. Then, data processor can save the one or more timestamps in a record. Furthermore, the data processor can select a subset of the data in the received data flow based on the one or more timestamps saved in the record.
Other objects and features of the present invention will become apparent by a review of the specification, claims, and appended figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings briefly described herein.

FIG. 1 is an exemplary illustration of a data recording system, in accordance with various embodiments of the present invention.

FIG. 2 is an exemplary illustration of selective data recording, in accordance with various embodiments of the present invention.

FIG. 3 is an exemplary illustration of using a buffer to support selective data recording, in accordance with various embodiments of the present invention.

FIG. 4 is an exemplary illustration of an asynchronous selective data recording, in accordance with various embodiments of the present invention.

FIG. 5 shows a flowchart of selective data recording, in accordance with various embodiments of the present invention.

FIG. 6 is an exemplary illustration of backtracking data recording, in accordance with various embodiments of the present invention.

FIG. 7 is an exemplary illustration of backtracking recording optimized in memory usage, in accordance with various embodiments of the present invention.

FIG. 8 is an exemplary illustration of backtracking recording optimized in computation load, in accordance with various embodiments of the present invention.

FIG. 9 shows a flowchart of backtracking data recording, in accordance with various embodiments of the present invention.

FIG. 10 is an exemplary illustration of data recording with tags, in accordance with various embodiments of the present invention.

FIG. 11 is an exemplary illustration of exporting from data recording with tags, in accordance with various embodiments of the present invention.

FIG. 12 shows a flowchart of data recording with tags, in accordance with various embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention is illustrated, by way of example and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” or “some” embodiment(s) in this disclosure are not necessarily to the same embodiment, and such references mean at least one.
The description of the invention as following uses multimedia data recording as example for data recording. It will be apparent to those skilled in the art that other types of data recording can be used without limitation.

An Exemplary Data Recording System

FIG. 1 is an exemplary illustration of a data recording system, in accordance with various embodiments of the present invention. As shown in FIG. 1, a data processing system 100 can include a data processor 102 that receives various types of data from a data source 101.
In accordance with various embodiments of the present invention, the data source 101 can be associated with a data capturing device 110. For example, an image sensor can be used for capturing the image or video information, while a microphone (MIC) can be used for capturing the audio information and a physical or virtual keyboard can be used for capturing the textual information.
Additionally, the data capturing device 110 can be placed on a stationary object and/or a movable object. For example, a stationary object may include a fixed monitoring device, while a movable object may include an unmanned aircraft, an unmanned vehicle, a hand held device (such as a camera), or a robot.
In accordance with various embodiments of the present invention, different communication protocols can be used to receive data from the one or more data sources. For example, the data source 101 can be physically connected to the data processor 102. Alternatively, the data source 101 can communicate with data processor 102 via wireless connection.
Different approaches can be used for controlling the data recording. The traditional approach is to start recording after a need is identified, such as when critical data is identified. This approach may not be ideal since the success of the recording depends on how fast the user can start the recording process and how responsive the system is.
Alternatively, the data processing system 100 can support selective backtracking data recording. For example, once started, the data capturing device 110 can be configured to keep on capturing data and pushes the captured data flow to the data processor 102 continuously. In the meantime, the data processor 102 may choose not to store the received data (i.e. abandoning the received data), unless an instruction, to perform otherwise, is received from a user or is prescribed by the system.
Then, the data processor 102 can export the selected data to a storage 103. For example, the data processor 102 can save the data into a storage medium that is physically connected. Alternatively, the data processor 102 may send the first data segment to a remote server via a network.
Thus, the selective backtracking data recording approach is beneficial, since there is no dependency on how fast the user can start the recording process and how responsive the system is.
Additionally, a user terminal 104 can be used for different purposes, such as controlling the data capturing device, and viewing the received data flow.
In accordance with various embodiments of the present invention, the data processor 102 can transmit a reduced set of the received data to a user terminal 104 for preview. For example, while the received data may contain data in multiple data types, the reduced set of the received data flow can only contain data in one data type.
Additionally, the user terminal 104, such as a handheld device, can be used for editing data captured by the data capture devices 110. For example, the user terminal 104 can be used for editing video captured by image capture devices on unmanned aerial vehicles (UAVs).
In some embodiments, the UAV may have an image capture device that captures video at a high definition and transmits the captured video to the handheld device at a low definition. The handheld device may receive and edit the captured video at the low definition and form a video edit request. The video edit request may be small in size and contain edited video data information for an edited video. The video edit request may then be transmitted to the UAV and/or image capture device. The UAV and/or image capture device may analyze the edited video file, generate a video file corresponding to the edited video data information, and transmit the generated video file to the handheld device or another device.
Therefore, the pressure on bandwidth for transmission and computation for video editing may be alleviated, since not all the high definition video is transmitted.

Selective Data Recording

FIG. 2 is an exemplary illustration of selective data recording, in accordance with various embodiments of the present invention. As shown in FIG. 2, a data processing system 200 can include a data processor 201 that can receive data in a data flow 202 from a data source 210.
In accordance with various embodiments of the present invention, the data flow 202 may include different types of information. For example, the data in a data flow 202 may contain audio data that are captured using a microphone; video data that are captured using an image sensor; and/or textual data that are captured using a physical or virtual keyboard.
Furthermore, the data flow 202 can be configured based on a time sequence 206. For example, the time sequence can be defined based on a clock time, and/or a system defined time (such as image frame counts).
As shown in FIG. 2, the data processor 201 can receive a control signal 203. The control signal 203 may be triggered by a button being pushed, a gesture of a being, a movement of an object, and/or a change of a state, and generated by a user terminal, a motion detector, a state monitor, and/or a data analyzer.
Furthermore, the control signal 203 is associated with a timestamp 204, which indicates (or represents) a time 214 in the time sequence 206. When the data processor 201 receives the control signal 203, the data processor 201 can apply the timestamp 204 on the data flow 202, to determine a data segment 211 that is associated with a time period 205 in the time sequence 206.
As shown in FIG. 2, the time period 205, from a starting time 207 to an ending time 208, includes the time 214. This indicates that the data segment 211 includes data received from both the time period before the time 214 (i.e. backtracking) and the time period after the time 214. Thus, the data processing system 200 can support selective backtracking data recording.
In accordance with various embodiments of the present invention, the data processing system 200 allows for the use of the reduced set of the received data flow, which may only contain data in one data type, for initiating the backtracking data recording.
Then, the data processor 201 can export the selected data segment 211 to a storage. For example, the data processor 201 can save the data segment into a storage medium that is physically connected. Alternatively, the data processor 201 may send the first data segment to a remote server via a network.
FIG. 3 is an exemplary illustration of using a buffer to support selective data recording, in accordance with various embodiments of the present invention. As shown in FIG. 3, a data processing system 300 can take advantage of a buffer 302 in a memory 310 for handling the received data 301.
The data processing system 300 can use the buffer 302 for storing the received data 301 before removing it from the memory 310. For example, the data processing system 300 can drop 303 (a portion of) the data stored in the buffer 302 periodically, e.g. when receiving a frame synchronization signal and the buffer 302 is full. Furthermore, the size of the buffer 302 can be preconfigured. Alternatively, the size of the buffer 302 can be configured dynamically, e.g. based on policies.
Furthermore, the data processing system 300 can determine a data segment 311 after receiving a control signal 306 for storing the related data. The data segment 311 can include both past data stored in the buffer 302 and the data arriving as the time progresses.
Then, the data processing system 300 can export the data segment 311 to a storage. For example, the data processing system 300 can save the data segment into a storage medium 304 that is physically connected. Alternatively, the data processing system 300 may send the first data segment to a remote server 305 via a network 307.
FIG. 4 is an exemplary illustration of an asynchronous selective data recording, in accordance with various embodiments of the present invention. As shown in FIG. 4, a data processing system 400 can take advantage of a buffer 402 in a memory 410 for handling the received data 401.
The data processing system 400 can use the buffer 402 for storing the received data 401 before removing it from the memory 410. For example, the data processing system 400 can drop 403 (a portion of) the data stored in the buffer 402 periodically, e.g. when receiving a frame synchronization signal and the buffer 402 is full. Furthermore, the size of the buffer 402 can be preconfigured. Alternatively, the size of the buffer 402 can be configured dynamically, e.g. based on policies.
As shown in FIG. 4, the data processing system 400 can create a copy 412 in the memory 410 for the data segment 411 when the data processing system 400 receives a control signal 406. Then, the data processing system 400 can export the copy of the data segment 411 at a later time (i.e. in an asynchronous fashion).
In accordance with various embodiments of the present invention, the data processing system 400 can export the data segment 411 to a storage. For example, the data processing system 400 can save the data segment into a storage medium 404 that is physically connected. Alternatively, the data processing system 400 may send the first data segment to a remote server 405 via a network 407.
Additionally, in order to save the overhead relating to establishing network connection, the data processing system 400 may be configured to export multiple data segments (each of which is associated with a different time period) in one action.
FIG. 5 shows a flowchart of selective data recording, in accordance with various embodiments of the present invention. As shown in FIG. 5, at step 501, a data processor can receive data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence. Then, at step 502, the data processor can receive a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time. Furthermore, at step 503, the data processor can determine a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.

Backtracking Data Recording

FIG. 6 is an exemplary illustration of backtracking data recording, in accordance with various embodiments of the present invention. As shown in FIG. 6, a data processing system 600 can take advantage of a buffer 606 in a memory 602 for handling the data 611, which are received from a data source 601. Additionally, the received data 611 can be associated with a time sequence 610. For example, the received data may contain one or more image frames, and the time sequence 610 can correspond to an image frame count.
In accordance with various embodiments of the present invention, the received data 611 can be stored in a memory 602, which may be based on a first-in-first-out (FIFO) protocol. Alternatively, other types of memory can be used without limitation. For example, the memory 602 can be based on a ring buffer or a list.
As shown in FIG. 6, the data processing system 600 can remove a portion of the data (e.g. one or more image frames) stored in the memory 602 periodically, e.g. after receiving a synchronization signal 620 (i.e. when new data arrives and the buffer 606 is full). Furthermore, the data processing system 600 can include a switch 603, which can be responsible for directing the data out of the memory 602.
The data processing system 600 can turn on the switch 603 after receiving a control signal. When the switch 603 is turned on, the data processing system 600 can export the data to a storage 605. The storage 605 can be based on a storage medium that connects to the memory 602 and/or a remote server that connects via a network. On the other hand, when the switch 603 is turned off, the data processing system 600 can drop 604 the data without storing it.
As shown in FIG. 6, the data processing system 600 receives a control signal at a time, tg. Once the switch 603 is turned on, the switch 603 can be maintained for a time period, T (i.e. from tg to tc in the time sequence 610). Here, the time period, T, can be preconfigured or prescribed. Alternatively, the time period, T, can be dynamically configured based on one or more policies.
In accordance with various embodiments of the present invention, the data processing system 600 can use a dynamically determined time, e.g. when the volume of a voice signal exceeds certain or when certain moving objects are detected in the scene, as the starting point and the ending point for the time period, T, as long as the memory is sufficient.
For example, the data processing system 600 can use video analysis technology to automatically identify the time when certain interesting object appears in the image. Then, the data processing system 600 can use this time as the starting point for the time period, T.
In accordance with various embodiments of the present invention, the data processing system 600 can support backtracking data recording. As shown in FIG. 6, the buffer 606 in the memory 602 can be used to store a number of image frames with variable sizes. There can be a number of images existing in the buffer 606, which may be configured to have a maximum image frame number, M.
Thus, for the time period, T, the stored data segment can include a past portion with a duration of Tr, which equals to the result of the maximum image frame number, M, divided by the image speed, fps, (i.e. M/fps). Additionally, the data segment can include a current portion with a duration of Tc, where Tc is the difference between the time period, T, and the duration for the past portion Tr (i.e. T−Tr).
In accordance with various embodiments of the present invention, the data processing system 600 can be optimized for different purposes, depending on how the memory 602 is configured. For example, the backtracking data recording can be configured for optimized memory usage or for optimized computation load (i.e. battery usage).
FIG. 7 is an exemplary illustration of backtracking recording optimized in memory usage, in accordance with various embodiments of the present invention. As shown in FIG. 7, a data processing system 700 can take advantage of a memory 708 for handling the data received from a data source.
For example, the data processing system 700 can receive various image information from a sensor 701. Then, the data processing system 700 can use an image signal processor (ISP) 703 for transforming the image information into raw video data.
Additionally, the data processing system 700 can receive various analog voice information from a microphone (MIC) 702. Then, the data processing system 700 can use an analog/digital (A/D) converter 704 to convert the analog voice information into digital audio data.
Furthermore, the data processing system 700 can use different encoder for encoding, or compressing, the received data before storing the encoded data in the memory 708. For example, a video encoder 705 can be used for encoding the raw video data from the ISP 703, and an audio encoder 706 can be used for encoding the audio data from the (A/D) converter 704. Additionally, the data processing system 700 can use a multiplexer, e.g. AV Mux 707, for combining the separate audio data and video data into an audio/video (A/V) data stream.
Thus, the memory usage is optimized, since the received data are encoded or compressed before storing the encoded data in the memory 708 (on the other hand, more computation load may be required).
In accordance with various embodiments of the present invention, the memory 708 can be based on a first-in-first-out (FIFO) protocol. Alternatively, other types of memory can be used without limitation. For example, the memory 708 can be based on a ring buffer or a list.
Furthermore, the data processing system 700 can remove a portion of the data (e.g. one or more image frames) stored in the memory 708 periodically, e.g. after receiving a synchronization signal 720 (i.e. when new data arrives and the buffer is full).
As shown in FIG. 7, the data processing system 700 can include a switch 707 and a switch 709, which is responsible for directing the video data and the audio data out of the memory 708. The data processing system 700 can turn on the switch 709 after receiving a control signal. When the switch 709 is turned on, the data processing system 700 can direct the data to a storage 712. On the other hand, when the switch 709 is turned off, the data processing system 700 can drop 711 the data without storing it.
As shown in FIG. 7, the data processing system 700 receives a control signal at a time, tg. Once the switch 709 is turned on, the data processing system 700 can maintain the switch 709 for a time period, T (i.e. from tg to tc in the time sequence 710).
Thus, the data processing system 700 can support backtracking data recording. A buffer in the memory 708 can be used for storing a number of image frames with variable sizes. There can be a number of images existing in the buffer, which may be configured to have a maximum image frame number, M.
Thus, for the time period, T, the stored data segment can include a past portion with a duration of Tr, which equals to the result of the maximum image frame number, M, divided by the image speed, fps, (i.e. M/fps). Additionally, the data segment can include a current portion with a duration of Tc, where Tc is the difference between the time period, T, and the duration for the past portion Tr (i.e. T−Tr).
Additionally, data processing system 700 can direct the received data to a user terminal for live view 713.
FIG. 8 is an exemplary illustration of backtracking recording optimized in computation load, in accordance with various embodiments of the present invention. As shown in FIG. 8, a data processing system 800 can take advantage of a memory 805 for handling the data received from a data source.
For example, the data processing system 800 can receive various image information from a sensor 801. Then, the data processing system 800 can use an image signal processor (ISP) 803 for transforming the image information into raw video data.
Additionally, the data processing system 800 can receive various analog voice information from a microphone (MIC) 802. Then, the data processing system 800 can use an analog/digital (A/D) converter 804 to convert the analog voice information into raw digital audio data.
In order to optimize the computation load, both the raw video data and raw digital audio data may be saved directly into the memory 802 (i.e. requires more memory). Here, the data processing system 800 can perform various image processing operations on the raw video data and raw digital audio data (e.g. playing backwards), since the raw video data and raw digital audio data has not be encode or compressed.
Furthermore, the data processing system 800 can remove a portion of the data (e.g. one or more image frames) stored in the memory 805 periodically, e.g. after receiving a synchronization signal 820 (i.e. when new data arrives and the buffer is full).
As shown in FIG. 8, the data processing system 800 can include a switch 807 and a switch 808, which are responsible for directing the video data and the audio data out of the memory 805, respectively.
The data processing system 800 can turn on the switches 807-808 after receiving a control signal. When the switches 807-808 are turned on, the data processing system 800 can direct the data to a storage 814. On the other hand, when the switches 807-808 are turned off, the data processing system 800 can drop 804 the data without storing it.
As shown in FIG. 8, the data processing system 800 receives a control signal at a time, tg. Once the switches 807-808 are turned on, the data processing system 800 can maintain the switches 807-808 for a time period, T (i.e. from tg to tc in the time sequence 810).
The data processing system 800 can support backtracking data recording. A buffer in the memory 805 can be used for storing a number of image frames with variable sizes. There can be a number of images existing in the buffer, which may be configured to have a maximum image frame number, M.
Thus, for the time period, T, the stored data segment can include a past portion with a duration of Tr, which equals to the result of the maximum image frame number, M, divided by the image speed, fps, (i.e. M/fps). Additionally, the data segment can include a current portion with a duration of Tc, where Tc is the difference between the time period, T, and the duration for the past portion Tr (i.e. T−Tr).
The data processing system 800 can use one or more encoders to encode the received data before actually saving the encoded data in the storage. Here, the storage 814 can be based on a storage medium that connects to the memory 805, and/or a remote server that connects via a network.
As shown in FIG. 8, the data processing system 800 can use a video encoder 805 for encoding the raw video data from the ISP 803, and use an audio encoder 806 for encoding the audio data from the (A/D) converter 804. Additionally, the data processing system 800 can use a multiplexer, e.g. AV Mux 813, to combine the separate audio data and video data into an audio/video (A/V) data stream.
Additionally, data processing system 800 can direct the received data to a user terminal for live view 806.
FIG. 9 shows a flowchart of backtracking data recording, in accordance with various embodiments of the present invention. As shown in FIG. 9, at step 901, a data processor can store, in a memory, data received from one or more data sources, wherein the received data is associate with a time sequence. Then, at step 902, the data processor can remove a portion of the data stored in the memory after receiving a synchronization signal. Furthermore, at step 903, the data processor can forward the removed data to a storage, when a switch is turned on, for a time period in the time sequence.

Data Recording with Tags

FIG. 10 is an exemplary illustration of data recording with tags, in accordance with various embodiments of the present invention. As shown in FIG. 10, a data processing system 1000 can receive a data flow 1008 from a data source.
For example, the data processing system 1000 can receive various image information from a sensor 1001. Then, the data processing system 1000 can use an image signal processor (ISP) 1003 for transforming the image information into raw video data.
Additionally, the data processing system 1000 can receive various analog voice information from a microphone (MIC) 1002. Then, the data processing system 1000 can use an analog/digital (A/D) converter 1004 to convert the analog voice information into digital audio data.
Furthermore, the data processing system 1000 can use various encoder for encoding, or compressing, the received digital data. For example, a video encoder 1005 is used for encoding the raw video data from the ISP 1003, and an audio encoder 1006 is used for encoding the audio data from the (A/D) converter 1004. Then, the data processing system 1000 can use a multiplexer, such as an AV Mux 1007, to combine the audio data and the video data into an audio/video (A/V) stream for the data flow 1008.
Additionally, the data processing system 1000 can export (or save) the audio/video (A/V) stream in the storage 1012. For example, the storage 1012 can be based on a storage medium that connects to the memory, and/or a remote server that connects via a network.
In accordance with various embodiments of the present invention, the data processing system 1000 can take advantage of one or more tags 1030 for handling the data in a data flow 1008 that are received from a data source. For example, the one or more tags 1030 may be applied by the user at the time of recording.
As shown in FIG. 10, the tags 1030 can be associated with one or more timestamps (e.g. tO to tn), which can be defined in a time sequence 1010. The time sequence 1010 can be defined using a clock time or based on a system defined time.
The data processing system 1000 can assign an image frame count, e.g. a recorded frame number, to each image frame in the received data flow. The recorded frame numbers, which may be defined based on a received frame synchronization signal 1020, can be used as a reference for the system defined time. Also, the timestamp associated with a tag 1030 may correspond to one or more recorded frame numbers.
When a user applies a tag 1030 on the received data flow 1008, the data processing system 1000 may receive a control signal, e.g. a Tigg signal. Alternatively, the data processing system 1000 may generate a control signal in responding to a user request.
Furthermore, the data processing system 1000 can save the timestamps, tO to tn, in a record 1011 (e.g. in a memory). Based on the timestamps saved in the record 1011, the data processing system 600 can select a subset of the data in the received data flow 1008. Then, the data processing system 1000 can use the selected data for different purposes, such as sharing, playing and storing.
Additionally, the data processing system 1000 can direct data in the received data flow to a user terminal for live view 1009.
FIG. 11 is an exemplary illustration of exporting from data recording with tags, in accordance with various embodiments of the present invention. As shown in FIG. 11, the data processing system 1100 can transcode data in the storage 1101 based the stored timestamps 1111 in the record 1110.
In accordance with various embodiments of the present invention, the data processing system 1000 can create a data segment, such as a clip 1121 in a clip memory 1120, based on a saved timestamp 1111. For example, the data processing system 1100 can use a decoder 1102 to decode the recorded data in the storage 1101, before creating the clip 1121 in the clip memory 1120 (e.g. in a memory).
As shown in FIG. 11, the data processing system 1100 can employ a searching step 1105 for creating the clip 1121, which may include one or more recorded image frames. For example, the timestamp 1111 may represent a time tg in the time sequence 1106. The data processing system 1100 can determine a time period, T (e.g. from T1 to T2), in the time sequence 1106) and the corresponding recorded frame counts. Then, the data processing system 1100 can determine the clip 1121, which includes one or more image frames recorded before the saved timestamp (i.e. the time period Tr), and/or one or more image frames recorded after the saved timestamp (i.e. the time period Tc).
Furthermore, the data processing system 1100 can traverse the different timestamps saved in the record, and select a data segment for each saved timestamp.
Then, the data processing system 1100 can use an encoder 1103 to create a data stream, e.g. AV stream 1104, based on the clips in the clip memory (i.e. the selected data segments).
FIG. 12 shows a flowchart of data recording with tags, in accordance with various embodiments of the present invention. As shown in FIG. 12, at step 1201, a data processor can associate one or more tags with a data flow received from one or more data sources, wherein said one or more tags are associated with one or more timestamps. Then, at step 1202, the data processor can save the one or more timestamps in a record. Furthermore, at step 1203, the data processor can select a subset of the data in the received data flow based on the one or more timestamps saved in the record.
In some embodiments, the methods and features described herein thus provide a data processing method comprising receiving data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence; receiving a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time; determining, via a data processor, a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.
In some embodiments, the one or more data sources may be associated with a data capturing device on a stationary object and/or a movable object.
In some embodiments, the data processing method may further comprise using different communication protocols to receive data from the one or more data sources.
In some embodiments, the data flow may contain audio data that are captured using a microphone, video data that are captured using an image sensor, and/or textual data that are captured using a physical or virtual keyboard.
In some embodiments, the time sequence may be defined based on a clock time and/or a system defined time.
In some embodiments, the data processing method may further comprise storing the received data into a buffer in a memory, and removing a portion of the received data stored in the buffer periodically when new data arrives and the buffer is full.
In some embodiments, a size of the buffer may be either preconfigured or dynamically configured based on one or more policies.
In some embodiments, the data processing method may further comprise creating a copy in the memory for the first data segment, before exporting the first data segment.
In some embodiments, the data processing method may further comprise exporting the first data segment by saving the first data segment into a storage medium that is physically connected to the memory, and/or sending the first data segment to a remote server via a network.
In some embodiments, the data processing method may further comprise exporting a plurality of data segments in the data flow, wherein each said data segment is associated with a different time period.
In some embodiments, the control signal may be triggered by a button being pushed, a gesture of a being, a movement of an object, and/or a change of a state, and generated by a user terminal, a motion detector, a state monitor, and/or a data analyzer.
In some embodiments, the data processing method may further comprise transmitting a reduced set of the received data flow to a user terminal for preview and/or initiating recording.
In some embodiments, the received data flow may contain data in multiple data types, while the reduced set of the received data flow may contain data only in one data type.
In some embodiments, the data processing method may further comprise configuring the data source to start transmitting the data flow.
In some embodiments, the systems and features described herein thus provide a data processing apparatus comprising one or more microprocessors and a data processor running on the one or more microprocessors, wherein the data processor operates to receive data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence; receive a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time; determine a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.
In some embodiments, the one or more data sources may be associated with a data capturing device on a stationary object and/or a movable object.
In some embodiments, different communication protocols may be used to receive data from the one or more data sources.
In some embodiments, the data flow may contain audio data that are captured using a microphone, video data that are captured using an image sensor, and/or textual data that are captured using a physical or virtual keyboard.
In some embodiments, the time sequence may be defined based on a clock time and/or a system defined time.
In some embodiments, the data processor may operate to store the received data into a buffer in a memory, and remove a portion of the data stored in the buffer periodically when new data arrives and the buffer is full.
In some embodiments, the size of the buffer may be either preconfigured or dynamically configured based on one or more policies.
In some embodiments, the data processor may operate to create a copy in the memory for the first data segment, before exporting the first data segment.
In some embodiments, the data processor may operate to export the first data segment by saving the first data segment into a storage medium that is physically connected to the memory, and/or send the first data segment to a remote server via a network.
In some embodiments, the data processor may operate to export a plurality of data segments in the data flow, wherein each said data segment is associated with a different time period.
In some embodiments, the control signal may be triggered by a button being pushed, a gesture of a being, a movement of an object, and/or a change of a state, and generated by a user terminal, a motion detector, a state monitor, and/or a data analyzer.
In some embodiments, the data processor may operate to transmit a reduced set of the received data flow to a user terminal for preview and/or initiate recording.
In some embodiments, the received data flow may contain data in multiple data types, while the reduced set of the received data flow may contain data only in one data type.
In some embodiments, the data processor may operate to configure the one or more data sources to start transmitting the data flow.
In some embodiments, the systems and features described herein also provide a non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, perform the steps comprising: receiving data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence; receiving a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time; determining, via a processor, a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.
In some embodiments, the systems and features described herein also provide a data processing system comprising: a data capturing device on a stationary object and/or a movable object; and a data processor, running on one or more microprocessors, wherein the data processor operates to receive data in a data flow from one or more data sources, wherein the data flow is configured based on a time sequence; receive a control signal, which is associated with a first timestamp, wherein the first timestamp indicates a first time; determine a first data segment by applying the first timestamp on the data flow, wherein the first data segment is associated with a time period in the time sequence that includes the first time.
In some embodiments, the methods and features described herein also provide a data processing method comprising: storing, in a memory, data received from one or more data sources, wherein the received data is associated with a time sequence; removing a portion of the data stored in the memory after receiving a synchronization signal; and forwarding the removed data to a storage, when a switch is turned on, for a time period in the time sequence.
In some embodiments, the one or more data sources may be associated with one or more data capturing devices on a stationary object and/or a movable object.
In some embodiments, the received data may contain video data captured using an image sensor, audio data captured using a microphone, and/or textual data captured using a key board.
In some embodiments, the data processing method may further comprise using different communication protocols to receive data from the one or more data sources.
In some embodiments, the memory may include one of a first-in-first-out (FIFO) protocol memory, a ring buffer, or a list.
In some embodiments, the data processing method may further comprise using one or more encoders to encode the received data before storing the encoded data in the memory, and using a multiplexer to combine audio data and video data into an audio/video (A/V) data stream.
In some embodiments, the data processing method may further comprise using one or more encoders to encode the received data before saving the encoded data in the storage, and using a multiplexer to combine audio data and video data into an audio/video (A/V) data stream.
In some embodiments, the data processing method may further comprise performing one or more image processing operations on the received data before the encoding step.
In some embodiments, the storage may be based on a storage medium that connects to the memory and a remote server that connects via a network.
In some embodiments, the data processing method may further comprise turning on the switch when receiving a control signal, wherein the control signal is triggered by a button being pushed, a gesture of a being, a movement of an object, and/or a change of a state, and generated by a user terminal, a system detector, a state monitor, and/or a data analyzer.
In some embodiments, the received data may contain one or more image frames, wherein the time sequence is associated with an image frame count.
In some embodiments, the data processing method may further comprise providing a buffer in the memory, wherein the buffer is used to store a number of image frames with variable sizes.
In some embodiments, the data processing method may further comprise creating a data segment, which includes a past portion and a current portion.
In some embodiments, the data processing method may further comprise directing the received data to a user terminal for live view.
In some embodiments, the systems and features described herein also provide a data processing apparatus comprising one or more microprocessors; a data processor, running on the one or more microprocessors, wherein the data processor operates to store, in a memory, data received from one or more data sources in a time sequence; remove a portion of the data stored in the memory after receiving a synchronization signal; and forward the removed data to a storage, when a switch is turned on, for a time period in the time sequence.
In some embodiments, the one or more data sources may be associated with one or more data capturing devices on a stationary object and/or a movable object.
In some embodiments, the received data may contain video data captured using an image sensor, audio data captured using a microphone, and/or textual data captured using a key board.
In some embodiments, different communication protocols may be used to receive data from the one or more data sources.
In some embodiments, the memory may be based on a first-in-first-out (FIFO) protocol, or the memory is a ring buffer or a list.
In some embodiments, one or more encoders may be used to encode the received data before storing the encoded data in the memory.
In some embodiments, one or more encoders may be used to encode the received data before saving the encoded data in the storage.
In some embodiments, a multiplexer may be used to combine audio data and video data into an audio/video (A/V) data stream.
In some embodiments, the storage may be based on a storage medium that connects to the memory and a remote server that connects via a network.
In some embodiments, the data processor may operate to turn on the switch when receiving a control signal, wherein the control signal is triggered by a button being pushed, a gesture of a being, a movement of an object, and/or a change of a state, and generated by a user terminal, a system detector, a state monitor, and/or a data analyzer.
In some embodiments, the received data may contain one or more image frames, wherein the time sequence is associated with an image frame count.
In some embodiments, the data processing apparatus may further comprise a buffer in the memory, wherein the buffer is used to store a number of image frames with variable sizes.
In some embodiments, the data processor may operate to create a data segment, which includes a past portion and a current portion.
In some embodiments, the data processor may operate to direct the received data to a user terminal for live view.
In some embodiments, the systems and features described herein also provide a non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, perform the steps comprising: storing, in a memory, data received from one or more data sources, wherein the received data is associate with a time sequence; removing a portion of the data stored in the memory after receiving a synchronization signal; and forwarding the removed data to a storage, when a switch is turned on, for a time period in the time sequence.
In some embodiments, the systems and features described herein also provide a data processing system comprising: a data capturing device on a stationary object and/or a movable object; and a data processor, running on one or more microprocessors, wherein the data processor operates to store, in a memory, data received from one or more data sources in a time sequence; remove a portion of the data stored in the memory after receiving a synchronization signal; and forward the removed data to a storage, when a switch is turned on, for a time period in the time sequence.
Many features of the present invention can be performed in, using, or with the assistance of hardware, software, firmware, or combinations thereof. Consequently, features of the present invention may be implemented using a processing system (e.g., including one or more processors). Exemplary processors can include, without limitation, one or more general purpose microprocessors (for example, single or multi-core processors), application-specific integrated circuits, application-specific instruction-set processors, graphics processing units, physics processing units, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, and the like.
Features of the present invention can be implemented in, using, or with the assistance of a computer program product which is a storage medium (media) or computer readable medium (media) having instructions stored thereon/in which can be used to program a processing system to perform any of the features presented herein. The storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
Stored on any one of the machine readable medium (media), features of the present invention can be incorporated in software and/or firmware for controlling the hardware of a processing system, and for enabling a processing system to interact with other mechanism utilizing the results of the present invention. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems and execution environments/containers.
Features of the invention may also be implemented in hardware using, for example, hardware components such as application specific integrated circuits (ASICs) and field-programmable gate array (FPGA) devices. Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art.
Additionally, the present invention may be conveniently implemented using one or more conventional general purpose or specialized digital computer, computing device, machine, or microprocessor, including one or more processors, memory and/or computer readable storage media programmed according to the teachings of the present disclosure. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art.
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention.
The present invention has been described above with the aid of functional building blocks illustrating the performance of specified functions and relationships thereof. The boundaries of these functional building blocks have often been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Any such alternate boundaries are thus within the scope and spirit of the invention.
The foregoing description of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments. Many modifications and variations will be apparent to the practitioner skilled in the art. The modifications and variations include any relevant combination of the disclosed features. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalence.

Claims

1-30. (canceled)

31. A method for backtracking data recording, comprising:

receiving a data flow associated with a time sequence from one or more data sources;

storing the data flow in a first storage medium;

receiving, at a time point, a control signal to transfer data from the first storage medium to a second storage medium;

determining a time period in the time sequence according to the time point;

selecting a subset of data in the data flow; and

transferring the subset of data from the first storage medium to the second storage medium;

wherein the time period includes a past portion and a current portion, and the past portion and the current portion are adjacent in the time sequence relative to the time point.

32. The method of claim 31, wherein the past portion is determined based on at least one of a characteristic of the data flow or a characteristic of the first storage medium and the current portion is a difference between the time period and the past portion.

33. The method of claim 31, wherein the past portion equals a maximum image frame number associated with the first storage medium divided by an image speed.

34. The method of claim 31, wherein the control signal is triggered by a button being pushed, a gesture of a user, a movement of an object, or a change of a state.

35. The method of claim 31, wherein the data flow comprises at least one of:

video data captured using an image sensor,

audio data captured using a microphone, or

textual data captured using a keyboard.

36. The method of claim 31, wherein the first storage medium is configured to store a maximum number of image frames, and wherein the time period is based on a maximum number of image frames associated with the first storage medium.

37. The method of claim 31, wherein the received data contains one or more image frames, and the time sequence is associated with an image frame count.

38. The method of claim 31, wherein the first storage medium is configured to store the data flow in a first-in, first-out buffer.

39. The method of claim 31, wherein the subset of data includes:

at least one image frame recorded before receipt of the control signal, and at least one image frame recorded after receipt of the control signal.

40. The method of claim 31, wherein the one or more data sources are configured to capture information for at least one of an unmanned aircraft, an unmanned vehicle, a hand held device, or a robot.

41. A system for backtracking data recording, comprising:

at least one non-transitory computer-readable medium configured to store instructions; and

at least one processor configured to execute the instructions to perform operations comprising:

storing the data flow in a first storage medium;

determining a time period in the time sequence according to the time point;

selecting a subset of data in the data flow; and

42. The system of claim 41, wherein the past portion is determined based on at least one of a characteristic of the data flow or a characteristic of the first storage medium and the current portion is a difference between the time period and the past portion.

43. The system of claim 41, wherein the past portion equals a maximum image frame number associated with the first storage medium divided by an image speed.

44. The system of claim 41, wherein the control signal is triggered by a button being pushed, a gesture of a user, a movement of an object, or a change of a state.

45. The system of claim 41, wherein the data flow comprises at least one of:

video data captured using an image sensor,

audio data captured using a microphone, or

textual data captured using a keyboard.

46. The system of claim 41, wherein the first storage medium is configured to store a maximum number of image frames, and wherein the time period is based on a maximum number of image frames associated with the first storage medium.

47. The system of claim 41, wherein the received data contains one or more image frames, and the time sequence is associated with an image frame count.

48. The system of claim 41, wherein the first storage medium is configured to store the data flow in a first-in, first-out buffer.

49. The system of claim 41, wherein the subset of data includes:

at least one image frame recorded before receipt of the control signal, and

at least one image frame recorded after receipt of the control signal.

50. The system of claim 41, wherein the one or more data sources are configured to capture information for at least one of an unmanned aircraft, an unmanned vehicle, a hand held device, or a robot.