WO2022094772A1

WO2022094772A1 - Position estimation method, following control method, device and storage medium

Info

Publication number: WO2022094772A1
Application number: PCT/CN2020/126243
Authority: WO
Inventors: 李然; 郭亚娜; 许中研; 王帅
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-11-03
Filing date: 2020-11-03
Publication date: 2022-05-12

Abstract

A position estimation method and apparatus, a following control method and apparatus, a device and a storage medium. The following control method comprises: sending to a control terminal a first image captured by a photographing apparatus at a first moment, so that the control terminal identifies a first position, in the first image, of a following target in the image and sends the first position to a movable platform (S101); acquiring first operation information of the movable platform at the first moment (S102); upon receiving the first position, acquiring second operation information of the movable platform at a second moment, and acquiring a second image captured by the photographing apparatus at the second moment (S103); generating an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information (S104); and following the following target according to the estimated position in the second image (S105).

Description

Position estimation method, following control method, device and storage medium

technical field

The present application relates to the technical field related to mobile devices, and in particular, to a position estimation method, a follow-up control method, an apparatus, a movable platform, and a storage medium.

Background technique

When using a movable platform to follow a target, such as using an aircraft to follow the target, the target position (position frame) of the following target is determined by analyzing and processing the images collected by the aircraft to achieve the following target. Due to the large amount of calculation in image processing, the aircraft needs to have high computing power, which increases the cost of the aircraft, and also increases the power consumption of the aircraft and affects the heat dissipation of the aircraft.

SUMMARY OF THE INVENTION

Based on this, the present application provides a position estimation method, a following control method, a device and a storage medium, so as to reduce the following cost and improve the accuracy of target following.

In a first aspect, the present application provides a follow-up control method, which is applied to a movable platform, wherein the movable platform is communicatively connected to a control terminal, the movable platform includes a pan-tilt, and the pan-tilt is equipped with a photographing device, so The photographing device is used to collect images in real time, and the method includes:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the following target in the image in the first image and sends the image to the control terminal. mobile platform;

obtaining the first operation information of the movable platform at the first moment;

When the first position is received, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment;

generating an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information;

The following target is followed according to the estimated position in the second image.

In a second aspect, the present application also provides a method for position estimation, which is applied to a movable platform, wherein the movable platform is communicatively connected to a control terminal, the movable platform includes a pan-tilt, and the pan-tilt is equipped with a photographing device, The photographing device is used to collect images in real time, and the method includes:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the photographing target in the image in the first image and sends it to the available mobile platform;

An estimated position of the shooting target in the second image is generated according to the first position, the first operation information, and the second operation information.

In a third aspect, the present application also provides a follow-up control device, the control device includes a memory and a processor:

the memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

In a fourth aspect, the present application also provides a position estimation device, the control device includes a memory and a processor:

the memory is used to store computer programs;

The processor is configured to execute the computer program and when executing the computer program, implement the following steps:

When receiving the first position, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment;

In a fifth aspect, the present application also provides a movable platform, the movable platform includes a memory and a processor:

the memory is used to store computer programs;

In a sixth aspect, the application further provides a follow-up control system, the flight control system includes a movable platform and a control terminal;

The control terminal is configured to receive the image sent by the movable platform, and identify the first position of the following target in the image in the image, so as to send the first position to the movable platform;

The movable platform is configured to send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first image of the following target in the first image. a position is sent to the movable platform;

The movable platform is used to obtain the first operation information of the movable platform at the first moment;

The movable platform is configured to acquire the second operation information of the movable platform at the second moment when the first position is received, and acquire the second operation information of the photographing device at the second moment. image;

The movable platform is configured to generate an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information;

The movable platform is used for following the following target according to the estimated position in the second image.

In a seventh aspect, the present application also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the above-mentioned following control method, and/or location estimation methods.

In the position estimation method, following control method, device, movable platform, and storage medium provided by the embodiments of the present application, when following the following target, the movable platform sends the collected first image to the control terminal, so that the control The terminal analyzes and processes the first image to obtain the first position of the following target in the first image, and then, when the movable platform receives the first position, obtains the second operation information of the movable platform at the second moment, and obtains The second image captured at the second moment, wherein the second moment is the moment when the movable platform receives the first position, and then conducts information according to the obtained first position, the first operation information and the second operation information After fusion, the estimated position of the following target in the second image is generated, and finally the following target is followed according to the obtained estimated position of the following target in the second image. When the target is following, the control terminal is used to analyze and process the image, which reduces the requirements for the performance of the movable platform, and when processing according to the received processing results, the current position of the following target is estimated by information fusion. , which improves the accuracy of following.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic block diagram of a follow-up control system provided by an embodiment of the present application;

2 is a schematic flowchart of steps of a follow-up control method provided by an embodiment of the present application;

3 is a schematic diagram of a target frame form provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of another target frame form provided by an embodiment of the present application.

5 is a schematic flowchart of a step of recording first operation information provided by an embodiment of the present application;

6 is a schematic diagram of a scene of a target frame position provided by an embodiment of the present application;

7 is a schematic flowchart of a step of following a follow target provided by an embodiment of the present application;

8 is a schematic flowchart of a step of determining a position of a follow target in a world coordinate system provided by an embodiment of the present application;

9 is a schematic diagram of a scene for calculating a target distance provided by an embodiment of the present application;

10 is a schematic flowchart of steps of a position estimation method provided by an embodiment of the present application;

FIG. 11 is a schematic block diagram of a movable platform provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The flowcharts shown in the figures are for illustration only, and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and features in the embodiments may be combined with each other without conflict.

The following control method of the movable platform provided by the embodiment of the present application specifically uses the control terminal to analyze and process the images collected by the movable platform, and then the movable platform performs corresponding data processing according to the result information fed back by the control terminal. Process to achieve following to the follow target.

Exemplarily, the movable platform includes an aircraft, a robot, or an autonomous vehicle, etc., and the movable platform is equipped with a photographing device. For ease of description, the present application takes an unmanned aerial vehicle as an example of a movable platform for detailed description.

Please refer to FIG. 1 . FIG. 1 is a schematic block diagram of a follow-up control system provided by an embodiment of the present application. The following control system in the embodiment of the present application will be described below with reference to FIG. 1 .

As shown in FIG. 1 , the following control system 10 includes a movable platform 100 and a control terminal 200 . The movable platform 100 is connected in communication with the control terminal 200 , wherein the movable platform 100 is used for sending the acquired image to the control terminal 200 . , and make a corresponding response according to the feedback information of the control terminal 200 , the control terminal 200 is used for analyzing and processing the received image, and sending the position obtained by the analysis and processing to the movable platform 100 .

The control terminal 200 includes a mobile communication device, which may be, for example, a mobile phone, a laptop computer, a watch, or the like. The movable platform 100 is a drone.

The control terminal 200 may include a processor, a display, and a display controller. The display controller is electrically connected to the processor and connected to the display for controlling the display to display corresponding content, such as displaying an image.

Wherein, the processor may be a central processing unit (Central Processing Unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (application specific integrated circuits, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The processor may communicate with the display controller, eg, to generate timing signals for triggering the display controller to control the display to display relevant content.

It can be understood that the display controller may be a part of the display, or may be separate from the display. The displays include LCD displays, LED displays, OLED displays, and the like.

It can be understood that the display may be a part of the control terminal 200 , for example, the control terminal 200 is a mobile phone, and the display is a display screen of the mobile phone; of course, it may also be an external display of the control terminal 200 . Correspondingly, the display controller may also be a part of the control terminal 200, or a separate part from the control terminal 200.

In addition, according to an embodiment of the present application, the control terminal 200 can also be replaced by other devices (eg, a controller) that can have data processing capabilities and can perform data feedback.

In the following control system, when the following target needs to be followed, the movable platform 100 uses the camera set by itself to collect images, obtain the first image collected and the first moment of collecting the first image, and then Send the collected first image to the control terminal 200, so that the control terminal 200 can identify and obtain the first position in the first image, and at the same time, the movable platform 100 will obtain the first operation of itself at the first moment information, and then the movable platform 100 receives the first position sent by the control terminal 200 at the second moment, and at the same time obtains its own second operation information at the second moment and the second image captured at the second moment, and then according to The obtained first position, first running information and second running information determine the estimated position of the following target in the second image, and finally follow the following target according to the determined estimated position of the following target in the second image.

In one embodiment, the number of movable platforms 100 is not limited, but each movable platform 100 independently communicates with the control terminal 200, so as to timely and accurately complete the respective following targets. When the number of movable platforms 100 is greater than one, for the control terminal, the image or data sent by each movable platform 100 can be marked, so that the data transmission will not be erroneously transmitted. The data of the mobile platform is transmitted to the second mobile platform.

Please refer to FIG. 2 , which is a schematic flowchart of steps of a follow-up control method provided by an embodiment of the present application. The following control method can be applied to a movable platform, so that when the movable platform follows a target, the following target can be followed accurately and timely.

As shown in FIG. 2 , the following control method includes steps S201 to S205.

S201. Send to the control terminal the first image captured by the photographing device at the first moment, so that the control terminal recognizes the first position of the following target in the first image and sends the image to the movable platform.

The movable platform is connected to the control terminal in communication, so that when the movable platform needs to perform image processing, the collected images are sent to the control terminal, so as to use the control terminal to analyze and process the image, and then receive the returned information from the control terminal. Analyzing and processing results to realize following the following target according to the received analysis and processing results.

In one embodiment, the movable platform uses the mounted photographing device to capture images in real time, so when the movable platform obtains the first image captured at the first moment, the first image is sent to the location associated with the movable platform. The associated control terminal, so that the control terminal identifies the first position of the following target in the first image in the first image, and then sends the identified first position and related information to the movable platform.

Because when the target is followed, the position of the following target in the collected image needs to be in a certain position, such as the middle of the image. At the same time, in order to ensure the display effect of the target following, the size of the following target in the image will also vary. Restriction, the size of the following target in the image cannot be too large or too small. Therefore, when the target is followed, the position and size of the following target in the image can be controlled, such as the position and size of the target frame corresponding to the following target.

Therefore, in an embodiment, when the control terminal analyzes and processes the first image to obtain the first position of the following target in the first image, the obtained first position includes the corresponding position of the following target in the first image. The position and size of the first target box.

For the first target frame corresponding to the following target in the first image, the following target needs to be completely included, and other interference information should be included as little as possible. When the target frame of the target is used, a square frame can be used as the target frame of the following target in the manner shown in FIG. 3 , or a circular frame can be used as the target frame of the following target in the manner shown in FIG. There is no specific limitation, and it is determined according to the actual selection and demand.

In one embodiment, when following a target, it is first necessary to determine the target currently being followed. Therefore, when determining the target to follow, it includes: acquiring an initial image captured by the photographing device, and sending the initial image to the control unit. terminal, so that the control terminal displays the initial image so as to determine the following target.

When determining the following target, the movable platform first collects and acquires an initial image, wherein the initial image contains the following target that needs to be followed, and then sends the collected following target to the control terminal, and then the control terminal passes the The initial image is displayed to determine the following target to be followed. When the following target to be followed is determined in the initial image, it may be selected by manual operation, or may be automatically selected by the control terminal.

For example, after the control terminal receives the initial image sent by the movable platform, the control terminal displays the initial image, and then the controller determines the following target to be followed in the initial image by selecting the target frame. The following target is framed to complete the determination of the following target when it is determined.

For another example, when the control terminal automatically realizes the determination of the following target, when the control terminal receives the initial image sent by the movable platform, it can receive the input target image in advance, and then determine the required basis in the initial image according to the target image. the follow target.

S202. Acquire first operation information of the movable platform at the first moment.

When the movable platform captures an image at the first moment, it will also acquire the first operation information of the movable platform. For the movable platform, it includes a pan/tilt, and the pan/tilt is also equipped with a related photographing device for capturing images in real time.

The collected first operation information includes the position information and attitude of the movable platform at the first moment, wherein the attitude of the movable platform includes the flight direction of the movable platform, and at the same time, the first operation information also includes The attitude information of the gimbal at the first moment.

In one embodiment, the operation information of the movable platform at any time will be recorded in the relevant storage space, that is, the first operation information is pre-recorded and stored, and when it needs to be used, The first operation information corresponding to the movable platform at the first moment can be directly obtained in the relevant storage space.

For the storage space for recording and storing the first operation information, the type and form of the storage space are not limited, and an appropriate type can be selected according to actual requirements.

For example, a circular buffer can be used to store the first operation information to save computing resources, and when the first operation information is stored in this way, since the length of the storage queue is fixed, it can automatically The first running information in the storage space is updated, specifically deleting the oldest data.

For another example, a conventional type of storage space may be used to store the first operation information, that is, any type of storage space may be used to store the first operation information. When using a conventional type of storage space for recording and storage, due to the limited memory of the storage space, data storage cannot be performed without limitation. Therefore, after the mobile platform obtains the first operation information, the first operation will be deleted. information. In order to make the storage space have more sufficient space for use, and avoid abnormal situations of data recording and storage.

In one embodiment, since the first operation information needs to be recorded and stored in advance, referring to FIG. 5 , FIG. 5 is a schematic flowchart of the steps of recording the first operation information provided by an embodiment of the present application.

Wherein, when the first operation information is stored, it includes:

Step S501, collecting the first operation information of the movable platform when the photographing device photographed the first image at the first moment;

Step S502, marking the first operation information according to the identification tag, and storing the first operation information with the identification tag in a preset storage queue;

The first operation information and the first image have the same identification tag, and are used to index the first operation information corresponding to the first image from the preset storage queue.

During the operation of the movable platform, each moment corresponds to different operation information. Therefore, when the photographing device of the movable platform captures the first image at the first moment, it will capture the available information at the first moment. The first operation information of the mobile platform is obtained, and the relevant identification tag is obtained to identify the first operation information, and then the first operation information after the marking is completed is stored in a preset storage queue.

At the same time, when the first operation information is marked with the identification tag, the first image collected at the first moment will also be marked with the identification tag, so that the identification tag can be directly used for indexing later.

The shooting device mounted on the movable platform is used for real-time shooting of images. In practical applications, when the movable platform completes one image shooting, it will record and store the operation information when the image is shot. The queue will store operation information corresponding to several times. The operation information at different times can be obtained through index query when it needs to be used.

In practical applications, when the camera captures an image, the movable platform will collect its own operation information when capturing the captured image. Taking capturing the first image information at the first moment as an example, when capturing the first image, the The first operation information of the mobile platform at the first moment, and then when it is stored, since the image acquisition is real-time, there will also be a lot of operation information. Therefore, when obtaining the first operation information, it is necessary to Distinguish the first operation information from the operation information that has been collected before and the operation information to be collected later, and the method of making the distinction can be determined according to the actual situation, which is not specifically limited.

In one embodiment, when distinguishing the first operation information, the first operation information is marked with a relevant identification mark, that is, each stored operation information is marked with an identification mark, and each operation information is marked with an identification mark. The identifications of the operation information are all different, and are recorded and stored after the marking of the first operation information is completed.

Wherein, the identification label may be a numerical number or other unique identification.

For example, when the identification label is a numerical number, Arabic numerals can be used to mark the collected operation information in sequence, for example, the first collected operation information is 1, and the second collected operation information is 2 , and so on, to mark the collected running information.

In one embodiment, the identification tag may be a time stamp. In practical applications, the operation information is collected at different times. Due to the uniqueness of time, the timestamp corresponding to the collection of the operation information can be used as the identification label of the operation information.

At the same time, when the first operation information is marked with the identification mark, the first image collected at the first moment will also be marked. During the actual operation of the movable platform, the images corresponding to different operation information There may be differences. In order to ensure the accuracy of the following, during the analysis and processing, it is necessary to accurately obtain the shooting status corresponding to the image currently being analyzed and processed, such as the operation information of the movable platform.

In one embodiment, when the tracking of the following target is realized, the first image collected is sent to the associated control terminal through the movable platform, and then the control terminal determines the following target by analyzing and processing the first image. At the first position corresponding to the first image, when the movable platform obtains the first position, it can follow the following target based on the first position and other relevant information. The analysis needs to be completed based on the first operation information of the movable platform at the first moment.

In addition, when the movable platform obtains the operation information, in addition to using the identification tag to mark the first operation information collected at the first moment, it will also use the identification tag to mark the first image, so that the movable platform can use the identification tag to mark the first image according to the The used identification tags are indexed accordingly.

Therefore, when the mobile platform sends the first image collected at the first moment to the control terminal, the first image needs to carry a corresponding identification label. Taking the identification label as a time stamp as an example, the mobile platform sends the first image to the control terminal. When an image is sent to the control terminal, it includes: acquiring a first image captured by the photographing device at a first moment, and using the first moment as a time stamp; marking the first image according to the time stamp, and The first image marked with the time stamp is sent to the control terminal.

The collected first image is marked with a timestamp, so that when receiving the analysis and processing result of the first image by the control terminal, an index can be performed in a preset storage queue according to the marked timestamp, so as to obtain an available value. The first operation information corresponding to the mobile platform at the first moment.

It should be noted that although the identification tags have a certain variety, they will not affect the actual index. When the identification tags are not limited, when the movable platform captures the first image, the acquisition of a certain identification tag will affect the first image. The image is marked, and the use status of the identification label is changed to in use, so as to avoid the situation that an identification label is used multiple times.

For example, if the identification label is a number, any number may be used as the identification label to mark the first image and the first operation information.

For another example, if the identification label is an English letter, any English letter may be used as the identification label to mark the first image and the first operation information.

When the identification label is a number or English letter, there will be no restrictions on the selection of the identification label. It only restricts each number or English letter to be used once. Only when the marked running information and image are invalid, it can be used. use again.

In one embodiment, the identification tag has a certain indexing function for the storage queue, that is, the operation information corresponding to different images is obtained through the identification tag. Therefore, when the identification tag is a timestamp, the first moment of When the first operation information of the movable platform includes: obtaining the timestamp corresponding to the first image; querying and obtaining the available time stamp from the preset storage queue according to the timestamp when the first image was taken. The first operation information of the mobile platform.

When the control terminal sends the first position obtained by the analysis processing to the movable platform, the first position will carry an identification tag for marking the first image, such as a time stamp. Therefore, when acquiring the first operation information, according to The timestamp carried in the first position is queried in a preset storage queue to obtain the first operation information of the movable platform when the first image is captured.

S203. When the first position is received, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment.

When the movable platform receives the first position sent by the control terminal, it obtains the second operation information of the movable platform at the second moment, and also obtains the first position captured by the photographing device of the movable platform at the second moment. Second image.

Wherein, the second moment is the time information when the movable platform receives the first position.

For the movable platform, the photographing device is collecting images in real time, so when the first position sent by the control terminal is received, the second operation information of the movable platform at this time can be obtained in time, and the second operation information of the movable platform can be obtained in time. A second image taken by the movable platform at the moment.

In practical applications, the mobile platform and the control terminal are connected by communication. When the mobile platform performs target tracking, the data processing process during target tracking is placed on the control terminal, and then the mobile platform is based on the data uploaded by the control terminal. The data processing result tracks the following target.

Since there is a certain delay in data transmission, that is, the mobile platform needs to transmit the data to the control terminal for processing after acquiring the relevant data, and the mobile platform is still in the running state at this time. When controlling the data processing result returned by the terminal, the position of the movable platform has changed. If the received result information is used to follow directly at this time, the following situation may not be accurate enough.

S204. Generate an estimated position of the following target in the second image according to the first position, the first operation information, and the second operation information.

After receiving the first position sent by the control terminal, and after acquiring the first operation information and the second operation information in the storage queue, the following target will be generated according to the first position, the first operation information and the second operation information. Estimated position in the second image. Wherein, the estimated position is the position and size of the target frame following the target in the second image.

Considering the delay of data transmission, when the mobile platform receives the data sent by the control terminal, if the mobile platform directly uses the received data as the current reference for following the target, the target will follow accurately. When the following target is followed, it is usually determined whether and how the movable platform needs to be adjusted currently according to the positional relationship between the following target and the movable platform.

Therefore, when the first position is received, the position corresponding to the following target needs to be determined, and then the movable platform can be controlled to follow the following target. For the position corresponding to the following target at this time, use the fusion of the first position, the first operation information and the second operation information to determine the estimated position of the following target in the second image, and then realize the following according to the estimated position. .

In the actual following process, the following method of the movable platform to the following target is pre-set, such as simple straight line following, or complex circle follow, spiral follow, or gradual distance follow, etc. When following the following target, the position of the target frame corresponding to the following target in the real-time captured image will be in a fixed position, such as in the middle of the image, so that the following scene can be better displayed .

At the same time, when following the target, it will always ensure that the target frame corresponding to the following target is in the middle of the image, and the size of the target frame also needs to be controlled at a fixed size, such as a square target frame with a fixed length and width, and a fixed radius. The circular target frame of , taking the target frame as a square target frame as an example, the position of the target frame in the image when accurately following is shown in Figure 6. When the relative positional relationship between the tracking target and the movable platform changes, the captured image and the position of the target frame in the image will also change accordingly, such as the movement of up, down, left and right and the change of the size of the target frame.

For the change of the target frame, the position change of the following target can be estimated and judged based on the principle of taking pictures. Specifically, when the distance between the movable platform and the following target is getting closer, the following target will become larger in the image, and also That is, the size of the target frame will become larger, and when the distance between the movable platform and the following target becomes farther, the following target will become smaller in the image, that is, the size of the target frame will become smaller. Therefore, the distance between the movable platform and the following target can be reflected by the size of the target frame.

In one embodiment, determining the estimated position of the following target in the second image is obtained by fusing the first position, the first operation information and the second operation information. In fact, during the target following process, the first position is determined based on the first image captured at the first moment, so the first position is related to the first running information at the first moment. Therefore, at the second moment The estimated position of the following target in the second image is also related to the second running information at the second moment, and then the estimated position of the following target in the second image at the second moment is determined through fusion between the three.

S205. Follow the following target according to the estimated position in the second image.

After the estimated position of the following target in the second image is determined, the following target will be followed according to the obtained estimated position. In fact, when the target is followed, the following is realized according to the actual position of the following target and the actual position of the movable platform itself.

Therefore, when following the following target according to the estimated position in the second image, it is first necessary to determine the actual position of the following target according to the estimated position, that is, the position in the world space, and then follow according to the actual position.

Specifically, referring to FIG. 7 , FIG. 7 is a schematic flowchart of a step of following a follow target provided by an embodiment of the present application. Specifically, it includes steps S701 to S702.

S701. Determine the position of the following target in the world coordinate system according to the estimated position and the second operation information;

S702. Follow the following target according to the position of the following target in the world coordinate system.

After determining the estimated position of the following target in the second image, since the target following of the movable platform is realized based on the actual position of the following target, after obtaining the estimated position, according to the estimated position and the second operation of the movable platform information, determine the position of the following target in the world coordinate system, and then follow the following target according to the obtained position of the following target in the world coordinate system.

In practical applications, the position of the movable platform in the world coordinate system can be obtained in real time, which is specifically included in the operation information of the movable platform. Therefore, when determining the position of the following target in the world coordinate system, read The position of the movable platform in the world coordinate system in the second operation information, and then by determining the relative positional relationship between the following target and the movable platform, the position of the following target in the world coordinate system is obtained.

In one embodiment, referring to FIG. 8 , when determining the position of the following target in the world coordinate system, it includes:

S801. Determine the target distance from the following target to the movable platform according to the attitude information of the PTZ and the position of the target frame in the second image;

S802. Determine the position of the following target in the world coordinate system according to the target distance and the position information of the movable platform.

When determining the position of the following target in the world coordinate system, it is obtained according to the second operation information at the second moment. Specifically, the attitude information of the gimbal included in the second operation information is obtained, and The position of the target frame in the included second image is to determine the target distance from the following target to the movable platform according to the obtained attitude information of the gimbal and the position of the target frame in the second image, and then according to the obtained The target distance and the position information of the movable platform determine the position of the following target in the world coordinate system, wherein the position information of the movable platform is the position of the movable platform in the world coordinate system.

In one embodiment, when acquiring the attitude information of the gimbal included in the second operation information, the obtained attitude information mainly includes the pitch angle of the gimbal, but for the attitude information of the gimbal, it also includes the yaw angle. and roll angles, respectively represent different directions in the three-dimensional coordinates, and when determining the position of the following target in the world coordinate system, the pitch angle and the actual position of the movable platform can be obtained.

When determining the target distance between the following target and the movable platform according to the attitude information of the gimbal and the position of the target frame in the second image, the relationship between the movable platform and the following target can be constructed, such as forming a horizontal line with a horizontal line. Triangular relationship, and then use the formed triangular relationship to determine the distance from the following target to the movable platform.

For example, in a following scene as shown in Figure 9, the distance between the following target and the movable platform can be obtained as follows, assuming that the height of the following target is L, that is, L=2AB, L is the experience height, which can be advanced in advance set up. The movable platform is located at point O, and the pitch angle in the gimbal posture of the movable platform is θ, that is, ∠AOC=θ, and ∠AOB can be accounted for by the size of the target frame in the second image as a whole in the second image. The ratio of the size is obtained, and the actual distance of the AO can be obtained according to the triangular relationship, that is, the target distance between the following target and the movable platform.

After the obtained target distance between the following target and the movable platform, the position of the following target in the world coordinate system is determined according to the position information of the movable platform. By obtaining the direction of the following target compared to the movable platform, the position of the following target in the world coordinate system is obtained according to the target distance, direction and position information.

When the position of the following target in the world coordinate system is determined, the movable platform will be controlled to follow the following target according to the obtained position according to the preset running trajectory, wherein the preset running trajectory and the pre-determined following method Correspondingly, for example, the preset running track includes one of a revolving track, a spiral track, and an increasing track.

Referring to FIG. 10 , FIG. 10 is a schematic flowchart of steps of a position estimation method provided by an embodiment of the present application. The position estimation method can not only reduce the calculation amount of the movable platform, but also overcome the delay between the movable platform and the control terminal, thereby improving the accuracy of the estimated position. Wherein, the method includes steps S1001 to S1004.

S1001. Send to the control terminal the first image captured by the photographing device at the first moment, so that the control terminal recognizes the first position of the photographing target in the image in the first image and sends the image to the control terminal. said removable platform;

S1002, acquiring first operation information of the movable platform at the first moment;

S1003. When the first position is received, obtain second operation information of the movable platform at a second moment, and obtain a second image captured by the photographing device at the second moment;

S1004. Generate an estimated position of the shooting target in the second image according to the first position, the first operation information, and the second operation information.

Wherein, during the position estimation, the position of the shooting target in the collected image is estimated to determine the actual position of the shooting target.

It should be noted that when performing position estimation, some steps and implementation methods are the same as those in the following control method of the previous solution, specifically steps S201 to S204. Therefore, the specific implementation process of the position estimation method can refer to steps S201 to S201. The embodiment described in S204 is not described in detail here.

In one embodiment, for the position estimation method, in addition to being used in the following control method described above to realize following the following target, it can also be applied in other different scenarios, such as target positioning by a drone or a robot. In the scene, such as the estimated scene of the running trajectory.

In the above-described position estimation method, following control method, device and storage medium, when following the following target, the movable platform sends the collected first image to the control terminal, so that the control terminal can monitor the first image Perform analysis and processing to obtain the first position of the following target in the first image, and then when the movable platform receives the first position, obtains the second operation information of the movable platform at the second moment, and obtains the second operation information at the second moment. The captured second image, wherein the second moment is the moment when the movable platform receives the first position, and then performs information fusion according to the obtained first position, the first operation information and the second operation information, and generates a follow-up The estimated position of the target in the second image, and finally the following target is followed according to the obtained estimated position of the following target in the second image. When the target is following, the control terminal is used to analyze and process the image, which reduces the performance requirements of the movable platform, and when processing according to the received processing results, the current position of the following target is estimated by information fusion. , which improves the accuracy of following.

In another embodiment, the following control method includes:

When receiving the first position, obtain second operation information of the movable platform at a second moment;

obtaining the position of the following target in the world coordinate system according to the first position and the first running information;

The following target is followed according to the position of the following target in the world coordinate system and the second running information.

Wherein, obtaining the position of the following target in the world coordinate system according to the first position and the first operation information can be obtained by referring to the above method;

That is, as shown in FIG. 9 , the distance between the following target and the movable platform can be obtained as follows, assuming that the height of the following target is L, that is, L=2AB, L is the empirical height, which can be set in advance. The movable platform is located at point O, and the pitch angle in the gimbal posture of the movable platform is θ, that is, ∠AOC=θ, and ∠AOB can be accounted for by the size of the target frame in the second image as a whole in the second image. The ratio of the size is obtained, and the actual distance of the AO can be obtained according to the triangular relationship, that is, the target distance between the following target and the movable platform. According to the GPS position of the movable platform, the position of the following target in the world coordinate system can be obtained.

In an embodiment, collecting the first operation information of the movable platform when the photographing device photographs the first image at the first moment;

Mark the first operation information according to the identification tag, and store the first operation information with the identification tag in a preset storage queue;

In one embodiment, the identification tag includes a time stamp.

In an embodiment, the sending to the control terminal the first image captured by the capturing device at the first moment includes:

acquiring a first image captured by the photographing device at a first moment, and using the first moment as a timestamp;

The first image is marked according to the time stamp, and the first image marked with the time stamp is sent to the control terminal.

In an embodiment, the first location sent by the control terminal includes a time stamp;

The acquiring the first operation information of the movable platform at the first moment includes:

obtaining the timestamp corresponding to the first image;

The first operation information of the movable platform when the first image is captured is queried and acquired from the preset storage queue according to the time stamp.

In one embodiment, after querying from the preset storage queue according to the timestamp and acquiring the first operation information of the movable platform when the first image is captured, the method includes:

Delete the first operation information.

In one embodiment, the storage queue includes a queue with a preset length for storing a preset number of first operation information.

In one embodiment, the movable platform comprises a drone or a robot.

In an embodiment, the following the following target according to the position of the following target in the world coordinate system includes:

According to the position of the following target in the world coordinate system, follow the following target according to the preset running track;

Wherein, the preset running track includes one of a circle track, a spiral track, and an increasing track.

Please refer to FIG. 11 , which is a schematic block diagram of a movable platform provided by an embodiment of the present application. The movable platform 21 includes a processor 211 and a memory 212, and the processor 211 and the memory 212 are connected through a bus, such as an I2C (Inter-integrated Circuit) bus.

Specifically, the processor 211 may be a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP) or the like.

Specifically, the memory 212 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, a mobile hard disk, and the like.

Wherein, the movable platform is connected to the control terminal in communication, the movable platform includes a pan-tilt, and the pan-tilt is equipped with a photographing device, and the photographing device is used for capturing images in real time, and the processor 211 is used for running storage A computer program in memory, and implementing the following steps when executing the computer program:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the following target in the first image in the first image and sends it to the control terminal. said removable platform;

In some embodiments, the first operation information and the second operation information include position information and attitude of the movable platform, and attitude information of the gimbal.

In some embodiments, when implementing the method, the processor 211 specifically implements:

collecting the first operation information of the movable platform when the photographing device photographed the first image at the first moment;

In some embodiments, the identification tag includes a time stamp.

In some embodiments, when the processor 211 implements the sending to the control terminal the first image captured by the photographing device at the first moment, the processor 211 specifically implements:

In some embodiments, the first position sent by the control terminal includes a timestamp, and when the processor 211 implements the acquiring of the first operating information of the movable platform at the first moment, the processor 211 specifically implements:

obtaining the timestamp corresponding to the first image;

In some embodiments, after the processor 211 implements the query from the preset storage queue according to the timestamp and obtains the first operation information of the movable platform when the first image is captured, Implementation:

Delete the first operation information.

In some embodiments, the storage queue includes a queue with a preset length for storing a preset number of first operation information.

The initial image captured by the photographing device is acquired, and the initial image is sent to the control terminal, so that the control terminal displays the initial image, so as to determine the following target.

In some embodiments, when the processor 211 implements the following to the following target according to the estimated position in the second image, the processor 211 specifically implements:

determining the position of the following target in the world coordinate system according to the estimated position and the second operation information;

The following target is followed according to the position of the following target in the world coordinate system.

In some embodiments, the first position includes the position and size of a first target frame corresponding to the following target in the first image, and the estimated position includes the following target in the second image The position and size of the target box in the corresponding second image.

In some embodiments, when the processor 211 determines the position of the following target in the world coordinate system according to the estimated position and the second operation information, the processor 211 specifically implements:

Determine the target distance from the following target to the movable platform according to the attitude information of the PTZ and the position of the target frame in the second image;

According to the target distance and the position information of the movable platform, the position of the following target in the world coordinate system is determined.

In some embodiments, when the processor 211 implements the following to the following target according to the position of the following target in the world coordinate system, the processor 211 specifically implements:

According to the position of the following target in the world coordinate system, follow the following target according to the preset running trajectory;

In some embodiments, the movable platform includes a drone or a robot.

An embodiment of the present application provides a follow-up control device, the follow-up control device includes a memory and a processor, and the memory is used for storing a computer program; the processor is used for executing the computer program and executing the computer program. When the computer program is executed, the following control method described above is realized.

An embodiment of the present application provides a position estimation apparatus, where the position estimation apparatus includes a memory and a processor, where the memory is used for storing a computer program; the processor is used for executing the computer program and executing the computer program. When the computer program is executed, the position estimation method described above is realized.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement the above implementation The example provides the steps for following the control method and/or position estimation.

The computer-readable storage medium may be an internal storage unit of the removable platform described in any of the foregoing embodiments, such as a hard disk or a memory of the removable platform. The computer-readable storage medium can also be an external storage device of the removable platform, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) equipped on the removable platform , SD) card, flash memory card (Flash Card), etc.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in the present application. Modifications or substitutions shall be covered by the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A following control method, characterized in that it is applied to a movable platform, the movable platform is connected to a control terminal in communication, the movable platform includes a pan/tilt, and the pan/tilt is equipped with a photographing device, and the photographing device uses In real-time acquisition of images, the method includes:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the following target in the first image in the first image and sends it to the control terminal. said removable platform;

obtaining the first operation information of the movable platform at the first moment;

When receiving the first position, acquiring the second operation information of the movable platform at the second moment, and acquiring the second image captured by the photographing device at the second moment;

generating an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information;

The following target is followed according to the estimated position in the second image.
The method according to claim 1, wherein the first operation information and the second operation information include position information and attitude of the movable platform, and attitude information of the PTZ.
The method of claim 1, wherein the method comprises:

collecting the first operation information of the movable platform when the photographing device photographed the first image at the first moment;

Mark the first operation information according to the identification tag, and store the first operation information with the identification tag in a preset storage queue;

The first operation information and the first image have the same identification tag, and are used to index the first operation information corresponding to the first image from the preset storage queue.
4. The method of claim 3, wherein the identification tag includes a time stamp.
The method according to claim 4, wherein the sending to the control terminal the first image captured by the capturing device at the first moment comprises:

Acquiring the first image taken by the photographing device at the first moment, and using the first moment as a time stamp;

The first image is marked according to the time stamp, and the first image marked with the time stamp is sent to the control terminal.
The method according to claim 3, wherein the first position sent by the control terminal includes a time stamp;

The acquiring the first operation information of the movable platform at the first moment includes:

obtaining the timestamp corresponding to the first image;

The first operation information of the movable platform when the first image is captured is queried and acquired from the preset storage queue according to the time stamp.
The method according to claim 6, wherein after querying from the preset storage queue according to the time stamp and acquiring the first operation information of the movable platform when the first image is captured, include:

Delete the first operation information.
The method according to claim 3, wherein the storage queue comprises a queue with a preset length for storing a preset number of first operation information.
The method of claim 1, wherein the method comprises:

The initial image captured by the photographing device is acquired, and the initial image is sent to the control terminal, so that the control terminal displays the initial image, so as to determine the following target.
The method according to claim 2, wherein the following the tracking target according to the estimated position in the second image comprises:

determining the position of the following target in the world coordinate system according to the estimated position and the second operation information;

The following target is followed according to the position of the following target in the world coordinate system.
The method according to claim 10, wherein the first position comprises the position and size of a first target frame corresponding to the following target in the first image, and the estimated position comprises the following The position and size of the target frame in the second image corresponding to the target in the second image.
The method according to claim 11, wherein the determining the position of the following target in the world coordinate system according to the estimated position and the second operation information comprises:

Determine the target distance from the following target to the movable platform according to the attitude information of the PTZ and the position of the target frame in the second image;

According to the target distance and the position information of the movable platform, the position of the following target in the world coordinate system is determined.
The method according to claim 10, wherein the following the following target according to the position of the following target in the world coordinate system comprises:

According to the position of the following target in the world coordinate system, follow the following target according to the preset running trajectory;

Wherein, the preset running track includes one of a circle track, a spiral track, and an increasing track.
The method of any one of claims 1 to 13, wherein the movable platform comprises a drone or a robot.
A position estimation method, characterized in that it is applied to a movable platform, the movable platform is connected to a control terminal in communication, the movable platform includes a pan/tilt, and the pan/tilt is equipped with a photographing device, and the photographing device uses In real-time acquisition of images, the method includes:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the photographing target in the image in the first image and sends it to the available mobile platform;

obtaining the first operation information of the movable platform at the first moment;

When the first position is received, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment;

An estimated position of the shooting target in the second image is generated according to the first position, the first operation information, and the second operation information.
The method according to claim 15, wherein the first operation information and the second operation information include position information and attitude of the movable platform, and attitude information of the PTZ.
The method of claim 16, wherein the method comprises:

collecting the first operation information of the movable platform when the photographing device photographed the first image at the first moment;

Mark the first operation information according to the identification tag, and store the first operation information with the identification tag in a preset storage queue;

Wherein, the first operation information and the first image have the same identification label, and are used to index the first operation information corresponding to the first image from the preset storage queue.
18. The method of claim 17, wherein the identification tag includes a time stamp.
The method according to claim 18, wherein the sending to the control terminal the first image captured by the capturing device at the first moment comprises:

acquiring a first image captured by the capturing device at the first moment, and using the time at which the first image was captured as a time stamp;

The first image is marked according to the time stamp, and the first image marked with the time stamp is sent to the control terminal.
The method according to claim 17, wherein the first position sent by the control terminal includes a time stamp;

The acquiring the first operation information of the movable platform at the first moment includes:

obtaining the timestamp corresponding to the first image;

The first operation information of the movable platform when the first image is captured is queried and acquired from the preset storage queue according to the time stamp.
The method according to claim 20, wherein after querying from the preset storage queue according to the time stamp and acquiring the first operation information of the movable platform when the first image is captured, include:

Delete the first operation information.
The method according to claim 17, wherein the storage queue comprises a queue with a preset length for storing a preset number of first operation information.
The method of claim 14, wherein the method comprises:

An initial image captured by the photographing device is acquired, and the initial image is sent to the control terminal, so that the control terminal displays the initial image, so as to determine a photographing target.
The method according to claim 15, wherein the first position includes the position and size of a target frame corresponding to the shooting target in the first image, and the estimated position includes the shooting target at The position and size of the corresponding target frame in the second image.
The method of any one of claims 15 to 24, wherein the movable platform comprises a drone or a robot.
A follow-up control device, characterized in that the control device comprises a memory and a processor:

the memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the following target in the image in the first image and sends the image to the control terminal. mobile platform;

obtaining the first operation information of the movable platform at the first moment;

When the first position is received, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment;

generating an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information;

The following target is followed according to the estimated position in the second image.
A position estimation device, characterized in that the control device includes a memory and a processor:

the memory is used to store computer programs;

The processor is configured to execute the computer program and implement the following steps when executing the computer program:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the photographing target in the image in the first image and sends it to the available mobile platform;

obtaining the first operation information of the movable platform at the first moment;

When receiving the first position, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment;

An estimated position of the shooting target in the second image is generated according to the first position, the first operation information, and the second operation information.
A movable platform, characterized in that the movable platform includes a memory and a processor:

the memory is used to store computer programs;

The processor is configured to execute the computer program and when executing the computer program, implement the following steps:

Send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first position of the following target in the image in the first image and sends the image to the control terminal. mobile platform;

obtaining the first operation information of the movable platform at the first moment;

When the first position is received, obtain the second operation information of the movable platform at the second moment, and obtain the second image captured by the photographing device at the second moment;

generating an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information;

The following target is followed according to the estimated position in the second image.
The movable platform according to claim 28, wherein the first operation information and the second operation information include position information and attitude of the movable platform, and attitude information of the pan/tilt head.
The movable platform of claim 28, wherein the method comprises:

collecting the first operation information of the movable platform when the photographing device photographed the first image at the first moment;

Mark the first operation information according to the identification tag, and store the first operation information with the identification tag in a preset storage queue;

The first operation information and the first image have the same identification tag, and are used to index the first operation information corresponding to the first image from the preset storage queue.
31. The movable platform of claim 30, wherein the identification tag includes a time stamp.
The movable platform according to claim 31, wherein the sending to the control terminal the first image captured by the capturing device at the first moment comprises:

acquiring a first image captured by the photographing device at a first moment, and using the first moment as a timestamp;

The first image is marked according to the time stamp, and the first image marked with the time stamp is sent to the control terminal.
The movable platform according to claim 30, wherein the first position sent by the control terminal includes a time stamp;

The acquiring the first operation information of the movable platform at the first moment includes:

obtaining the timestamp corresponding to the first image;

The first operation information of the movable platform when the first image is captured is queried and acquired from the preset storage queue according to the time stamp.
The movable platform according to claim 33, wherein the first operation information of the movable platform when the first image is captured is inquired and acquired from the preset storage queue according to the timestamp. After that, include:

Delete the first operation information.
The movable platform according to claim 30, wherein the storage queue comprises a queue with a preset length for storing a preset number of first operation information.
The movable platform of claim 28, wherein the method comprises:

The initial image captured by the photographing device is acquired, and the initial image is sent to the control terminal, so that the control terminal displays the initial image, so as to determine the following target.
The movable platform according to claim 29, wherein the following the tracking target according to the estimated position in the second image comprises:

determining the position of the following target in the world coordinate system according to the estimated position and the second operation information;

The following target is followed according to the position of the following target in the world coordinate system.
The movable platform according to claim 37, wherein the first position includes the position and size of the first target frame corresponding to the following target in the first image, and the estimated position includes the the position and size of the target frame in the second image corresponding to the following target in the second image.
The movable platform according to claim 38, wherein determining the position of the following target in the world coordinate system according to the estimated position and the second operation information comprises:

Determine the target distance from the following target to the movable platform according to the attitude information of the PTZ and the position of the target frame in the second image;

According to the target distance and the position information of the movable platform, the position of the following target in the world coordinate system is determined.
The movable platform according to claim 37, wherein, the following target is followed according to the position of the following target in the world coordinate system, comprising:

According to the position of the following target in the world coordinate system, follow the following target according to the preset running trajectory;

Wherein, the preset running track includes one of a circle track, a spiral track, and an increasing track.
The movable platform of any one of claims 28 to 40, wherein the movable platform comprises a drone or a robot.
A following control system, characterized in that the flight control system includes a movable platform and a control terminal;

The control terminal is configured to receive the image sent by the movable platform, and identify the first position of the following target in the image in the image, so as to send the first position to the movable platform;

The movable platform is configured to send the first image captured by the photographing device at the first moment to the control terminal, so that the control terminal recognizes the first image of the following target in the first image. a position is sent to the movable platform;

The movable platform is used to obtain the first operation information of the movable platform at the first moment;

The movable platform is configured to acquire the second operation information of the movable platform at the second moment when the first position is received, and acquire the second information captured by the photographing device at the second moment. image;

The movable platform is configured to generate an estimated position of the following target in the second image according to the first position, the first operation information and the second operation information;

The movable platform is used for following the following target according to the estimated position in the second image.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following control according to any one of claims 1 to 14 is realized method, and/or implement a position estimation method as claimed in claims 15 to 25.