WO2021127888A1

WO2021127888A1 - Control method, smart glasses, mobile platform, gimbal, control system, and computer-readable storage medium

Info

Publication number: WO2021127888A1
Application number: PCT/CN2019/127562
Authority: WO
Inventors: 黄敏
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-12-23
Filing date: 2019-12-23
Publication date: 2021-07-01
Also published as: CN113508351A

Abstract

Provided in an embodiment of the present application are a control method, smart glasses, a mobile platform, a gimbal, a control system, and a storage medium. The method is applicable to an apparatus having a collection device. The collection device is configured to collect in real time posture data of a user. The method comprises: acquiring posture data from the collection device, comparing the same with specified posture data, and determining whether to generate a control instruction according to the comparison result; if so, sending the generated control instruction to a mobile platform, wherein the mobile platform is installed with a camera thereon, and the control instruction is configured to adjust a field of view range of the camera. The present embodiment uses body-sensing data to realize the adjustment of a camera, reducing the operation steps of a user and improving user experience.

Description

Control method, smart glasses, movable platform, pan-tilt, control system and computer readable storage medium

Technical field

This application relates to the field of human-computer interaction technology, and in particular to a control method, smart glasses, a movable platform, a pan-tilt, a control system, and a computer-readable storage medium.

Background technique

In order to get a better shooting experience and better shooting works, people are not satisfied with the process of shooting only through the camera of a mobile terminal such as a mobile phone. With the development of network technology, auxiliary cameras or other shooting devices are used to shoot. The number of equipment is also increasing, such as tracking and shooting through unmanned vehicles, or aerial photography through drones and so on. Although better shooting effects can be obtained with the help of auxiliary equipment, the user needs to control the auxiliary equipment through the relevant remote control device during the shooting process. In some scenarios, the user needs to control the auxiliary equipment while watching the real-time picture, which requires high operating requirements. , The operation is cumbersome, resulting in poor user experience.

Summary of the invention

In view of this, one of the objectives of this application is to provide a control method, smart glasses, a movable platform, a pan-tilt, a control system, and a computer-readable storage medium.

First, according to the first aspect of the embodiments of the present application, a control method is provided, which is applied to a device equipped with a collection device, the collection device is used for real-time collection of user posture data, and the method includes:

Obtain the posture data from the acquisition device and compare it with the specified posture data, and determine whether to generate a control instruction according to the comparison result;

If yes, the generated control instruction is sent to a movable platform; a camera is installed on the movable platform, and the control instruction is used to adjust the field of view captured by the camera. According to a second aspect of the embodiments of the present application, a smart glasses are provided, including:

processor;

A memory for storing processor executable instructions;

A collection device for real-time collection of user posture data;

Wireless communication device

Wherein, the processor calls the executable instruction, and when the executable instruction is executed, it is used to execute the control method according to any one of the first aspect.

According to a third aspect of the embodiments of the present application, a movable platform is provided, the movable platform is installed with a camera, and the movable platform includes:

Body

A power system, installed in the body, used to provide power to the movable platform;

A wireless communication system is installed in the body and used to receive control instructions sent by the smart glasses as described in the second aspect; and a control system is installed in the body and used to adjust the control instructions according to the control instructions. The field of view captured by the camera.

According to a fourth aspect of the embodiments of the present application, a pan-tilt is provided, including:

Gimbal axis

Angle sensor used to collect the angle information of the pan/tilt axis;

A communication device for receiving a control instruction sent by the movable platform according to any one of the third aspect; and,

The processor is configured to control the rotation of the pan-tilt axis according to the control instruction and the angle information to move from the current joint angle to the target joint angle.

According to a fifth aspect of the embodiments of the present application, a control system is provided, including the smart glasses according to any one of the second aspect, the movable platform according to any one of the third aspect, and a camera; the camera is installed On the movable platform.

According to a sixth aspect of the embodiments of the present application, there is provided a computer-readable storage medium having computer instructions stored thereon, and when the instructions are executed by a processor, the method described in any one of the first aspects is implemented.

The embodiments of the application have beneficial effects if:

The device collects the user's posture data in real time through the collecting device, and then determines to generate a control instruction for adjusting the field of view shot by the camera based on the comparison result of the posture data and the specified posture data, and realizes the adjustment process by using somatosensory without the user's additional Manual operation reduces the user's operation steps, which is beneficial to improve the user's use experience, and there is no need to separately design corresponding remote control equipment, which is beneficial to save hardware expenditures.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure of this specification.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained from these drawings without creative labor.

Fig. 1A is a flowchart of a control method according to an exemplary embodiment of the application.

Fig. 1B is a schematic diagram of a head-mounted display device according to an exemplary embodiment of the application.

Fig. 2 is a flowchart of a second control method according to an exemplary embodiment of the application.

Fig. 3 is a structural diagram of a smart glasses according to an exemplary embodiment of the application.

Fig. 4 is a structural diagram of a movable platform according to an exemplary embodiment of the application.

Fig. 5 is a structural diagram of a camera according to an exemplary embodiment of the present application.

Fig. 6 is a structural diagram of a pan/tilt head according to an exemplary embodiment of the application.

Fig. 7 is a structural diagram of a control system according to an exemplary embodiment of the application.

Fig. 8 is a structural diagram of another control system according to an exemplary embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The flowchart shown in the drawings is only an example, and does not necessarily include all contents and operations/steps, nor does it have to be executed in the described order. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to actual conditions.

Hereinafter, some embodiments of this specification will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

In response to problems in related technologies, the embodiments of the present application provide a control method, which can be used on a device equipped with a collection device. The device can be a wearable device with a display function, such as smart glasses, helmets, and smart glasses. For bracelets, watches, chest straps, armbands, tops, waistbands or protective belts, etc., please refer to FIG. 1A, which is a flowchart of a control method according to an exemplary embodiment of the present application. The method includes:

In step S101, the posture data is acquired from the acquisition device and compared with the specified posture data, and whether to generate a control instruction is determined according to the comparison result.

In step S102, if yes, the generated control instruction is sent to a camera, a movable platform; a camera is installed on the movable platform, and the control instruction is used to adjust the field of view captured by the camera.

In an exemplary application scenario, a camera is installed on the movable platform, and the camera is used to take real-time pictures of the movable platform during the movement and transmit it to the device. The device includes a display. The device receives the real-time image transmitted by the camera and displays it on the display, so that the user can view the real-time image of the first angle of view collected by the camera through the display, wherein the real-time image of the acquisition device on the device Collect the user's posture data, compare the posture data with the specified data, and send the generated control instruction to the movable platform in the case of determining to generate a control instruction according to the comparison result, and use somatosensory to achieve The control of the movable platform realizes the adjustment of the field of view shot by the camera and reduces the user's operation steps, so that the user can see the images in the different field of view shot by the camera through the display of the device, and there is no need to set Another remote control device is helpful to reduce the cost of hardware expenditure.

As an example, referring to FIG. 1B, the device may be a head-mounted display device such as smart glasses, and the screen shot by the camera is displayed in real time through the display, thereby enriching the user's visual experience.

In another exemplary application scenario, a camera is installed on the movable platform, and the camera is used for real-time shooting of a designated user. The movable platform realizes tracking and shooting of the designated user, taking into account that the user is moving The camera may go out of the field of view captured by the camera, so that the camera cannot capture the specified user. Therefore, in this embodiment, the user’s posture data can be collected in real time by the collection device on the device, and the posture The data is compared with the specified data, and in the case that the control instruction is determined to be generated according to the comparison result, the generated control instruction is sent to the movable platform, and the somatosensory is used to realize the control of the movable platform, thereby realizing the comparison The adjustment of the field of view captured by the camera reduces the user's operation steps, so that the adjusted camera can capture the designated user, and achieve continuous tracking of the designated user, and there is no need to set up additional remote control equipment, which is beneficial to reduce the cost of hardware expenditures .

Of course, the embodiment of the application does not impose any restrictions on its installation method, and can be specifically set according to actual conditions. As an example, the camera can be fixedly installed on the movable platform, or can be detachably installed on the platform. The movable platform; wherein, the movable platform includes, but is not limited to, unmanned aerial vehicles, unmanned vehicles, unmanned boats, and mobile robots.

In an embodiment, the collecting device collects the posture data of the user wearing the device in real time. The posture data may be at least one of a pitch angle, a yaw angle, and a roll angle, and the posture data reflects the user’s The posture of the head or limbs relative to the ground; it is understood that this application does not impose any restrictions on the specific type of the collection device, and can be specifically set according to actual application scenarios. As an example, the collection device may be an inertial measurement device. Unit, the inertial measurement unit includes three single-axis accelerometers and three single-axis gyroscopes, the accelerometer detects the user's independent three-axis acceleration signal in the carrier coordinate system, and the gyroscope detects the carrier relative to the navigation coordinate The angular velocity signal of the system, so that the inertial measurement unit measures the angular velocity and acceleration of the user in the three-dimensional space, and calculates the user’s posture based on this, and obtains the user’s posture data; in another example, the acquisition device may also be A posture sensor, the posture sensor includes a three-axis gyroscope, a three-axis accelerometer, a three-axis electronic compass and other motion sensors.

Exemplarily, the device may obtain the attitude data of the pitch angle, yaw angle, roll angle, or any combination of the three of the user's head or limbs sensed by the inertial measurement unit, and the device may be pre-stored There is a correspondence between designated posture data and control instructions. After acquiring the posture data, the device can compare the posture data with the designated posture data, if the posture data is the same as the designated posture data , Or the posture data is within the preset range of the designated posture data, the device can generate corresponding control instructions based on the designated posture data and the pre-stored corresponding relationship and send them to the movable platform, so A camera is installed on the movable platform, and the control instruction is used to adjust the field of view shot by the camera.

It is understandable that the embodiments of this application do not impose any restrictions on the specific settings of the preset range, and specific settings can be made according to actual conditions; in addition, the movable platform may establish a connection with the device in advance, so that the For data transmission between the movable platform and the device, the embodiment of the application does not impose any restrictions on the connection mode between the movable platform and the device. As an example, the connection between the movable platform and the device The connection can be achieved by accessing a wireless network based on a communication standard, such as WiFi, 3G or 4G, or a combination thereof; as an example, the mobile platform receives the broadcast signal or broadcast related information from the device via a broadcast channel, Thus, a connection is established; as an example, the mobile platform and the device can establish a connection through near field communication (NFC), for example, based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) ) Technology, Bluetooth (BT) technology and other technologies.

In another exemplary embodiment, the device may also establish an instruction classification model in advance, and the instruction classification model is used to determine the corresponding control instruction according to the comparison result of the posture data and the specified posture data. The device may input the posture data into the instruction classification model to obtain the control instruction.

Wherein, the instruction classification model can be obtained by training the deep neural network to convergence according to a preset sample set. The sample set includes a number of designated posture data and corresponding control instructions. Therefore, the trained instruction classification model can more accurately determine the corresponding control instruction based on the collected posture data.

In an embodiment, the camera is fixedly installed on the movable platform, and the control instruction is specifically used to: control the movable platform to adjust its own movement mode; after the movable platform receives the control instruction , Can adjust its own movement mode according to the movable platform, such as changing the moving direction, etc., thereby indirectly adjusting the field of view taken by the camera.

In another embodiment, the movable platform is equipped with a rotatable pan/tilt, the pan/tilt is used to support the camera, and the movable platform sends the control instruction after receiving the control instruction To the pan-tilt, the control instruction is specifically used to: control the pan-tilt to move from the current joint angle to the target joint angle; after receiving the control instruction, the pan-tilt moves from the current joint according to the control instruction To the target joint angle, thereby indirectly adjusting the field of view captured by the camera.

Wherein, the movable platform and the pan/tilt may be connected in a wired or wireless manner. The embodiment of the present application does not impose any restriction on this, and specific settings can be made according to actual application scenarios.

In this embodiment, the device collects the user's posture data in real time through the collecting device, and then determines based on the posture data to generate a control instruction for adjusting the field of view shot by the camera, and uses somatosensory to realize the control of the movable platform or the pan/tilt. No additional manual operation by the user is required, which reduces the user's operation steps, which is beneficial to improve the user's use experience, and there is no need to separately design a corresponding remote control device, which is beneficial to save the cost of hardware expenditure.

It should be noted that, considering that the actual types of movable platforms are different, there are also differences in control instructions related to movement. Therefore, the embodiment of the present application does not impose any restrictions on the specific instruction types of control instructions, which can be based on actual application scenarios. Make specific settings.

Exemplarily, the control instruction may include at least one of a movement control instruction and a direction adjustment instruction. The movement control instruction is used to control the movement of the movable platform, and the direction adjustment instruction is used to adjust the movable platform. For the movement direction of the platform, the electronic device determines the movement control instruction or direction adjustment instruction according to the attitude data after acquiring attitude data such as pitch angle, yaw angle, roll angle, or any combination of the three from the collecting device.

As one of the implementation manners, the device compares the posture data with the pre-stored designated posture data, and then determines a movement control instruction or a direction adjustment instruction according to the comparison result and the pre-stored corresponding relationship, and The generated movement control instruction or direction adjustment instruction is sent to the movable platform, so that the movable platform can move according to the movement control instruction, or adjust its own movement direction according to the direction adjustment instruction; this embodiment is based on The user's posture data controls the movement mode of the movable platform, which is beneficial to reduce the user's operation steps and improve the user's experience.

In one embodiment, when the user's head or limbs perform different movements, in the posture data collected by the collecting device, the changes between the different parameters included in the posture data are also different, the implementation of this application For example, there is no restriction on the control instructions generated by the posture data determination, and the correspondence between the designated posture data corresponding to different actions and different control instructions can be set according to actual application scenarios.

Exemplarily, when the user nods or raises his head, the collection device in the device can collect pitch angle data with a large variation, while other angle data may not change or the variation is small, and the device can be pre-stored with Correspondence between the designated posture data corresponding to nodding or tilting the head and the control instructions (such as movement control instructions); or when the user shakes or sways his head left and right, the collecting device in the device can collect the yaw angle with a large variation range Data or roll angle data, while other angle data may not change or the magnitude of change is not large. The device can pre-store the correspondence between the designated posture data corresponding to the left and right shaking or head shaking and the control instructions (such as direction adjustment instructions), and then After the device collects the user's posture data, it can compare with the specified posture data to determine whether to generate a corresponding control instruction, so as to realize the control of the movable platform or the pan-tilt.

It should be noted that the above is only an example and does not constitute a limitation to the embodiment of the present application. Other actions such as squatting, walking and other actions corresponding to the designated posture data corresponding to the corresponding relationship of the control instruction can also be set. Do not make any restrictions.

In another exemplary application scenario, a camera is installed on the movable platform, and the camera is used for real-time shooting of a designated user, and the movable platform realizes tracking and shooting of the designated user. Further, based on the actual situation Application requirements. The camera is used to photograph parts of the designated user, such as the face, hands, upper body or lower body of the designated user. In this scenario, since the field of view captured by the camera is small, During the shooting process, if the user performs a fast movement or other large-scale actions, the user is easily out of the shooting range of the camera, resulting in that the camera cannot capture the specified user, thereby failing to achieve the purpose of tracking and shooting.

Based on this, as one of the implementation manners, the device in the embodiment of the present application may also send the posture data obtained from the collection device to the movable platform, and the movable platform obtains the user image taken by the camera. , And after receiving the posture data, according to the posture data and the user image taken by the camera, the control instruction is generated, so as to adjust the field of view taken by the camera according to the control instruction, which is implemented in this embodiment The adjustment process based on user posture data and user images. Since the camera is mainly used to photograph parts of the designated user, it is adjusted according to user-related content (user posture data and user data), which is beneficial to improve tracking The accuracy of shooting.

Wherein, the movable platform may establish a connection with the camera in advance to facilitate the data transmission between the movable platform and the camera. The embodiment of the present application relates to the connection between the movable platform and the camera. There is no restriction on the method. As an example, the mobile platform and the camera can be connected to a wireless network based on a communication standard, such as WiFi, 3G, or 4G, or a combination thereof.

In an embodiment, the camera is fixedly installed on the movable platform, and the control instruction is specifically used to: control the movable platform to adjust its own movement mode; after the movable platform receives the control instruction , Can adjust its own movement mode according to the movable platform, such as changing the moving direction, etc., thereby indirectly adjusting the field of view taken by the camera and realizing accurate tracking of the user.

In another embodiment, the movable platform is equipped with a rotatable pan/tilt, and the pan/tilt is used to support the camera. After the control instruction is generated by the movable platform, the control instruction is sent to For the pan/tilt head, the control instruction is specifically used to control the pan/tilt head to move from the current joint angle to the target joint angle; after receiving the control instruction, the pan/tilt head moves from the current joint to the target joint angle according to the control instruction. The target joint angle indirectly adjusts the field of view captured by the camera and realizes accurate tracking of the user.

As another implementation manner, the device may obtain posture data from the collection device and obtain a user image taken by a camera, and then generate the control instruction based on the posture data and the user image, and send it to the station The movable platform enables the movable platform to adjust the field of view taken by the camera according to the control instruction. In this embodiment, the adjustment process based on user posture data and user images is implemented, because the camera is mainly used In shooting a part of the designated user, the adjustment is made according to user-related content (user posture data and user data), which is beneficial to improve the accuracy of tracking and shooting.

Please refer to FIG. 2, which is a flowchart of a second control method according to an exemplary embodiment of this application. The method includes:

In step S201, the posture data is acquired from the acquisition device and compared with the specified posture data, and whether to generate a control instruction is determined according to the comparison result. It is similar to step S101 and will not be repeated here.

In step S202, if yes, the generated control instruction is sent to a movable platform; a camera is installed on the movable platform, and the control instruction is used to adjust the field of view shot by the camera. It is similar to step S102 and will not be repeated here.

In step S203, a user image taken by the camera is acquired; the camera is used to take a picture of a designated user.

In step S204, the control instruction sent to the movable platform is generated according to the user gesture recognized from the user image.

In an embodiment, the movable platform may track a designated user in real time through the camera, and the camera is used to photograph the designated user in real time, and then transmit the captured user image to the device, then the device may Acquire a user image taken by the camera, and then perform image recognition on the image to recognize the user gesture, and generate the control instruction sent to the movable platform according to the user gesture recognized from the user image, In this way, the field of view captured by the camera is adjusted; in this embodiment, there is no need to set up a remote control device to control the mobile platform, and no manual operation by the user is required. The control of the movable platform can be achieved by recognizing the user's gestures, reducing the number of users. The operation steps are conducive to improving the user experience and also conducive to saving hardware expenditure costs.

As one of the implementation manners, the corresponding relationship between the designated gesture and the control instruction may be pre-stored on the device. Different designated gestures correspond to different control instructions. After the device acquires the user image taken by the camera, The user gesture is recognized in the user image, and the user gesture is compared with the designated gesture, and the corresponding control instruction is generated according to the comparison result and the corresponding relationship between the designated gesture and the control instruction, and the corresponding control instruction is generated according to the comparison result and the corresponding relationship between the designated gesture and the control instruction. The control instruction controls the movement mode of the movable platform or the posture of the pan/tilt; as an example, the control instruction includes at least one of a movement control instruction, a direction adjustment instruction, and a posture adjustment instruction. The movement control instruction is used to control the movement of the platform. The movable platform moves, the direction adjustment instruction is used to adjust the moving direction of the movable platform, and the posture adjustment instruction is used to control the pan/tilt head to move from the current joint to the target joint angle.

It is understandable that the embodiment of the present application does not impose any restrictions on the number of the designated users and the number of user gestures obtained from the user images. It can include one designated user, or two and two. The above designated user; can include one user gesture, or two or more user gestures.

Specifically, after acquiring the user image, the device performs hand detection in the entire image, and outputs the specific position of the hand. For example, a target frame may be used to identify the position of the hand. The target frame includes the coordinates of the upper left corner and the lower right corner of the target frame on the captured image; or the target frame includes the coordinates of the center of the target frame on the captured image and the width and height of the target frame.

After that, the device crops a picture of the hand area from the user image according to the specific position of the hand. Then input the picture of the hand area into the gesture detection neural network to obtain the user gesture in the picture; wherein, the gesture detection neural network can be obtained by training the deep neural network according to the gesture sample set to convergence. The gesture sample set includes multiple hand pictures including different gestures. Therefore, the trained gesture detection neural network can more accurately detect specific gestures from the picture of the hand area.

In another embodiment, it is considered that there may be a sending delay problem in the process of sending user images from the camera to the device, in order to further shorten the delay, so that the movable platform can respond to the user's gesture in real time. Control instructions, the device may include one or more photographing devices, and the target user extends his hand and is located within the photographing range of the photographing device, so that the photographing device can acquire the user's hand image in real time, so that the device The control instruction sent to the movable platform may be generated according to the user gesture recognized from the hand image to control the movable platform to adjust its own movement mode or the pan-tilt to adjust its own posture; In this embodiment, the hand image of the target user is captured by the camera set by itself, and then based on the user gesture in the recognized user image, a control instruction for controlling the movement mode of the movable platform is generated, which is beneficial to shorten the time delay , Improve the response speed, but also reduce the user's operation steps, which is conducive to improving the user's experience.

In an embodiment, the device further includes at least one input device such as a five-dimensional key, a virtual key, or a touch screen, and the user can also control the movable platform or the pan-tilt through the input device, In this way, the field of view captured by the camera can be adjusted without additional equipment such as a remote control, which is beneficial to reduce the cost of hardware expenditures.

In an exemplary embodiment, the device may obtain the movement speed indication information input by the user on the input device, where the movement speed indication information is used to indicate the target speed of the movable platform, and then the device is based on The movement speed indication information generates a speed adjustment instruction sent to the movable platform, and after obtaining the speed adjustment instruction, the movable platform adjusts from the current speed to the target speed according to the speed adjustment instruction; In the embodiment, by integrating the corresponding speed control function on the device, there is no need to set up additional devices such as a remote control, which is beneficial to reduce the cost of hardware expenditure.

As an example, the movable platform may be an unmanned aerial vehicle. Before the unmanned aerial vehicle takes off, the user can input initial speed indication information through the input device, and the initial speed indication information is used to indicate the unmanned aerial vehicle. The device generates a speed setting instruction sent to the unmanned aerial vehicle according to the initial speed instruction information, and the unmanned aerial vehicle receives the speed setting instruction according to the direction of the speed setting instruction The initial speed of flight.

In an exemplary embodiment, the movable platform may be an unmanned aerial vehicle, and the device may also obtain the flying height indication information input by the user on the input device, and the movement speed indication information is used to indicate the The target altitude of the unmanned aerial vehicle, the device may generate an altitude adjustment instruction sent to the unmanned aerial vehicle according to the flight altitude indication information, so that the unmanned aerial vehicle can, after receiving the altitude adjustment instruction, perform the altitude adjustment instruction according to the altitude adjustment instruction. The height adjustment instruction is adjusted from the current height to the target height; in this embodiment, by integrating the corresponding height control function on the device, no additional equipment such as a remote control is required, which is beneficial to reduce hardware expenditure costs.

In an exemplary embodiment, the movable platform may be an unmanned aerial vehicle, and the device further includes a hovering button (which may be a virtual control or a physical button), and the user can use the hovering button according to actual needs. Press the button to control the hovering of the UAV (that is, stay in a certain position in the air). If the device detects that the hovering control is triggered, it generates a hovering instruction sent to the UAV, and the hovering The instruction is used to control the unmanned aerial vehicle to stay in place. After receiving the hovering instruction, the unmanned aerial vehicle executes the operation of staying in place according to the hovering instruction; The corresponding hover control function is integrated on the upper, so there is no need to set up additional equipment such as a remote control, which is conducive to reducing hardware expenditures.

In an embodiment, the device may be a wearable device with a display function, such as smart glasses, helmets, smart bracelets, watches, chest straps, armbands, tops, waistbands or protective belts, etc., please refer to Figure 3. Taking the device as the smart glasses 300 as an example for description, FIG. 3 is a structural diagram of a smart glasses 300 according to an exemplary embodiment of the present application. The smart glasses 300 include:

Processor 301.

A memory 302 for storing instructions executable by the processor 301.

A collecting device 303 for collecting posture data of a user in real time.

Wireless communication device 304.

Wherein, the processor 301 calls the executable instruction, and when the executable instruction is executed, it is used to perform the following operations:

Obtain the posture data from the acquisition device 303, compare it with the specified posture data, and determine whether to generate a control instruction according to the comparison result;

If yes, generate control instructions;

The wireless communication device 304 is configured to send the generated control instruction to a movable platform; a camera is installed on the movable platform, and the control instruction is used to adjust the field of view captured by the camera.

The processor 301 may be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (ASIC), Ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

The memory 302 stores a computer program of executable instructions of the control method. The memory 302 may include at least one type of storage medium. The storage medium includes a flash memory, a hard disk, a multimedia card, and a card-type memory (for example, SD or DX memory). Etc.), random access memory (RAM), static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory , Disks, CDs, etc. Also, the smart glasses 300 may cooperate with a network storage device that performs the storage function of the memory 302 through a network connection. The memory 302 may be an internal storage unit of the smart glasses 300, such as a hard disk or a memory of the smart glasses 300. The memory 302 may also be an external storage device of the smart glasses 300, such as a plug-in hard disk equipped on the smart glasses 300, a smart memory card (Smart Media Card, SMC), a Secure Digital (SD) card, and a flash memory card (Flash). Card) and so on. Further, the memory 302 may also include both an internal storage unit of the smart glasses 300 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the smart glasses 300. The memory 302 can also be used to temporarily store data that has been output or will be output.

The collection device 303 may be an inertial measurement unit (IMU, Inertial Measurement Unit), a posture sensor, an acceleration sensor, or a gyroscope or other equipment used to collect posture data.

The wireless communication device 304 implements the communication between the smart glasses 300 and the camera and the mobile platform. This embodiment has any restrictions on its specific communication methods. The smart glasses 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination of them. In an exemplary embodiment, the wireless communication device 304 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the wireless communication device 304 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

The various embodiments described herein may be implemented using a computer-readable medium such as computer software, hardware, or any combination thereof. For hardware implementation, the implementation described here can be implemented by using application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays ( It is implemented by at least one of an FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. For software implementation, implementations such as procedures or functions may be implemented with a separate software module that allows execution of at least one function or operation. The software codes can be implemented by software applications (or programs) written in any suitable programming language, and the software codes can be stored in the memory 302 and executed by the processor 301.

In an embodiment, the control instruction is specifically used to control the movable platform to adjust its own movement mode.

In an embodiment, the movable platform is equipped with a pan/tilt, and the pan/tilt is used to support the camera.

After receiving the control instruction, the movable platform sends the control instruction to the pan/tilt. The control instruction is specifically used to control the movement of the pan/tilt from the current joint angle to the target joint angle. In an embodiment, the attitude data includes at least one of the following: a pitch angle, a yaw angle, and a roll angle.

In an embodiment, the collection device 303 includes an inertial measurement unit.

The inertial measurement unit is used to sense the pitch angle and/or the yaw angle and/or the roll angle of the user's head or limbs.

The processor 301 is further configured to obtain the pitch angle and/or yaw angle and/or roll angle.

In an embodiment, the control instruction includes at least one of the following: a movement control instruction and a direction adjustment instruction.

The movement control instruction is generated according to the pitch angle, the yaw angle, and/or the roll angle and pitch angle, and is used to control the movement of the movable platform.

The direction adjustment instruction is generated according to the pitch angle, the yaw angle and/or the roll angle and the yaw angle, and is used to adjust the moving direction of the movable platform. In an embodiment, it further includes a display; the display is used to display the images transmitted by the camera in real time.

In an embodiment, an input device is also included.

The processor 301 is further configured to: obtain movement speed indication information input by the user on the input device, where the movement speed indication information is used to indicate the target speed of the movable platform; and generate according to the movement speed indication information A speed adjustment instruction sent to the movable platform to adjust the movable platform from the current speed to the target speed.

In an embodiment, the input device includes at least one of the following: a five-dimensional key, a virtual key, or a touch screen.

In an embodiment, the movable platform includes an unmanned aerial vehicle, an unmanned ship, an unmanned vehicle, and a mobile robot.

In an embodiment, the movable platform is an unmanned aerial vehicle.

The processor 301 is further configured to: obtain flight altitude indication information input by the user on the input device, where the movement speed indication information is used to indicate the target speed and altitude of the unmanned aerial vehicle; and according to the flight altitude indication information Generate an altitude adjustment instruction sent to the unmanned aerial vehicle to adjust the unmanned aerial vehicle from the current speed altitude to the target altitude.

In one embodiment, it also includes a hover button control.

The processor 301 is further configured to: if it is detected that the hover button control is triggered, generate a hover instruction sent to the unmanned aerial vehicle to control the unmanned aerial vehicle to stay in place.

In an embodiment, the camera is used to photograph a designated user.

The processor 301 is further configured to: acquire a user image taken by the camera shooting device; and generate the control instruction sent to the movable platform according to the user gesture recognized from the user image.

In an embodiment, the camera is used to photograph a designated user.

The processor 301 is further configured to send the posture data to the movable platform;

The movable platform is used for generating the control instruction according to the posture data and the user image taken by the camera.

In an embodiment, the camera is used to photograph a designated user.

The processor 301 is further configured to: obtain a user image taken by the camera shooting device; and generate the control instruction according to the posture data and the user image taken by the camera.

Correspondingly, please refer to FIG. 4, which is a structural diagram of a movable platform 400 according to an exemplary embodiment of the present application. The movable platform 400 includes:

Body 401.

The power system 402 is installed in the body 401 and used to provide power for the movable platform 400.

The wireless communication system 403 is installed in the body 401 for receiving the control instructions sent by the smart glasses 300; and,

The control system 404 is installed in the body 401 and is used to adjust the field of view captured by the camera according to the control instruction.

Those skilled in the art can understand that FIG. 4 is only an example of the movable platform 400, and does not constitute a limitation on the movable platform 400. It may include more or less components than those shown in the figure, or combine certain components, or different components. For example, the mobile platform 400 may also include input and output devices, network access devices, and so on.

As an example, the movable platform 400 includes: an unmanned aerial vehicle, an unmanned vehicle, an unmanned boat, and a mobile robot.

In an embodiment, the control system is specifically configured to: control the power system according to the control instruction to control the movable platform to adjust its own movement mode.

In an embodiment, the control system is specifically configured to send the control instruction to the pan/tilt head, so that the pan/tilt head moves from the current joint angle to the target joint angle according to the control instruction.

In an embodiment, the camera is used to photograph a designated user.

The wireless communication system is also used to receive the posture data sent by the smart glasses.

The control system is further configured to generate the control instruction according to the posture data and the user image taken by the camera.

In an exemplary embodiment, the movable platform may be an unmanned aerial vehicle, and the unmanned aerial vehicle further includes at least one of the following measuring equipment: a TOF image sensor and a GPS device; the control system is also used to A measurement result of the measurement device controls the power system to maintain its own flying height, so as to achieve the purpose of automatic altitude following.

In an exemplary embodiment, the control system is further configured to determine whether there is an obstacle in the current flight direction according to the shooting result of the camera, and if so, control the power system to adjust the flight direction or stay in place, So as to avoid damage to the unmanned aerial vehicle.

Correspondingly, please refer to FIG. 5, which is a structural diagram of a camera 500 according to an exemplary embodiment of this application. The camera 500 includes:

壳501。 Case 501.

The lens assembly 502 is arranged inside the housing 501.

The sensor assembly 503 is arranged inside the housing 501 and is used to sense light passing through the lens assembly 502 and generate electrical signals.

The processor 504 is arranged inside the housing 501 and is used to process the electrical signal to form an image.

The wireless communication component 505 is arranged inside the housing 501 and is used to send the image to the smart glasses 300.

Those skilled in the art can understand that FIG. 5 is only an example of the camera 500, and does not constitute a limitation on the camera 500. It may include more or less components than shown, or a combination of certain components, or different components, such as The camera 500 may also include input and output devices, network access devices, and so on.

Correspondingly, please refer to FIG. 6, which is a structural diagram of a pan-tilt 600 according to an exemplary embodiment of this application. The pan-tilt 600 includes:

PTZ axis 601.

An angle sensor 602 for collecting the angle information of the pan/tilt axis.

The communication device 603 is configured to receive the control instruction sent by the movable platform 400 described above.

The processor 604 is configured to control the rotation of the pan-tilt axis according to the control instruction and the angle information, so as to move from the current joint angle to the target joint angle.

In one embodiment, the processor 604, upon receiving the control instruction, controls the rotation of the pan/tilt axis according to the control instruction, and determines whether the pan/tilt axis has moved to the desired angle based on the angle information collected in real time. Target joint angle.

Those skilled in the art can understand that FIG. 6 is only an example of the pan-tilt 600, and does not constitute a limitation on the pan-tilt 600. It may include more or less components than shown in the figure, or a combination of certain components, or different components For example, the pan-tilt 600 may also include input and output devices, network access devices, and so on.

Wherein, the PTZ and the movable platform can be connected in a wired manner or a wireless manner. The embodiment of the present application does not impose any restriction on this, and can be specifically set according to actual conditions.

Correspondingly, please refer to FIG. 7, which is a structural diagram of a control system according to an exemplary embodiment of this application. The control system includes the smart glasses 300, the movable platform 400, and the camera 500; The camera 500 is installed on the movable platform 400.

In an exemplary application scenario, the movable platform 400 moves under the control of the user, that is, based on the comparison between the posture data collected by the smart glasses 300 and the specified posture data, it is determined to generate a control instruction based on the The control instructions control the movable platform 400 to adjust its own movement mode, thereby indirectly adjusting the field of view taken by the camera 500. The camera 500 installed on the movable platform 400 takes pictures along the way in real time and captures the pictures taken. It is transmitted to the smart glasses 300, and the captured images are displayed in real time by the display on the smart glasses 300.

In another exemplary application scenario, the camera 500 is used to photograph a part of a designated user, and send the captured user image to the smart glasses 300. During the tracking and shooting process, the smart glasses 300 can capture The obtained posture data and the user image are generated, the control instruction sent to the movable platform is generated, and the movable platform 400 is controlled to adjust its own movement mode based on the control instruction, thereby indirectly adjusting the shooting of the camera 500 The field of view can achieve accurate tracking and shooting.

Please refer to FIG. 8, which is a structural diagram of another control system according to an exemplary embodiment of this application. The control system further includes the pan/tilt 600, which is used to support the camera 500, The pan-tilt 600 is connected to the movable platform 400; that is, the camera 500 is installed on the movable platform 400 through the pan-tilt 600.

In an exemplary embodiment, based on the comparison of the posture data collected by the smart glasses 300 with the specified posture data, it is determined to generate a control instruction, and the control instruction is sent to the movable platform 400 so that the The movable platform 400 sends the control instruction to the pan/tilt 600 to control the movement of the pan/tilt 600 from the current joint angle to the target joint angle, thereby indirectly adjusting the field of view captured by the camera 500, and is installed in the pan/tilt head 600. The camera 500 on the mobile platform 400 takes pictures along the way in real time, and transmits the taken pictures to the smart glasses 300, and the display on the smart glasses 300 displays the taken pictures in real time.

In another exemplary application scenario, the camera 500 is used to photograph a part of a designated user, and send the captured user image to the smart glasses 300. During the tracking and shooting process, the smart glasses 300 can capture The received posture data and the user image are generated, and the control instruction sent to the movable platform is generated, so that the movable platform 400 sends the control instruction to the pan/tilt 600 to control the pan/tilt 600 Move from the current joint angle to the target joint angle, thereby indirectly adjusting the field of view captured by the camera 500 to achieve accurate tracking and shooting.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, for storing a computer program, the computer program including instructions for executing the above method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Wherein, when the instructions in the storage medium are executed by the smart glasses 300, the device can execute the aforementioned control method.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply one of these entities or operations. There is any such actual relationship or order between. The terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements that are not explicitly listed. Elements, or also include elements inherent to such processes, methods, articles, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment that includes the element.

The methods and devices provided by the embodiments of the present invention are described in detail above. Specific examples are used in this article to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and methods of the present invention. The core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation of the present invention .

Claims

A control method, characterized in that it is applied to a device equipped with a collection device, the collection device is used for real-time collection of user posture data, and the method includes:

Obtain the posture data from the acquisition device and compare it with the specified posture data, and determine whether to generate a control instruction according to the comparison result;

If yes, the generated control instruction is sent to a movable platform; a camera is installed on the movable platform, and the control instruction is used to adjust the field of view captured by the camera.
The method according to claim 1, wherein the control instruction is specifically used to control the movable platform to adjust its own movement mode.
The method according to claim 1, wherein the movable platform is equipped with a pan/tilt, and the pan/tilt is used to support the camera;

After receiving the control instruction, the movable platform sends the control instruction to the pan/tilt. The control instruction is specifically used to control the movement of the pan/tilt from the current joint angle to the target joint angle.
The method according to claim 2, wherein the attitude data includes at least one of the following: a pitch angle, a yaw angle, and a roll angle.
The method according to claim 4, wherein the acquisition device comprises an inertial measurement unit;

The acquiring posture data from the acquisition device includes:

Acquire the pitch angle, yaw angle and/or roll angle of the user's head or limb sensed by the inertial measurement unit.
The method according to claim 4, wherein the control instruction includes at least one of the following: a movement control instruction and a direction adjustment instruction;

The movement control instruction is generated according to the pitch angle, yaw angle and/or roll angle, and is used to control the movement of the movable platform;

The direction adjustment instruction is generated according to the pitch angle, yaw angle and/or roll angle, and is used to adjust the moving direction of the movable platform.
The method according to claim 1, wherein the device further comprises an input device;

The method further includes:

Acquiring movement speed indication information input by the user on the input device, where the movement speed indication information is used to indicate the target speed of the movable platform;

Generate a speed adjustment instruction sent to the movable platform according to the movement speed indication information, so as to adjust the movable platform from the current speed to the target speed.
The method according to claim 7, wherein the input device comprises at least one of the following:

Five-dimensional keys, virtual keys or touch screen.
The method according to claim 7, wherein the movable platform includes an unmanned aerial vehicle, an unmanned vehicle, an unmanned boat, and a mobile robot.
The method according to claim 7, wherein the movable platform is an unmanned aerial vehicle;

The method also includes:

Acquiring flight altitude indication information input by the user on the input device, where the movement speed indication information is used to indicate the target altitude of the unmanned aerial vehicle;

Generate an altitude adjustment instruction sent to the unmanned aerial vehicle according to the flight altitude indication information, so as to adjust the unmanned aerial vehicle from the current altitude to the target altitude.
The method of claim 10, wherein the device further comprises a hover control;

The method further includes:

If it is detected that the hovering control is triggered, a hovering instruction sent to the unmanned aerial vehicle is generated to control the unmanned aerial vehicle to stay in place.
The method according to claim 1, wherein the device is a wearable device, and the device further comprises a display;

The method also includes:

The picture transmitted in real time by the camera is received and displayed on the display.
The method according to claim 1, wherein the camera is used to photograph a designated user;

The method also includes:

Acquiring a user image taken by the camera;

According to the user gesture recognized from the user image, the control instruction sent to the movable platform is generated.
The method according to any one of claims 1 to 13, wherein the device is smart glasses.
A smart glasses, characterized by comprising:

processor;

A memory for storing processor executable instructions;

A collection device for real-time collection of user posture data;

Wireless communication device

Wherein, the processor calls the executable instruction, and when the executable instruction is executed, it is used to execute the control method according to any one of claims 1 to 14.
A movable platform, characterized in that a camera is installed on the movable platform, and the movable platform includes:

Body

A power system, installed in the body, used to provide power to the movable platform;

A wireless communication system installed in the body for receiving control instructions sent by the smart glasses according to claim 15; and,

The control system is installed in the body and used to adjust the field of view shot by the camera according to the control instruction.
The method according to claim 16, wherein the movable platform comprises: an unmanned aerial vehicle, an unmanned vehicle, an unmanned boat, and a mobile robot.
The method according to claim 16, wherein the control system is specifically configured to control the power system according to the control instruction to control the movable platform to adjust its own movement mode.

The method according to claim 16, wherein the control system is specifically configured to: send the control instruction to the pan/tilt, so that the pan/tilt moves from the current joint angle to the target joint according to the control instruction angle.
A pan-tilt, characterized in that it comprises:

Gimbal axis

Angle sensor used to collect the angle information of the pan/tilt axis;

A communication device for receiving a control instruction sent by the movable platform according to any one of claims 16, 17 and 18; and,

The processor is configured to control the rotation of the pan-tilt axis according to the control instruction and the angle information to move from the current joint angle to the target joint angle.
A control system, characterized by comprising the smart glasses according to claim 15, the movable platform according to any one of claims 16 to 18, and a camera; the camera is installed on the movable platform.
The control system according to claim 20, further comprising the pan/tilt according to claim 19, the pan/tilt is connected to the movable platform, and the pan/tilt is used to support the camera.
A computer-readable storage medium, characterized in that computer instructions are stored thereon, and when the instructions are executed by a processor, the method according to any one of claims 1 to 14 is realized.