WO2023179264A1

WO2023179264A1 - Air input method, device and system

Info

Publication number: WO2023179264A1
Application number: PCT/CN2023/077011
Authority: WO
Inventors: 郑文植; 何梦佳; 柏忠嘉; 张行; 程林松
Original assignee: 华为技术有限公司
Priority date: 2022-03-25
Filing date: 2023-02-18
Publication date: 2023-09-28
Also published as: CN116841385A

Abstract

The present application relates to an air input method, device and system. The method comprises: determining a fixed-point starting point in a display area; in a three-dimensional fixed-point movement in air, a first fixed-point device determines, according to the pose information of the first fixed-point device collected by a first vision sensor, the pose information of the first fixed-point device collected by a first inertial measurement unit in the first fixed-point device, and the mapping relation between the first pose information and a first fixed-point trajectory, the fixed-point trajectory of the first fixed-point device in air, wherein the first pose information comprises the pose information of a second fixed-point device collected by a second vision sensor and the pose information of the second fixed-point device collected by a second inertial measurement unit; and determining a fixed-point trace in the display area according to the fixed-point starting point and the fixed-point trajectory. The solution can improve the accuracy of determining the fixed-point trajectory in air.

Description

Air-to-air input methods, devices and systems

Technical field

The present application relates to electronic devices, and in particular, to an air-to-air input method, device and system.

Background technique

The air-to-air input system can meet the needs of team collaboration scenarios such as teaching, meetings, and large-scale office work. It is a new terminal category for team collaboration. At present, the air-to-air input system is widely used in the financial industry, education and training industry, medical industry, etc., playing an important role in simplifying the complexity of office work in these industries.

As shown in FIG. 1 , the air-to-air input system includes an air-to-air input device 110 and a pointing device 120 . Among them, the air-to-air input device 110 may include but is not limited to: smart screen, electronic whiteboard, projection device, conference tablet, etc. The following will take a smart screen as an example for explanation. The user can input on the smart screen through the pointing device 120 through the air. The user's input on the smart screen through the pointing device 120 means that the pointing device 120 is not in contact with the smart screen. The user generates a fixed-point trajectory in the air through the pointing device 120. Correspondingly, the user can display the fixed-point trajectory on the smart screen. The fixed point trace corresponding to the trajectory. Here, the distance between the pointing device 120 and the smart screen can be 50 centimeters, 1 meter, 2 meters, 3 meters or even more. In an ideal state, the shape of the fixed-point trajectory drawn by the user in the air through the pointing device 120 and the fixed-point trace displayed on the smart screen should be the same, and the size can be proportional. For example, fixed-point trajectories and fixed-point traces are 1:2 circles, fixed-point trajectories and fixed-point traces are 1:1 squares, etc., fixed-point trajectories and fixed-point traces are 2:1 triangles, etc. It should be understood that the above fixed-point trajectories and fixed-point traces are explained using regular graphics as examples. In practical applications, fixed-point trajectories and fixed-point traces can also be irregular graphics, numbers, characters, letters, words, etc., which are not discussed here. Specific limitations.

Through the air-distance input function of the pointing device 120 , the user can complete input on the smart screen through the air at a location far away from the smart screen without having to go to the front of the smart screen for input, which provides users with great benefits. of convenience. For example, when the teacher displays the speech on the smart screen on the podium, if a student raises a question about a certain part of it, the student can use the pointing device 120 to mark the speech on the smart screen at his or her seat without having to use his or her own computer to mark the speech. Walk to your seat in front of the smart screen and mark the speech document.

However, due to the problem of drift error in the inertial measurement unit (IMU) in the pointing device 120, when using the pointing device containing the IMU for air input, the fixed-point trajectory of the pointing device 120 in the air is different from the one displayed on the smart screen. Fixed-point traces are often different. For example, as shown in Figure 1, the fixed-point trace written clockwise by the user in the air with point A as the starting point is a standard circle. However, the fixed-point trace displayed on the smart screen starts with A' Dot the irregular shape written clockwise.

Contents of the invention

This application provides an air-to-air input method, device and system, which can improve the accuracy of determining fixed-point trajectories in the air.

The first aspect provides an air-to-air input method, including:

The air input device determines the starting point of the fixed point in the display area; during the three-dimensional fixed point movement of the first pointing device in the air, the position and orientation information of the first pointing device collected by the first visual sensor, the first fixed point The posture information of the first pointing device collected by the first IMU in the pointing device and the mapping relationship between the first posture information and the first pointing trajectory Determine the fixed-point trajectory of the first pointing device in the air, wherein the first attitude information includes the position and attitude information of the second pointing device collected by the second visual sensor, and the second position and attitude information collected by the second IMU. Position information of the pointing device; determine the fixed point trace in the display area based on the fixed point starting point and the fixed point trajectory.

Here, the first visual sensor and the second visual sensor may be the same visual sensor, or they may be two different visual sensors. For example, the first visual sensor and the second visual sensor may be the same stereo camera, or they may be two different visual sensors. different stereo cameras. Alternatively, the first visual sensor can be a stereo camera, the second visual sensor a lidar, and so on. Similarly, the first pointing device and the second pointing device may be the same device, or may not be the same device. When the first pointing device and the second pointing device are the same device, the first IMU and the second IMU are the same IMU. When the first pointing device and the second pointing device are two different devices, the first The IMU and the second IMU are two different IMUs. In some possible scenarios, the first visual sensor and the second visual sensor are the same visual sensor, but the first pointing device and the second pointing device are different pointing devices. Alternatively, the first visual sensor and the second visual sensor are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device. Alternatively, the first visual sensor and the second visual sensor are different visual sensors, but the first pointing device and the second pointing device are the same pointing device. Alternatively, the first visual sensor and the second visual sensor are different visual sensors, and the first pointing device and the second pointing device are also different pointing devices.

Obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory may occur at the same time or not at the same time. When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory occur at the same time, training to obtain the mapping relationship and using the mapping relationship to determine the fixed-point trajectory occur in the same time and space. Therefore, while using the mapping relationship to determine the fixed-point trajectory, mapping The relationship can continue to change to continuously improve the accuracy of the mapping relationship in the process of using the mapping relationship. When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory do not occur at the same time, the mapping relationship can be obtained through training first, and then the mapping relationship is used to determine the fixed-point trajectory. That is, obtaining the mapping relationship through training occurred in the past time and space, while using the mapping relationship to determine the fixed-point trajectory occurred in the present time and space. Therefore, while using the mapping relationship to determine the fixed-point trajectory, the mapping relationship can no longer change. . Therefore, the air-to-air input device does not need to undertake the task of training to obtain the mapping relationship, reducing the load of the air-to-air input device. Of course, according to the actual situation, the mapping relationship can still be obtained by training through the air input device.

In the above solution, when determining the fixed-point trajectory of the first pointing device in the air, the pose information of the first pointing device collected by the first visual sensor and the first IMU collection of the first pointing device are simultaneously referred to. The position and attitude information of the first pointing device, that is, the position and attitude information of the first pointing device collected by multi-angle sensors is used to determine the fixed point trajectory of the first pointing device in the air, thereby improving the accuracy of determining the position and attitude of the first pointing device in the air. Accuracy of fixed-point trajectories.

In some possible designs, the pose information of the first pointing device collected by the first visual sensor includes three-dimensional coordinates of points in the three-dimensional model of the first pointing device, and, the first The rotation angle of the wrist part in the three-dimensional model of the pointing device relative to the user's three-dimensional model. The three-dimensional model of the user and the three-dimensional model of the first pointing device are the user's three-dimensional model collected according to the first visual sensor. The three-dimensional data and the three-dimensional data of the first pointing device are established.

Here, the point in the three-dimensional model of the first pointing device may be any point in the three-dimensional model of the first pointing device, for example, it may be the end point, midpoint, etc. of the first pointing device. In a more specific embodiment, when the first pointing device is a writing pen, the point in the three-dimensional model of the first pointing device may be the tip of the writing pen.

In the above solution, the first visual sensor collects the user's three-dimensional data and the three-dimensional data of the first pointing device to establish a three-dimensional model, and determines the first position through the user's three-dimensional model and the three-dimensional model of the first pointing device. pointing device Because the three-dimensional data modeling collected through the visual sensor has the characteristics of very high accuracy, the accuracy of the collected posture information of the first pointing device is also very high. Therefore, it is determined that the first pointing device is in the air. The accuracy of the fixed-point trajectory is also very high.

In some possible designs, the starting point of the fixed point is the intersection point of the normal vector of a specific part in the three-dimensional model of the user or the three-dimensional model of the first pointing device and the display area.

Here, the specific part of the user's three-dimensional model may be any part of the user's three-dimensional model, for example, the eye part, the nose tip part, the finger tip part, etc. Moreover, the priority of specific parts of the user's three-dimensional model can be set based on different conditions. For example, when the user points to the display area with a finger, the tip of the finger is used first to determine the starting point of the fixed point. When the user does not point to the display area with a finger, , using the eye part first to determine the starting point of the fixed point. The specific part in the three-dimensional model of the first pointing device may be any part in the three-dimensional model of the first pointing device, for example, an endpoint part, a center part, etc.

In the existing technical solution, the starting point of the fixed point is determined based on the pose information of the IMU in the first pointing device. Therefore, the starting point of the fixed point determined based on the pose information of the IMU is often not the starting point of the fixed point desired by the user. Users often need to adjust the pointing device multiple times to reach the starting point they want, resulting in very low efficiency. In this solution, a visual sensor is used to determine the fixed starting point, which can more accurately and efficiently determine the fixed starting point desired by the user. In addition, because the three-dimensional data modeling collected by the visual sensor has the characteristics of very high accuracy, it is also accurate to determine the starting point of the fixed point through the user's three-dimensional model established by the three-dimensional data collected by the visual sensor or the three-dimensional model of the first pointing device. very high.

In some possible designs, the starting point of the fixed point is the intersection point of the normal vector of the eye part in the user's three-dimensional model and the display area.

In the above solution, the starting point of the fixed point is determined through the intersection of the normal vector of the eye part in the user's three-dimensional model and the display area. It not only has the characteristics of high accuracy, but also the starting point of the fixed point is wherever the user's eyes look. Very convenient and user-friendly.

In some possible designs, before determining the fixed-point trajectory of the first pointing device in the air, the method further includes:

Receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is obtained by the robotic arm controlling the second fixed-pointing device to perform fixed-point movement in the air;

Receive first attitude information of the second pointing device, wherein the first attitude information is the position of the second pointing device when the robot arm controls the second pointing device to perform a fixed point movement in the air. posture information;

The neural network is trained through the first posture information and the first fixed point trajectory to obtain the mapping relationship.

In the above scheme, the fixed-point trajectory generated by the robotic arm is more accurate and stable than the fixed-point trajectory generated manually. Using the robotic arm to generate the fixed-point trajectory during training can obtain a more accurate mapping relationship.

In some possible designs, the first visual sensor includes a stereo camera, lidar, depth camera or monocular camera. Here, in scenarios where the accuracy of fixed-point trajectories is high, lidar, depth cameras and stereo cameras can be used as first-view sensors. When the accuracy of fixed-point trajectories is not high, a monocular camera can be used. To reduce costs, however, before using a monocular camera, the monocular camera needs to be trained so that the monocular camera has the ability to acquire three-dimensional data.

In some possible designs, the first visual sensor is disposed in the display area, or may be disposed outside the display area. Here, when the first visual sensor is disposed in the display area, the angle between the line of sight axis of the first visual sensor and the normal line of the display area is zero. At this time, when determining the user's angle established based on the first viewing angle sensor three-dimensional When the normal vector of a specific part of the three-dimensional model of the model and the first pointing device intersects with the display area, the complexity of the calculation can be simplified. When the first visual sensor is set outside the display area, the degree of freedom in setting the first visual sensor can be increased. Especially in some specific occasions, when it is inconvenient to set the first viewing angle sensor in the display area, it can also be set in the display area. outside the display area.

In a second aspect, an air-to-air input device is provided, including: a processor and a display unit, the processor being connected to the display unit,

The processor is used to determine the starting point of a fixed point in the display area generated by the display unit. During the three-dimensional fixed point movement of the first pointing device in the air, the position and orientation of the first pointing device collected by the first visual sensor are Information, the posture information of the first pointing device collected by the IMU in the first pointing device, and the mapping relationship between the first posture information and the first pointing trajectory determine the first pointing device A fixed-point trajectory in the air, wherein the first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU; according to The fixed point starting point and the fixed point trajectory determine the fixed point trace in the display area;

The display unit is used to display the fixed point trace in the display area.

In some possible designs, the device further includes a receiver, the receiver is configured to receive the posture information of the first pointing device collected by the first visual sensor, wherein the first visual sensor collects The pose information of the first pointing device includes the three-dimensional coordinates of the point in the three-dimensional model of the first pointing device, and the three-dimensional model of the first pointing device is relative to the user's wrist in the three-dimensional model. The rotation angle of the part, the three-dimensional model of the user and the three-dimensional model of the first pointing device are established based on the three-dimensional data of the user and the three-dimensional data of the first pointing device collected by the first visual sensor of.

In some possible designs, the receiver is also used to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is the second fixed-pointing device controlled by the robotic arm in the air. Obtained by performing fixed-point movement; the receiver is also used to receive the first attitude information of the second pointing device, wherein the first attitude information is obtained by controlling the second pointing device in the air by the robotic arm. The posture information of the second fixed-point device when performing fixed-point movement; the processor is also configured to train a neural network through the first posture information and the first fixed-point trajectory to obtain the mapping relationship.

In some possible designs, the first visual sensor includes a stereo camera, lidar, depth camera or monocular camera.

In the third aspect, an air-to-air input system is provided, including:

A first pointing device, used for performing three-dimensional fixed-point movement in the air, and collecting the pose information of the first pointing device through the first IMU in the first pointing device;

A first visual sensor used to collect position and orientation information of the first pointing device;

An air-to-air input device is used to determine the starting point of a fixed point in the display area generated by the air-to-air input device. During the three-dimensional fixed-point movement of the first pointing device in the air, according to the first fixed point collected by the first visual sensor Determination of the mapping relationship between the posture information of the pointing device, the posture information of the first pointing device collected by the first IMU in the first pointing device, and the first posture information and the first pointing trajectory The fixed-point trajectory of the first pointing device in the air, wherein the first pointing device One posture information includes the posture information of the second fixed-point device collected by the second visual sensor, and the posture information of the second fixed-point device collected by the second IMU; according to the fixed-point starting point and the fixed-point trajectory, it is determined fixed-point traces in the display area;

The air-to-air input device is also used to display the fixed-point trace in the display area.

Here, the air-to-air input device can be a smart screen, or a projector with a fixed-point trace determination function, etc. When the air-to-air input device is a smart screen, the display area generated by the air-to-air input device refers to the display area of the smart screen. When the air-to-air input device is a projector with a fixed-point trace determination function, the air-to-air input device The display area generated by the device refers to the projection area generated by the projector.

In a possible implementation, the fixed-point trace determination function is implemented by a peripheral fixed-point trace determination device and is not integrated with the projector. The fixed-point trace determination function includes receiving the position and orientation information of the first pointing device collected by the first visual sensor and the position and orientation information of the first pointing device collected by the first IMU, and based on the first fixed-pointing device collected by the first visual sensor. The posture information of the pointing device and the first IMU collect the posture information of the first pointing device to determine the pointing trace. A projector can be used to display the fixed point traces in the projection area.

In some possible designs, the air-to-air input device is also used to receive the first fixed point trajectory sent by the robotic arm, wherein the first fixed point trajectory is the second fixed point controlled by the robotic arm. Obtained by the equipment performing three-dimensional fixed-point movement in the air;

The air-to-air input device is also used to receive the first attitude information of the second pointing device, wherein the first attitude information is the robot arm controlling the second pointing device to perform fixed-point movement in the air. The pose information of the second fixed-point device;

The air-to-air input device is also used to train a neural network through the first attitude information and the first fixed point trajectory to obtain the mapping relationship.

The fourth aspect provides an air-to-air input device, including:

The starting point determination unit is used to determine the fixed point starting point in the display area;

A trajectory determination unit, configured to determine the position and orientation information of the first pointing device and the first pointing device in the first pointing device according to the position and orientation information collected by the first visual sensor during the three-dimensional pointing movement of the first pointing device in the air. The position and orientation information of the first pointing device collected by an IMU and the mapping relationship between the first position information and the first fixed point trajectory determine the fixed point trajectory of the first pointing device in the air, wherein, The first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU;

A trace determination unit is configured to determine fixed-point traces in the display area according to the fixed-point starting point and the fixed-point trajectory.

In some possible designs, the device further includes a training unit configured to receive the first fixed-point trajectory sent by the robotic arm, where the first fixed-point trajectory is controlled by the robotic arm. The second fixed-point device is obtained by performing three-dimensional fixed-point movement in the air; receiving the first attitude information, training the neural network through the first attitude information and the first fixed-point trajectory, and obtaining the mapping relationship .

In a fifth aspect, a computer-readable storage medium is provided, which is characterized in that it includes instructions that, when the instructions are run on an air-to-air input device, cause the air-to-air input device to execute any one of the first aspects. method described.

Description of the drawings

In order to more clearly explain the technical solutions in the embodiments of the present application or the background technology, the drawings required to be used in the embodiments or the background technology of the present application will be described below.

Figure 1 is a schematic diagram of an air-to-air input scenario involved in this application;

Figure 2 is a schematic structural diagram of an air-to-air input system involved in this application;

Figure 3 is a schematic structural diagram of a smart screen provided by this application;

Figure 4 is a schematic structural diagram of a pointing device provided by this application;

Figure 5 is a schematic structural diagram of an IMU provided by this application;

Figure 6 is a schematic structural diagram of an air-to-air input system provided by this application;

Figure 7 is a schematic structural diagram of a projection device provided by this application;

Figure 8 is a schematic flow chart of an air-to-air input method provided by this application;

Figure 9 is a schematic diagram of the pose information of the fixed-point device collected by the IMU provided by this application;

Figure 10 is a schematic diagram of determining the starting point of a fixed point based on the eye part of the user's three-dimensional model provided by this application;

Figure 11 is a schematic diagram of a trajectory prediction model provided by this application;

Figure 12 shows a schematic structural diagram of an air-to-air input device provided by this application;

Figure 13 shows a schematic structural diagram of an air-to-air input device provided by this application.

Detailed ways

Refer to Figure 2, which is a schematic structural diagram of an air-to-air input system provided by the present application. As shown in Figure 2, this application The provided air-to-air input system includes: a smart screen 110, a pointing device 120, and a visual sensor 130.

As shown in Figure 3, the smart screen 110 may include: a processor 112, a memory 113, a wireless communication module 114, a power switch 115, a wired local area network (LAN) communication module 116, and a high definition multimedia interface. HDMI) communication module 117, universal serial bus (universal serial bus, USB) communication module 118 and display 119. in:

Processor 112 may be used to read and execute computer-readable instructions. In specific implementation, the processor 112 may mainly include a controller, arithmetic unit, and a register. Among them, the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations. It can also perform address operations and conversions. Registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution.

In some embodiments, processor 112 may be used to parse signals received by wireless communication module 114 and/or wired LAN communication module 116 . The processor 112 may be configured to perform corresponding processing operations according to the parsing results, such as generating a detection response, driving the display 119 to perform display according to the display request or display instruction, and so on.

In some embodiments, the processor 112 can also be used to generate signals sent externally by the wireless communication module 114 and/or the wired LAN communication module 116, such as Bluetooth broadcast signals, beacon signals, or signals sent to electronic devices. Feedback the signal of display status (such as display success, display failure, etc.).

Memory 113 is coupled to processor 112 for storing various software programs and/or sets of instructions. In specific implementations, the memory 113 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 113 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems. Memory 113 may also store communications programs that may be used to communicate with one or more servers, or additional devices.

The wireless communication module 114 may include one or more of a Bluetooth (bluetooth) communication module 114A and a wireless local area networks (WLAN) communication module 114B.

In some embodiments, one or more of the Bluetooth (BT) communication module 114A and the WLAN communication module 114B can monitor signals transmitted by other devices, such as detection requests, scanning signals, etc., and can send response signals, such as The detection response, scanning response, etc. enable other devices to discover the smart screen 110, establish wireless communication connections with other devices, and communicate with other devices through one or more wireless communication technologies in Bluetooth or WLAN.

In other embodiments, one or more of the Bluetooth communication module 114A and the WLAN communication module 114B can also transmit signals, such as broadcasting Bluetooth signals and beacon signals, so that other devices can discover the smart screen 110 and communicate with other devices. Establish a wireless communication connection to communicate with other devices through one or more wireless communication technologies such as Bluetooth or WLAN.

The wireless communication module 114 may also include: Bluetooth, WLAN, near field communication (NFC), ultra wide band (UWB), infrared, and so on.

The power switch 115 can be used to control the power supply to the smart screen 110 .

The wired LAN communication module 116 can be used to communicate with other devices in the same LAN through the wired LAN, and can also be used to connect to the wide area network through the wired LAN and communicate with devices in the wide area network.

HDMI communication module 117 may be used to communicate with other devices through an HDMI interface (not shown).

USB communication module 118 may be used to communicate with other devices through a USB interface (not shown).

Display 119 may be used to display images, videos, etc. The display 119 can be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or an active matrix display. Active-matrix organic light emitting diode (AMOLED) display, flexible light-emitting diode (FLED) display, quantum dot light emitting diode (QLED) display etc.

In some embodiments, the smart screen 110 may also include an audio module (not shown). The audio module can be used to output audio signals through the audio output interface, so that the smart screen 110 supports audio playback. The audio module can also be used to receive audio data through the audio input interface. The smart screen 110 can be a media playback device such as a television.

In some embodiments, the smart screen 110 may also include a serial interface such as an RS-232 interface. The serial interface can be connected to other devices, such as speakers and other audio external amplifiers, so that the smart screen 110 and the audio external amplifiers can cooperate to play audio and video.

It can be understood that the structure illustrated in Figure 3 does not constitute a specific limitation on the smart screen 110. In other embodiments of the present application, the smart screen 110 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange different components. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

As shown in FIG. 4 , the pointing device 120 may be a drawing device provided with an IMU. The pointing device can be a pen-shaped device or a device of other shapes. Pointing devices can be pens, controllers, etc. The pointing device 120 may include: a processor 122, a memory 123, a wireless communication module 124, a charging management module 125, a USB interface 126, a battery 127, a power management module 128 and an IMU 129.

Processor 122 may be used to read and execute computer-readable instructions. In specific implementation, the processor 122 may mainly include a controller, arithmetic unit, and a register. Among them, the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations. It can also perform address operations and conversions. Registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In addition, the processor 122 may also adopt a heterogeneous architecture, such as an ARM+DSP architecture, an ARM+ASIC architecture, an ARM+AI chip architecture, and so on.

Memory 123 is coupled to processor 122 for storing various software programs and/or sets of instructions. In specific implementations, the memory 123 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 123 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems.

The wireless communication module 124 may include one or more of a Bluetooth (BT) communication module, a WLAN communication module, near field communication (NFC), ultra wide band (UWB), infrared, and the like.

The charge management module 125 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 125 may receive charging input from the wired charger through the USB interface 126 . In some wireless charging embodiments, the charging management module 125 may receive wireless charging input through a wireless charging coil. While the charging management module 125 charges the battery 127, it can also provide power to the pointing device through the power management module 128.

The power management module 128 is used to connect the battery 127, the charging management module 125 and the processor 122. The power management module 128 receives input from the battery 127 and/or the charging management module 125 and supplies power to the processor 122, the memory 123, the wireless communication module 124, the IMU 129, and the like. The power management module 128 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters. In some other embodiments, the power management module 128 may also be provided in the processor 122 . In other embodiments, the power management module 128 and the charging management module 125 can also be provided in the same device.

IMU 129 is a device that measures the three-axis angular velocity and acceleration of an object. An IMU may be equipped with a three-axis gyroscope and a three-axis accelerometer to measure the angular velocity and acceleration of an object in three-dimensional space. In the strict sense, an IMU only provides users with three-axis angular velocity and three-axis acceleration data. The vertical reference unit (VRU) is based on the IMU, uses the gravity vector as a reference, and uses algorithms such as Kalman or complementary filtering to provide users with pitch angles, roll angles referenced by the gravity vector, and angles without reference standards. Heading. The so-called 6-axis attitude module belongs to this type of system. There is no reference for the heading angle. No matter where the module is facing, the heading angle will be 0° (or a set constant) after startup. As the working time of the module increases, the heading angle will slowly accumulate errors. Since the pitch angle and roll angle have a gravity vector reference, there will be no cumulative error for a long time under low maneuvering conditions. The attitude and heading reference system (AHRS) system adds a magnetometer or optical flow sensor to the VRU, and uses algorithms such as Kalman or complementary filtering to provide users with absolute reference pitch angles and roll angles. As well as heading angle equipment, this type of system is used to provide accurate and reliable attitude and navigation information for the aircraft. The 9-axis attitude sensor we usually call belongs to this type of system. Because the heading angle has a reference to the geomagnetic field, it will not drift. However, the geomagnetic field is very weak and is often interfered by surrounding magnetic objects. Therefore, how to resist various magnetic interferences under high maneuverability has become a hot topic in AHRS research. As shown in Figure 5, accelerometers, gyroscopes, magnetometers, etc. are used as basic inputs, and the attitude information, position information and speed required by the user are output through the data acquisition unit, calibration and compensation unit, data fusion unit, output and configuration unit information.

It can be understood that the structure illustrated in FIG. 4 does not constitute a specific limitation on the pointing device 120 . In other embodiments of the present application, the pointing device 120 may include more or fewer components than shown in the figures, or some components may be combined, or some components may be separated, or may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The visual sensor 130 may be a monocular camera, a stereo camera, a depth camera or a laser radar that has been trained to acquire three-dimensional data, and is not specifically limited here. In a specific embodiment, the visual sensor may be integrated with the air-to-air input device, or may be provided separately, which is not specifically limited here.

It can be understood that the structure illustrated in Figure 2 does not constitute a specific limitation on the air-to-air input system. In other embodiments of the present application, the air-to-air input system may include more or less components than shown in the figures, or some components may be combined, or some components may be separated, or may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

Referring to Figure 6, Figure 6 is a schematic structural diagram of an air-to-air input system provided by the present application. As shown in Figure 6, the air-to-air input system provided by this application includes: a projector 210 with a function of determining fixed-point traces, a pointing device 220, and a visual sensor 230.

As shown in Figure 7, the projector 210 with fixed-point trace determination function may include: a processor 212, a memory 213, a wireless communication module 214, a power switch 215, a wired LAN communication module 216, an HDMI communication module 217, and a light source controller 218. and image projector 219. in:

Processor 212 may be used to read and execute computer-readable instructions. In specific implementation, the processor 212 may mainly include a controller, arithmetic unit, and a register. Among them, the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations. It can also perform address operations and conversions. Registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution.

In some embodiments, the processor 212 may be used to parse signals received by the wireless communication module 214 and/or the wired LAN communication module 216, such as broadcast detection requests, projection requests, projection instructions sent by the server of the cloud projection service provider, etc. The processor 212 may be configured to perform corresponding processing operations according to the parsing results, such as generating a detection response, or according to The projection request or projection instruction drives the light source controller 218 and the image projector to perform projection operations, and so on.

In some embodiments, the processor 212 can also be used to generate signals sent externally by the wireless communication module 214 and/or the wired LAN communication module 216, such as Bluetooth broadcast signals, beacon signals, or for feedback sent to electronic devices. Signal of projection status (such as projection success, projection failure, etc.).

Memory 213 is coupled to processor 212 for storing various software programs and/or sets of instructions. In specific implementations, the memory 213 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 213 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems. Memory 213 may also store communications programs that may be used to communicate with one or more servers, or additional devices.

The wireless communication module 214 may include one or more of a Bluetooth (BT) communication module 214A and a WLAN communication module 214B.

In some embodiments, one or more of the Bluetooth (BT) communication module and the WLAN communication module can monitor signals transmitted by other devices, such as detection requests, scanning signals, etc., and can send response signals, such as detection responses. , scanning response, etc., so that other devices can discover the projector 210, establish wireless communication connections with other devices, and communicate with other devices through one or more wireless communication technologies in Bluetooth or WLAN.

In other embodiments, one or more of the Bluetooth (BT) communication module and the WLAN communication module can also transmit signals, such as broadcasting Bluetooth signals and beacon signals, so that other devices can discover the projector 210 and communicate with other devices. The device establishes a wireless communication connection and communicates with other devices through one or more wireless communication technologies such as Bluetooth or WLAN.

Wireless communication module 214 may also include a cellular mobile communication module (not shown). The cellular mobile communication module can communicate with other devices (such as servers) through cellular mobile communication technology.

The power switch 215 may be used to control power supply to the projector 210 .

The wired LAN communication module 216 can be used to communicate with other devices in the same LAN through the wired LAN, and can also be used to connect to the wide area network through the wired LAN and communicate with devices in the wide area network.

HDMI communication module 217 may be used to communicate with other devices through an HDMI interface (not shown).

The image projector 219 may have a light source (not shown), may modulate light emitted from the light source according to image data and project an image on the screen.

The light source controller 218 may be used to control lighting of the light source provided by the image projector 219 .

It can be understood that the structure illustrated in FIG. 6 does not constitute a specific limitation on the projector 210 with the function of determining fixed point traces. In other embodiments of the present application, the projector 210 with fixed point trace determination function may include more or less components than shown in the figure, or combine some components, or split some components, or arrange different components. . The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The pointing device 220 and the visual sensor 230 may refer to the pointing device 120 and the visual sensor 130 shown in FIG. 2 .

In addition, the fixed-point trace determination function can be implemented by a peripheral fixed-point trace determination device without being integrated with the projector. The fixed-point trace determination function includes receiving the position and orientation information of the first pointing device collected by the first visual sensor and the position and orientation information of the first pointing device collected by the first IMU, and based on the first fixed-pointing device collected by the first visual sensor. The posture information of the pointing device and the first IMU collect the posture information of the first pointing device to determine the pointing trace. A projector can be used to display the fixed point traces in the projection area.

Refer to Figure 8, which is a schematic flow chart of an air-to-air input method provided by the present application. As shown in Figure 8, this application Provided air-to-air input methods include:

S101: The first pointing device performs three-dimensional pointing movement in the air, and collects the pose information of the first pointing device through the first IMU in the first pointing device.

In a specific embodiment, the first pointing device may be a pointing device in the air-to-air input system shown in FIG. 2 or FIG. 6 . Please refer to Figure 2 or Figure 6 and related descriptions for the specific structure of the first pointing device, and will not be described in detail here.

In a specific embodiment, the pose information of the first pointing device collected by the first IMU includes: the three-dimensional coordinates of the first pointing device and the angle of the first pointing device. Combined with Figure 5, the first pointing device measures data through the gyroscope, accelerometer and magnetometer in the first IMU, and inputs the measured data into the data acquisition unit, calibration and compensation unit, data fusion unit and output and The configuration unit performs acquisition, calibration and compensation, data fusion and output configuration to obtain the three-dimensional coordinates of the first pointing device (position information in Figure 5) and the angle of the first pointing device (attitude information in Figure 5).

The following will be described with reference to the specific embodiment in FIG. 9 . The pose information of the first pointing device collected by the first IMU is (x_pen, y_pen, z_pen, yaw, roll, pitch), where x_pen is the coordinate value of the first pointing device on the x-axis relative to the starting time, y_pen is the coordinate value of the first fixed point device on the y axis relative to the starting time, z_pen is the coordinate value of the first fixed point device on the z axis relative to the starting time, yaw is the coordinate value of the first fixed point device relative to the starting time is the pitch angle, roll is the roll angle of the first pointing device relative to the starting time, and pitch is the heading angle of the first pointing device relative to the starting time. Among them, the original data collected by the first IMU are the acceleration of the x-axis, y-axis and z-axis and the angular velocity of the x-axis, y-axis and z-axis. The pose information x_pen is obtained by integrating the acceleration of the x-axis twice. The pose information y_pen is obtained by integrating the acceleration of the y-axis twice. The pose information z_pen is obtained by integrating the acceleration of the z-axis twice. The pose information yaw is obtained by integrating the angular velocity of the x-axis once. Obtained, the pose information roll is obtained by integrating the angular velocity of the y-axis once, and the pose information pitchl is obtained by integrating the angular velocity of the z-axis. Because the acceleration of the x-axis, y-axis, and z-axis and the angular velocity of the x-axis, y-axis, and z-axis collected by the first IMU all have systematic errors and random errors, and the integration process will amplify these errors, resulting in the obtained pose information. The error is relatively large.

S102: The first visual sensor collects the pose information of the first pointing device.

In a specific embodiment, the first visual sensor may be the visual sensor in the air-to-air input system shown in FIG. 2 or FIG. 6 . Please refer to Figure 2 or Figure 6 and related descriptions for the specific structure of the first visual sensor, which will not be described again here.

In a specific embodiment, the first visual sensor may be disposed in the display area or may be disposed outside the display area. Here, when the first visual sensor is disposed in the display area, the angle between the line of sight axis of the first visual sensor and the normal line of the display area is zero. In this case, the complexity of subsequent calculations can be simplified. When the first visual sensor is set outside the display area, the degree of freedom in setting the first visual sensor can be increased. Especially in some specific occasions, when it is inconvenient to set the first viewing angle sensor in the display area, it can also be set in the display area. outside the display area.

In a specific embodiment, the first visual sensor collects the three-dimensional data of the user and the three-dimensional data of the first pointing device. At this time, the three-dimensional data of the user and the three-dimensional data of the first pointing device are collected based on the first visual sensor. The viewpoint of the sensor is used as the origin of the coordinates, and the coordinate transformation relationship is needed to convert the user's three-dimensional data with the visual center of the first vision sensor as the coordinate origin and the three-dimensional data of the first pointing device into a coordinate origin with the center of the display area. The three-dimensional data of the user and the three-dimensional data of the first pointing device. A three-dimensional model of the user is established based on the converted three-dimensional data of the user, and a three-dimensional model of the first pointing device is established based on the converted three-dimensional data of the first pointing device. Then, determine the pose information of the first pointing device according to the three-dimensional model of the user and the three-dimensional model of the first pointing device. First vision sensor collection The pose information of the first pointing device includes the three-dimensional coordinates of the point in the three-dimensional model of the first pointing device, and the three-dimensional model of the first pointing device is relative to the user's wrist in the three-dimensional model. The rotation angle of the part.

When the first visual sensor is installed in the smart screen, that is, when the first visual sensor is integrated in the smart screen, the positional relationship between the first visual sensor and the center of the display area of the smart screen will not change. Therefore, , it can be calibrated in advance before leaving the factory to determine the coordinate transformation relationship between the visual center of the first visual sensor and the center of the display area. This coordinate transformation relationship will not change after leaving the factory.

When the first visual sensor is set outside the smart screen, that is, the first visual sensor is an external device of the smart screen, one or several placement positions can be set on the outer frame of the smart screen. The one or several placement positions and The positional relationship between the centers of the display areas of the smart screen will not change. Therefore, it can be calibrated in advance before leaving the factory to determine the distance between the first visual sensor set at one or several placement positions and the display area. The coordinate transformation relationship between centers will not change after leaving the factory. Alternatively, the first visual sensor is set at any position that can capture the display area of the smart screen, and then the calibration image is played in the display area of the smart screen, and the calibration is performed based on the calibration image to determine the visual center of the first visual sensor. Displays the coordinate transformation relationship between the centers of the areas.

When the first visual sensor is set outside the projector with fixed-point trace determination function, one or several placement positions can be set by formulating the projection boundary. For example, the first visual sensor can be set at the upper left of the projection boundary. corner or upper right corner. The relative positional relationship between the one or several placement positions and the center of the display area of the projector is clear. Therefore, it can be determined between the first visual sensor disposed at the one or several placement positions and the center of the display area. coordinate transformation relationship between them. Alternatively, the first visual sensor is set at any position that can capture the display area of the projector, and then the calibration image is played in the display area of the projector, and calibration is performed based on the calibration image to determine the visual center of the first visual sensor. Displays the coordinate transformation relationship between the centers of the areas. It can be understood that when the projector with the fixed-point trace determination function is replaced by a combination of the fixed-point trace determination device and the projector, the calibration method of the coordinate transformation relationship is also similar, and will not be described here.

Taking the center point of the smart screen as the coordinate origin (0, 0, 0), the pose information of the first pointing device collected by the first vision sensor is (x_tip, y_tip, z_tip, θ_x, θ_y, θ_z), where, x_tip is the coordinate value of the point in the three-dimensional model of the first pointing device relative to the coordinate origin (0, 0, 0) on the x-axis, y_tip is the point in the three-dimensional model of the first pointing device relative to the coordinate origin (0, 0, 0) is the coordinate value on the y-axis, z_tip is the coordinate value of the point in the three-dimensional model of the first pointing device relative to the coordinate origin (0, 0, 0) on the z-axis, θ_x is the coordinate value of the first pointing device The pitch angle of the three-dimensional model relative to the user's wrist in the three-dimensional model, θ_y is the roll angle of the three-dimensional model of the first pointing device relative to the wrist in the user's three-dimensional model, and θ_z is the relative angle of the three-dimensional model of the first pointing device. The heading angle of the wrist in the user's 3D model.

In a specific embodiment, the wrist part in the first user's three-dimensional model can be extracted by inputting the first user's three-dimensional model into an extraction model, where the extraction model can be a deep learning (DL) network. , convolutional neural networks (CNN) and so on. The input extraction model may be trained using a large number of three-dimensional models of the second user and wrist parts of the three-dimensional model of the second user. The three-dimensional model of the first user is collected by the first visual sensor, the three-dimensional model of the second user can be collected by the second visual sensor, and the three-dimensional model of the second user can be manually annotated. Here, the first visual sensor and the second visual sensor may be the same visual sensor, or they may be two different visual sensors. For example, the first visual sensor and the second visual sensor may be the same stereo camera, or they may be two different visual sensors. different stereo cameras. Alternatively, the first visual sensor can be a stereo camera, the second visual sensor a lidar, and so on. Similarly, the first IMU and the second IMU can So the same visual sensor can also be two different visual sensors.

In a specific embodiment, the rotation angle of the three-dimensional model of the first pointing device relative to the wrist in the three-dimensional model of the user can be determined in the following manner: converting the points in the three-dimensional model of the first pointing device Connect a line with the point in the wrist of the user's three-dimensional model, and then find the rotation angle of the connecting line relative to the coordinate system with the point in the wrist as the coordinate origin.

In a specific embodiment, the point in the three-dimensional model of the first pointing device may be any point in the three-dimensional model of the first pointing device, for example, it may be a vertex of the three-dimensional model of the first pointing device (for example, , the pen tip in the three-dimensional model of the writing pen), the center of mass of the three-dimensional model of the first pointing device and the end point of the three-dimensional model of the first pointing device, and so on. In the embodiments of the present application, the pen tip in the three-dimensional model of the first pointing device is taken as an example for explanation.

In a specific embodiment, the point in the wrist part of the user's three-dimensional model can be any point in the wrist part of the user's three-dimensional model. For example, it can be the center point of the wrist part in the user's three-dimensional model. etc. Wherein, the center point of the wrist part in the user's three-dimensional model may be obtained by averaging the coordinate values of each point of the wrist part in the user's three-dimensional model. In the above examples, the point in the user's wrist in the three-dimensional model is used as one point for explanation. In other embodiments, it can also be two points, three points or even more points, which are not specifically limited here.

It can be understood that the first visual sensor can provide relatively accurate absolute pose information relative to the center point of the display area, but the first visual sensor is usually unable to provide continuous absolute pose information. Therefore, in the embodiment of the present invention, a first IMU is introduced, which can continuously provide relatively accurate relative pose information relative to the starting position. Through the cooperation of the two sensors, it can not only provide continuous pose information, but also avoid the systematic errors and random errors caused by simply using the IMU, and improve the accuracy of determining the fixed-point trajectory in the air.

S103: The first pointing device sends the pose information of the first pointing device collected by the first IMU to the air-to-air input device. Correspondingly, the air-to-air input device receives the pose information of the first pointing device collected by the first IMU and sent by the first pointing device.

In a specific embodiment, with reference to Figures 3, 4 and 7, the first pointing device is provided with one or more of a wireless communication module or a USB interface. Correspondingly, the air-to-air input device is also provided with one or more of a wireless communication module and a USB communication module. When the first pointing device and the air-to-air input device communicate in a wireless manner, there is no data connection line between the first pointing device and the air-to-air input device, which can make the use of the first pointing device more convenient. When the first pointing device and the air-to-air input device communicate using USB, the data communication between the first pointing device and the air-to-air input device can be made smoother. The first pointing device may be provided with one or more of a wireless communication module or a USB interface.

S104: The pose information of the first pointing device collected by the first visual sensor is sent to the air-to-air input device. Correspondingly, the air-to-air input device receives the position and orientation information of the first pointing device collected by the visual sensor and sent by the first visual sensor.

In a specific embodiment, the first visual sensor is provided with one or more of a wireless communication module, a USB interface, a wired LAN communication module, and an HDMI communication module. Correspondingly, the air-to-air input device is also provided with a wireless communication module and a USB communication module. When the first visual sensor and the air-to-air input device communicate in a wireless manner, there is no data connection line between the first visual sensor and the air-to-air input device, which can make the use of the first visual sensor more convenient. When the first vision sensor and the air-to-air input device communicate using USB, wired LAN or HDMI, the data communication between the first vision sensor and the air-to-air input device can be made smoother.

S105: The air-to-air input device determines the starting point of the fixed point in the display area.

In a specific embodiment, the air-to-air input device may be the smart screen in Figure 2 or the projector with fixed-point trace determination function in Figure 6, or an air-to-air input device composed of a fixed-point trace determination device and a projector. etc. Please refer to Figure 2 or Figure 6 and related descriptions for the specific structure of the air-to-air input device, which will not be described again here.

In a specific embodiment, the display area may be an area capable of displaying a fixed-point trajectory, for example, it may be a displayable area of a smart screen, or a projection area of a projector, or the like.

In a specific embodiment, the air-to-air input device determines the starting point of the fixed point in the display area, specifically: the processor of the air-to-air input device obtains the user's three-dimensional image collected from the first visual sensor from the memory of the air-to-air input device. Data, three-dimensional data of the first pointing device. The processor of the air-to-air input device respectively establishes a three-dimensional model of the user and a three-dimensional model of the first pointing device based on the three-dimensional data of the user and the three-dimensional data of the first pointing device. Then, the processor of the air-to-air input device determines the intersection of the normal vector of the specific part in the user's three-dimensional model or the three-dimensional model of the first pointing device and the display area as the starting point of the fixed point. Here, the specific part of the user's three-dimensional model may be any part of the user's three-dimensional model, for example, the eye part, the nose tip part, the finger tip part, etc. Moreover, the priority of specific parts of the user's three-dimensional model can be set based on different conditions. For example, when the user points to the display area with a finger, the tip of the finger is used first to determine the starting point of the fixed point. When the user does not point to the display area with a finger, , using the eye part first to determine the starting point of the fixed point. The specific part in the three-dimensional model of the first pointing device may be any part in the three-dimensional model of the first pointing device, for example, an endpoint part, a center part, etc. In a more specific embodiment, when the first pointing device is a writing pen, the point in the three-dimensional model of the first pointing device may be the tip of the writing pen. The following will be described with reference to the specific embodiment in FIG. 10 . Taking the center point of the smart screen as the coordinate origin (0, 0, 0), the coordinates of the eye part in the user's three-dimensional model are (x_eye, y_eye, z_eye), and the normal vector is (x_n, y_n, z_n), then the starting point of the fixed point is The intersection point (x, y, 0) of the normal vector (x_n, y_n, z_n) of the user's eye part in the user's three-dimensional model and the smart screen. In the existing technical solution, the starting point of the fixed point is determined based on the pose information of the IMU in the first pointing device. Therefore, the starting point of the fixed point determined based on the pose information of the IMU is often not the starting point of the fixed point desired by the user. Users often need to adjust the pointing device multiple times to reach the starting point they want, resulting in very low efficiency. In this application, visual sensors are used to determine the part where people's eyes are looking at to determine the drawing, which can more accurately and efficiently determine the starting point of the fixed point that the user wants. Moreover, wherever the user's eyes look, the starting point of the fixed point is located. , it is very convenient and user-friendly to use. In addition, because the three-dimensional data modeling collected by the visual sensor has the characteristics of very high accuracy, the user's three-dimensional model established by the three-dimensional data collected by the visual sensor or the three-dimensional model of the first pointing device is used to determine the accuracy of the starting point of the fixed point. Also very high.

S106: The airborne input device determines the fixed-point trajectory of the first pointing device in the air.

In a specific embodiment, the air-to-air input device determines the fixed-point trajectory of the first pointing device in the air, specifically: during the movement of the first pointing device in the air, the processor of the air-to-air input device determines the fixed-point trajectory of the first pointing device in the air according to the first The posture information of the first pointing device collected by a visual sensor, the posture information of the first pointing device collected by the IMU in the first pointing device, and the first posture information is consistent with the first pointing device. The mapping relationship between point trajectories determines the fixed-point trajectory of the first pointing device in the air. Wherein, the first posture information in the mapping relationship includes the posture information collected by the second visual sensor and the posture information collected by the second IMU.

In a more specific embodiment, the processor of the air-to-air input device uses the posture information of the first pointing device collected by the first visual sensor and all the information collected by the first IMU in the first pointing device. The posture information and trajectory prediction model of the first pointing device determine the fixed-point trajectory of the first pointing device in the air. Among them, the trajectory prediction model is trained based on the first posture information collected by the second vision sensor, the posture information collected by the second IMU, and the first fixed point trajectory. Arrived. The trajectory prediction model can be (deep neural networks, DNN), linear regression model, etc.

In a specific embodiment, the trajectory prediction model can be expressed as:
y=f(State ₁ , State ₂ )

Among them, y is the fixed-point trajectory of the first pointing device in the air, state ₁ is the pose information of the first pointing device collected by the first visual sensor, and state ₂ is the position and orientation information of the first pointing device collected by the first IMU. Pose information, f() is the mapping relationship. In a specific embodiment, state ₁ is the pose information of the first pointing device collected by the first visual sensor (x_tip, y_tip, z_tip, θ_x, θ_y, θ_z), and tats ₂ is the position information of the first pointing device collected by the first IMU. The pose information of the device at a certain point is (x_pen, y_pen, z_pen, yaw, roll, pitch).

In a more specific embodiment, as shown in Figure 11, the trajectory prediction model may include an input layer, a hidden layer and an output layer.

Input layer:

Assume that the inputs of the input layer are S ₁ and S ₂ , where S ₁ is the pose information of the first pointing device collected by the first visual sensor, and S ₂ is the position of the first pointing device collected by the first IMU. pose information, the output and input are equal, that is, no processing is performed on the input. For simplicity of description, it is assumed here that the input layer does not perform any processing. However, in actual applications, the input layer can be normalized and processed without specific limitations here.

Hidden layer:

Take S ₁ and S ₂ output by the input layer as the input of the hidden layer. Assume a total of L (L≥2) hidden layers. Let Z ^l represent the output result of the lth layer. When l = 1, Among them, 1≤l≤L, then the relationship between the lth layer and the l+1th layer is:

in, For the l level The weight vector of , W ₂ ^l is the weight vector of Z ₂ ^l of the l-th layer, b ^l is the bias vector of the l-th layer, a ^l+1 is the intermediate vector of the l+1-th layer, is the first excitation function of layer l+1, is the second excitation function of the l+1th layer, and Z ^l+1 is the hidden layer result of the l+1th layer. The first excitation function and the second excitation function may be any one of sigmoid function, hyperbolic tangent function, Relu function, etc.

Output layer:

Assume that the output result Z ^L of layer L is specifically:

in, For the L-1 floor The weight vector of , W ₂ ^l is the weight vector of Z ₂ ^l of the l-th layer, b ^l is the bias vector of the l-th layer,

In a specific embodiment, before using the trajectory prediction model, the trajectory prediction model needs to be trained first. The specific process of training the trajectory prediction model is to obtain a large amount of first attitude information and the corresponding first fixed point trajectory. The first posture information includes the posture information of the second pointing device collected by the second visual sensor, the posture information of the second pointing device collected by the second IMU, and the corresponding first pointing trajectory. Then, input the pose information of the second fixed-point device collected by the second visual sensor, the pose information of the second fixed-point device collected by the second IMU and the corresponding first fixed-point trajectory multiple times into the untrained trajectory prediction model. Repeat the training until the trajectory prediction model can correctly predict the fixed-point trajectory. For a single training, because the output of the trajectory prediction model is as close as possible to the value that is actually expected to be predicted, the pose information of the second fixed-point device collected by the second visual sensor and the second position and orientation information collected by the second IMU can be combined. The pose information of the fixed-point device is input into the trajectory prediction model to obtain the predicted value of the data, and the first fixed-point trajectory is used as the real desired target value, which is more accurate. The weight vector of each layer of the deep neural network in the trajectory prediction model is updated based on the difference between the previous predicted value and the really desired target value (of course, there is usually a The initialization process is to pre-configure parameters for each layer in the trajectory prediction model). For example, if the prediction value of the trajectory prediction model is high, adjust the weight vector to make it predict a lower value. Continue to adjust until the trajectory prediction model can predict Find the target value you really want. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value". This is the loss function (loss function) or objective function (objective function), which is used to measure the difference between the predicted value and the target value. Important equations. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the trajectory prediction model becomes a process of reducing this loss as much as possible.

Here, the first visual sensor and the second visual sensor may be the same visual sensor, or they may be two different visual sensors. For example, the first visual sensor and the second visual sensor may be the same stereo camera, or they may be two different visual sensors. different stereo cameras. Alternatively, the first visual sensor can be a stereo camera, the second visual sensor a lidar, and so on. Similarly, the first pointing device and the second pointing device may be the same device, or may not be the same device. When the first pointing device and the second pointing device are the same device, the first IMU and the second IMU are the same IMU. When the first pointing device and the second pointing device are two different devices, the first The IMU and the second IMU are two different IMUs. In some possible scenarios, the first visual sensor and the second visual sensor are the same visual sensor, but the first pointing device and the second pointing device are different pointing devices. Alternatively, the first visual sensor and the second visual sensor are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device. Alternatively, the first visual sensor and the second visual sensor are different visual sensors, but the first pointing device and the second pointing device are the same pointing device.

Obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory may occur at the same time or not at the same time.

When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory occur at the same time, training to obtain the mapping relationship and using the mapping relationship to determine the fixed-point trajectory occur in the same time and space. Therefore, while using the mapping relationship to determine the fixed-point trajectory, mapping The relationship can continue to change to continuously improve the accuracy of the mapping relationship in the process of using the mapping relationship. In this scenario, the posture information of the first positioning device collected by the first visual sensor and the posture information of the first positioning device collected by the first IMU can be used as the third positioning device when determining the fixed-point trajectory using the mapping relationship. Based on the attitude information, the fixed-point trajectory determined using the mapping relationship can be used as the first fixed-point trajectory to train the trajectory prediction model in the air-to-air input device.

When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory do not occur at the same time, the mapping relationship can be obtained through training first, and then the mapping relationship is used to determine the fixed-point trajectory. That is, obtaining the mapping relationship through training occurs in the past time and space, while using the mapping relationship to determine the fixed-point trajectory occurs in the present time and space. At this time, a specialized training device can be tasked with training to obtain the mapping relationship to reduce the load on the air-to-air input device. Of course, according to the actual situation, the mapping relationship can still be obtained by training through the air input device. In other words, the mapping relationship is obtained through training using historically collected data. In this scenario, a robotic arm can be used to control the second pointing device to perform fixed-point movement in the air. That is, a robotic arm can be used to simulate a human arm's three-dimensional fixed-point movement in the air. In order to better simulate the human arm, the robotic arm can use a bionic arm that is comparable to a human arm. The trajectory obtained by the robot arm controlling the second fixed-point device to perform three-dimensional fixed-point movement in the air is the first fixed-point trajectory. When the robotic arm controls the second fixed-point device to perform fixed-point movement in the air, the second visual sensor collects the three-dimensional data of the robotic arm and the three-dimensional data of the second fixed-point device, and establishes a three-dimensional model of the robotic arm based on the three-dimensional data of the robotic arm. Based on the three-dimensional data of the second fixed-point device, Establish a three-dimensional model of the second pointing device, and then determine the pose information of the second pointing device based on the three-dimensional model of the robotic arm and the three-dimensional model of the second pointing device (for details, please refer to step S102 in which the first visual sensor determines the first The process of position information of the pointing device); and, when the robot arm controls the second pointing device to perform fixed-point movement in the air, the second IMU in the second pointing device collects the pose information of the second pointing device (for details, please refer to the steps The process in which the first IMU of the first pointing device collects the pose information of the first pointing device in S101). The pose information of the second pointing device collected by the second visual sensor and the pose information of the second pointing device collected by the second IMU in the second pointing device are collectively referred to as the first pose information. The robotic arm sends the first fixed-point trajectory to the air-to-air input device or training device, and the second visual sensor sends the pose information of the second fixed-point device collected by the second visual sensor to the air-to-air input device or training device. The pointing device sends the pose information of the second pointing device collected by the second IMU in the second pointing device to the air-to-air input device or the training device to train the trajectory prediction model.

In the above example, it is assumed that the positions of the first visual sensor and the second visual sensor remain unchanged. For example, the first visual sensor can be placed in the middle of the upper edge of the smart screen, and the second visual sensor can also be placed. In the middle of the upper edge of the smart screen. However, in actual applications, the positions of the first visual sensor and the second visual sensor may also change. For example, the first visual sensor can be disposed in the center of the upper edge of the smart screen, and the second visual sensor can be disposed in the upper left corner of the smart screen. At this time, it is necessary to recalibrate the conversion relationship between the second perspective sensor and the center of the display area, and use this conversion relationship to combine the three-dimensional data of the robotic arm collected by the second perspective sensor with the visual center of the second perspective sensor as the coordinate origin and the second perspective sensor. The three-dimensional data of the fixed-point device is converted into the three-dimensional data of the robot arm with the center of the display area as the coordinate origin and the three-dimensional data of the second fixed-point device.

When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory occur at the same time, the first visual sensor and the second visual sensor often used are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device. . When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory do not occur at the same time, the first visual sensor and the second visual sensor may be the same visual sensor, but the first pointing device and the second pointing device may be different. Pointing device. Alternatively, the first visual sensor and the second visual sensor are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device. Alternatively, the first visual sensor and the second visual sensor are different visual sensors, but the first pointing device and the second pointing device are the same pointing device. Alternatively, the first visual sensor and the second visual sensor are different visual sensors, and the first pointing device and the second pointing device are also different pointing devices.

S107: The air-to-air input device determines the fixed-point trace in the display area based on the fixed-point starting point and the fixed-point trajectory.

In a specific embodiment, the processor of the air-to-air input device determines the fixed-point trace in the display area based on the fixed-point starting point and the fixed-point trajectory.

S108: The air-to-air input device displays fixed-point traces in the display area.

In a specific embodiment, when the air-to-air input device is a smart screen, fixed-point traces can be displayed in the display area of the display of the smart screen; when the air-to-air input device is a projector with a fixed-point trace determination function or a fixed-point trace When a device with a trace determination function and an air-to-air input device composed of a projector are used, fixed-point traces can be projected to the projection area through the image projector of the projector.

In the above embodiment, the air-to-air input device can receive the posture information of the first pointing device collected by the first visual sensor and the first IMU collects the posture information of the first pointing device, and performs the processing according to the first The position and orientation information of the first pointing device collected by the visual sensor and the position and orientation information of the first pointing device collected by the first IMU determine the fixed point trace, and project The instrument can be used to display the fixed-point traces in the projection area, and the steps of displaying the fixed-point traces in the display area are explained as examples. In a possible embodiment, the air-to-air input device may include a fixed-point trace determination device and a projector, wherein the fixed-point trace determination device is configured to receive the pose information of the first pointing device collected by the first visual sensor and the first The IMU collects the pose information of the first pointing device, and determines the fixing trace based on the pose information of the first pointing device collected by the first visual sensor and the pose information of the first pointing device collected by the first IMU. , the projector can be used to display the fixed point trace in the projection area, which is not specifically limited here.

In the above embodiment, in steps S102 and S104, the first visual sensor collects the three-dimensional data of the user and the three-dimensional data of the first pointing device, and then determines the first fixed point based on the three-dimensional data of the user and the three-dimensional data of the first pointing device. For the pose information of the device, in other embodiments, the first visual sensor can also collect the three-dimensional data of the user and the three-dimensional data of the first fixed point device, and the three-dimensional data of the user and the first fixed point collected by the first visual sensor can be used. The three-dimensional data of the device is sent to the air-to-air input device, and the air-to-air input device determines the pose information of the first pointing device based on the user's three-dimensional data and the three-dimensional data of the first pointing device. In the same way, the second vision sensor can also collect the three-dimensional data of the robotic arm and the three-dimensional data of the second fixed-point device and send it to the air-to-air input device or training device, and the air-to-air input device or training device can collect the three-dimensional data of the robotic arm and the three-dimensional data of the second fixed-point device. The three-dimensional data of the second pointing device determines the pose information of the second pointing device.

In the above embodiment, steps S101 and S102 are executed in no particular order. Step S101 may be executed first and then step S102, or step S102 may be executed first and then step S101, or step S101 and step S102 may be executed simultaneously. The execution order of step S103 and step S104 is in no particular order. Step S103 may be executed first and then step S104, or step S103 may be executed first and then step S104, or step S103 and step S104 may be executed simultaneously.

Please refer to Figure 12, which shows a schematic structural diagram of an air-to-air input device provided by the present application. As shown in Figure 12, air-to-air input devices include:

The starting point determining unit 310 is used to determine a fixed point starting point in the display area.

The trajectory determination unit 320 is configured to determine, according to the posture information of the first pointing device collected by the first visual sensor, the third point in the first pointing device during the three-dimensional pointing movement of the first pointing device in the air. The position and orientation information of the first pointing device collected by an IMU and the mapping relationship between the first position information and the first fixed point trajectory determine the fixed point trajectory of the first pointing device in the air, wherein, The first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU.

The trace determining unit 330 is configured to determine the fixed point trace in the display area according to the fixed point starting point and the fixed point trajectory.

Among them, the starting point determination unit 310, the trajectory determination unit 320, and the trace determination unit 330 work together to implement the steps performed by the air-to-air input device in S104. Specifically, the starting point determining unit 310 is used to perform the step of determining the fixed point starting point in S105, the trajectory determining unit 320 is used to perform the step of determining the fixed point trajectory of the first pointing device in the air in S106, and the trace determining unit 330 is used to perform The step of determining fixed-point traces in the display area in step 106 above.

Optionally, the air-to-air input device may also include a training unit (not shown), the training unit being configured to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is the The mechanical arm controls the second fixed-point device to perform three-dimensional fixed-point movement in the air; receives the first attitude information, and trains the neural network through the first attitude information and the first fixed-point trajectory to obtain the mapping relationship.

Optionally, the air-to-air input device may also include a receiving unit (not shown) for receiving the information collected by the first visual sensor. The pose information of the first pointing device (or the user's three-dimensional data collected by the first visual sensor and the three-dimensional data of the first pointing device), and the pose information of the first pointing device collected by the first IMU. In addition, it can also be used to receive the pose information of the second pointing device collected by the second visual sensor (or the three-dimensional data of the mechanical arm and the three-dimensional data of the second pointing device collected by the second visual sensor). The pose information of the second fixed-pointing device, the first fixed-point trajectory sent by the robot arm, etc.

Optionally, the air-to-air input device may also include a display unit (not shown) for displaying fixed-point traces in the display area.

Optionally, the air-to-air input device may include a fixed-point trace determination device and a projector, wherein the starting point determination unit 310, the trajectory determination unit 320 and the trace determination unit 330 are provided in the trace determination device, and the display unit is provided in the projector, where There are no specific limitations.

In addition, for the position of the fixed point starting point determined by the starting point determining unit 310 and the method of determining the fixed point starting point, please refer to the relevant introduction in step S105 in FIG. 8 . The posture information of the first pointing device collected by the first visual sensor used in the trajectory determination unit 320, the posture information of the first pointing device collected by the first IMU, the first posture information, the third For the definition of a certain point trajectory, the acquisition method, and the training method of the mapping relationship, please refer to the relevant introduction in steps S101, S102 and S105 in Figure 8, and will not be described here.

Please refer to Figure 13, which shows a schematic structural diagram of an air-to-air input device provided by the present application. The air-to-air input device is used to perform the steps performed by the air-to-air input device in the above-mentioned air to air input method.

As shown in FIG. 13 , the air-to-air input device includes a memory 410 , a processor 420 , a communication interface 430 and a bus 440 . Among them, the memory 410, the processor 420, and the communication interface 430 implement communication connections between each other through the bus 440.

The memory 410 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 410 may store computer instructions, such as: computer instructions in the starting point determination unit 310, computer instructions in the trajectory determination unit 320, computer instructions in the trace determination unit 330, etc. When the computer instructions stored in the memory 410 are executed by the processor 420, the processor 420 and the communication interface 430 are used to execute part or all of the method described in the above steps S104 to S106. The memory 410 can also store data, for example: intermediate data or result data generated by the processor 420 during execution, for example, the user's three-dimensional data collected by the first visual sensor, the three-dimensional data of the first pointing device, the user's three-dimensional data. Model, three-dimensional model of the first pointing device, mapping relationship, pose information collected by the first visual sensor, pose information collected by the first IMU, etc.

The processor 420 may be a CPU, a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

The processor 420 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, part or all of the functions of the air-to-air input device may be completed by instructions in the form of hardware integrated logic circuits or software in the processor 420 . The processor 420 may also be a general processor, a digital signal process (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, Discrete hardware components are used to implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or the processor can be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be executed using decoding processing. The combination of hardware and software modules in the device is executed. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 410. The processor 420 reads the information in the memory 410 and completes steps S105 to S108 in the above-mentioned air-to-air input method in conjunction with its hardware.

The communication interface 430 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device and other devices (eg, camera device, microphone, server).

Bus 440 may include a path for transmitting information between various components in the air-to-air input device (eg, memory 410, processor 420, communication interface 430).

In some embodiments, the air-to-air input device can be used as a terminal device for team collaboration and communication. Therefore, optionally, the air-to-air input device may also include a camera device 450 and a microphone 460 for real-time collection of image signals and sound signals. Alternatively, the air-to-air input device can also be connected to the camera device 450 and the microphone 460 through the communication interface 430 for real-time collection of image signals and sound signals.

It can be understood that the structure of the fixed-point trace determining device is similar to the air-to-air input device shown in Figure 13. However, the fixed-point trace determining device does not need to perform the step of displaying fixed-point traces in the display area in the above-mentioned air-to-air input method.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disk, storage disk, tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), etc.

Claims

An air-to-air input method, characterized by including:

Determine the starting point of the fixed point in the display area;

During the three-dimensional fixed-point movement of the first pointing device in the air, the pose information of the first pointing device collected by the first visual sensor and the position and orientation information collected by the first inertial measurement unit IMU in the first pointing device are The position and attitude information of the first pointing device and the mapping relationship between the first position information and the first fixed point trajectory determine the fixed point trajectory of the first pointing device in the air, wherein the first attitude The information includes the pose information of the second pointing device collected by the second visual sensor, and the pose information of the second pointing device collected by the second IMU;

According to the fixed point starting point and the fixed point trajectory, a fixed point trace in the display area is determined.
The method of claim 1, wherein the pose information of the first pointing device collected by the first visual sensor includes three-dimensional coordinates of points in the three-dimensional model of the first pointing device, And, the rotation angle of the three-dimensional model of the first pointing device relative to the wrist in the three-dimensional model of the user, the three-dimensional model of the user and the three-dimensional model of the first pointing device are determined according to the first visual sensor The collected three-dimensional data of the user and the three-dimensional data of the first pointing device are established.
The method according to claim 2, characterized in that the fixed point starting point is an intersection of a normal vector of a specific part in the user's three-dimensional model or the three-dimensional model of the first pointing device and the display area.
The method according to claim 3, characterized in that the fixed point starting point is the intersection point of the normal vector of the eye part in the user's three-dimensional model and the display area.
The method according to any one of claims 1 to 4, characterized in that before determining the fixed-point trajectory of the first pointing device in the air, the method further includes:

Receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is obtained by the robotic arm controlling the second fixed-pointing device to perform fixed-point movement in the air;

Receive first attitude information of the second pointing device, wherein the first attitude information is the position of the second pointing device when the robot arm controls the second pointing device to perform a fixed point movement in the air. posture information;

The neural network is trained through the first posture information and the first fixed point trajectory to obtain the mapping relationship.
An air-to-air input device, characterized by comprising: a processor and a display unit, the processor being connected to the display unit,

The processor is used to determine the starting point of a fixed point in the display area generated by the display unit. During the three-dimensional fixed point movement of the first pointing device in the air, the position and orientation of the first pointing device collected by the first visual sensor are Information, the posture information of the first pointing device collected by the IMU in the first pointing device, and the mapping relationship between the first posture information and the first pointing trajectory determine the first pointing device A fixed-point trajectory in the air, wherein the first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU; according to The fixed point starting point and the fixed point trajectory determine the fixed point trace in the display area;

The display unit is used to display the fixed point trace in the display area.
The device of claim 6, further comprising a receiver,

The receiver is configured to receive the pose information of the first pointing device collected by the first visual sensor, wherein the pose information of the first pointing device collected by the first visual sensor includes the The three-dimensional coordinates of the point in the three-dimensional model of the first pointing device, and the rotation angle of the wrist part in the three-dimensional model of the first pointing device relative to the user's three-dimensional model, the user's three-dimensional model and the third The three-dimensional model of the fixed-point device is established based on the three-dimensional data of the user collected by the first visual sensor and the three-dimensional data of the first fixed-point device.
The device according to claim 7, characterized in that:

The receiver is also used to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is obtained by the robotic arm controlling the second fixed-pointing device to perform fixed-point movement in the air;

The receiver is further configured to receive first attitude information of the second pointing device, wherein the first attitude information is obtained when the robotic arm controls the second pointing device to perform fixed-point movement in the air. The pose information of the second pointing device;

The processor is also configured to train a neural network through the first attitude information and the first fixed point trajectory to obtain the mapping relationship.
An air-to-air input system is characterized by including:

A first pointing device, used for performing three-dimensional fixed-point movement in the air, and collecting the pose information of the first pointing device through the first IMU in the first pointing device;

A first visual sensor used to collect position and orientation information of the first pointing device;

An air-to-air input device is used to determine the starting point of a fixed point in the display area generated by the air-to-air input device. During the three-dimensional fixed-point movement of the first pointing device in the air, according to the first fixed point collected by the first visual sensor Determination of the mapping relationship between the posture information of the pointing device, the posture information of the first pointing device collected by the first IMU in the first pointing device, and the first posture information and the first pointing trajectory The fixed-point trajectory of the first pointing device in the air, wherein the first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the second fixed point collected by the second IMU The pose information of the device; determining the fixed point trace in the display area according to the fixed point starting point and the fixed point trajectory;

The air-to-air input device is also used to display the fixed-point trace in the display area.
The system of claim 9, wherein the pose information of the first pointing device includes three-dimensional coordinates of points in the three-dimensional model of the first pointing device, and the first pointing device The rotation angle of the wrist in the three-dimensional model of the device relative to the user's three-dimensional model. The three-dimensional model of the user and the three-dimensional model of the first pointing device are based on the three-dimensional data of the user collected by the first visual sensor. and established with the three-dimensional data of the first pointing device.
The system according to claim 9 or 10, characterized in that,

The air-to-air input device is also used to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is the three-dimensional fixed-point movement of the second fixed-point device in the air controlled by the robotic arm. owned;

The air-to-air input device is also used to receive the first attitude information of the second pointing device, wherein the first attitude information is the robot arm controlling the second pointing device to perform fixed-point movement in the air. The pose information of the second fixed-point device;

The separated input device is also used to train a neural network through the first posture information and the first fixed point trajectory to obtain the mapping relationship.