WO2023179264A1 - Procédé, dispositif et système de saisie en l'air - Google Patents

Procédé, dispositif et système de saisie en l'air Download PDF

Info

Publication number
WO2023179264A1
WO2023179264A1 PCT/CN2023/077011 CN2023077011W WO2023179264A1 WO 2023179264 A1 WO2023179264 A1 WO 2023179264A1 CN 2023077011 W CN2023077011 W CN 2023077011W WO 2023179264 A1 WO2023179264 A1 WO 2023179264A1
Authority
WO
WIPO (PCT)
Prior art keywords
pointing device
point
fixed
air
visual sensor
Prior art date
Application number
PCT/CN2023/077011
Other languages
English (en)
Chinese (zh)
Inventor
郑文植
何梦佳
柏忠嘉
张行
程林松
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023179264A1 publication Critical patent/WO2023179264A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03542—Light pens for emitting or receiving light

Definitions

  • the present application relates to electronic devices, and in particular, to an air-to-air input method, device and system.
  • the air-to-air input system can meet the needs of team collaboration scenarios such as teaching, meetings, and large-scale office work. It is a new terminal category for team collaboration. At present, the air-to-air input system is widely used in the financial industry, education and training industry, medical industry, etc., playing an important role in simplifying the complexity of office work in these industries.
  • the air-to-air input system includes an air-to-air input device 110 and a pointing device 120 .
  • the air-to-air input device 110 may include but is not limited to: smart screen, electronic whiteboard, projection device, conference tablet, etc.
  • the following will take a smart screen as an example for explanation.
  • the user can input on the smart screen through the pointing device 120 through the air.
  • the user's input on the smart screen through the pointing device 120 means that the pointing device 120 is not in contact with the smart screen.
  • the user generates a fixed-point trajectory in the air through the pointing device 120.
  • the user can display the fixed-point trajectory on the smart screen.
  • the distance between the pointing device 120 and the smart screen can be 50 centimeters, 1 meter, 2 meters, 3 meters or even more.
  • the shape of the fixed-point trajectory drawn by the user in the air through the pointing device 120 and the fixed-point trace displayed on the smart screen should be the same, and the size can be proportional.
  • fixed-point trajectories and fixed-point traces are 1:2 circles
  • fixed-point trajectories and fixed-point traces are 1:1 squares, etc.
  • fixed-point trajectories and fixed-point traces are 2:1 triangles, etc.
  • the above fixed-point trajectories and fixed-point traces are explained using regular graphics as examples. In practical applications, fixed-point trajectories and fixed-point traces can also be irregular graphics, numbers, characters, letters, words, etc., which are not discussed here. Specific limitations.
  • the user can complete input on the smart screen through the air at a location far away from the smart screen without having to go to the front of the smart screen for input, which provides users with great benefits. of convenience. For example, when the teacher displays the speech on the smart screen on the podium, if a student raises a question about a certain part of it, the student can use the pointing device 120 to mark the speech on the smart screen at his or her seat without having to use his or her own computer to mark the speech. Walk to your seat in front of the smart screen and mark the speech document.
  • the fixed-point trajectory of the pointing device 120 in the air is different from the one displayed on the smart screen.
  • Fixed-point traces are often different. For example, as shown in Figure 1, the fixed-point trace written clockwise by the user in the air with point A as the starting point is a standard circle. However, the fixed-point trace displayed on the smart screen starts with A' Dot the irregular shape written clockwise.
  • This application provides an air-to-air input method, device and system, which can improve the accuracy of determining fixed-point trajectories in the air.
  • the first aspect provides an air-to-air input method, including:
  • the air input device determines the starting point of the fixed point in the display area; during the three-dimensional fixed point movement of the first pointing device in the air, the position and orientation information of the first pointing device collected by the first visual sensor, the first fixed point.
  • the posture information of the first pointing device collected by the first IMU in the pointing device and the mapping relationship between the first posture information and the first pointing trajectory Determine the fixed-point trajectory of the first pointing device in the air, wherein the first attitude information includes the position and attitude information of the second pointing device collected by the second visual sensor, and the second position and attitude information collected by the second IMU.
  • Position information of the pointing device determine the fixed point trace in the display area based on the fixed point starting point and the fixed point trajectory.
  • the first visual sensor and the second visual sensor may be the same visual sensor, or they may be two different visual sensors.
  • the first visual sensor and the second visual sensor may be the same stereo camera, or they may be two different visual sensors. different stereo cameras.
  • the first visual sensor can be a stereo camera, the second visual sensor a lidar, and so on.
  • the first pointing device and the second pointing device may be the same device, or may not be the same device.
  • the first IMU and the second IMU are the same IMU.
  • the first The IMU and the second IMU are two different IMUs.
  • the first visual sensor and the second visual sensor are the same visual sensor, but the first pointing device and the second pointing device are different pointing devices.
  • the first visual sensor and the second visual sensor are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device.
  • the first visual sensor and the second visual sensor are different visual sensors, but the first pointing device and the second pointing device are the same pointing device.
  • the first visual sensor and the second visual sensor are different visual sensors, and the first pointing device and the second pointing device are also different pointing devices.
  • mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory may occur at the same time or not at the same time.
  • training to obtain the mapping relationship and using the mapping relationship to determine the fixed-point trajectory occur in the same time and space. Therefore, while using the mapping relationship to determine the fixed-point trajectory, mapping The relationship can continue to change to continuously improve the accuracy of the mapping relationship in the process of using the mapping relationship.
  • mapping relationship can be obtained through training first, and then the mapping relationship is used to determine the fixed-point trajectory.
  • mapping relationship can no longer change. . Therefore, the air-to-air input device does not need to undertake the task of training to obtain the mapping relationship, reducing the load of the air-to-air input device. Of course, according to the actual situation, the mapping relationship can still be obtained by training through the air input device.
  • the pose information of the first pointing device collected by the first visual sensor and the first IMU collection of the first pointing device are simultaneously referred to.
  • the position and attitude information of the first pointing device that is, the position and attitude information of the first pointing device collected by multi-angle sensors is used to determine the fixed point trajectory of the first pointing device in the air, thereby improving the accuracy of determining the position and attitude of the first pointing device in the air. Accuracy of fixed-point trajectories.
  • the pose information of the first pointing device collected by the first visual sensor includes three-dimensional coordinates of points in the three-dimensional model of the first pointing device, and, the first The rotation angle of the wrist part in the three-dimensional model of the pointing device relative to the user's three-dimensional model.
  • the three-dimensional model of the user and the three-dimensional model of the first pointing device are the user's three-dimensional model collected according to the first visual sensor.
  • the three-dimensional data and the three-dimensional data of the first pointing device are established.
  • the point in the three-dimensional model of the first pointing device may be any point in the three-dimensional model of the first pointing device, for example, it may be the end point, midpoint, etc. of the first pointing device.
  • the point in the three-dimensional model of the first pointing device may be the tip of the writing pen.
  • the first visual sensor collects the user's three-dimensional data and the three-dimensional data of the first pointing device to establish a three-dimensional model, and determines the first position through the user's three-dimensional model and the three-dimensional model of the first pointing device.
  • pointing device Because the three-dimensional data modeling collected through the visual sensor has the characteristics of very high accuracy, the accuracy of the collected posture information of the first pointing device is also very high. Therefore, it is determined that the first pointing device is in the air. The accuracy of the fixed-point trajectory is also very high.
  • the starting point of the fixed point is the intersection point of the normal vector of a specific part in the three-dimensional model of the user or the three-dimensional model of the first pointing device and the display area.
  • the specific part of the user's three-dimensional model may be any part of the user's three-dimensional model, for example, the eye part, the nose tip part, the finger tip part, etc.
  • the priority of specific parts of the user's three-dimensional model can be set based on different conditions. For example, when the user points to the display area with a finger, the tip of the finger is used first to determine the starting point of the fixed point. When the user does not point to the display area with a finger, , using the eye part first to determine the starting point of the fixed point.
  • the specific part in the three-dimensional model of the first pointing device may be any part in the three-dimensional model of the first pointing device, for example, an endpoint part, a center part, etc.
  • the starting point of the fixed point is determined based on the pose information of the IMU in the first pointing device. Therefore, the starting point of the fixed point determined based on the pose information of the IMU is often not the starting point of the fixed point desired by the user. Users often need to adjust the pointing device multiple times to reach the starting point they want, resulting in very low efficiency.
  • a visual sensor is used to determine the fixed starting point, which can more accurately and efficiently determine the fixed starting point desired by the user.
  • the three-dimensional data modeling collected by the visual sensor has the characteristics of very high accuracy, it is also accurate to determine the starting point of the fixed point through the user's three-dimensional model established by the three-dimensional data collected by the visual sensor or the three-dimensional model of the first pointing device. very high.
  • the starting point of the fixed point is the intersection point of the normal vector of the eye part in the user's three-dimensional model and the display area.
  • the starting point of the fixed point is determined through the intersection of the normal vector of the eye part in the user's three-dimensional model and the display area. It not only has the characteristics of high accuracy, but also the starting point of the fixed point is wherever the user's eyes look. Very convenient and user-friendly.
  • the method before determining the fixed-point trajectory of the first pointing device in the air, the method further includes:
  • first attitude information of the second pointing device wherein the first attitude information is the position of the second pointing device when the robot arm controls the second pointing device to perform a fixed point movement in the air. posture information;
  • the neural network is trained through the first posture information and the first fixed point trajectory to obtain the mapping relationship.
  • the fixed-point trajectory generated by the robotic arm is more accurate and stable than the fixed-point trajectory generated manually.
  • Using the robotic arm to generate the fixed-point trajectory during training can obtain a more accurate mapping relationship.
  • the first visual sensor includes a stereo camera, lidar, depth camera or monocular camera.
  • lidar, depth cameras and stereo cameras can be used as first-view sensors.
  • a monocular camera can be used. To reduce costs, however, before using a monocular camera, the monocular camera needs to be trained so that the monocular camera has the ability to acquire three-dimensional data.
  • the first visual sensor is disposed in the display area, or may be disposed outside the display area.
  • the angle between the line of sight axis of the first visual sensor and the normal line of the display area is zero.
  • the complexity of the calculation can be simplified.
  • the first visual sensor is set outside the display area, the degree of freedom in setting the first visual sensor can be increased.
  • it can also be set in the display area. outside the display area.
  • an air-to-air input device including: a processor and a display unit, the processor being connected to the display unit,
  • the processor is used to determine the starting point of a fixed point in the display area generated by the display unit.
  • the position and orientation of the first pointing device collected by the first visual sensor are Information, the posture information of the first pointing device collected by the IMU in the first pointing device, and the mapping relationship between the first posture information and the first pointing trajectory determine the first pointing device A fixed-point trajectory in the air, wherein the first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU; according to The fixed point starting point and the fixed point trajectory determine the fixed point trace in the display area;
  • the display unit is used to display the fixed point trace in the display area.
  • the device further includes a receiver, the receiver is configured to receive the posture information of the first pointing device collected by the first visual sensor, wherein the first visual sensor collects
  • the pose information of the first pointing device includes the three-dimensional coordinates of the point in the three-dimensional model of the first pointing device, and the three-dimensional model of the first pointing device is relative to the user's wrist in the three-dimensional model.
  • the rotation angle of the part, the three-dimensional model of the user and the three-dimensional model of the first pointing device are established based on the three-dimensional data of the user and the three-dimensional data of the first pointing device collected by the first visual sensor of.
  • the starting point of the fixed point is the intersection point of the normal vector of a specific part in the three-dimensional model of the user or the three-dimensional model of the first pointing device and the display area.
  • the starting point of the fixed point is the intersection point of the normal vector of the eye part in the user's three-dimensional model and the display area.
  • the receiver is also used to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is the second fixed-pointing device controlled by the robotic arm in the air. Obtained by performing fixed-point movement; the receiver is also used to receive the first attitude information of the second pointing device, wherein the first attitude information is obtained by controlling the second pointing device in the air by the robotic arm.
  • the posture information of the second fixed-point device when performing fixed-point movement; the processor is also configured to train a neural network through the first posture information and the first fixed-point trajectory to obtain the mapping relationship.
  • the first visual sensor includes a stereo camera, lidar, depth camera or monocular camera.
  • an air-to-air input system including:
  • a first pointing device used for performing three-dimensional fixed-point movement in the air, and collecting the pose information of the first pointing device through the first IMU in the first pointing device;
  • a first visual sensor used to collect position and orientation information of the first pointing device
  • An air-to-air input device is used to determine the starting point of a fixed point in the display area generated by the air-to-air input device.
  • the first fixed point collected by the first visual sensor Determination of the mapping relationship between the posture information of the pointing device, the posture information of the first pointing device collected by the first IMU in the first pointing device, and the first posture information and the first pointing trajectory
  • the fixed-point trajectory of the first pointing device in the air wherein the first pointing device
  • One posture information includes the posture information of the second fixed-point device collected by the second visual sensor, and the posture information of the second fixed-point device collected by the second IMU; according to the fixed-point starting point and the fixed-point trajectory, it is determined fixed-point traces in the display area;
  • the air-to-air input device is also used to display the fixed-point trace in the display area.
  • the air-to-air input device can be a smart screen, or a projector with a fixed-point trace determination function, etc.
  • the display area generated by the air-to-air input device refers to the display area of the smart screen.
  • the air-to-air input device is a projector with a fixed-point trace determination function
  • the air-to-air input device The display area generated by the device refers to the projection area generated by the projector.
  • the fixed-point trace determination function is implemented by a peripheral fixed-point trace determination device and is not integrated with the projector.
  • the fixed-point trace determination function includes receiving the position and orientation information of the first pointing device collected by the first visual sensor and the position and orientation information of the first pointing device collected by the first IMU, and based on the first fixed-pointing device collected by the first visual sensor.
  • the posture information of the pointing device and the first IMU collect the posture information of the first pointing device to determine the pointing trace.
  • a projector can be used to display the fixed point traces in the projection area.
  • the pose information of the first pointing device collected by the first visual sensor includes three-dimensional coordinates of points in the three-dimensional model of the first pointing device, and, the first The rotation angle of the wrist part in the three-dimensional model of the pointing device relative to the user's three-dimensional model.
  • the three-dimensional model of the user and the three-dimensional model of the first pointing device are the user's three-dimensional model collected according to the first visual sensor.
  • the three-dimensional data and the three-dimensional data of the first pointing device are established.
  • the starting point of the fixed point is the intersection point of the normal vector of a specific part in the three-dimensional model of the user or the three-dimensional model of the first pointing device and the display area.
  • the starting point of the fixed point is the intersection point of the normal vector of the eye part in the user's three-dimensional model and the display area.
  • the air-to-air input device is also used to receive the first fixed point trajectory sent by the robotic arm, wherein the first fixed point trajectory is the second fixed point controlled by the robotic arm. Obtained by the equipment performing three-dimensional fixed-point movement in the air;
  • the air-to-air input device is also used to receive the first attitude information of the second pointing device, wherein the first attitude information is the robot arm controlling the second pointing device to perform fixed-point movement in the air.
  • the air-to-air input device is also used to train a neural network through the first attitude information and the first fixed point trajectory to obtain the mapping relationship.
  • the first visual sensor includes a stereo camera, lidar, depth camera or monocular camera.
  • the fourth aspect provides an air-to-air input device, including:
  • the starting point determination unit is used to determine the fixed point starting point in the display area
  • a trajectory determination unit configured to determine the position and orientation information of the first pointing device and the first pointing device in the first pointing device according to the position and orientation information collected by the first visual sensor during the three-dimensional pointing movement of the first pointing device in the air.
  • the position and orientation information of the first pointing device collected by an IMU and the mapping relationship between the first position information and the first fixed point trajectory determine the fixed point trajectory of the first pointing device in the air, wherein,
  • the first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU;
  • a trace determination unit is configured to determine fixed-point traces in the display area according to the fixed-point starting point and the fixed-point trajectory.
  • the pose information of the first pointing device collected by the first visual sensor includes three-dimensional coordinates of points in the three-dimensional model of the first pointing device, and, the first The rotation angle of the wrist part in the three-dimensional model of the pointing device relative to the user's three-dimensional model.
  • the three-dimensional model of the user and the three-dimensional model of the first pointing device are the user's three-dimensional model collected according to the first visual sensor.
  • the three-dimensional data and the three-dimensional data of the first pointing device are established.
  • the starting point of the fixed point is the intersection point of the normal vector of a specific part in the three-dimensional model of the user or the three-dimensional model of the first pointing device and the display area.
  • the starting point of the fixed point is the intersection point of the normal vector of the eye part in the user's three-dimensional model and the display area.
  • the device further includes a training unit configured to receive the first fixed-point trajectory sent by the robotic arm, where the first fixed-point trajectory is controlled by the robotic arm.
  • the second fixed-point device is obtained by performing three-dimensional fixed-point movement in the air; receiving the first attitude information, training the neural network through the first attitude information and the first fixed-point trajectory, and obtaining the mapping relationship .
  • the first visual sensor includes a stereo camera, lidar, depth camera or monocular camera.
  • a computer-readable storage medium which is characterized in that it includes instructions that, when the instructions are run on an air-to-air input device, cause the air-to-air input device to execute any one of the first aspects. method described.
  • Figure 1 is a schematic diagram of an air-to-air input scenario involved in this application
  • FIG. 2 is a schematic structural diagram of an air-to-air input system involved in this application.
  • Figure 3 is a schematic structural diagram of a smart screen provided by this application.
  • Figure 4 is a schematic structural diagram of a pointing device provided by this application.
  • FIG. 5 is a schematic structural diagram of an IMU provided by this application.
  • Figure 6 is a schematic structural diagram of an air-to-air input system provided by this application.
  • Figure 7 is a schematic structural diagram of a projection device provided by this application.
  • Figure 8 is a schematic flow chart of an air-to-air input method provided by this application.
  • Figure 9 is a schematic diagram of the pose information of the fixed-point device collected by the IMU provided by this application.
  • Figure 10 is a schematic diagram of determining the starting point of a fixed point based on the eye part of the user's three-dimensional model provided by this application;
  • Figure 11 is a schematic diagram of a trajectory prediction model provided by this application.
  • Figure 12 shows a schematic structural diagram of an air-to-air input device provided by this application.
  • Figure 13 shows a schematic structural diagram of an air-to-air input device provided by this application.
  • FIG 2 is a schematic structural diagram of an air-to-air input system provided by the present application.
  • the provided air-to-air input system includes: a smart screen 110, a pointing device 120, and a visual sensor 130.
  • the smart screen 110 may include: a processor 112, a memory 113, a wireless communication module 114, a power switch 115, a wired local area network (LAN) communication module 116, and a high definition multimedia interface.
  • HDMI communication module 117, universal serial bus (universal serial bus, USB) communication module 118 and display 119. in:
  • Processor 112 may be used to read and execute computer-readable instructions.
  • the processor 112 may mainly include a controller, arithmetic unit, and a register.
  • the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions.
  • the arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations. It can also perform address operations and conversions.
  • Registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution.
  • processor 112 may be used to parse signals received by wireless communication module 114 and/or wired LAN communication module 116 .
  • the processor 112 may be configured to perform corresponding processing operations according to the parsing results, such as generating a detection response, driving the display 119 to perform display according to the display request or display instruction, and so on.
  • the processor 112 can also be used to generate signals sent externally by the wireless communication module 114 and/or the wired LAN communication module 116, such as Bluetooth broadcast signals, beacon signals, or signals sent to electronic devices. Feedback the signal of display status (such as display success, display failure, etc.).
  • Memory 113 is coupled to processor 112 for storing various software programs and/or sets of instructions.
  • the memory 113 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices.
  • the memory 113 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems.
  • Memory 113 may also store communications programs that may be used to communicate with one or more servers, or additional devices.
  • the wireless communication module 114 may include one or more of a Bluetooth (bluetooth) communication module 114A and a wireless local area networks (WLAN) communication module 114B.
  • a Bluetooth (bluetooth) communication module 114A and a wireless local area networks (WLAN) communication module 114B.
  • WLAN wireless local area networks
  • one or more of the Bluetooth (BT) communication module 114A and the WLAN communication module 114B can monitor signals transmitted by other devices, such as detection requests, scanning signals, etc., and can send response signals, such as The detection response, scanning response, etc. enable other devices to discover the smart screen 110, establish wireless communication connections with other devices, and communicate with other devices through one or more wireless communication technologies in Bluetooth or WLAN.
  • BT Bluetooth
  • WLAN wireless local area network
  • one or more of the Bluetooth communication module 114A and the WLAN communication module 114B can also transmit signals, such as broadcasting Bluetooth signals and beacon signals, so that other devices can discover the smart screen 110 and communicate with other devices. Establish a wireless communication connection to communicate with other devices through one or more wireless communication technologies such as Bluetooth or WLAN.
  • the wireless communication module 114 may also include: Bluetooth, WLAN, near field communication (NFC), ultra wide band (UWB), infrared, and so on.
  • the power switch 115 can be used to control the power supply to the smart screen 110 .
  • the wired LAN communication module 116 can be used to communicate with other devices in the same LAN through the wired LAN, and can also be used to connect to the wide area network through the wired LAN and communicate with devices in the wide area network.
  • HDMI communication module 117 may be used to communicate with other devices through an HDMI interface (not shown).
  • USB communication module 118 may be used to communicate with other devices through a USB interface (not shown).
  • Display 119 may be used to display images, videos, etc.
  • the display 119 can be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or an active matrix display.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED Active-matrix organic light emitting diode
  • FLED flexible light-emitting diode
  • QLED quantum dot light emitting diode
  • the smart screen 110 may also include an audio module (not shown).
  • the audio module can be used to output audio signals through the audio output interface, so that the smart screen 110 supports audio playback.
  • the audio module can also be used to receive audio data through the audio input interface.
  • the smart screen 110 can be a media playback device such as a television.
  • the smart screen 110 may also include a serial interface such as an RS-232 interface.
  • the serial interface can be connected to other devices, such as speakers and other audio external amplifiers, so that the smart screen 110 and the audio external amplifiers can cooperate to play audio and video.
  • the structure illustrated in Figure 3 does not constitute a specific limitation on the smart screen 110.
  • the smart screen 110 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange different components.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • the pointing device 120 may be a drawing device provided with an IMU.
  • the pointing device can be a pen-shaped device or a device of other shapes. Pointing devices can be pens, controllers, etc.
  • the pointing device 120 may include: a processor 122, a memory 123, a wireless communication module 124, a charging management module 125, a USB interface 126, a battery 127, a power management module 128 and an IMU 129.
  • Processor 122 may be used to read and execute computer-readable instructions.
  • the processor 122 may mainly include a controller, arithmetic unit, and a register.
  • the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions.
  • the arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations. It can also perform address operations and conversions.
  • Registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution.
  • the processor 122 may also adopt a heterogeneous architecture, such as an ARM+DSP architecture, an ARM+ASIC architecture, an ARM+AI chip architecture, and so on.
  • Memory 123 is coupled to processor 122 for storing various software programs and/or sets of instructions.
  • the memory 123 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices.
  • the memory 123 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems.
  • the wireless communication module 124 may include one or more of a Bluetooth (BT) communication module, a WLAN communication module, near field communication (NFC), ultra wide band (UWB), infrared, and the like.
  • BT Bluetooth
  • WLAN wireless local area network
  • NFC near field communication
  • UWB ultra wide band
  • infrared and the like.
  • the charge management module 125 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 125 may receive charging input from the wired charger through the USB interface 126 .
  • the charging management module 125 may receive wireless charging input through a wireless charging coil. While the charging management module 125 charges the battery 127, it can also provide power to the pointing device through the power management module 128.
  • the power management module 128 is used to connect the battery 127, the charging management module 125 and the processor 122.
  • the power management module 128 receives input from the battery 127 and/or the charging management module 125 and supplies power to the processor 122, the memory 123, the wireless communication module 124, the IMU 129, and the like.
  • the power management module 128 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters.
  • the power management module 128 may also be provided in the processor 122 .
  • the power management module 128 and the charging management module 125 can also be provided in the same device.
  • IMU 129 is a device that measures the three-axis angular velocity and acceleration of an object.
  • An IMU may be equipped with a three-axis gyroscope and a three-axis accelerometer to measure the angular velocity and acceleration of an object in three-dimensional space.
  • an IMU only provides users with three-axis angular velocity and three-axis acceleration data.
  • the vertical reference unit (VRU) is based on the IMU, uses the gravity vector as a reference, and uses algorithms such as Kalman or complementary filtering to provide users with pitch angles, roll angles referenced by the gravity vector, and angles without reference standards. Heading.
  • the so-called 6-axis attitude module belongs to this type of system. There is no reference for the heading angle.
  • the heading angle will be 0° (or a set constant) after startup. As the working time of the module increases, the heading angle will slowly accumulate errors. Since the pitch angle and roll angle have a gravity vector reference, there will be no cumulative error for a long time under low maneuvering conditions.
  • the attitude and heading reference system (AHRS) system adds a magnetometer or optical flow sensor to the VRU, and uses algorithms such as Kalman or complementary filtering to provide users with absolute reference pitch angles and roll angles. As well as heading angle equipment, this type of system is used to provide accurate and reliable attitude and navigation information for the aircraft.
  • the 9-axis attitude sensor we usually call belongs to this type of system. Because the heading angle has a reference to the geomagnetic field, it will not drift.
  • accelerometers, gyroscopes, magnetometers, etc. are used as basic inputs, and the attitude information, position information and speed required by the user are output through the data acquisition unit, calibration and compensation unit, data fusion unit, output and configuration unit information.
  • the structure illustrated in FIG. 4 does not constitute a specific limitation on the pointing device 120 .
  • the pointing device 120 may include more or fewer components than shown in the figures, or some components may be combined, or some components may be separated, or may be arranged differently.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • the visual sensor 130 may be a monocular camera, a stereo camera, a depth camera or a laser radar that has been trained to acquire three-dimensional data, and is not specifically limited here.
  • the visual sensor may be integrated with the air-to-air input device, or may be provided separately, which is not specifically limited here.
  • the structure illustrated in Figure 2 does not constitute a specific limitation on the air-to-air input system.
  • the air-to-air input system may include more or less components than shown in the figures, or some components may be combined, or some components may be separated, or may be arranged differently.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • Figure 6 is a schematic structural diagram of an air-to-air input system provided by the present application.
  • the air-to-air input system provided by this application includes: a projector 210 with a function of determining fixed-point traces, a pointing device 220, and a visual sensor 230.
  • the projector 210 with fixed-point trace determination function may include: a processor 212, a memory 213, a wireless communication module 214, a power switch 215, a wired LAN communication module 216, an HDMI communication module 217, and a light source controller 218. and image projector 219. in:
  • Processor 212 may be used to read and execute computer-readable instructions.
  • the processor 212 may mainly include a controller, arithmetic unit, and a register.
  • the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions.
  • the arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations. It can also perform address operations and conversions.
  • Registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution.
  • the processor 212 may be used to parse signals received by the wireless communication module 214 and/or the wired LAN communication module 216, such as broadcast detection requests, projection requests, projection instructions sent by the server of the cloud projection service provider, etc.
  • the processor 212 may be configured to perform corresponding processing operations according to the parsing results, such as generating a detection response, or according to
  • the projection request or projection instruction drives the light source controller 218 and the image projector to perform projection operations, and so on.
  • the processor 212 can also be used to generate signals sent externally by the wireless communication module 214 and/or the wired LAN communication module 216, such as Bluetooth broadcast signals, beacon signals, or for feedback sent to electronic devices.
  • Signal of projection status (such as projection success, projection failure, etc.).
  • Memory 213 is coupled to processor 212 for storing various software programs and/or sets of instructions.
  • the memory 213 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices.
  • the memory 213 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems.
  • Memory 213 may also store communications programs that may be used to communicate with one or more servers, or additional devices.
  • the wireless communication module 214 may include one or more of a Bluetooth (BT) communication module 214A and a WLAN communication module 214B.
  • BT Bluetooth
  • WLAN WLAN
  • one or more of the Bluetooth (BT) communication module and the WLAN communication module can monitor signals transmitted by other devices, such as detection requests, scanning signals, etc., and can send response signals, such as detection responses. , scanning response, etc., so that other devices can discover the projector 210, establish wireless communication connections with other devices, and communicate with other devices through one or more wireless communication technologies in Bluetooth or WLAN.
  • BT Bluetooth
  • WLAN wireless communication
  • one or more of the Bluetooth (BT) communication module and the WLAN communication module can also transmit signals, such as broadcasting Bluetooth signals and beacon signals, so that other devices can discover the projector 210 and communicate with other devices.
  • the device establishes a wireless communication connection and communicates with other devices through one or more wireless communication technologies such as Bluetooth or WLAN.
  • Wireless communication module 214 may also include a cellular mobile communication module (not shown).
  • the cellular mobile communication module can communicate with other devices (such as servers) through cellular mobile communication technology.
  • the power switch 215 may be used to control power supply to the projector 210 .
  • the wired LAN communication module 216 can be used to communicate with other devices in the same LAN through the wired LAN, and can also be used to connect to the wide area network through the wired LAN and communicate with devices in the wide area network.
  • HDMI communication module 217 may be used to communicate with other devices through an HDMI interface (not shown).
  • the image projector 219 may have a light source (not shown), may modulate light emitted from the light source according to image data and project an image on the screen.
  • the light source controller 218 may be used to control lighting of the light source provided by the image projector 219 .
  • the structure illustrated in FIG. 6 does not constitute a specific limitation on the projector 210 with the function of determining fixed point traces.
  • the projector 210 with fixed point trace determination function may include more or less components than shown in the figure, or combine some components, or split some components, or arrange different components.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • the pointing device 220 and the visual sensor 230 may refer to the pointing device 120 and the visual sensor 130 shown in FIG. 2 .
  • the fixed-point trace determination function can be implemented by a peripheral fixed-point trace determination device without being integrated with the projector.
  • the fixed-point trace determination function includes receiving the position and orientation information of the first pointing device collected by the first visual sensor and the position and orientation information of the first pointing device collected by the first IMU, and based on the first fixed-pointing device collected by the first visual sensor.
  • the posture information of the pointing device and the first IMU collect the posture information of the first pointing device to determine the pointing trace.
  • a projector can be used to display the fixed point traces in the projection area.
  • FIG 8 is a schematic flow chart of an air-to-air input method provided by the present application.
  • this application Provided air-to-air input methods include:
  • the first pointing device performs three-dimensional pointing movement in the air, and collects the pose information of the first pointing device through the first IMU in the first pointing device.
  • the first pointing device may be a pointing device in the air-to-air input system shown in FIG. 2 or FIG. 6 . Please refer to Figure 2 or Figure 6 and related descriptions for the specific structure of the first pointing device, and will not be described in detail here.
  • the pose information of the first pointing device collected by the first IMU includes: the three-dimensional coordinates of the first pointing device and the angle of the first pointing device.
  • the first pointing device measures data through the gyroscope, accelerometer and magnetometer in the first IMU, and inputs the measured data into the data acquisition unit, calibration and compensation unit, data fusion unit and output and
  • the configuration unit performs acquisition, calibration and compensation, data fusion and output configuration to obtain the three-dimensional coordinates of the first pointing device (position information in Figure 5) and the angle of the first pointing device (attitude information in Figure 5).
  • the pose information of the first pointing device collected by the first IMU is (x_pen, y_pen, z_pen, yaw, roll, pitch), where x_pen is the coordinate value of the first pointing device on the x-axis relative to the starting time, y_pen is the coordinate value of the first fixed point device on the y axis relative to the starting time, z_pen is the coordinate value of the first fixed point device on the z axis relative to the starting time, yaw is the coordinate value of the first fixed point device relative to the starting time is the pitch angle, roll is the roll angle of the first pointing device relative to the starting time, and pitch is the heading angle of the first pointing device relative to the starting time.
  • the original data collected by the first IMU are the acceleration of the x-axis, y-axis and z-axis and the angular velocity of the x-axis, y-axis and z-axis.
  • the pose information x_pen is obtained by integrating the acceleration of the x-axis twice.
  • the pose information y_pen is obtained by integrating the acceleration of the y-axis twice.
  • the pose information z_pen is obtained by integrating the acceleration of the z-axis twice.
  • the pose information yaw is obtained by integrating the angular velocity of the x-axis once.
  • the pose information roll is obtained by integrating the angular velocity of the y-axis once
  • the pose information pitchl is obtained by integrating the angular velocity of the z-axis. Because the acceleration of the x-axis, y-axis, and z-axis and the angular velocity of the x-axis, y-axis, and z-axis collected by the first IMU all have systematic errors and random errors, and the integration process will amplify these errors, resulting in the obtained pose information. The error is relatively large.
  • the first visual sensor collects the pose information of the first pointing device.
  • the first visual sensor may be the visual sensor in the air-to-air input system shown in FIG. 2 or FIG. 6 . Please refer to Figure 2 or Figure 6 and related descriptions for the specific structure of the first visual sensor, which will not be described again here.
  • the first visual sensor may be disposed in the display area or may be disposed outside the display area.
  • the angle between the line of sight axis of the first visual sensor and the normal line of the display area is zero. In this case, the complexity of subsequent calculations can be simplified.
  • the first visual sensor is set outside the display area, the degree of freedom in setting the first visual sensor can be increased.
  • it can also be set in the display area. outside the display area.
  • the first visual sensor collects the three-dimensional data of the user and the three-dimensional data of the first pointing device. At this time, the three-dimensional data of the user and the three-dimensional data of the first pointing device are collected based on the first visual sensor.
  • the viewpoint of the sensor is used as the origin of the coordinates, and the coordinate transformation relationship is needed to convert the user's three-dimensional data with the visual center of the first vision sensor as the coordinate origin and the three-dimensional data of the first pointing device into a coordinate origin with the center of the display area.
  • the three-dimensional data of the user and the three-dimensional data of the first pointing device are used as the origin of the coordinates, and the coordinate transformation relationship is needed to convert the user's three-dimensional data with the visual center of the first vision sensor as the coordinate origin and the three-dimensional data of the first pointing device into a coordinate origin with the center of the display area.
  • a three-dimensional model of the user is established based on the converted three-dimensional data of the user, and a three-dimensional model of the first pointing device is established based on the converted three-dimensional data of the first pointing device. Then, determine the pose information of the first pointing device according to the three-dimensional model of the user and the three-dimensional model of the first pointing device.
  • First vision sensor collection The pose information of the first pointing device includes the three-dimensional coordinates of the point in the three-dimensional model of the first pointing device, and the three-dimensional model of the first pointing device is relative to the user's wrist in the three-dimensional model. The rotation angle of the part.
  • the first visual sensor When the first visual sensor is installed in the smart screen, that is, when the first visual sensor is integrated in the smart screen, the positional relationship between the first visual sensor and the center of the display area of the smart screen will not change. Therefore, , it can be calibrated in advance before leaving the factory to determine the coordinate transformation relationship between the visual center of the first visual sensor and the center of the display area. This coordinate transformation relationship will not change after leaving the factory.
  • the first visual sensor When the first visual sensor is set outside the smart screen, that is, the first visual sensor is an external device of the smart screen, one or several placement positions can be set on the outer frame of the smart screen.
  • the one or several placement positions and The positional relationship between the centers of the display areas of the smart screen will not change. Therefore, it can be calibrated in advance before leaving the factory to determine the distance between the first visual sensor set at one or several placement positions and the display area.
  • the coordinate transformation relationship between centers will not change after leaving the factory.
  • the first visual sensor is set at any position that can capture the display area of the smart screen, and then the calibration image is played in the display area of the smart screen, and the calibration is performed based on the calibration image to determine the visual center of the first visual sensor. Displays the coordinate transformation relationship between the centers of the areas.
  • the first visual sensor When the first visual sensor is set outside the projector with fixed-point trace determination function, one or several placement positions can be set by formulating the projection boundary.
  • the first visual sensor can be set at the upper left of the projection boundary. corner or upper right corner.
  • the relative positional relationship between the one or several placement positions and the center of the display area of the projector is clear. Therefore, it can be determined between the first visual sensor disposed at the one or several placement positions and the center of the display area. coordinate transformation relationship between them.
  • the first visual sensor is set at any position that can capture the display area of the projector, and then the calibration image is played in the display area of the projector, and calibration is performed based on the calibration image to determine the visual center of the first visual sensor. Displays the coordinate transformation relationship between the centers of the areas. It can be understood that when the projector with the fixed-point trace determination function is replaced by a combination of the fixed-point trace determination device and the projector, the calibration method of the coordinate transformation relationship is also similar, and will not be described here.
  • the pose information of the first pointing device collected by the first vision sensor is (x_tip, y_tip, z_tip, ⁇ _x, ⁇ _y, ⁇ _z), where, x_tip is the coordinate value of the point in the three-dimensional model of the first pointing device relative to the coordinate origin (0, 0, 0) on the x-axis, y_tip is the point in the three-dimensional model of the first pointing device relative to the coordinate origin (0, 0, 0) is the coordinate value on the y-axis, z_tip is the coordinate value of the point in the three-dimensional model of the first pointing device relative to the coordinate origin (0, 0, 0) on the z-axis, ⁇ _x is the coordinate value of the first pointing device The pitch angle of the three-dimensional model relative to the user's wrist in the three-dimensional model, ⁇ _y is the roll angle of the three-dimensional model of the first pointing device relative to the wrist in
  • the wrist part in the first user's three-dimensional model can be extracted by inputting the first user's three-dimensional model into an extraction model, where the extraction model can be a deep learning (DL) network. , convolutional neural networks (CNN) and so on.
  • the input extraction model may be trained using a large number of three-dimensional models of the second user and wrist parts of the three-dimensional model of the second user.
  • the three-dimensional model of the first user is collected by the first visual sensor
  • the three-dimensional model of the second user can be collected by the second visual sensor
  • the three-dimensional model of the second user can be manually annotated.
  • the first visual sensor and the second visual sensor may be the same visual sensor, or they may be two different visual sensors.
  • the first visual sensor and the second visual sensor may be the same stereo camera, or they may be two different visual sensors. different stereo cameras.
  • the first visual sensor can be a stereo camera
  • the second visual sensor a lidar, and so on.
  • the first IMU and the second IMU can So the same visual sensor can also be two different visual sensors.
  • the rotation angle of the three-dimensional model of the first pointing device relative to the wrist in the three-dimensional model of the user can be determined in the following manner: converting the points in the three-dimensional model of the first pointing device Connect a line with the point in the wrist of the user's three-dimensional model, and then find the rotation angle of the connecting line relative to the coordinate system with the point in the wrist as the coordinate origin.
  • the point in the three-dimensional model of the first pointing device may be any point in the three-dimensional model of the first pointing device, for example, it may be a vertex of the three-dimensional model of the first pointing device (for example, , the pen tip in the three-dimensional model of the writing pen), the center of mass of the three-dimensional model of the first pointing device and the end point of the three-dimensional model of the first pointing device, and so on.
  • the pen tip in the three-dimensional model of the first pointing device is taken as an example for explanation.
  • the point in the wrist part of the user's three-dimensional model can be any point in the wrist part of the user's three-dimensional model.
  • it can be the center point of the wrist part in the user's three-dimensional model.
  • the center point of the wrist part in the user's three-dimensional model may be obtained by averaging the coordinate values of each point of the wrist part in the user's three-dimensional model.
  • the point in the user's wrist in the three-dimensional model is used as one point for explanation. In other embodiments, it can also be two points, three points or even more points, which are not specifically limited here.
  • the first visual sensor can provide relatively accurate absolute pose information relative to the center point of the display area, but the first visual sensor is usually unable to provide continuous absolute pose information. Therefore, in the embodiment of the present invention, a first IMU is introduced, which can continuously provide relatively accurate relative pose information relative to the starting position. Through the cooperation of the two sensors, it can not only provide continuous pose information, but also avoid the systematic errors and random errors caused by simply using the IMU, and improve the accuracy of determining the fixed-point trajectory in the air.
  • the first pointing device sends the pose information of the first pointing device collected by the first IMU to the air-to-air input device.
  • the air-to-air input device receives the pose information of the first pointing device collected by the first IMU and sent by the first pointing device.
  • the first pointing device is provided with one or more of a wireless communication module or a USB interface.
  • the air-to-air input device is also provided with one or more of a wireless communication module and a USB communication module.
  • the first pointing device and the air-to-air input device communicate in a wireless manner, there is no data connection line between the first pointing device and the air-to-air input device, which can make the use of the first pointing device more convenient.
  • the first pointing device and the air-to-air input device communicate using USB, the data communication between the first pointing device and the air-to-air input device can be made smoother.
  • the first pointing device may be provided with one or more of a wireless communication module or a USB interface.
  • S104 The pose information of the first pointing device collected by the first visual sensor is sent to the air-to-air input device.
  • the air-to-air input device receives the position and orientation information of the first pointing device collected by the visual sensor and sent by the first visual sensor.
  • the first visual sensor is provided with one or more of a wireless communication module, a USB interface, a wired LAN communication module, and an HDMI communication module.
  • the air-to-air input device is also provided with a wireless communication module and a USB communication module.
  • the first visual sensor and the air-to-air input device communicate in a wireless manner, there is no data connection line between the first visual sensor and the air-to-air input device, which can make the use of the first visual sensor more convenient.
  • the first vision sensor and the air-to-air input device communicate using USB, wired LAN or HDMI, the data communication between the first vision sensor and the air-to-air input device can be made smoother.
  • the air-to-air input device determines the starting point of the fixed point in the display area.
  • the air-to-air input device may be the smart screen in Figure 2 or the projector with fixed-point trace determination function in Figure 6, or an air-to-air input device composed of a fixed-point trace determination device and a projector. etc. Please refer to Figure 2 or Figure 6 and related descriptions for the specific structure of the air-to-air input device, which will not be described again here.
  • the display area may be an area capable of displaying a fixed-point trajectory, for example, it may be a displayable area of a smart screen, or a projection area of a projector, or the like.
  • the air-to-air input device determines the starting point of the fixed point in the display area, specifically: the processor of the air-to-air input device obtains the user's three-dimensional image collected from the first visual sensor from the memory of the air-to-air input device. Data, three-dimensional data of the first pointing device. The processor of the air-to-air input device respectively establishes a three-dimensional model of the user and a three-dimensional model of the first pointing device based on the three-dimensional data of the user and the three-dimensional data of the first pointing device.
  • the processor of the air-to-air input device determines the intersection of the normal vector of the specific part in the user's three-dimensional model or the three-dimensional model of the first pointing device and the display area as the starting point of the fixed point.
  • the specific part of the user's three-dimensional model may be any part of the user's three-dimensional model, for example, the eye part, the nose tip part, the finger tip part, etc.
  • the priority of specific parts of the user's three-dimensional model can be set based on different conditions. For example, when the user points to the display area with a finger, the tip of the finger is used first to determine the starting point of the fixed point.
  • the specific part in the three-dimensional model of the first pointing device may be any part in the three-dimensional model of the first pointing device, for example, an endpoint part, a center part, etc.
  • the point in the three-dimensional model of the first pointing device may be the tip of the writing pen. The following will be described with reference to the specific embodiment in FIG. 10 .
  • the starting point of the fixed point is The intersection point (x, y, 0) of the normal vector (x_n, y_n, z_n) of the user's eye part in the user's three-dimensional model and the smart screen.
  • the starting point of the fixed point is determined based on the pose information of the IMU in the first pointing device.
  • the starting point of the fixed point determined based on the pose information of the IMU is often not the starting point of the fixed point desired by the user. Users often need to adjust the pointing device multiple times to reach the starting point they want, resulting in very low efficiency.
  • visual sensors are used to determine the part where people's eyes are looking at to determine the drawing, which can more accurately and efficiently determine the starting point of the fixed point that the user wants.
  • the starting point of the fixed point is located. , it is very convenient and user-friendly to use.
  • the user's three-dimensional model established by the three-dimensional data collected by the visual sensor or the three-dimensional model of the first pointing device is used to determine the accuracy of the starting point of the fixed point. Also very high.
  • the airborne input device determines the fixed-point trajectory of the first pointing device in the air.
  • the air-to-air input device determines the fixed-point trajectory of the first pointing device in the air, specifically: during the movement of the first pointing device in the air, the processor of the air-to-air input device determines the fixed-point trajectory of the first pointing device in the air according to the first
  • the posture information of the first pointing device collected by a visual sensor, the posture information of the first pointing device collected by the IMU in the first pointing device, and the first posture information is consistent with the first pointing device.
  • the mapping relationship between point trajectories determines the fixed-point trajectory of the first pointing device in the air.
  • the first posture information in the mapping relationship includes the posture information collected by the second visual sensor and the posture information collected by the second IMU.
  • the processor of the air-to-air input device uses the posture information of the first pointing device collected by the first visual sensor and all the information collected by the first IMU in the first pointing device.
  • the posture information and trajectory prediction model of the first pointing device determine the fixed-point trajectory of the first pointing device in the air.
  • the trajectory prediction model is trained based on the first posture information collected by the second vision sensor, the posture information collected by the second IMU, and the first fixed point trajectory. Arrived.
  • the trajectory prediction model can be (deep neural networks, DNN), linear regression model, etc.
  • y is the fixed-point trajectory of the first pointing device in the air
  • state 1 is the pose information of the first pointing device collected by the first visual sensor
  • state 2 is the position and orientation information of the first pointing device collected by the first IMU.
  • Pose information, f() is the mapping relationship.
  • state 1 is the pose information of the first pointing device collected by the first visual sensor (x_tip, y_tip, z_tip, ⁇ _x, ⁇ _y, ⁇ _z)
  • tats 2 is the position information of the first pointing device collected by the first IMU.
  • the pose information of the device at a certain point is (x_pen, y_pen, z_pen, yaw, roll, pitch).
  • the trajectory prediction model may include an input layer, a hidden layer and an output layer.
  • the inputs of the input layer are S 1 and S 2 , where S 1 is the pose information of the first pointing device collected by the first visual sensor, and S 2 is the position of the first pointing device collected by the first IMU. pose information, the output and input are equal, that is, no processing is performed on the input.
  • S 1 is the pose information of the first pointing device collected by the first visual sensor
  • S 2 is the position of the first pointing device collected by the first IMU. pose information
  • the output and input are equal, that is, no processing is performed on the input.
  • the input layer does not perform any processing.
  • the input layer can be normalized and processed without specific limitations here.
  • the weight vector of , W 2 l is the weight vector of Z 2 l of the l-th layer
  • b l is the bias vector of the l-th layer
  • a l+1 is the intermediate vector of the l+1-th layer
  • Z l+1 is the hidden layer result of the l+1th layer.
  • the first excitation function and the second excitation function may be any one of sigmoid function, hyperbolic tangent function, Relu function, etc.
  • W 2 l is the weight vector of Z 2 l of the l-th layer
  • b l is the bias vector of the l-th layer
  • the trajectory prediction model before using the trajectory prediction model, needs to be trained first.
  • the specific process of training the trajectory prediction model is to obtain a large amount of first attitude information and the corresponding first fixed point trajectory.
  • the first posture information includes the posture information of the second pointing device collected by the second visual sensor, the posture information of the second pointing device collected by the second IMU, and the corresponding first pointing trajectory.
  • the pose information of the second fixed-point device collected by the second visual sensor and the second position and orientation information collected by the second IMU can be combined.
  • the pose information of the fixed-point device is input into the trajectory prediction model to obtain the predicted value of the data, and the first fixed-point trajectory is used as the real desired target value, which is more accurate.
  • the weight vector of each layer of the deep neural network in the trajectory prediction model is updated based on the difference between the previous predicted value and the really desired target value (of course, there is usually a
  • the initialization process is to pre-configure parameters for each layer in the trajectory prediction model).
  • the prediction value of the trajectory prediction model is high, adjust the weight vector to make it predict a lower value.
  • the trajectory prediction model can predict Find the target value you really want. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value".
  • This is the loss function (loss function) or objective function (objective function), which is used to measure the difference between the predicted value and the target value.
  • the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the trajectory prediction model becomes a process of reducing this loss as much as possible.
  • the first visual sensor and the second visual sensor may be the same visual sensor, or they may be two different visual sensors.
  • the first visual sensor and the second visual sensor may be the same stereo camera, or they may be two different visual sensors. different stereo cameras.
  • the first visual sensor can be a stereo camera, the second visual sensor a lidar, and so on.
  • the first pointing device and the second pointing device may be the same device, or may not be the same device.
  • the first IMU and the second IMU are the same IMU.
  • the first The IMU and the second IMU are two different IMUs.
  • the first visual sensor and the second visual sensor are the same visual sensor, but the first pointing device and the second pointing device are different pointing devices.
  • the first visual sensor and the second visual sensor are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device.
  • the first visual sensor and the second visual sensor are different visual sensors, but the first pointing device and the second pointing device are the same pointing device.
  • Obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory may occur at the same time or not at the same time.
  • mapping The relationship can continue to change to continuously improve the accuracy of the mapping relationship in the process of using the mapping relationship.
  • the posture information of the first positioning device collected by the first visual sensor and the posture information of the first positioning device collected by the first IMU can be used as the third positioning device when determining the fixed-point trajectory using the mapping relationship.
  • the fixed-point trajectory determined using the mapping relationship can be used as the first fixed-point trajectory to train the trajectory prediction model in the air-to-air input device.
  • the mapping relationship can be obtained through training first, and then the mapping relationship is used to determine the fixed-point trajectory. That is, obtaining the mapping relationship through training occurs in the past time and space, while using the mapping relationship to determine the fixed-point trajectory occurs in the present time and space.
  • a specialized training device can be tasked with training to obtain the mapping relationship to reduce the load on the air-to-air input device.
  • the mapping relationship can still be obtained by training through the air input device. In other words, the mapping relationship is obtained through training using historically collected data.
  • a robotic arm can be used to control the second pointing device to perform fixed-point movement in the air. That is, a robotic arm can be used to simulate a human arm's three-dimensional fixed-point movement in the air. In order to better simulate the human arm, the robotic arm can use a bionic arm that is comparable to a human arm.
  • the trajectory obtained by the robot arm controlling the second fixed-point device to perform three-dimensional fixed-point movement in the air is the first fixed-point trajectory.
  • the second visual sensor collects the three-dimensional data of the robotic arm and the three-dimensional data of the second fixed-point device, and establishes a three-dimensional model of the robotic arm based on the three-dimensional data of the robotic arm.
  • the pose information of the second pointing device collected by the second visual sensor and the pose information of the second pointing device collected by the second IMU in the second pointing device are collectively referred to as the first pose information.
  • the robotic arm sends the first fixed-point trajectory to the air-to-air input device or training device
  • the second visual sensor sends the pose information of the second fixed-point device collected by the second visual sensor to the air-to-air input device or training device.
  • the pointing device sends the pose information of the second pointing device collected by the second IMU in the second pointing device to the air-to-air input device or the training device to train the trajectory prediction model.
  • the positions of the first visual sensor and the second visual sensor remain unchanged.
  • the first visual sensor can be placed in the middle of the upper edge of the smart screen, and the second visual sensor can also be placed. In the middle of the upper edge of the smart screen.
  • the positions of the first visual sensor and the second visual sensor may also change.
  • the first visual sensor can be disposed in the center of the upper edge of the smart screen, and the second visual sensor can be disposed in the upper left corner of the smart screen.
  • the three-dimensional data of the fixed-point device is converted into the three-dimensional data of the robot arm with the center of the display area as the coordinate origin and the three-dimensional data of the second fixed-point device.
  • the first visual sensor and the second visual sensor When obtaining the mapping relationship through training and using the mapping relationship to determine the fixed-point trajectory occur at the same time, the first visual sensor and the second visual sensor often used are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device. .
  • the first visual sensor and the second visual sensor may be the same visual sensor, but the first pointing device and the second pointing device may be different. Pointing device. Alternatively, the first visual sensor and the second visual sensor are the same visual sensor, and the first pointing device and the second pointing device are the same pointing device.
  • first visual sensor and the second visual sensor are different visual sensors, but the first pointing device and the second pointing device are the same pointing device.
  • first visual sensor and the second visual sensor are different visual sensors, and the first pointing device and the second pointing device are also different pointing devices.
  • the air-to-air input device determines the fixed-point trace in the display area based on the fixed-point starting point and the fixed-point trajectory.
  • the processor of the air-to-air input device determines the fixed-point trace in the display area based on the fixed-point starting point and the fixed-point trajectory.
  • the air-to-air input device displays fixed-point traces in the display area.
  • fixed-point traces can be displayed in the display area of the display of the smart screen; when the air-to-air input device is a projector with a fixed-point trace determination function or a fixed-point trace When a device with a trace determination function and an air-to-air input device composed of a projector are used, fixed-point traces can be projected to the projection area through the image projector of the projector.
  • the air-to-air input device can receive the posture information of the first pointing device collected by the first visual sensor and the first IMU collects the posture information of the first pointing device, and performs the processing according to the first
  • the position and orientation information of the first pointing device collected by the visual sensor and the position and orientation information of the first pointing device collected by the first IMU determine the fixed point trace, and project
  • the instrument can be used to display the fixed-point traces in the projection area, and the steps of displaying the fixed-point traces in the display area are explained as examples.
  • the air-to-air input device may include a fixed-point trace determination device and a projector, wherein the fixed-point trace determination device is configured to receive the pose information of the first pointing device collected by the first visual sensor and the first The IMU collects the pose information of the first pointing device, and determines the fixing trace based on the pose information of the first pointing device collected by the first visual sensor and the pose information of the first pointing device collected by the first IMU.
  • the projector can be used to display the fixed point trace in the projection area, which is not specifically limited here.
  • the first visual sensor collects the three-dimensional data of the user and the three-dimensional data of the first pointing device, and then determines the first fixed point based on the three-dimensional data of the user and the three-dimensional data of the first pointing device.
  • the first visual sensor can also collect the three-dimensional data of the user and the three-dimensional data of the first fixed point device, and the three-dimensional data of the user and the first fixed point collected by the first visual sensor can be used.
  • the three-dimensional data of the device is sent to the air-to-air input device, and the air-to-air input device determines the pose information of the first pointing device based on the user's three-dimensional data and the three-dimensional data of the first pointing device.
  • the second vision sensor can also collect the three-dimensional data of the robotic arm and the three-dimensional data of the second fixed-point device and send it to the air-to-air input device or training device, and the air-to-air input device or training device can collect the three-dimensional data of the robotic arm and the three-dimensional data of the second fixed-point device.
  • the three-dimensional data of the second pointing device determines the pose information of the second pointing device.
  • steps S101 and S102 are executed in no particular order.
  • Step S101 may be executed first and then step S102, or step S102 may be executed first and then step S101, or step S101 and step S102 may be executed simultaneously.
  • the execution order of step S103 and step S104 is in no particular order.
  • Step S103 may be executed first and then step S104, or step S103 may be executed first and then step S104, or step S103 and step S104 may be executed simultaneously.
  • FIG 12 shows a schematic structural diagram of an air-to-air input device provided by the present application.
  • air-to-air input devices include:
  • the starting point determining unit 310 is used to determine a fixed point starting point in the display area.
  • the trajectory determination unit 320 is configured to determine, according to the posture information of the first pointing device collected by the first visual sensor, the third point in the first pointing device during the three-dimensional pointing movement of the first pointing device in the air.
  • the position and orientation information of the first pointing device collected by an IMU and the mapping relationship between the first position information and the first fixed point trajectory determine the fixed point trajectory of the first pointing device in the air, wherein,
  • the first posture information includes the posture information of the second pointing device collected by the second visual sensor, and the posture information of the second pointing device collected by the second IMU.
  • the trace determining unit 330 is configured to determine the fixed point trace in the display area according to the fixed point starting point and the fixed point trajectory.
  • the starting point determination unit 310, the trajectory determination unit 320, and the trace determination unit 330 work together to implement the steps performed by the air-to-air input device in S104.
  • the starting point determining unit 310 is used to perform the step of determining the fixed point starting point in S105
  • the trajectory determining unit 320 is used to perform the step of determining the fixed point trajectory of the first pointing device in the air in S106
  • the trace determining unit 330 is used to perform The step of determining fixed-point traces in the display area in step 106 above.
  • the air-to-air input device may also include a training unit (not shown), the training unit being configured to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is the The mechanical arm controls the second fixed-point device to perform three-dimensional fixed-point movement in the air; receives the first attitude information, and trains the neural network through the first attitude information and the first fixed-point trajectory to obtain the mapping relationship.
  • a training unit (not shown)
  • the training unit being configured to receive the first fixed-point trajectory sent by the robotic arm, wherein the first fixed-point trajectory is the The mechanical arm controls the second fixed-point device to perform three-dimensional fixed-point movement in the air; receives the first attitude information, and trains the neural network through the first attitude information and the first fixed-point trajectory to obtain the mapping relationship.
  • the air-to-air input device may also include a receiving unit (not shown) for receiving the information collected by the first visual sensor.
  • the pose information of the first pointing device (or the user's three-dimensional data collected by the first visual sensor and the three-dimensional data of the first pointing device), and the pose information of the first pointing device collected by the first IMU.
  • it can also be used to receive the pose information of the second pointing device collected by the second visual sensor (or the three-dimensional data of the mechanical arm and the three-dimensional data of the second pointing device collected by the second visual sensor).
  • the air-to-air input device may also include a display unit (not shown) for displaying fixed-point traces in the display area.
  • the air-to-air input device may include a fixed-point trace determination device and a projector, wherein the starting point determination unit 310, the trajectory determination unit 320 and the trace determination unit 330 are provided in the trace determination device, and the display unit is provided in the projector, where There are no specific limitations.
  • the relevant introduction in step S105 in FIG. 8 For the position of the fixed point starting point determined by the starting point determining unit 310 and the method of determining the fixed point starting point, please refer to the relevant introduction in step S105 in FIG. 8 .
  • the relevant introduction in steps S101, S102 and S105 in Figure 8 please refer to the relevant introduction in steps S101, S102 and S105 in Figure 8, and will not be described here.
  • FIG 13 shows a schematic structural diagram of an air-to-air input device provided by the present application.
  • the air-to-air input device is used to perform the steps performed by the air-to-air input device in the above-mentioned air to air input method.
  • the air-to-air input device includes a memory 410 , a processor 420 , a communication interface 430 and a bus 440 .
  • the memory 410, the processor 420, and the communication interface 430 implement communication connections between each other through the bus 440.
  • the memory 410 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).
  • the memory 410 may store computer instructions, such as: computer instructions in the starting point determination unit 310, computer instructions in the trajectory determination unit 320, computer instructions in the trace determination unit 330, etc.
  • the processor 420 and the communication interface 430 are used to execute part or all of the method described in the above steps S104 to S106.
  • the memory 410 can also store data, for example: intermediate data or result data generated by the processor 420 during execution, for example, the user's three-dimensional data collected by the first visual sensor, the three-dimensional data of the first pointing device, the user's three-dimensional data. Model, three-dimensional model of the first pointing device, mapping relationship, pose information collected by the first visual sensor, pose information collected by the first IMU, etc.
  • the processor 420 may be a CPU, a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.
  • ASIC application specific integrated circuit
  • GPU graphics processing unit
  • the processor 420 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, part or all of the functions of the air-to-air input device may be completed by instructions in the form of hardware integrated logic circuits or software in the processor 420 .
  • the processor 420 may also be a general processor, a digital signal process (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, Discrete hardware components are used to implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of this application.
  • DSP digital signal process
  • FPGA field programmable gate array
  • Discrete hardware components are used to implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of this application.
  • the general-purpose processor can be a microprocessor or the processor can be any conventional processor, etc.
  • the steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be executed using decoding processing.
  • the combination of hardware and software modules in the device is executed.
  • the software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field.
  • the storage medium is located in the memory 410.
  • the processor 420 reads the information in the memory 410 and completes steps S105 to S108 in the above-mentioned air-to-air input method in conjunction with its hardware.
  • the communication interface 430 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device and other devices (eg, camera device, microphone, server).
  • a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device and other devices (eg, camera device, microphone, server).
  • Bus 440 may include a path for transmitting information between various components in the air-to-air input device (eg, memory 410, processor 420, communication interface 430).
  • the air-to-air input device can be used as a terminal device for team collaboration and communication. Therefore, optionally, the air-to-air input device may also include a camera device 450 and a microphone 460 for real-time collection of image signals and sound signals. Alternatively, the air-to-air input device can also be connected to the camera device 450 and the microphone 460 through the communication interface 430 for real-time collection of image signals and sound signals.
  • the structure of the fixed-point trace determining device is similar to the air-to-air input device shown in Figure 13. However, the fixed-point trace determining device does not need to perform the step of displaying fixed-point traces in the display area in the above-mentioned air-to-air input method.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated.
  • the available media may be magnetic media (eg, floppy disk, storage disk, tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Position Input By Displaying (AREA)

Abstract

La présente demande concerne un procédé, un dispositif et un système de saisie en l'air. Le procédé comprend les étapes suivantes : détermination d'un point de départ de point fixe dans une zone d'affichage ; dans un mouvement de point fixe tridimensionnel en l'air, un premier dispositif de point fixe détermine, en fonction d'informations de pose du premier dispositif de point fixe collectées par un premier capteur de vision, d'informations de pose du premier dispositif de point fixe collectées par une première unité de mesure inertielle dans le premier dispositif de point fixe, et d'une relation de correspondance entre des premières informations de pose et une première trajectoire de point fixe, la trajectoire de point fixe du premier dispositif de point fixe en l'air, les premières informations de pose comprenant les informations de pose d'un second dispositif de point fixe collectées par un second capteur de vision et les informations de pose du second dispositif de point fixe collectées par une seconde unité de mesure inertielle ; et détermination d'une trace de point fixe dans la zone d'affichage en fonction du point de départ de point fixe et de la trajectoire de point fixe. La solution peut améliorer la précision de détermination de la trajectoire de point fixe en l'air.
PCT/CN2023/077011 2022-03-25 2023-02-18 Procédé, dispositif et système de saisie en l'air WO2023179264A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210302541.4A CN116841385A (zh) 2022-03-25 2022-03-25 隔空输入方法、设备以及系统
CN202210302541.4 2022-03-25

Publications (1)

Publication Number Publication Date
WO2023179264A1 true WO2023179264A1 (fr) 2023-09-28

Family

ID=88099884

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/077011 WO2023179264A1 (fr) 2022-03-25 2023-02-18 Procédé, dispositif et système de saisie en l'air

Country Status (2)

Country Link
CN (1) CN116841385A (fr)
WO (1) WO2023179264A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3678388B2 (ja) * 1996-10-11 2005-08-03 株式会社リコー ペン型入力装置及びペン型入力装置のパターン認識方法
US20120013578A1 (en) * 2010-07-15 2012-01-19 Avermedia Information, Inc. Pen-shaped pointing device and shift control method thereof
CN104007846A (zh) * 2014-05-22 2014-08-27 深圳市宇恒互动科技开发有限公司 一种三维图形的生成方法和电子白板系统
JP2017027472A (ja) * 2015-07-24 2017-02-02 株式会社リコー 座標入力システム、座標入力装置、座標入力方法およびプログラム
CN113052078A (zh) * 2021-03-25 2021-06-29 Oppo广东移动通信有限公司 空中书写轨迹识别方法、装置、存储介质及电子设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3678388B2 (ja) * 1996-10-11 2005-08-03 株式会社リコー ペン型入力装置及びペン型入力装置のパターン認識方法
US20120013578A1 (en) * 2010-07-15 2012-01-19 Avermedia Information, Inc. Pen-shaped pointing device and shift control method thereof
CN104007846A (zh) * 2014-05-22 2014-08-27 深圳市宇恒互动科技开发有限公司 一种三维图形的生成方法和电子白板系统
JP2017027472A (ja) * 2015-07-24 2017-02-02 株式会社リコー 座標入力システム、座標入力装置、座標入力方法およびプログラム
CN113052078A (zh) * 2021-03-25 2021-06-29 Oppo广东移动通信有限公司 空中书写轨迹识别方法、装置、存储介质及电子设备

Also Published As

Publication number Publication date
CN116841385A (zh) 2023-10-03

Similar Documents

Publication Publication Date Title
US11321929B2 (en) System and method for spatially registering multiple augmented reality devices
US11625841B2 (en) Localization and tracking method and platform, head-mounted display system, and computer-readable storage medium
US20230037922A1 (en) Image display method and apparatus, computer device, and storage medium
CN111026314B (zh) 控制显示设备的方法及便携设备
TW201820077A (zh) 移動設備及求取移動設備姿態之方法
WO2021004412A1 (fr) Dispositif d'entrée portatif, et procédé et appareil de commande de la position d'affichage d'une icône d'indication de celui-ci
TWI453462B (zh) 智慧型電子裝置之虛擬望遠系統及其方法
US20200089311A1 (en) Tracking System and Tacking Method Using the Same
US20240144617A1 (en) Methods and systems for anchoring objects in augmented or virtual reality
US20210263592A1 (en) Hand and totem input fusion for wearable systems
CN114332423A (zh) 虚拟现实手柄追踪方法、终端及计算可读存储介质
CN107193380A (zh) 一种低成本高精度虚拟现实定位及交互系统
KR20200020295A (ko) 로봇과 인터랙션하는 증강현실 서비스 제공 장치 및 방법
CN113784767A (zh) 热电堆阵列融合跟踪
WO2023179264A1 (fr) Procédé, dispositif et système de saisie en l'air
US20230089061A1 (en) Space recognition system, space recognition method, information terminal, and server apparatus
WO2019119999A1 (fr) Procédé et appareil de présentation d'un processus d'expansion d'une figure solide, et dispositif et support d'informations
WO2021190421A1 (fr) Procédé de suivi de boule lumineuse de dispositif de commande fondé sur la réalité virtuelle et dispositif de réalité virtuelle
CN105739684A (zh) 具有手势校准机制的电子系统及其操作方法
US11200741B1 (en) Generating high fidelity spatial maps and pose evolutions
Vincze et al. What-You-See-Is-What-You-Get Indoor Localization for Physical Human-Robot Interaction Experiments
CN114373016A (zh) 一种在扩展现实技术场景中实现点的定位方法
KR20180106178A (ko) 무인 비행체, 전자 장치 및 그에 대한 제어 방법
JP7346977B2 (ja) 制御装置、電子機器、制御システム、制御方法、およびプログラム
CN103034345B (zh) 一种真实空间中的地理虚拟仿真三维鼠标笔

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23773515

Country of ref document: EP

Kind code of ref document: A1