WO2022000149A1 - 交互方法及电子设备 - Google Patents

交互方法及电子设备 Download PDF

Info

Publication number
WO2022000149A1
WO2022000149A1 PCT/CN2020/098643 CN2020098643W WO2022000149A1 WO 2022000149 A1 WO2022000149 A1 WO 2022000149A1 CN 2020098643 W CN2020098643 W CN 2020098643W WO 2022000149 A1 WO2022000149 A1 WO 2022000149A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinates
finger
camera
coordinate system
vehicle
Prior art date
Application number
PCT/CN2020/098643
Other languages
English (en)
French (fr)
Inventor
韩振华
黄为
何彦杉
徐文康
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080004859.3A priority Critical patent/CN112771480B/zh
Priority to PCT/CN2020/098643 priority patent/WO2022000149A1/zh
Priority to EP20942766.5A priority patent/EP4086731A4/en
Publication of WO2022000149A1 publication Critical patent/WO2022000149A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/113—Recognition of static hand signs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30268—Vehicle interior

Definitions

  • the embodiments of the present application relate to the field of automobiles, and in particular, to an interaction method and an electronic device.
  • the function of human-machine interaction is introduced in the car cockpit. That is, the driver or passenger in the car cockpit can interact with the devices in the car through gestures to achieve different functions. For example, when playing music, the driver or passenger can switch between songs through hand movements (eg, swiping left or right). For another example, when music or broadcasting is paused, the driver or passenger can start the playback function through a hand gesture (for example, an OK gesture or a V-shaped gesture, etc.).
  • a hand gesture for example, an OK gesture or a V-shaped gesture, etc.
  • the above hand movements or hand styles are all recognized by cameras in the car cockpit. At present, the hand movements or hand styles that can be recognized by the cameras in the car cockpit are limited. Therefore, the driver or passenger interacts with the device in the car through gestures, and realizes fewer and single functions.
  • the embodiments of the present application provide an interaction method and an electronic device, which can identify the direction pointed by the finger, so that the user can interact with the device according to the direction pointed by the finger, and realize rich functions.
  • an embodiment of the present application provides an interaction method, which is applied to a smart cockpit.
  • the method includes: acquiring a first spatial coordinate of a finger, where the first spatial coordinate is a coordinate in a three-dimensional coordinate system; according to the first spatial coordinate , determine the first direction pointed by the finger; according to the first direction, adjust the parameters of the target device, or display the information of the target object.
  • the first spatial coordinate of the finger can be obtained, the first direction pointed by the finger can be determined according to the first spatial coordinate, and the parameters of the target device or the information of the target object can be adjusted according to the first direction.
  • the finger in the smart cockpit, can be pointed in any direction, and different directions can correspond to different functions of the smart cockpit. Take the smart cockpit as an example of a car cockpit, the finger pointing up can be used to close the window, and the finger pointing down can be used to close the window. It is used to open the car window, the finger pointing forward can be used to open the sunroof, and the finger pointing backward can be used to close the sunroof.
  • the method provided by the embodiments of the present application can also display the information of the target object according to the direction pointed by the finger, which increases the way of human-computer interaction.
  • the method before acquiring the first spatial coordinates of the finger, the method further includes: acquiring the second spatial coordinates of the finger, where the second spatial coordinates are coordinates in the three-dimensional coordinate system; The second space coordinate determines the second direction pointed by the finger; and the device in the second direction is determined as the target device.
  • the target device before acquiring the first spatial coordinate of the finger, the target device may be determined according to the second spatial coordinate of the finger, so as to adjust the parameters of the target device according to the first direction subsequently.
  • acquiring the first spatial coordinates of the finger includes: within a preset time period, detecting that the direction pointed by the finger is in the first In the case of being within a range, the first spatial coordinate of the finger is obtained; wherein, the first range is the range in which the angle with the initial direction of the finger is less than or equal to the first threshold, and the initial direction is within the preset time period At the initial moment, the detected direction of the finger. Based on the above method, the first spatial coordinate of the finger can be obtained when it is detected that the direction pointed by the finger has not changed within a preset time period, so as to avoid triggering the acquisition of the first spatial coordinate of the finger due to misoperation.
  • the acquiring the first spatial coordinates of the finger includes: acquiring the image coordinates of the finger in the image captured by the camera, the The image coordinates are the coordinates in the two-dimensional coordinate system; the first spatial coordinates are obtained according to the image coordinates of the finger, the internal parameter matrix of the camera and the physical transformation matrix, and the internal parameter matrix of the camera and the physical transformation matrix are used to indicate the two.
  • the transformation relationship between the three-dimensional coordinate system and the three-dimensional coordinate system includes: acquiring the image coordinates of the finger in the image captured by the camera, the The image coordinates are the coordinates in the two-dimensional coordinate system; the first spatial coordinates are obtained according to the image coordinates of the finger, the internal parameter matrix of the camera and the physical transformation matrix, and the internal parameter matrix of the camera and the physical transformation matrix are used to indicate the two.
  • the transformation relationship between the three-dimensional coordinate system and the three-dimensional coordinate system includes: acquiring the image coordinates of the finger in the image captured by the camera, the The image coordinates are the coordinates in the two-dimensional
  • the image coordinates of the finger in the image captured by the camera can be obtained, and the first spatial coordinates can be obtained according to the image coordinates of the finger and the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system.
  • the first direction can be quickly obtained according to the first spatial coordinate.
  • the method further includes: acquiring image coordinates of multiple estimated points in the image captured by the camera and the coordinates of the multiple estimated points Spatial coordinates, the image coordinates of the multiple estimated points are the coordinates in the two-dimensional coordinate system, and the spatial coordinates of the multiple estimated points are the coordinates in the three-dimensional coordinate system; according to the camera's internal parameter matrix, the multiple estimated points The physical transformation matrix is obtained by the image coordinates of and the spatial coordinates of the plurality of estimated points.
  • the image coordinates of the multiple estimated points and the spatial coordinates of the multiple estimated points can be obtained, and the physical transformation matrix can be obtained according to the internal parameter matrix, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points.
  • the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system can be known, and the image coordinates of the finger in the image captured by the camera can be converted into the first spatial coordinates according to the transformation relationship, so that the direction pointed by the finger can be determined.
  • the physical transformation matrix is preset. Based on the above method, the physical transformation matrix may be preset in the smart cockpit, so that before acquiring the first spatial coordinates, the physical transformation matrix does not need to be calculated, which reduces the calculation amount of the equipment in the car cockpit.
  • the camera is a monocular camera, a binocular camera, or a depth camera. Based on the above method, images can be captured by various types of cameras, eg, a monocular camera, a binocular camera, or a depth camera.
  • the target object is an object in an interaction area; the interaction area is an area covered by the first direction. Based on the above method, the information of the objects in the area covered by the first direction can be displayed according to the first direction, so that the user can know the information of the target object and the user experience is improved.
  • the two-dimensional coordinate system is the coordinate system corresponding to the camera;
  • the three-dimensional coordinate system is the coordinate corresponding to the three-dimensional model of the smart cockpit system, or the coordinate system corresponding to the three-dimensional drawing of the smart cockpit.
  • the smart cockpit is a cockpit in a vehicle
  • adjusting the parameters of the target device includes: according to the first direction In one direction, the parameters of the window in the vehicle are adjusted; or, according to the first direction, the parameters of the sunroof in the vehicle are adjusted; or, according to the first direction, the parameters of the air conditioner in the vehicle are adjusted.
  • the windows, sunroofs and air conditioners in the vehicle can be adjusted according to the direction pointed by the finger, which can save hardware costs compared to adjusting the above devices by means of keys.
  • an embodiment of the present application provides an electronic device, which can implement the method in the first aspect or any possible implementation manner of the first aspect.
  • the electronic device comprises corresponding units or components for carrying out the above method.
  • the units included in the electronic device may be implemented by means of software and/or hardware.
  • an embodiment of the present application provides an electronic device, including: a processor, the processor is coupled to a memory, and the memory is used to store a program or an instruction, when the program or the instruction is executed by the processor , so that the electronic device implements the first aspect or the method described in any possible implementation manner of the first aspect.
  • an embodiment of the present application provides an electronic device, where the electronic device is configured to implement the method described in the first aspect or any possible implementation manner of the first aspect.
  • an embodiment of the present application provides a computer-readable medium on which a computer program or instruction is stored, and when the computer program or instruction is executed, enables a computer to execute the first aspect or any possibility of the first aspect.
  • an embodiment of the present application provides a computer program product, which includes computer program code, and when the computer program code is run on a computer, causes the computer to execute the first aspect or any possible implementation of the first aspect. method described in the method.
  • an embodiment of the present application provides a chip, including: a processor, where the processor is coupled to a memory, and the memory is used to store a program or an instruction, and when the program or instruction is executed by the processor, The chip is made to implement the method described in the first aspect or any possible implementation manner of the first aspect.
  • an embodiment of the present application provides a vehicle, including: the electronic device described in the second aspect; or the electronic device described in the third aspect; or the electronic device described in the fourth aspect; The chip described in the seventh aspect.
  • any of the electronic devices, chips, computer-readable media, computer program products or vehicles provided above are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to. The beneficial effects in the corresponding method will not be repeated here.
  • FIG. 1 is a schematic diagram of a transformation relationship between an image coordinate system and a space coordinate system provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of an interaction system provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 4 is a schematic flowchart 1 of an interaction method provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an image captured by a camera according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram 1 of a direction pointed by a finger according to an embodiment of the present application.
  • FIG. 7 is a schematic diagram 2 of a direction pointed by a finger according to an embodiment of the present application.
  • FIG. 8 is a schematic diagram 3 of a direction pointed by a finger according to an embodiment of the present application.
  • FIG. 9 is a second schematic flowchart of an interaction method provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of an estimated point provided by an embodiment of the present application.
  • FIG. 11 is a third schematic flowchart of an interaction method provided by an embodiment of the present application.
  • FIG. 12 is a fourth schematic flowchart of an interaction method provided by an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 14 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • the image coordinate system is the coordinate system established on the image captured by the camera.
  • the image coordinate system may take the center of the image captured by the camera as the origin, and the image coordinate system is a two-dimensional coordinate system.
  • the space coordinate system is the coordinate system corresponding to the three-dimensional drawing of the smart cockpit, or the coordinate system corresponding to the three-dimensional model of the smart cockpit.
  • the space coordinate system is the coordinate system of the computer aided design (CAD) drawing of the intelligent cockpit.
  • CAD computer aided design
  • the space coordinate system is a three-dimensional coordinate system, and the space coordinate system may also be called the coordinate system or the three-dimensional coordinate system of the three-dimensional model of the smart cockpit.
  • the information about the 3D drawings or the information about the 3D models can be pre-stored in the device in the smart cockpit; or, the information about the 3D drawings or the 3D models can be downloaded from the network.
  • the information of the three-dimensional drawing is used to indicate the coordinates of the device or equipment in the smart cockpit in the coordinate system corresponding to the three-dimensional drawing.
  • the information of the three-dimensional model of the smart cockpit is used to indicate the coordinates of the device or device in the smart cockpit in the coordinate system corresponding to the three-dimensional model.
  • the transformation relationship between the image coordinate system and the space coordinate system can be shown in Figure 1.
  • the two-dimensional coordinate system o-xy is the image coordinate system of the image q captured by the camera.
  • the intersection of the x-axis and the y-axis is o, that is, o is the origin of the image coordinate system, and the coordinates of any point on the image plane q can be represented by image coordinates (x, y).
  • the three-dimensional coordinate system O-XYZ is a space coordinate system, and any plane in the space coordinate system can be called a space plane.
  • the space plane Q is any plane in the space coordinate system (that is, the space plane Q is any plane in O-XYZ), and the coordinates of any point on the space plane Q can be represented by space coordinates (X, Y, Z).
  • q represents the image coordinates of the point in the image plane
  • Q represents the spatial coordinates of the point in the space plane
  • s is the scale factor
  • H homography matrix
  • M is the internal parameter matrix of the camera
  • f x represents the product of the physical focal length of the lens in the camera and the size of each unit imaged in the x direction
  • f y is the product of the physical focal length of the lens in the camera and the size of each unit imaged in the y direction
  • c x represents the offset of the projection coordinate center relative to the optical axis in the x direction
  • c y represents the offset of the projected coordinate center relative to the optical axis in the y direction.
  • the size of each unit imaged can be one pixel in the image captured by the camera.
  • W [R, t] W is the extrinsic parameter matrix of the camera.
  • R is a 3*3 orthogonal identity matrix, also called a rotation matrix
  • t is a three-dimensional translation vector.
  • Q 1 is a rectangle on the space plane Q
  • the rectangle q 1 is a rectangle mapped by the rectangle Q 1 on the image plane q according to the transformation relationship between the image plane and the space plane.
  • the image coordinates of the four corners when the rectangular q 1 respectively (x 1, y 1), (x 2, y 2), (x 3, y 3) and (x 4, y 4), the rectangle Q 4 1 of
  • the spatial coordinates of the corners are (X 1 , Y 1 , Z 1 ), (X 2 , Y 2 , Z 2 ), (X 3 , Y 3 , Z 3 ) and (X 4 , Y 4 , Z 4 )
  • the internal parameter matrix of the camera is M, then according to the transformation relationship between the image plane and the space plane, the overdetermined equation with sH as the unknown can be obtained: And use the constrained least squares method or random sampling consistency method to solve the overdetermined equation, you can get sH, and then divide sH with the camera's internal
  • the method provided by the embodiment of the present application can be applied to any interactive system that needs to recognize the direction of a finger.
  • the interactive system may be an interactive system in a smart cockpit, or other interactive systems that need to recognize the direction of a finger, such as a smart home system.
  • the smart cockpit may be a smart cockpit of a vehicle, a smart cockpit of a train, or a smart cockpit of an airplane, or the like.
  • the methods provided in the embodiments of the present application are introduced by taking the intelligent cockpit as an example of a vehicle.
  • FIG. 2 is a schematic structural diagram of an interaction system provided by an embodiment of the present application.
  • the interactive system shown in FIG. 2 includes a vehicle-machine control device 201 and a visual perception device 202 .
  • the vehicle control device 201 and the visual perception device 202 are connected through wired or wireless connection (eg, Bluetooth communication).
  • the in-vehicle control device 201 can control or adjust the in-vehicle equipment.
  • the vehicle machine control device 201 can adjust the air outlet of the air conditioner in the vehicle.
  • the vehicle controller 201 may control an augmented reality (AR) head-up display (HUD) to display corresponding information.
  • AR augmented reality
  • HUD head-up display
  • the visual perception device 202 may be a conventional camera or a smart camera.
  • a conventional camera is a camera with video capabilities.
  • smart cameras also have image processing functions and artificial intelligence algorithm functions.
  • the visual perception device 202 may be deployed above the rear-view mirror, or elsewhere in the vehicle's smart cockpit, without limitation.
  • the visual perception device 202 is a monocular camera, a binocular camera or a depth camera. It can be understood that the visual perception device 202 can also be a trinocular camera or a multi-eye camera, which is not limited.
  • the visual perception device 202 is a traditional camera
  • the vehicle control device 201 can obtain the first spatial coordinate of the finger, determine the first direction pointed by the finger according to the first spatial coordinate, and adjust the adjustment according to the first direction.
  • the vehicle-machine control device 201 is stored or pre-configured with information of a three-dimensional drawing of the intelligent cockpit of the vehicle, or information of a three-dimensional model of the intelligent cockpit of the vehicle.
  • the vehicle-machine control device 201 can also download the intelligent cockpit of the vehicle from the network before using the coordinate system information corresponding to the three-dimensional drawing of the intelligent cockpit of the vehicle, or the coordinate system information corresponding to the three-dimensional model of the intelligent cockpit of the vehicle.
  • the visual perception device 202 is a smart camera, and the camera can acquire the first spatial coordinates of the finger, and send the information of the first spatial coordinates to the vehicle control device 201 .
  • the vehicle-machine control device 201 receives the information of the first space coordinate, and can determine the first direction pointed by the finger according to the information of the first space coordinate, and adjust the parameters of the target device according to the first direction; or, display the information of the target object . Specifically, reference may be made to the method shown in FIG. 11 or FIG. 12 below.
  • the vehicle-machine control device 201 and the visual perception device 202 are stored or pre-configured with information about the three-dimensional drawing of the smart cockpit of the vehicle, or information about the three-dimensional model of the smart cockpit of the vehicle.
  • the vehicle-machine control device 201 and the visual perception device 202 can also obtain information from the network before using the coordinate system information corresponding to the three-dimensional drawing of the intelligent cockpit of the vehicle, or the coordinate system information corresponding to the three-dimensional model of the intelligent cockpit of the vehicle. Download the coordinate system information corresponding to the 3D drawing of the smart cockpit of the vehicle, or the coordinate system information corresponding to the 3D model of the smart cockpit of the vehicle.
  • the interactive system shown in FIG. 2 is only used for example, and is not used to limit the technical solution of the present application. Those skilled in the art should understand that, in a specific implementation process, the interaction system may also include other devices.
  • each device in FIG. 2 in this embodiment of the present application may be a functional module in one device.
  • the functional module can be an element in a hardware device, such as a communication chip or a communication component in a terminal device or a network device, or a software functional module running on the hardware, or a platform (such as a cloud Virtualization functions instantiated on the platform).
  • FIG. 3 is a schematic diagram of a hardware structure of an electronic device applicable to an embodiment of the present application.
  • the electronic device 300 may include at least one processor 301 , communication lines 302 , memory 303 and at least one communication interface 304 .
  • the processor 301 can be a general-purpose central processing unit (central processing unit, CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more processors for controlling the execution of the programs of the present application. integrated circuit.
  • CPU central processing unit
  • ASIC application-specific integrated circuit
  • Communication line 302 may include a path, such as a bus, to transfer information between the components described above.
  • Communication interface 304 using any transceiver-like device, for communicating with other devices or communication networks, such as Ethernet interfaces, radio access network (RAN), wireless local area networks (wireless local area networks, WLAN), etc.
  • RAN radio access network
  • WLAN wireless local area networks
  • Memory 303 may be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of information and instructions It can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, CD-ROM storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of being executed by a computer Access any other medium without limitation.
  • the memory may exist independently and be connected to the processor through communication line 302 .
  • the memory can also be integrated with the processor.
  • the memory provided by the embodiments of the present application may generally be non-volatile.
  • the memory 303 is used for storing the computer-executed instructions involved in executing the solution of the present application, and the execution is controlled by the processor 301 .
  • the processor 301 is configured to execute the computer-executed instructions stored in the memory 303, so as to implement the method provided by the embodiments of the present application.
  • the computer-executed instructions in the embodiment of the present application may also be referred to as application code, which is not specifically limited in the embodiment of the present application.
  • the processor 301 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 3 .
  • the electronic device 300 may include multiple processors, such as the processor 301 and the processor 307 in FIG. 3 .
  • processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor.
  • a processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (eg, computer program instructions).
  • the electronic device 300 may further include an output device 305 and an input device 306 .
  • the output device 305 is in communication with the processor 301 and can display information in a variety of ways.
  • the output device 305 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector (projector) Wait.
  • Input device 306 is in communication with processor 301 and can receive user input in a variety of ways.
  • the input device 306 may be a mouse, a keyboard, a touch screen device, a sensor device, or the like.
  • the visual perception device is a smart camera, and the visual perception device is implemented by the electronic device 300, the CPU in the electronic device 300 can be replaced by a graphics processing unit (graphics processing unit, GPU).
  • graphics processing unit graphics processing unit
  • the interaction method provided by the embodiment of the present application will be described in detail below with reference to FIG. 1 , FIG. 2 , and FIG. 3 .
  • the devices in the following embodiments may have the components shown in FIG. 3 .
  • the visual perception device or the vehicle control device may execute some or all of the steps in the embodiments of the present application, these steps are only examples, and the embodiments of the present application may also execute other steps or various steps deformation. In addition, various steps may be performed in different orders presented in the embodiments of the present application, and it may not be necessary to perform all the steps in the embodiments of the present application.
  • the specific structure of the execution body of an interactive method is not particularly limited in the embodiment of the present application, as long as the program that records the code of the interactive method of the embodiment of the present application can be executed according to the present application. It is sufficient to communicate with an interaction method of the embodiment of the application.
  • the execution subject of an interaction method provided by the embodiment of the application may be a visual perception device, or a component applied to a visual perception device, such as a chip or a processor , which is not limited in this application.
  • the execution body of an interaction method provided in this embodiment of the present application may be a vehicle-machine control device, or a component applied in the vehicle-machine control device, such as a chip or a processor, which is not limited in this application.
  • the execution bodies of an interaction method are a visual perception device and a vehicle-machine control device respectively.
  • the interaction method provided by the embodiment of the present application is introduced by taking the visual perception device as a traditional camera as an example.
  • FIG. 4 is an interaction method provided by an embodiment of the present application.
  • the interaction method includes steps 401-403.
  • Step 401 the vehicle-machine control device acquires the first spatial coordinates of the finger.
  • the vehicle-machine control device may be the vehicle-machine control device 201 in FIG. 2 .
  • the first space coordinates are coordinates in a three-dimensional coordinate system.
  • the first space coordinate is a coordinate in a coordinate system corresponding to a three-dimensional drawing (such as a CAD drawing) of the smart cockpit of the vehicle, or a coordinate in a coordinate system corresponding to a three-dimensional model of the smart cockpit of the vehicle.
  • the vehicle is the vehicle where the vehicle-machine control device is located.
  • the first spatial coordinates include spatial coordinates of at least two points on the finger.
  • the vehicle-machine control device acquires the first spatial coordinates of the finger when the vehicle-machine control device detects that the direction pointed by the finger is within the first range within a preset time period.
  • the first range is a range in which the included angle with the initial direction of the finger is less than or equal to the first threshold.
  • the initial direction of the finger is the direction of the finger detected at the initial moment of the preset time period.
  • the first threshold may be set before leaving the factory, for example, an appropriate first threshold may be set according to big data statistics.
  • the first threshold value can also be set by the user after delivery.
  • the vehicle-machine control device does not trigger the operation of acquiring the first spatial coordinates of the finger. If at any time within 2 seconds, the included angle between the direction the finger points to and the initial direction detected by the vehicle control device are less than or equal to 10°, the vehicle control device obtains the first spatial coordinates of the finger.
  • the vehicle control device obtains the first space coordinates in the following manner: the vehicle control device obtains the image coordinates of the finger in the image captured by the camera; obtains the first space according to the image coordinates of the finger, the internal parameter matrix of the camera and the physical transformation matrix. coordinate.
  • the image coordinates of the finger are coordinates in a two-dimensional coordinate system.
  • the image coordinates of the finger are the coordinates of the finger in the coordinate system of the camera.
  • the coordinate system of the camera may also be referred to as the coordinate system established on the image captured by the camera.
  • the camera may be the visual perception device 202 in FIG. 2 .
  • the camera's internal parameter matrix and physical transformation matrix can be used to indicate the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system.
  • the vehicle control device obtains the image coordinates of the finger in the image captured by the camera, including: the vehicle control device receives the image captured by the camera from the camera; the vehicle control device obtains the image captured by the camera according to the image captured by the camera. The image coordinates of the finger.
  • the image captured by the camera may be as shown in (a) of FIG. 5 .
  • the image coordinates of the finger are the coordinates of the key pixel points of the finger in the image captured by the camera.
  • the key pixel points of the finger may include a plurality of pixel points that can reflect the direction pointed by the finger.
  • the key pixel points of the finger include at least two pixel points on the finger joints.
  • the key pixel points of the finger can be as shown in (b) of FIG. 5 .
  • the vehicle-machine control device determines the key pixel points of the finger in the image captured by the camera, and obtains the image coordinates of the finger. For example, the vehicle control device can determine the key pixel points of the finger in the image captured by the camera through methods such as deep learning.
  • the vehicle control device may first cut out the hand in the image captured by the camera, and then obtain the image coordinates of the finger according to the cut out hand.
  • the car-machine control device determines the hand frame in the image captured by the camera, and cuts the hand frame; the car-machine control device determines the key pixel points of the finger in the hand frame, and obtains the image coordinates of the finger.
  • the hand frame has the wrist and hand.
  • the hand frame determined by the vehicle control device may be shown in (a) of FIG. 5
  • the hand frame cut by the vehicle control device may be shown as (b) of FIG. 5 .
  • the vehicle control device can determine the position of the hand frame in the image captured by the camera through a target detector, such as SSD (single shot multibox detector) or version 3 (v3) of YOLO (you only look once), etc. .
  • a target detector such as SSD (single shot multibox detector) or version 3 (v3) of YOLO (you only look once), etc.
  • SSD single shot multibox detector
  • v3 version 3 of YOLO (you only look once)
  • the vehicle-machine control device can also determine the position of the hand frame through other target detectors, which is not limited.
  • the vehicle-machine control apparatus may determine the key pixel points of the finger based on the Openpose model. It can be understood that, in practical applications, the vehicle-machine control device can also determine the key pixel points of the finger through other models or software, which is not limited.
  • the vehicle-machine control device obtains the image coordinates of the finger in the image captured by the camera in the following situations:
  • the vehicle control device receives the image captured by the camera from the camera, and obtains the image coordinates of the finger in the image captured by the camera. In this case, there are hands in the image captured by the camera. That is, when the camera detects that there is a hand in the captured image, it sends the captured image to the vehicle control device.
  • the vehicle-machine control device receives the photographed image, and obtains the image coordinates of the finger in the photographed image.
  • Scenario 2 When the vehicle control device receives the image captured by the camera from the camera and detects that there is a hand in the image captured by the camera, it obtains the image coordinates of the finger in the image captured by the camera. In this case, there is not necessarily a hand in the image captured by the camera. That is to say, when the camera captures an image, the captured image is sent to the vehicle control device.
  • the internal parameter matrix of the camera can reflect the camera's own properties, for example, the internal parameters of the camera.
  • the camera's internal parameters may include the camera's field of vision (FOV), focal length, and distortion.
  • FOV field of vision
  • the internal parameter matrix of the camera can be obtained by Zhang Zhengyou's checkerboard grid method.
  • the physical transformation matrix is sW, that is, the physical transformation matrix is the product of the scale factor and the extrinsic parameter matrix of the camera.
  • the extrinsic parameter matrix of the camera can reflect the extrinsic properties of the camera, for example, the extrinsic parameters of the camera.
  • the external parameters of the camera include but are not limited to the position of the camera, the height of the camera, and the pitch angle of the camera.
  • the physical transformation matrix can be obtained by the following methods: the vehicle-machine control device obtains the internal parameter matrix of the camera; the vehicle-machine control device obtains the image coordinates of multiple estimated points in the image captured by the camera and the multiple estimated points The spatial coordinates of the point; the vehicle-machine control device obtains the physical transformation matrix according to the internal parameter matrix of the camera, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points. Specifically, reference may be made to the method shown in FIG. 9 below, which will not be repeated.
  • the process of obtaining the physical transformation matrix by the vehicle-machine control device is a preprocessing process, that is, the vehicle-machine control device obtains the physical transformation matrix, which is performed before the vehicle-machine control device executes the method shown in FIG. 4 . It can be understood that if the external parameters of the camera do not change, the vehicle-machine control device does not need to acquire the physical transformation matrix again.
  • the physical transformation matrix may also be preset in the vehicle-machine control device. That is to say, when the vehicle leaves the factory, the physical transformation matrix can be preset in the vehicle-machine control device. Subsequently, if the external parameters of the camera change, the vehicle-machine control device can re-acquire the physical transformation matrix through the method shown in FIG. 9 .
  • the vehicle-machine control device obtains the first space coordinates according to the image coordinates of the finger, the internal parameter matrix of the camera, and the physical transformation matrix, including: The matrix performs coordinate transformation to obtain the first space coordinate.
  • the first spatial coordinates include spatial coordinates of at least two points on the finger.
  • the image coordinates of the finger are in one-to-one correspondence with the first spatial coordinates.
  • the coordinates of any key pixel in the image coordinates of the finger, and the spatial coordinates corresponding to the coordinates of the key pixel satisfy the formula: Among them, (x, y) is the coordinate of any key pixel point in the image coordinates of the finger, (X, Y, Z) is the spatial coordinate corresponding to the coordinate of the key pixel point, sW physical transformation matrix, M is the camera the internal parameter matrix.
  • the first spatial coordinates include (X 1 , Y 1 , Z 1 ) and (X 2 , Y 2 , Z 2 ), where (X 1 , Y 1 , Z 1 ) satisfies (X 2 , Y 2 , Z 2 ) satisfies
  • Step 402 The vehicle-machine control device determines the first direction pointed by the finger according to the first spatial coordinate.
  • the vehicle-machine control device determines the first direction pointed by the finger according to the first spatial coordinate, including: the vehicle-machine control device fits a straight line according to the spatial coordinates of at least two points included in the first spatial coordinate, the direction of the straight line. for the first direction.
  • the direction of the straight line is that among the at least two points, the point closer to the wrist points to the point farther from the wrist.
  • a straight line fitted by the vehicle-machine control device according to the spatial coordinates of at least two points included in the first spatial coordinate may be as shown in FIG. 6 .
  • the first space coordinate includes the space coordinates of three points.
  • the vehicle control device fits a straight line according to the space coordinates of the three points.
  • the direction pointed by the finger is the distance from the point close to the wrist among the three points. wrist farther away.
  • the more points included in the first space coordinate the closer the straight line fitted by the vehicle control device is to the posture of the finger, and the more accurate the direction pointed by the finger determined by the vehicle control device. Since the first spatial coordinates correspond to the image coordinates of the finger one-to-one, the more the coordinates of the key pixel points of the finger in the image coordinates of the finger obtained by the vehicle-machine control device in step 401, the more accurate the direction the finger points.
  • Step 403 The vehicle-machine control device adjusts the parameters of the target device according to the first direction, or displays the information of the target object.
  • the target device may be determined in any of the following ways: according to the second direction pointed by the finger; or, the user selects the target device by pressing a key; or, the user selects the target device by voice interaction.
  • the second direction pointed by the finger may be obtained according to the second spatial coordinate of the finger used to determine the target device.
  • the vehicle control device obtains the second spatial coordinate of the finger; the vehicle control device determines the second direction pointed by the finger according to the second spatial coordinate; the vehicle control device determines the device in the second direction as the device in the second direction. target device.
  • the second space coordinate is a coordinate in a three-dimensional coordinate system.
  • the process of determining the second direction pointed by the finger according to the second spatial coordinates of the finger by the vehicle control device can be obtained by referring to the vehicle control device in the above steps 401 to 402 to obtain the first direction. process is not repeated.
  • the vehicle-machine control apparatus may determine the first device in the second direction as the target device. For example, the vehicle machine control device determines that the first device in the second direction is the car window, and uses the car window as the target device.
  • adjusting the parameters of the target device according to the first direction includes: adjusting the parameters of the window in the vehicle according to the first direction; or, adjusting the parameters of the sunroof in the vehicle according to the first direction; or, according to the first direction. In one direction, the parameters of the air conditioner in the vehicle are adjusted.
  • the direction of the finger shown in (a) of FIG. 7 can adjust the parameters of the vehicle window, for example, the parameter controlling the degree of opening and closing of the vehicle window can be adjusted smaller.
  • the direction of the finger shown in (b) of FIG. 7 can adjust the parameters of the sunroof of the vehicle, for example, the parameter that controls the degree of opening and closing of the sunroof can be adjusted smaller.
  • the direction of the finger shown in (c) of FIG. 7 can adjust the parameters of the air conditioner, for example, the parameter for controlling the air outlet direction of the air conditioner can be increased.
  • the target object is an object in the interaction area; or, the target object is the first object in the first direction.
  • the interaction area is the area covered by the first direction.
  • the interaction area is an area whose included angle with the first direction is less than or equal to the second threshold.
  • the interaction area may be shown as 803 in FIG. 8 .
  • the target object is all or part of the objects in the interaction area.
  • the target object is an object in the interaction area whose included angle with the first direction is less than or equal to the third threshold.
  • the second threshold is greater than the third threshold.
  • the above-mentioned second threshold or third threshold may be set before delivery, for example, an appropriate second or third threshold may be set according to big data statistics.
  • the second threshold or the third threshold may also be set by the user after delivery.
  • the second threshold may be the same as or different from the first threshold.
  • the third threshold may be the same as or different from the first threshold.
  • the target object is an object in the interaction area
  • the vehicle-machine control device controls the AR-HUD to display the relevant information of the object on the display interface (such as the windshield of the vehicle) according to the first direction.
  • the area 801 is the image displayed by the AR-HUD on the windshield, and the vehicle control device controls the AR-HUD to display the relevant information of the target object 802 on the windshield according to the first direction. , such as ** hotel and so on.
  • the target object is the first object in the first direction
  • the vehicle control device controls the AR-HUD to display the information of the target object on the display interface (such as the windshield of the vehicle) according to the first direction .
  • the vehicle control device controls the AR-HUD to display the interior temperature, interior humidity, and air outlet mode on the windshield.
  • the vehicle control device can obtain the first space coordinates, determine the first direction pointed by the finger according to the first space coordinates, and adjust the parameters of the target device or display the information of the target object according to the first direction.
  • the finger in the smart cockpit of the vehicle, can point in any direction, and different directions can correspond to different functions of the smart cockpit of the vehicle, so different functions can be realized according to the direction the finger points.
  • the method provided by the embodiments of the present application can also display the information of the target object according to the direction pointed by the finger, thereby increasing the way of human-computer interaction.
  • the physical transformation matrix can be obtained through steps 404-406:
  • Step 404 The vehicle-machine control device acquires the internal parameter matrix of the camera.
  • the vehicle-machine control device can obtain the internal parameter matrix of the camera through Zhang Zhengyou's checkerboard grid method.
  • Step 405 the vehicle-machine control device acquires the image coordinates of the multiple estimated points in the image captured by the camera and the spatial coordinates of the multiple estimated points.
  • the image coordinates of the plurality of estimated points are coordinates in a two-dimensional coordinate system.
  • the image coordinates of the multiple estimated points are the coordinates of the multiple estimated points in the image captured by the camera.
  • the spatial coordinates of the plurality of estimated points are coordinates in a three-dimensional coordinate system.
  • the spatial coordinates of the multiple estimated points are the coordinates of the multiple estimated points in the coordinate system corresponding to the three-dimensional drawing of the smart cockpit of the vehicle, or the coordinates in the coordinate system corresponding to the three-dimensional model of the smart cockpit of the vehicle.
  • the information on the three-dimensional drawing or the information on the three-dimensional model is stored in the vehicle-machine control device.
  • the plurality of estimated points may be easily identifiable points in the intelligent cabin of the vehicle, for example, corner points of components in the intelligent cabin of the vehicle, so that errors in calculating the physical transformation matrix can be reduced.
  • a plurality of estimation points may be shown in FIG. 10 .
  • the vehicle control device determines a plurality of estimated points; the vehicle control device acquires an image captured by a camera, and obtains the image coordinates of the multiple estimated points according to the image captured by the camera; The 3D drawings of multiple estimated points are obtained in space coordinates.
  • the embodiment of the present application does not limit the number of estimation points. However, those skilled in the art should understand that the greater the number of estimated points, the more accurate the calculated physical transformation matrix.
  • Step 406 The vehicle-machine control device obtains a physical transformation matrix according to the internal parameter matrix of the camera, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points.
  • the vehicle control device obtains a physical transformation matrix according to the internal parameter matrix of the camera, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points, including: the vehicle control device is based on the image coordinates of the multiple estimated points, The spatial coordinates of a plurality of estimated points and the internal parameter matrix of the camera are used to obtain an equation system with the physical transformation matrix as the unknown. By solving the equation system, the physical transformation matrix corresponding to the external parameters of the camera is obtained.
  • the number of multiple estimated points is 8, and the spatial coordinates of the 8 estimated points are respectively (X 1 , Y 1 , Z 1 ), (X 2 , Y 2 , Z 2 ), (X 3 , Y 3 , Z 3 ), (X 4 , Y 4 , Z 4 ), (X 5 , Y 5 , Z 5 ), (X 6 , Y 6 , Z 6 ), (X 7 , Y 7 , Z 7 ) and (X 8 , Y 8 , Z 8 ), the image coordinates of the 8 estimated points are (x 1 , y 1 ), (x 2 , y 2 ), (x 3 , y 3 ), (x 4 ) respectively , y 4 ), (x 5 , y 5 ), (x 6 , y 6 ), (x 7 , y 7 ) and (x 8 , y 8 , Z 8 ), the image coordinates of the 8 estimated points are (x 1 , y 1 ), (x 2
  • the vehicle-machine control device can acquire the internal parameter matrix of the camera, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points.
  • the spatial coordinates of the point get the physical transformation matrix.
  • the vehicle-machine control device can determine the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, and can subsequently convert the image coordinates of the finger in the image captured by the camera into the spatial coordinates of the finger according to the transformation relationship, so as to determine the position of the finger. pointing direction.
  • the visual perception device takes the visual perception device as an example of a traditional camera, and introduces the interaction method provided by the embodiment of the present application.
  • the visual perception device may also be a smart camera.
  • the interaction method provided by the embodiment of the present application is described below by taking the visual perception device as an intelligent camera as an example.
  • the interaction method includes steps 1101-1104.
  • Step 1101 The camera acquires the first spatial coordinates of the finger.
  • the camera may be the visual perception device 202 in FIG. 2 .
  • the camera may be the visual perception device 202 in FIG. 2 .
  • step 1101 reference may be made to the corresponding description in the foregoing step 401, which will not be repeated.
  • Step 1102 the camera sends the information of the first spatial coordinates to the vehicle control device.
  • the vehicle-machine control device may be the vehicle-machine control device 201 in FIG. 2 .
  • the information of the first space coordinate may be used to indicate the first space coordinate.
  • Step 1103 The vehicle-machine control device receives the information of the first spatial coordinate from the camera, and determines the first direction pointed by the finger according to the first spatial coordinate.
  • step 402 For the process of determining the first direction pointed by the finger according to the first spatial coordinate, the corresponding description in the above step 402 may be referred to, and the details will not be repeated.
  • Step 1104 The vehicle-machine control device adjusts the parameters of the target device according to the first direction, or displays the information of the target object.
  • step 1104 For the introduction of step 1104, reference may be made to the above-mentioned step 403, which will not be repeated.
  • the camera can acquire the first spatial coordinates, and send the information of the first spatial coordinates to the vehicle-machine control device.
  • the vehicle control device can determine the first direction pointed by the finger according to the first space coordinate, and adjust the parameters of the target device or display the information of the target object according to the first direction.
  • the finger in the smart cockpit of the vehicle, can point in any direction, and different directions can correspond to different functions of the smart cockpit of the vehicle, so different functions can be realized according to the direction the finger points.
  • the method provided by the embodiments of the present application can also display the information of the target object according to the direction pointed by the finger, thereby increasing the way of human-computer interaction.
  • the physical transformation matrix can be obtained through steps 1105-1107:
  • Step 1105 The camera acquires the camera's internal parameter matrix.
  • Step 1106 The camera acquires the image coordinates of multiple estimated points in the image captured by the camera and the spatial coordinates of the multiple estimated points.
  • Step 1107 The camera obtains a physical transformation matrix according to the camera's internal parameter matrix, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points.
  • steps 1105 to 1107 For the introduction of steps 1105 to 1107, reference may be made to the corresponding introductions in the foregoing steps 404 to 406, which will not be repeated.
  • the camera can obtain the camera's internal parameter matrix, the image coordinates of multiple estimated points, and the spatial coordinates of multiple estimated points.
  • the coordinates get the physical transformation matrix.
  • the camera can determine the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, and subsequently, according to the transformation relationship, the image coordinates of the finger in the image captured by the camera can be converted into first space coordinates, so that the first space coordinates can be converted into
  • the information is sent to the vehicle control device so that the vehicle control device can determine the direction the finger is pointing.
  • the above-mentioned vehicle-machine control device or camera, etc. includes corresponding hardware structures and/or software modules for executing each function.
  • the unit and algorithm operations of each example described in conjunction with the embodiments disclosed herein can be implemented in hardware or in the form of a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.
  • the vehicle control device or the camera can be divided into functional modules according to the above method examples.
  • each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. It should be noted that, the division of modules in the embodiments of the present application is schematic, and is only a logical function division, and there may be other division manners in actual implementation.
  • FIG. 13 shows a schematic structural diagram of an electronic device.
  • the electronic device may be a chip or a system-on-chip in a vehicle control device or a camera, and the electronic device may be used to execute the functions of the vehicle control device or the camera involved in the above embodiments.
  • the electronic device shown in FIG. 13 includes: an acquisition module 1301 and a determination module 1302 .
  • the obtaining module 1301 is configured to obtain first spatial coordinates of the finger, where the first spatial coordinates are coordinates in a three-dimensional coordinate system.
  • the determining module 1302 is configured to determine the first direction pointed by the finger according to the first spatial coordinate.
  • the determining module 1302 is further configured to adjust the parameters of the target device according to the first direction, or display the information of the target object.
  • the obtaining module 1301 is further configured to obtain the second spatial coordinates of the finger, and the second spatial coordinates are the coordinates in the three-dimensional coordinate system; the determining module 1302 is further configured to determine the second spatial coordinates according to the second spatial coordinates. The second direction pointed by the finger; the determining module 1302 is further configured to determine the device in the second direction as the target device.
  • the obtaining module 1301 is specifically configured to obtain the first spatial coordinates of the finger when it is detected that the direction pointed by the finger is within a first range within a preset time period; wherein, the first range is The included angle with the initial direction of the finger is less than or equal to the range of the first threshold, and the initial direction is the direction of the finger detected at the initial moment of the preset time period.
  • the acquisition module 1301 is specifically used to acquire the image coordinates of the finger in the image captured by the camera, and the image coordinates of the finger are coordinates in a two-dimensional coordinate system; the acquisition module 1301 is also specifically used to obtain the image coordinates of the finger according to the image.
  • the coordinates, the camera's internal parameter matrix and the physical transformation matrix obtain the first spatial coordinate, and the camera's internal parameter matrix and the physical transformation matrix are used to indicate the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system.
  • the acquisition module 1301 is further configured to acquire the image coordinates of multiple estimated points in the image captured by the camera and the spatial coordinates of the multiple estimated points, where the image coordinates of the multiple estimated points are in the two-dimensional coordinate system.
  • the coordinates of the multiple estimated points are the coordinates in the three-dimensional coordinate system;
  • the determining module 1302 is further configured to determine the coordinates of the multiple estimated points according to the internal parameter matrix of the camera, the image coordinates of the multiple estimated points, and the spatial coordinates of the multiple estimated points The coordinates get this physical transformation matrix.
  • the physical transformation matrix is preset.
  • the camera is a monocular camera, a binocular camera or a depth camera.
  • the target object is an object in the interaction area; the interaction area is an area covered by the first direction.
  • the two-dimensional coordinate system is the coordinate system corresponding to the camera;
  • the three-dimensional coordinate system is the coordinate system corresponding to the three-dimensional model of the smart cockpit, or the coordinate system corresponding to the three-dimensional drawing of the smart cockpit.
  • the determining module 1302 is specifically configured to adjust the parameters of the window in the vehicle according to the first direction; or, the determining module 1302 is specifically configured to adjust the parameters of the sunroof in the vehicle according to the first direction; or , the determining module 1302 is specifically configured to adjust the parameters of the air conditioner in the vehicle according to the first direction.
  • the electronic device is presented in the form of dividing each functional module in an integrated manner.
  • Module herein may refer to a specific ASIC, circuit, processor and memory executing one or more software or firmware programs, integrated logic circuit, and/or other device that may provide the functions described above.
  • the electronic device can take the form shown in FIG. 3 .
  • the processor 301 in FIG. 3 may invoke the computer-executed instructions stored in the memory 303 to cause the electronic device to execute the interaction method in the foregoing method embodiment.
  • the function/implementation process of the acquiring module 1301 and the determining module 1302 in FIG. 13 may be implemented by the processor 301 in FIG. 3 calling the computer-executed instructions stored in the memory 303 .
  • FIG. 14 is a schematic structural diagram of a chip according to an embodiment of the present application.
  • Chip 140 includes one or more processors 1401 and interface circuits 1402 .
  • the chip 140 may further include a bus 1403 . in:
  • the processor 1401 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method may be completed by an integrated logic circuit of hardware in the processor 1401 or an instruction in the form of software.
  • the above-mentioned processor 1401 may be a general purpose processor, a digital communicator (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components .
  • DSP digital communicator
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the interface circuit 1402 is used for sending or receiving data, instructions or information.
  • the processor 1401 can use the data, instructions or other information received by the interface circuit 1402 to perform processing, and can send the processing completion information through the interface circuit 1402 .
  • the chip 140 further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor.
  • a portion of the memory may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory stores executable software modules or data structures
  • the processor 1401 may execute corresponding operations by calling operation instructions stored in the memory (the operation instructions may be stored in the operating system).
  • the chip 140 may be used in the electronic device involved in the embodiments of the present application.
  • the interface circuit 1402 can be used to output the execution result of the processor 1401 .
  • processor 1401 and the interface circuit 1402 can be implemented by hardware design, software design, or a combination of software and hardware, which is not limited here.
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be Incorporation may either be integrated into another device, or some features may be omitted, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or may be distributed to multiple different places . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium.
  • the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , including several instructions to make a device (may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes: a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk and other mediums that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本申请实施例涉及一种交互方法及电子设备，涉及汽车领域，可以识别手指所指的方向，使得用户可以根据手指所指的方向与设备交互，实现丰富的功能。该方法包括：获取手指的第一空间坐标，其中，第一空间坐标为三维坐标系中的坐标；根据第一空间坐标，确定手指所指的第一方向；根据第一方向，调节目标设备的参数，或者，显示目标物体的信息。

Description

交互方法及电子设备 技术领域
本申请实施例涉及汽车领域，尤其涉及交互方法及电子设备。
背景技术
随着现代经济技术的发展，汽车已经成为人们日常生活中最重要的交通工具。为了提高驾驶人或乘车人的车内体验，在汽车座舱中引入了人机交互的功能。也就是说，驾驶人或乘车人在汽车座舱中可以通过手势与汽车内的设备交互，以实现不同的功能。例如，在播放音乐时，驾驶人或乘车人能够通过手部动作(例如，向左滑动或向右滑动)实现歌曲的切换。又例如，在音乐或广播暂停播放时，驾驶人或乘车人能够通过手部样式(例如，OK手势或V字形手势等)开启播放功能。
上述手部动作或手部样式都是通过汽车座舱中的摄像机识别的。目前，汽车座舱中的摄像机能够识别的手部动作或手部样式是有限的，因此，驾驶人或乘车人通过手势与汽车内的设备交互，实现的功能较少也较单一。
发明内容
本申请实施例提供一种交互方法及电子设备，可以识别手指所指的方向，使得用户可以根据手指所指的方向与设备交互，实现丰富的功能。
为达到上述目的，本申请实施例采用如下技术方案：
第一方面，本申请实施例提供一种交互方法，应用于智能座舱，该方法包括：获取手指的第一空间坐标，该第一空间坐标为三维坐标系中的坐标；根据该第一空间坐标，确定该手指所指的第一方向；根据该第一方向，调节目标设备的参数，或者，显示目标物体的信息。
基于上述第一方面所述的方法，可以获取手指的第一空间坐标，根据第一空间坐标确定手指所指的第一方向，并根据第一方向调节目标设备的参数或者显示目标物体的信息。一方面，在智能座舱中，手指可以指向任一方向，不同方向可以对应智能座舱的不同功能，以智能座舱为汽车的座舱为例，手指向上指可以用于关闭车窗，手指向下指可以用于打开车窗，手指向前方指可以用于打开天窗，手指向后指可以用于关闭天窗。因此根据手指所指的方向可以实现不同的功能。另一方面，相比于根据手势对智能座舱内的设备进行调节，本申请实施例提供的方法还可以根据手指所指的方向显示目标物体的信息，增加了人机交互的方式。
结合第一方面，在一种可能的实现方式中，获取手指的第一空间坐标之前，该方法还包括：获取手指的第二空间坐标，第二空间坐标为该三维坐标系中的坐标；根据该第二空间坐标，确定该手指所指的第二方向；将该第二方向上的设备，确定为该目标设备。基于上述方法，在获取手指的第一空间坐标之前，可以先根据手指的第二空间坐标确定目标设备，以便后续根据第一方向调节目标设备的参数。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，获 取手指的第一空间坐标，包括：在预设时间段内，检测到该手指所指方向在第一范围内的情况下，获取该手指的第一空间坐标；其中，该第一范围为与手指的初始方向的夹角小于或等于第一阈值的范围，该初始方向为在该预设时间段的初始时刻，检测到的该手指的方向。基于上述方法，可以预设时间段内，检测到手指所指方向未发生变化的情况下，获取手指的第一空间坐标，以避免由于误操作触发获取手指的第一空间坐标。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，该获取手指的第一空间坐标，包括：获取摄像机拍摄的图像中该手指的图像坐标，该手指的图像坐标为该二维坐标系中的坐标；根据该手指的图像坐标、该摄像机的内参矩阵和物理变换矩阵得到该第一空间坐标，该摄像机的内参矩阵和该物理变换矩阵用于指示该二维坐标系与该三维坐标系之间的变换关系。基于上述方法，可以获取摄像机拍摄的图像中手指的图像坐标，根据手指的图像坐标，以及二维坐标系和三维坐标系之间的变换关系，得到第一空间坐标。如此，根据第一空间坐标可以快速得到第一方向。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，该方法还包括：获取该摄像机拍摄的图像中多个估计点的图像坐标和该多个估计点的空间坐标，该多个估计点的图像坐标为该二维坐标系中的坐标，该多个估计点的空间坐标为该三维坐标系中的坐标；根据该摄像机的内参矩阵、该多个估计点的图像坐标和该多个估计点的空间坐标得到该物理变换矩阵。基于上述方法，可以获取多个估计点的图像坐标和多个估计点的空间坐标，根据内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标得到物理变换矩阵。如此，可以知道二维坐标系与三维坐标系之间的变换关系，后续可以根据该变换关系将摄像机拍摄的图像中手指的图像坐标转换为第一空间坐标，从而可以确定手指所指的方向。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，物理变换矩阵为预设置的。基于上述方法，物理变换矩阵可以是预设置在智能座舱中的，这样在获取第一空间坐标之前，就不需要计算物理变换矩阵了，减少了汽车座舱中设备的计算量。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，该摄像机为单目摄像机、双目摄像机或深度摄像机。基于上述方法，可以通过多种类型的摄像机，例如，单目摄像机、双目摄像机或深度摄像机来拍摄图像。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，该目标物体为交互区域中的物体；该交互区域为该第一方向覆盖的区域。基于上述方法，可以根据第一方向，显示第一方向覆盖的区域中的物体的信息，使得用户了解到目标物体的信息，提高了用户体验。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，该二维坐标系为摄像机对应的坐标系；该三维坐标系为该智能座舱的三维模型对应的坐标系，或者该智能座舱的三维图纸对应的坐标系。基于上述方法，可以将摄像机对应的坐标系和智能座舱的三维模型对应的坐标系的关联起来，或者，将摄像机对应的坐标系和智能座舱的三维图纸对应的坐标系关联起来，如此可以根据手指的图像坐标得 到第一空间坐标。
结合第一方面以及第一方面各种可能的实现方式，在一种可能的实现方式中，该智能座舱为车辆中的座舱，根据该第一方向，调节目标设备的参数，包括：根据该第一方向，调节该车辆中车窗的参数；或者，根据该第一方向，调节该车辆中天窗的参数；或者，根据该第一方向，调节该车辆中空调的参数。基于上述方法，可以根据手指所指的方向调节车辆中的车窗、天窗和空调，相比于通过按键的方式调节上述设备，可以节省硬件成本。另外，在调节车辆中的设备时，不需要持续性动态调节，通过手指指出一个方向，就可以一步到位调节好，用户体验较好。例如，可以通过手指向上指，逐渐关闭车窗，到达想要关闭的程度时，移开手指即可停止关闭操作。
第二方面，本申请实施例提供一种电子设备，可以实现上述第一方面、或第一方面任一种可能的实施方式中的方法。该电子设备包括用于执行上述方法的相应的单元或部件。该电子设备包括的单元可以通过软件和/或硬件方式实现。
第三方面，本申请实施例提供一种电子设备，包括：处理器，所述处理器与存储器耦合，所述存储器用于存储程序或指令，当所述程序或指令被所述处理器执行时，使得该电子设备实现上述第一方面、或第一方面任一种可能的实施方式中所述的方法。
第四方面，本申请实施例提供一种电子设备，该电子设备用于实现上述第一方面、或第一方面任一种可能的实施方式中所述的方法。
第五方面，本申请实施例提供一种计算机可读介质，其上存储有计算机程序或指令，所述计算机程序或指令被执行时使得计算机执行上述第一方面、或第一方面任一种可能的实施方式中所述的方法。
第六方面，本申请实施例提供一种计算机程序产品，其包括计算机程序代码，所述计算机程序代码在计算机上运行时，使得计算机执行上述第一方面、或第一方面任一种可能的实施方式中所述的方法。
第七方面，本申请实施例提供一种芯片，包括：处理器，所述处理器与存储器耦合，所述存储器用于存储程序或指令，当所述程序或指令被所述处理器执行时，使得该芯片实现上述第一方面、或第一方面任一种可能的实施方式中所述的方法。
第八方面，本申请实施例提供一种车，包括：上述第二方面所述的电子设备；或者上述第三方面所述的电子设备；或者上述第四方面所述的电子设备；或者上述第七方面所述的芯片。
可以理解的，上述提供的任一种电子设备、芯片、计算机可读介质、计算机程序产品或车等均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考对应的方法中的有益效果，此处不再赘述。
附图说明
图1为本申请实施例提供的图像坐标系与空间坐标系间的变换关系的示意图；
图2为本申请实施例提供的交互系统的示意图；
图3为本申请实施例提供的电子设备的硬件结构示意图；
图4为本申请实施例提供的交互方法的流程示意图一；
图5为本申请实施例提供的摄像机拍摄的图像的示意图；
图6为本申请实施例提供的手指所指方向的示意图一；
图7为本申请实施例提供的手指所指方向的示意图二；
图8为本申请实施例提供的手指所指方向的示意图三；
图9为本申请实施例提供的交互方法的流程示意图二；
图10为本申请实施例提供的估计点的示意图；
图11为本申请实施例提供的交互方法的流程示意图三；
图12为本申请实施例提供的交互方法的流程示意图四；
图13为本申请实施例提供的电子设备的结构示意图；
图14为本申请实施例提供的芯片的结构示意图。
具体实施方式
为方便理解本申请实施例的方案，首先对图像坐标系和空间坐标系之间的变换关系进行介绍：
图像坐标系是在摄像机拍摄的图像上建立的坐标系。该图像坐标系可以以摄像机拍摄的图像的中心为原点，上述图像坐标系是二维坐标系。空间坐标系是智能座舱的三维图纸对应的坐标系，或者智能座舱的三维模型对应的坐标系。例如，空间坐标系为智能座舱的计算机辅助设计(computer aided design，CAD)图纸的坐标系。空间坐标系是三维坐标系，空间坐标系也可以称为智能座舱三维模型的坐标系或者三维坐标系。
进一步的，上述三维图纸的信息或三维模型的信息可以预存储在智能座舱中的设备中；或者，上述三维图纸的信息或三维模型的信息可以从网络中下载。其中，三维图纸的信息用于指示智能座舱中装置或设备在该三维图纸对应的坐标系中的坐标。智能座舱的三维模型的信息用于指示智能座舱中装置或设备在该三维模型对应的坐标系中的坐标。
图像坐标系与空间坐标系间的变换关系可以如图1所示。图1中，二维坐标系o-xy为摄像机拍摄的图像q的图像坐标系。其中，x轴和y轴的交点为o，即o为图像坐标系的原点，图像平面q上任一点的坐标可以用图像坐标(x，y)表示。三维坐标系O-XYZ为空间坐标系，空间坐标系中的任一平面可以称为空间平面。其中，空间平面Q为空间坐标系中的任一平面(即空间平面Q为O-XYZ中的任一平面)，空间平面Q上任一点的坐标可以用空间坐标(X，Y，Z)表示。其中，图像平面q与空间平面Q有如下变换关系：q＝sHQ。
其中，q表示图像平面中的点的图像坐标，Q表示空间平面中的点的空间坐标，s为尺度比例因子，单应矩阵H＝MW。
其中，
Figure PCTCN2020098643-appb-000001
M是摄像机的内参矩阵，f x表示摄像机中透镜物理焦距长度与成像的每个单元尺寸在x方向的乘积。f y表示摄像机中透镜物理焦距长度与成像的每个单元尺寸在y方向的乘积。c x表示投影坐标中心相对光轴在x方向的偏移，c y表示投影坐标中心相对光轴在y方向的偏移。成像的每个单元尺寸可以是摄像机拍摄的图像中的一个像素。
其中，W＝[R，t]，W是摄像机的外参矩阵。R为3*3的正交单位矩阵，也称为旋 转矩阵，t为三维的平移向量。
在本申请实施例中，上述变换关系主要应用在以下两种示例情况：
情况1：若图像平面q与空间平面Q有上述变换关系，并且已知q上某一点的图像坐标，尺度比例因子以及单应矩阵H，可以根据上述图像平面与空间平面的变换关系计算该点在Q上的空间坐标。
例如，图1中，平面q上某一点的图像坐标为(x 1，y 1)，若摄像机的内参矩阵为M，摄像机的外参矩阵为W，尺度比例因子为s，则根据图像平面与空间平面的变换关系，可以得到以X 1，Y 1，Z 1为未知数的方程：
Figure PCTCN2020098643-appb-000002
其中，H＝MW，对该方程求解，可以得到该点在空间平面Q上的坐标(X 1，Y 1，Z 1)。
情况2：若图像平面q与空间平面Q有上述变换关系，并且已知q中多个点的图像坐标、该多个点在空间平面Q中的空间坐标以及摄像机的内参矩阵，可以根据上述图像平面与空间平面的变换关系计算出尺度比例因子与摄像机的外参矩阵的乘积。
例如，图1中，Q 1为空间平面Q上的一个矩形，矩形q 1为矩形Q 1根据上述图像平面与空间平面的变换关系，在图像平面q上映射的矩形。若矩形q 1的4个角的图像坐标分别为(x 1，y 1)，(x 2，y 2)，(x 3，y 3)以及(x 4，y 4)，矩形Q 1的4个角的空间坐标分别为(X 1，Y 1，Z 1)，(X 2，Y 2，Z 2)，(X 3，Y 3，Z 3)以及(X 4，Y 4，Z 4)，摄像机的内参矩阵为M，则根据图像平面与空间平面的变换关系可以得到以sH为未知数的超定方程：
Figure PCTCN2020098643-appb-000003
并利用约束最小二乘法或随机抽样一致性等方法对该超定方程求解，可以得到sH，再将sH与摄像机的内参矩阵M做除法，得到尺度比例因子与摄像机的外参矩阵的乘积。
下面基于上述图像坐标系和空间坐标系之间的变换关系，结合附图对本申请实施例的实施方式进行详细描述。
本申请实施例提供的方法可以应用在任一需要识别手指方向的交互系统中。例如，该交互系统可以为智能座舱中的交互系统，或者其他需要识别手指方向的交互系统，例如智能家居系统等。其中，该智能座舱可以为车辆的智能座舱、火车的智能座舱或飞机的智能座舱等。本申请实施例提供的方法是以智能座舱为车辆的智能座舱为例介绍的，本申请实施例提供的方法应用在其他类型的交互系统的情况可以参考本申请实施例中的介绍，不予赘述。
下面仅以图2所示的交互系统为例，对本申请实施例提供的方法进行描述。
请参考图2，图2为本申请实施例提供的交互系统的架构示意图。图2所示的交互系统包括车机控制装置201和视觉感知装置202。车机控制装置201和视觉感知装置202通过有线或者无线连接(如蓝牙通信)。
其中，车机控制装置201可以控制或调节车内设备。例如，车机控制装置201可以调节车内空调的出风口。又例如，车机控制装置201可以控制增强现实(augmented reality，AR)平视显示器(head up display，HUD)显示相应的信息。
视觉感知装置202可以是传统摄像机或者智能摄像机。传统摄像机是具备摄像能力的摄像机。智能摄像机除了具备摄像能力，还具备图像处理功能和人工智能算法功能。视觉感知装置202可以部署在后视镜上方，或者车辆的智能座舱中的其他位置，不予限制。
可选的，视觉感知装置202为单目摄像机、双目摄像机或深度摄像机。可以理解的视觉感知装置202还可以是三目摄像机或多目摄像机，不予限制。
在一些实施例中，视觉感知装置202是传统摄像机，车机控制装置201可以获取手指的第一空间坐标，根据第一空间坐标，确定手指所指的第一方向，并根据第一方向，调节目标设备的参数；或者，显示目标物体的信息。具体的，可以参考下述图4或图9所示方法。
可选的，在这种情况下，车机控制装置201中存储或预配置有车辆的智能座舱的三维图纸的信息，或者车辆的智能座舱的三维模型的信息。
可以理解的，车机控制装置201也可以在使用到车辆的智能座舱的三维图纸对应的坐标系信息，或者车辆的智能座舱的三维模型对应的坐标系信息之前，从网络中下载车辆的智能座舱的三维图纸对应的坐标系信息，或者车辆的智能座舱的三维模型对应的坐标系信息。
在一些实施例中，视觉感知装置202是智能摄像机，该摄像机可以获取手指的第一空间坐标，并向车机控制装置201发送第一空间坐标的信息。车机控制装置201接收到第一空间坐标的信息，可以根据第一空间坐标的信息确定手指所指的第一方向，并根据第一方向，调节目标设备的参数；或者，显示目标物体的信息。具体的，可以参考下述图11或图12所示的方法。
可选的，在这种情况下，车机控制装置201和视觉感知装置202中存储或预配置有车辆的智能座舱的三维图纸的信息，或者车辆的智能座舱的三维模型的信息。
可以理解的，车机控制装置201和视觉感知装置202也可以在使用到车辆的智能座舱的三维图纸对应的坐标系信息，或者车辆的智能座舱的三维模型对应的坐标系信息之前，从网络中下载车辆的智能座舱的三维图纸对应的坐标系信息，或者车辆的智能座舱的三维模型对应的坐标系信息。
应注意，图2所示的交互系统仅用于举例，并非用于限制本申请的技术方案。本领域技术人员应当明白，在具体实现过程中，该交互系统还可以包括其他设备。
可选的，本申请实施例图2中的各装置，例如车机控制装置201或视觉感知装置202，可以是一个装置内的一个功能模块。可以理解的是，该功能模块既可以是硬件设备中的元件，例如终端设备或网络设备中的通信芯片或通信部件，也可以是在硬件上运行的软件功能模块，或者是平台(例如，云平台)上实例化的虚拟化功能。
例如，图2中的各装置均可以通过图3中的电子设备300来实现。图3所示为可适用于本申请实施例的电子设备的硬件结构示意图。该电子设备300可以包括至少一个处理器301，通信线路302，存储器303以及至少一个通信接口304。
处理器301可以是一个通用中央处理器(central processing unit，CPU)，微处理器，特定应用集成电路(application-specific integrated circuit，ASIC)，或一个或多个用于控制本申请方案程序执行的集成电路。
通信线路302可包括一通路，在上述组件之间传送信息，例如总线。
通信接口304，使用任何收发器一类的装置，用于与其他设备或通信网络通信，如以太网接口，无线接入网接口(radio access network，RAN)，无线局域网接口(wireless local area networks，WLAN)等。
存储器303可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(electrically erasable programmable read-only memory，EEPROM)、只读光盘(compact disc read-only memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过通信线路302与处理器相连接。存储器也可以和处理器集成在一起。本申请实施例提供的存储器通常可以具有非易失性。其中，存储器303用于存储执行本申请方案所涉及的计算机执行指令，并由处理器301来控制执行。处理器301用于执行存储器303中存储的计算机执行指令，从而实现本申请实施例提供的方法。
可选的，本申请实施例中的计算机执行指令也可以称之为应用程序代码，本申请实施例对此不作具体限定。
在具体实现中，作为一种实施例，处理器301可以包括一个或多个CPU，例如图3中的CPU0和CPU1。
在具体实现中，作为一种实施例，电子设备300可以包括多个处理器，例如图3中的处理器301和处理器307。这些处理器中的每一个可以是一个单核(single-CPU)处理器，也可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的处理核。
在具体实现中，作为一种实施例，电子设备300还可以包括输出设备305和输入设备306。输出设备305和处理器301通信，可以以多种方式来显示信息。例如，输出设备305可以是液晶显示器(liquid crystal display，LCD)，发光二级管(light emitting diode，LED)显示设备，阴极射线管(cathode ray tube，CRT)显示设备，或投影仪(projector)等。输入设备306和处理器301通信，可以以多种方式接收用户的输入。例如，输入设备306可以是鼠标、键盘、触摸屏设备或传感设备等。
需要说明的是，若视觉感知装置为智能摄像机，并且该视觉感知装置是通过电子设备300来实现的，则电子设备300中的CPU可以替换为图形处理器(graphics processing unit，GPU)。
下面结合图1、图2和图3对本申请实施例提供的交互方法进行具体阐述。其中，下述实施例中的装置可以具备图3所示部件。
需要说明的是，本申请下述实施例中，“示例性的”、“可选的”或者“例如”等词用于表示作例子、例证或说明。本申请下述实施例中被描述为“示例性的”、“可选的”或者“例如”的任何实施例或设计方案不应被解释为比其他实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”、“可选的”或者“例如”等词旨 在以具体方式呈现相关概念。
需要说明的是，在本申请的描述中，“第一”、或“第二”等词汇，仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。本申请中的“第一方向”等具有不同编号的方向，该编号仅为用于上下文行文方便，不同的次序编号本身不具有特定技术含义，比如，第一方向，第二方向等，可以理解为是一系列方向中的一个或者任一个。
可以理解的，本申请实施例中同一个步骤或者具有相同功能的步骤或者消息在不同实施例之间可以互相参考借鉴。
可以理解的，本申请实施例中，视觉感知装置或车机控制装置可以执行本申请实施例中的部分或全部步骤，这些步骤仅是示例，本申请实施例还可以执行其它步骤或者各种步骤的变形。此外，各个步骤可以按照本申请实施例呈现的不同的顺序来执行，并且有可能并非要执行本申请实施例中的全部步骤。
在本申请实施例中，一种交互方法的执行主体的具体结构，本申请实施例并未特别限定，只要可以通过运行记录有本申请实施例的一种交互方法的代码的程序，以根据本申请实施例的一种交互方法进行通信即可，例如，本申请实施例提供的一种交互方法的执行主体可以是视觉感知装置，或者为应用于视觉感知装置的部件，例如，芯片或处理器，本申请对此不进行限定。或者，本申请实施例提供的一种交互方法的执行主体可以是车机控制装置，或者为应用于车机控制装置中的部件，例如，芯片或处理器，本申请对此不进行限定。下述实施例以一种交互方法的执行主体分别为视觉感知装置、车机控制装置为例进行描述。
首先，以视觉感知装置为传统摄像机为例，介绍本申请实施例提供的交互方法。
请参考图4，为本申请实施例提供的一种交互方法。该交互方法包括步骤401-步骤403。
步骤401：车机控制装置获取手指的第一空间坐标。
其中，车机控制装置可以是图2中的车机控制装置201。第一空间坐标为三维坐标系中的坐标。例如，第一空间坐标为车辆的智能座舱的三维图纸(如CAD图纸)对应的坐标系中的坐标，或者车辆的智能座舱的三维模型对应的坐标系中的坐标。该车辆为车机控制装置所在的车辆。第一空间坐标包括手指上至少两个点的空间坐标。
可选的，为了避免误操作，车机控制装置在预设时间段内，车机控制装置检测到手指所指方向在第一范围内的情况下，获取手指的第一空间坐标。
其中，第一范围为与手指的初始方向的夹角小于或等于第一阈值的范围。手指的初始方向为在预设时间段的初始时刻，检测到的手指的方向。第一阈值可以在出厂前设置，例如，根据大数据统计来设置合适的第一阈值。该第一阈值也可以在出厂后由用户设置。
示例性的，以预设时间段为2秒，上述第一阈值为10°为例，若在2秒内的某一时刻，车机控制装置检测到的手指所指方向与初始方向的夹角为30°，则车机控制装置不触发获取手指的第一空间坐标的操作。若在2秒内的任一时刻，车机控制装置检测到的手指所指方向与初始方向的夹角都小于或等于10°，则车机控制装置获取手指的第一空间坐标。
可选的，车机控制装置通过如下方式获取第一空间坐标：车机控制装置获取摄像机拍摄的图像中手指的图像坐标；根据手指的图像坐标、摄像机的内参矩阵和物理变换矩阵得到第一空间坐标。
其中，手指的图像坐标为二维坐标系中的坐标。例如，手指的图像坐标为手指在摄像机的坐标系中的坐标。摄像机的坐标系也可以称为在摄像机拍摄的图像上建立的坐标系。摄像机可以是图2中的视觉感知装置202。摄像机的内参矩阵和物理变换矩阵可以用于指示二维坐标系与三维坐标系之间的变换关系。
首先，介绍一种车机控制装置获取摄像机拍摄的图像中手指的图像坐标的示例性过程：
一种可能的实现方式，车机控制装置获取摄像机拍摄的图像中手指的图像坐标，包括：车机控制装置接收来自摄像机的摄像机拍摄的图像；车机控制装置根据摄像机拍摄的图像，得到摄像机拍摄的图像中，手指的图像坐标。
示例性的，摄像机拍摄的图像可以如图5中的(a)所示。手指的图像坐标为摄像机拍摄的图像中手指的关键像素点的坐标。手指的关键像素点可以包括能够体现手指所指方向的多个像素点。示例性的，手指的关键像素点包括手指关节上至少两个像素点。例如，手指的关键像素点可以如图5中的(b)所示。
进一步的，车机控制装置确定摄像机拍摄的图像中手指的关键像素点，并获取手指的图像坐标。例如，车机控制装置可以通过深度学习等方法确定摄像机拍摄的图像中手指的关键像素点。
进一步的，为了减小车机控制装置的计算量，车机控制装置可以先将摄像机拍摄的图像中的手部裁剪出来，再根据裁剪出的手部获取手指的图像坐标。示例性的，车机控制装置在摄像机拍摄的图像中确定手部框，并将手部框裁剪下来；车机控制装置确定手部框中手指的关键像素点，并获取手指的图像坐标。
其中，手部框中有手腕和手。例如，车机控制装置确定的手部框可以如图5中的(a)所示，车机控制装置裁剪下来的手部框可以如图5中的(b)所示。
示例性的，车机控制装置可以通过目标检测器，例如SSD(single shot multibox detector)或YOLO(you only look once)的版本3(v3)等，在摄像机拍摄的图像中确定手部框的位置。可以理解的，上述SSD、YOLO v3仅是示例性的，车机控制装置还可以通过其他目标检测器确定手部框的位置，不予限制。
示例性的，车机控制装置可以基于Openpose模型确定手指的关键像素点。可以理解的，在实际应用中，车机控制装置还可以通过其他模型或软件确定手指的关键像素点，不予限制。
一种可能的实现方式，车机控制装置在以下情形下获取摄像机拍摄的图像中手指的图像坐标：
情形1：车机控制装置接收到来自摄像机的摄像机拍摄的图像，获取摄像机拍摄的图像中手指的图像坐标。在这种情形下，摄像机拍摄的图像中有手部。也就是说，摄像机在检测到拍摄的图像中有手部时，向车机控制装置发送拍摄的图像。车机控制设备装置接收到拍摄的图像，获取拍摄的图像中手指的图像坐标。
情形2：车机控制装置接收到来自摄像机的摄像机拍摄的图像，并检测到摄像机拍摄 的图像中有手部时，获取摄像机拍摄的图像中手指的图像坐标。在这种情形下，摄像机拍摄的图像中不一定有手部。也就是说，摄像机拍摄到图像就向车机控制装置发送拍摄的图像。
下面介绍根据手指的图像坐标、摄像机的内参矩阵和物理变换矩阵得到第一空间坐标的具体过程：
其中，摄像机的内参矩阵可以反映摄像机的自身属性，例如，摄像机的内参。摄像机的内参可以包括摄像机的视场角(field of vision，FOV)、焦距和畸变量等。示例性的，摄像机的内参矩阵可以通过张正友的棋盘网格法得到。
其中，物理变换矩阵为sW，即物理变换矩阵为尺度比例因子与摄像机的外参矩阵的乘积。摄像机的外参矩阵可以反映摄像机的外部属性，例如，摄像机的外参。摄像机的外参包括但不限于摄像机的位置，摄像机高度，以及摄像机的俯仰角等。
在一种可能的实现方式中，物理变换矩阵可以通过如下方法得到：车机控制装置获取摄像机的内参矩阵；车机控制装置获取摄像机拍摄的图像中多个估计点的图像坐标和该多个估计点的空间坐标；车机控制装置根据摄像机的内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标，得到物理变换矩阵。具体的，可以参考下述图9所示方法中所述，不予赘述。
需要说明的是，车机控制装置获取物理变换矩阵的过程为预处理过程，即车机控制装置获取物理变换矩阵，是在车机控制装置执行图4所示方法之前执行。可以理解的，若摄像机的外参没有变化的情况下，车机控制装置无需再次获取物理变换矩阵。
可以理解的，物理变换矩阵也可以是预设置在车机控制装置中的。也就是说，车辆在出厂时，可以将物理变换矩阵预设置在车机控制装置中。后续，若摄像机的外参发生变化，车机控制装置可以通过图9所示的方法重新获取物理变换矩阵。
一种可能的实现方式，车机控制装置根据手指的图像坐标、摄像机的内参矩阵和物理变换矩阵得到第一空间坐标，包括：车机控制装置将手指的图像坐标、物理变换矩阵和摄像机的内参矩阵进行坐标变换，得到第一空间坐标。
其中，第一空间坐标包括手指上至少两个点的空间坐标。手指的图像坐标与第一空间坐标一一对应。手指的图像坐标中的任一关键像素点的坐标，与该关键像素点的坐标对应的空间坐标满足公式：
Figure PCTCN2020098643-appb-000004
其中，(x，y)为手指的图像坐标中的任一关键像素点的坐标，(X，Y，Z)为与该关键像素点的坐标对应的空间坐标，sW物理变换矩阵，M为摄像机的内参矩阵。
示例性的，以手指的图像坐标包括(x 1，y 1)和(x 2，y 2)，物理变换矩阵为sW，摄像机的内参矩阵为M为例，第一空间坐标包括(X 1，Y 1，Z 1)和(X 2，Y 2，Z 2)，其中，(X 1，Y 1，Z 1)满足
Figure PCTCN2020098643-appb-000005
(X 2，Y 2，Z 2)满足
Figure PCTCN2020098643-appb-000006
步骤402：车机控制装置根据第一空间坐标确定手指所指的第一方向。
可选的，车机控制装置根据第一空间坐标确定手指所指的第一方向，包括：车机控制装置根据第一空间坐标包括的至少两个点的空间坐标拟合直线，该直线的方向为该第一方向。该直线的方向为该至少两个点中，距手腕近的点指向距手腕远的点。
示例性的，车机控制装置根据第一空间坐标包括的至少两个点的空间坐标拟合的直线可以如图6所示。图6中第一空间坐标包括3个点的空间坐标，车机控制装置根据该3个点的空间坐标拟合直线，手指所指的方向为该3个点中，距手腕近的点指向距手腕远的点。
可以理解的，第一空间坐标包括的点的个数越多，车机控制装置拟合的直线与手指的姿态越相近，车机控制装置确定的手指所指的方向越准确。又因为第一空间坐标与手指的图像坐标一一对应，所以步骤401中车机控制装置获取的手指的图像坐标中手指的关键像素点的坐标越多，手指所指的方向越准确。
步骤403：车机控制装置根据第一方向，调节目标设备的参数，或者，显示目标物体的信息。
首先，介绍车机控制装置根据第一方向调节目标设备的参数的示例性的过程：
可选的，目标设备可以通过如下任一种方式确定：根据手指所指的第二方向确定；或者，用户通过按键的方式选择目标设备；或者，用户通过语音交互的方式选择目标设备。
进一步的，手指所指的第二方向可以根据用于确定目标设备的手指的第二空间坐标得到。示例性的，车机控制装置获取手指的第二空间坐标；车机控制装置根据第二空间坐标，确定手指所指的第二方向；车机控制装置将该第二方向上的设备，确定为目标设备。其中，第二空间坐标为三维坐标系中的坐标。
可以理解的，车机控制装置获取手指的第二空间坐标，根据第二空间坐标，确定手指所指的第二方向的过程，可以参考上述步骤401-步骤402中车机控制装置得到第一方向的过程，不予赘述。
进一步的，车机控制装置可以将第二方向上第一个设备确定为目标设备。例如，车机控制装置确定第二方向上第一个设备为车窗，将车窗作为目标设备。
一种可能的实现方式，根据第一方向调节目标设备的参数，包括：根据第一方向，调节车辆中车窗的参数；或者，根据第一方向，调节车辆中天窗的参数；或者，根据第一方向，调节车辆中空调的参数。
示例性的，如图7所示，图7中的(a)中所示的手指的方向可以调节车窗的参数，例如，将控制车窗开关程度的参数调小。图7中的(b)中所示的手指的方向可以调节车辆天窗的参数，例如，将控制天窗开关程度的参数调小。图7中的(c)中所示的手指的方向可以调节空调的参数，例如，将控制空调出风方向的参数调大。
下面介绍车机控制装置根据第一方向显示目标物体的信息的具体过程：
其中，目标物体为交互区域中的物体；或者，目标物体为第一方向上第一个物体。交互区域为第一方向覆盖的区域。示例性的，交互区域为与第一方向夹角小于或等于第二阈值的区域。例如，交互区域可以如图8中的803所示。
可选的，目标物体为交互区域中的全部或部分物体。目标物体为交互区域中的部分物体时，该目标物体为交互区域中，与第一方向夹角小于或等于第三阈值的物体。 第二阈值大于第三阈值。
上述第二阈值或第三阈值可以在出厂前设置，例如，根据大数据统计来设置合适的第二阈值或第三阈值。第二阈值或第三阈值也可以在出厂后由用户设置。
可以理解的，第二阈值可以和第一阈值相同也可以不同。第三阈值可以和第一阈值相同也可以不同。
一种可能的实现方式，目标物体为交互区域中的物体，车机控制装置根据第一方向，控制AR-HUD在显示界面(例如车辆的挡风玻璃)上显示该物体的相关信息。
示例性的，如图8所示，区域801为AR-HUD在挡风玻璃上显示的图像，车机控制装置根据第一方向，控制AR-HUD在挡风玻璃上显示目标物体802的相关信息，如**酒店等等。
另一种可能的实现方式，目标物体为第一方向上第一个物体，车机控制装置根据第一方向，控制AR-HUD在显示界面(例如车辆的挡风玻璃)上显示目标物体的信息。例如，若第一方向上第一个物体为空调，车机控制装置控制AR-HUD在挡风玻璃上显示车内温度、车内湿度和出风模式等等。
基于图4所示的方法，车机控制装置可以获取第一空间坐标，根据第一空间坐标确定手指所指的第一方向，并根据第一方向调节目标设备的参数或者显示目标物体的信息。一方面，在车辆的智能座舱中，手指可以指向任一方向，不同方向可以对应车辆的智能座舱的不同功能，因此根据手指所指的方向可以实现不同的功能。另一方面，相比于根据手势对车辆的智能座舱内的设备进行调节，本申请实施例提供的方法还可以根据手指所指的方向显示目标物体的信息，增加了人机交互的方式。
进一步可选的，如图9所示，图4所示方法中，物理变换矩阵可以通过步骤404-步骤406得到：
步骤404：车机控制装置获取摄像机的内参矩阵。
在一种可能的实现方式中，车机控制装置可以通过张正友的棋盘网格法获取摄像机的内参矩阵。
步骤405：车机控制装置获取摄像机拍摄的图像中多个估计点的图像坐标和该多个估计点的空间坐标。
其中，多个估计点的图像坐标为二维坐标系中的坐标。例如，多个估计点的图像坐标为该多个估计点在摄像机拍摄的图像中的坐标。多个估计点的空间坐标为三维坐标系中的坐标。例如，该多个估计点的空间坐标为该多个估计点在车辆的智能座舱的三维图纸对应的坐标系中的坐标，或者，车辆的智能座舱的三维模型对应的坐标系中的坐标。在这种情况下，车机控制装置中存储有三维图纸的信息或三维模型的信息。
该多个估计点可以为车辆的智能座舱中易于识别的点，例如，车辆的智能座舱中部件的角点，如此可以减少计算物理变换矩阵的误差。示例性的，多个估计点可以如图10中所示。
在一种可能的实现方式中，车机控制装置确定多个估计点；车机控制装置获取摄像机拍摄的图像，并根据摄像机拍摄的图像获取多个估计点的图像坐标；车机控制装置根据车辆的三维图纸获取多个估计点的空间坐标。
需要说明的是，本申请实施例不限制估计点的个数。但是本领域技术人员应理解， 估计点的个数越多，计算出的物理变换矩阵越准确。
步骤406：车机控制装置根据摄像机的内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标，得到物理变换矩阵。
可选的，车机控制装置根据摄像机的内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标，得到物理变换矩阵，包括：车机控制装置根据多个估计点的图像坐标、多个估计点的空间坐标以及摄像机的内参矩阵，得到以物理变换矩阵为未知数的方程组，通过求解该方程组，得到摄像机的外参对应的该物理变换矩阵。
示例性的，以多个估计点的个数为8，该8个估计点的空间坐标分别为(X 1，Y 1，Z 1)、(X 2，Y 2，Z 2)、(X 3，Y 3，Z 3)、(X 4，Y 4，Z 4)、(X 5，Y 5，Z 5)、(X 6，Y 6，Z 6)、(X 7，Y 7，Z 7)以及(X 8，Y 8，Z 8)，该8个估计点的图像坐标分别为(x 1，y 1)、(x 2，y 2)、(x 3，y 3)、(x 4，y 4)、(x 5，y 5)、(x 6，y 6)、(x 7，y 7)以及(x 8，y 8)为例，则根据多个估计点的空间坐标、多个估计点的图像坐标以及摄像机的内参矩阵(下述方程组中的M)，可以得到如下以物理变换矩阵为未知数的方程组：
Figure PCTCN2020098643-appb-000007
利用约束最小二乘法或随机抽样一致性等方法可以对上述方程进行求解，得到sW，即，物理变换矩阵。
基于图9所示的方法，车机控制装置可以获取摄像机的内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标，根据内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标得到物理变换矩阵。如此，车机控制装置可以确定二维坐标系与三维坐标系之间的变换关系，后续可以根据该变换关系将摄像机拍摄的图像中手指的图像坐标转换为手指的空间坐标，从而可以确定手指所指的方向。
上述是以视觉感知装置为传统摄像机为例，介绍的本申请实施例提供的交互方法，除此之外，视觉感知装置还可以是智能摄像机。下面以视觉感知装置为智能摄像机为例，介绍的本申请实施例提供的交互方法。
如图11所示，为本申请实施例提供的又一种交互方法。该交互方法包括步骤1101-步骤1104。
步骤1101：摄像机获取手指的第一空间坐标。
其中，该摄像机可以是图2中的视觉感知装置202。步骤1101的介绍可以参考上述步骤401中对应的描述，不予赘述。
步骤1102：摄像机向车机控制装置发送第一空间坐标的信息。
其中，车机控制装置可以是图2中的车机控制装置201。第一空间坐标的信息可以用于指示第一空间坐标。
步骤1103：车机控制装置接收来自摄像机的第一空间坐标的信息，根据第一空间坐标确定手指所指的第一方向。
车机控制装置根据第一空间坐标确定手指所指的第一方向的过程，可以参考上述步骤402中对应的描述，不予赘述。
步骤1104：车机控制装置根据第一方向，调节目标设备的参数，或者，显示目标物体的信息。
步骤1104的介绍可以参考上述步骤403中所述，不予赘述。
基于图11所示的方法，摄像机可以获取第一空间坐标，并向车机控制装置发送第一空间坐标的信息。车机控制装置接收到该第一空间坐标的信息，可以根据第一空间坐标确定手指所指的第一方向，并根据第一方向调节目标设备的参数或者显示目标物体的信息。一方面，在车辆的智能座舱中，手指可以指向任一方向，不同方向可以对应车辆的智能座舱的不同功能，因此根据手指所指的方向可以实现不同的功能。另一方面，相比于根据手势对车辆的智能座舱内的设备进行调节，本申请实施例提供的方法还可以根据手指所指的方向显示目标物体的信息，增加了人机交互的方式。
进一步可选的，如图12所示，图11所示方法中，物理变换矩阵可以通过步骤1105-步骤1107得到：
步骤1105：摄像机获取摄像机的内参矩阵。
步骤1106：摄像机获取摄像机拍摄的图像中多个估计点的图像坐标和该多个估计点的空间坐标。
步骤1107：摄像机根据摄像机的内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标，得到物理变换矩阵。
步骤1105-步骤1107的介绍可以参考上述步骤404-步骤406中对应的介绍，不予赘述。
基于图12所示的方法，摄像机可以获取摄像机的内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标，根据内参矩阵、多个估计点的图像坐标和多个估计点的空间坐标得到物理变换矩阵。如此，摄像机可以确定二维坐标系与三维坐标系之间的变换关系，后续可以根据该变换关系将摄像机拍摄的图像中手指的图像坐标转换为第一空间坐标，从而可以将第一空间坐标的信息发送给车机控制装置，以便车机控制装置确定手指所指的方向。
上述主要从各个网元之间交互的角度对本申请实施例提供的方案进行了介绍。可以理解的是，上述车机控制装置或者摄像机等为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法操作，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例可以根据上述方法示例对车机控制装置或者摄像机进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本申请实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
比如，以采用集成的方式划分各个功能模块的情况下，图13示出了一种电子设备的结构示意图。该电子设备可以为车机控制装置或者摄像机中的芯片或者片上系统， 该电子设备可以用于执行上述实施例中涉及的车机控制装置或者摄像机的功能。
作为一种可能的实现方式，图13所示的电子设备包括：获取模块1301和确定模块1302。
获取模块1301，用于获取手指的第一空间坐标，该第一空间坐标为三维坐标系中的坐标。
确定模块1302，用于根据该第一空间坐标，确定该手指所指的第一方向。
确定模块1302，还用于根据该第一方向，调节目标设备的参数，或者，显示目标物体的信息。
可选的，获取模块1301，还用于获取该手指的第二空间坐标，该第二空间坐标为该三维坐标系中的坐标；确定模块1302，还用于根据该第二空间坐标，确定该手指所指的第二方向；确定模块1302，还用于将该第二方向上的设备，确定为该目标设备。
可选的，获取模块1301，具体用于在预设时间段内，检测到该手指所指方向在第一范围内的情况下，获取该手指的第一空间坐标；其中，该第一范围为与该手指的初始方向的夹角小于或等于第一阈值的范围，该初始方向为在该预设时间段的初始时刻，检测到的该手指的方向。
可选的，获取模块1301，具体用于获取摄像机拍摄的图像中该手指的图像坐标，该手指的图像坐标为二维坐标系中的坐标；获取模块1301，还具体用于根据该手指的图像坐标、该摄像机的内参矩阵和物理变换矩阵得到该第一空间坐标，该摄像机的内参矩阵和该物理变换矩阵用于指示该二维坐标系与该三维坐标系之间的变换关系。
可选的，获取模块1301，还用于获取该摄像机拍摄的图像中多个估计点的图像坐标和该多个估计点的空间坐标，该多个估计点的图像坐标为该二维坐标系中的坐标，该多个估计点的空间坐标为该三维坐标系中的坐标；确定模块1302，还用于根据该摄像机的内参矩阵、该多个估计点的图像坐标和该多个估计点的空间坐标得到该物理变换矩阵。
可选的，该物理变换矩阵为预设置的。
可选的，该摄像机为单目摄像机、双目摄像机或深度摄像机。
可选的，该目标物体为交互区域中的物体；该交互区域为该第一方向覆盖的区域。
可选的，该二维坐标系为摄像机对应的坐标系；该三维坐标系为该智能座舱的三维模型对应的坐标系，或者该智能座舱的三维图纸对应的坐标系。
可选的，确定模块1302，具体用于根据该第一方向，调节该车辆中车窗的参数；或者，确定模块1302，具体用于根据该第一方向，调节该车辆中天窗的参数；或者，确定模块1302，具体用于根据该第一方向，调节该车辆中空调的参数。
其中，上述方法实施例涉及的各操作的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赘述。
在本实施例中，该电子设备以采用集成的方式划分各个功能模块的形式来呈现。这里的“模块”可以指特定ASIC，电路，执行一个或多个软件或固件程序的处理器和存储器，集成逻辑电路，和/或其他可以提供上述功能的器件。在一个简单的实施例中，本领域的技术人员可以想到该电子设备可以采用图3所示的形式。
比如，图3中的处理器301可以通过调用存储器303中存储的计算机执行指令， 使得电子设备执行上述方法实施例中的交互方法。
示例性的，图13中的获取模块1301和确定模块1302的功能/实现过程可以通过图3中的处理器301调用存储器303中存储的计算机执行指令来实现。
由于本实施例提供的电子设备可执行上述的交互方法，因此其所能获得的技术效果可参考上述方法实施例，在此不再赘述。
图14为本申请实施例提供的一种芯片的结构示意图。芯片140包括一个或多个处理器1401以及接口电路1402。可选的，所述芯片140还可以包含总线1403。其中：
处理器1401可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器1401中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器1401可以是通用处理器、数字通信器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或者其它可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
接口电路1402用于数据、指令或者信息的发送或者接收。处理器1401可以利用接口电路1402接收的数据、指令或者其它信息，进行加工，可以将加工完成信息通过接口电路1402发送出去。
可选的，芯片140还包括存储器，存储器可以包括只读存储器和随机存取存储器，并向处理器提供操作指令和数据。存储器的一部分还可以包括非易失性随机存取存储器(NVRAM)。
可选的，存储器存储了可执行软件模块或者数据结构，处理器1401可以通过调用存储器存储的操作指令(该操作指令可存储在操作系统中)，执行相应的操作。
可选的，芯片140可以使用在本申请实施例涉及的电子设备中。可选的，接口电路1402可用于输出处理器1401的执行结果。关于本申请的一个或多个实施例提供的交互方法可参考前述各个实施例，这里不再赘述。
需要说明的，处理器1401、接口电路1402各自对应的功能既可以通过硬件设计实现，也可以通过软件设计来实现，还可以通过软硬件结合的方式来实现，这里不作限制。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分 布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (22)

  1. 一种交互方法，其特征在于，应用于智能座舱，所述方法包括：
    获取手指的第一空间坐标，所述第一空间坐标为三维坐标系中的坐标；
    根据所述第一空间坐标，确定所述手指所指的第一方向；
    根据所述第一方向，调节目标设备的参数，或者，显示目标物体的信息。
  2. 根据权利要求1所述的方法，其特征在于，所述获取手指的第一空间坐标之前，所述方法还包括：
    获取所述手指的第二空间坐标，所述第二空间坐标为所述三维坐标系中的坐标；
    根据所述第二空间坐标，确定所述手指所指的第二方向；
    将所述第二方向上的设备，确定为所述目标设备。
  3. 根据权利要求1或2所述的方法，其特征在于，所述获取手指的第一空间坐标，包括：
    在预设时间段内，检测到所述手指所指方向在第一范围内的情况下，获取所述手指的第一空间坐标；其中，所述第一范围为与所述手指的初始方向的夹角小于或等于第一阈值的范围，所述初始方向为在所述预设时间段的初始时刻，检测到的所述手指的方向。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述获取手指的第一空间坐标，包括：
    获取摄像机拍摄的图像中所述手指的图像坐标，所述手指的图像坐标为二维坐标系中的坐标；
    根据所述手指的图像坐标、所述摄像机的内参矩阵和物理变换矩阵得到所述第一空间坐标，所述摄像机的内参矩阵和所述物理变换矩阵用于指示所述二维坐标系与所述三维坐标系之间的变换关系。
  5. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    获取所述摄像机拍摄的图像中多个估计点的图像坐标和所述多个估计点的空间坐标，所述多个估计点的图像坐标为所述二维坐标系中的坐标，所述多个估计点的空间坐标为所述三维坐标系中的坐标；
    根据所述摄像机的内参矩阵、所述多个估计点的图像坐标和所述多个估计点的空间坐标得到所述物理变换矩阵。
  6. 根据权利要求4所述的方法，其特征在于，所述物理变换矩阵为预设置的。
  7. 根据权利要求4-6中任一项所述的方法，其特征在于，所述摄像机为单目摄像机、双目摄像机或深度摄像机。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述目标物体为交互区域中的物体；所述交互区域为所述第一方向覆盖的区域。
  9. 根据权利要求4或5所述的方法，其特征在于，所述二维坐标系为摄像机对应的坐标系；所述三维坐标系为所述智能座舱的三维模型对应的坐标系，或者所述智能座舱的三维图纸对应的坐标系。
  10. 根据权利要求1-9中任一项所述的方法，其特征在于，所述智能座舱为车辆中的座舱，所述根据所述第一方向，调节目标设备的参数，包括：
    根据所述第一方向，调节所述车辆中车窗的参数；或者，
    根据所述第一方向，调节所述车辆中天窗的参数；或者，
    根据所述第一方向，调节所述车辆中空调的参数。
  11. 一种电子设备，其特征在于，所述电子设备应用于智能座舱，所述电子设备包括：获取模块和确定模块；
    所述获取模块，用于获取手指的第一空间坐标，所述第一空间坐标为三维坐标系中的坐标；
    所述确定模块，用于根据所述第一空间坐标，确定所述手指所指的第一方向；
    所述确定模块，还用于根据所述第一方向，调节目标设备的参数，或者，显示目标物体的信息。
  12. 根据权利要求11所述的电子设备，其特征在于，
    所述获取模块，还用于获取所述手指的第二空间坐标，所述第二空间坐标为所述三维坐标系中的坐标；
    所述确定模块，还用于根据所述第二空间坐标，确定所述手指所指的第二方向；
    所述确定模块，还用于将所述第二方向上的设备，确定为所述目标设备。
  13. 根据权利要求11或12所述的电子设备，其特征在于，
    所述获取模块，具体用于在预设时间段内，检测到所述手指所指方向在第一范围内的情况下，获取所述手指的第一空间坐标；其中，所述第一范围为与所述手指的初始方向的夹角小于或等于第一阈值的范围，所述初始方向为在所述预设时间段的初始时刻，检测到的所述手指的方向。
  14. 根据权利要求11-13中任一项所述的电子设备，其特征在于，
    所述获取模块，具体用于获取摄像机拍摄的图像中所述手指的图像坐标，所述手指的图像坐标为二维坐标系中的坐标；
    所述获取模块，还具体用于根据所述手指的图像坐标、所述摄像机的内参矩阵和物理变换矩阵得到所述第一空间坐标，所述摄像机的内参矩阵和所述物理变换矩阵用于指示所述二维坐标系与所述三维坐标系之间的变换关系。
  15. 根据权利要求14所述的电子设备，其特征在于，
    所述获取模块，还用于获取所述摄像机拍摄的图像中多个估计点的图像坐标和所述多个估计点的空间坐标，所述多个估计点的图像坐标为所述二维坐标系中的坐标，所述多个估计点的空间坐标为所述三维坐标系中的坐标；
    所述确定模块，还用于根据所述摄像机的内参矩阵、所述多个估计点的图像坐标和所述多个估计点的空间坐标得到所述物理变换矩阵。
  16. 根据权利要求14所述的电子设备，其特征在于，所述物理变换矩阵为预设置的。
  17. 根据权利要求14-16中任一项所述的电子设备，其特征在于，所述摄像机为单目摄像机、双目摄像机或深度摄像机。
  18. 根据权利要求11-17中任一项所述的电子设备，其特征在于，所述目标物体为交互区域中的物体；所述交互区域为所述第一方向覆盖的区域。
  19. 根据权利要求14或15所述的电子设备，其特征在于，所述二维坐标系为摄 像机对应的坐标系；所述三维坐标系为所述智能座舱的三维模型对应的坐标系，或者所述智能座舱的三维图纸对应的坐标系。
  20. 根据权利要求11-19中任一项所述的电子设备，其特征在于，所述智能座舱为车辆中的座舱，
    所述确定模块，具体用于根据所述第一方向，调节所述车辆中车窗的参数；或者，
    所述确定模块，具体用于根据所述第一方向，调节所述车辆中天窗的参数；或者，
    所述确定模块，具体用于根据所述第一方向，调节所述车辆中空调的参数。
  21. 一种电子设备，其特征在于，包括：处理器，所述处理器与存储器耦合，所述存储器用于存储程序或指令，当所述程序或指令被所述处理器执行时，使得所述电子设备执行如权利要求1至10中任一项所述的方法。
  22. 一种车，其特征在于，包括：如权利要求11-20中任一项所述的电子设备。
PCT/CN2020/098643 2020-06-28 2020-06-28 交互方法及电子设备 WO2022000149A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202080004859.3A CN112771480B (zh) 2020-06-28 2020-06-28 交互方法及电子设备
PCT/CN2020/098643 WO2022000149A1 (zh) 2020-06-28 2020-06-28 交互方法及电子设备
EP20942766.5A EP4086731A4 (en) 2020-06-28 2020-06-28 METHOD OF INTERACTION AND ELECTRONIC DEVICE

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/098643 WO2022000149A1 (zh) 2020-06-28 2020-06-28 交互方法及电子设备

Publications (1)

Publication Number Publication Date
WO2022000149A1 true WO2022000149A1 (zh) 2022-01-06

Family

ID=75699481

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/098643 WO2022000149A1 (zh) 2020-06-28 2020-06-28 交互方法及电子设备

Country Status (3)

Country Link
EP (1) EP4086731A4 (zh)
CN (1) CN112771480B (zh)
WO (1) WO2022000149A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115061565A (zh) * 2022-05-10 2022-09-16 华为技术有限公司 调节显示设备的方法和装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102012019649A1 (de) * 2012-10-06 2014-04-10 Leopold Kostal Gmbh & Co. Kg Verfahren zur Erkennung von im freien Raum ausgeführten Körpergesten durch einen optischen Sensor und Vorrichtung zur Durchführung des Verfahrens
CN104254817A (zh) * 2012-04-16 2014-12-31 高通股份有限公司 快速手势重新约定
CN104765444A (zh) * 2014-01-03 2015-07-08 哈曼国际工业有限公司 车载手势交互空间音频系统
CN111158491A (zh) * 2019-12-31 2020-05-15 苏州莱孚斯特电子科技有限公司 一种应用于车载hud的手势识别人机交互方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE60133386T2 (de) * 2000-05-17 2009-04-02 Koninklijke Philips Electronics N.V. Vorrichtung und verfahren zur anzeige eines ziels mittels bildverarbeitung ohne drei dimensionales modellieren
US8994652B2 (en) * 2013-02-15 2015-03-31 Intel Corporation Model-based multi-hypothesis target tracker
CN104978012B (zh) * 2014-04-03 2018-03-16 华为技术有限公司 一种指向交互方法、装置及系统
KR101630153B1 (ko) * 2014-12-10 2016-06-24 현대자동차주식회사 제스처 인식 장치, 그를 가지는 차량 및 차량의 제어 방법
CN106648039B (zh) * 2015-10-30 2019-07-16 富泰华工业（深圳）有限公司 手势控制系统及方法
KR101847577B1 (ko) * 2016-09-30 2018-04-10 포항공과대학교 산학협력단 표시 장치용 3차원 인터랙션 방법
CN110045825B (zh) * 2018-03-27 2022-05-13 杭州凌感科技有限公司 用于车辆交互控制的手势识别系统
CN110335292B (zh) * 2019-07-09 2021-04-30 北京猫眼视觉科技有限公司 一种基于图片跟踪实现模拟场景跟踪的方法、系统及终端
CN111158478B (zh) * 2019-12-26 2023-02-03 维沃移动通信有限公司 响应方法及电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104254817A (zh) * 2012-04-16 2014-12-31 高通股份有限公司 快速手势重新约定
DE102012019649A1 (de) * 2012-10-06 2014-04-10 Leopold Kostal Gmbh & Co. Kg Verfahren zur Erkennung von im freien Raum ausgeführten Körpergesten durch einen optischen Sensor und Vorrichtung zur Durchführung des Verfahrens
CN104765444A (zh) * 2014-01-03 2015-07-08 哈曼国际工业有限公司 车载手势交互空间音频系统
CN111158491A (zh) * 2019-12-31 2020-05-15 苏州莱孚斯特电子科技有限公司 一种应用于车载hud的手势识别人机交互方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4086731A4 *

Also Published As

Publication number Publication date
EP4086731A4 (en) 2023-04-05
EP4086731A1 (en) 2022-11-09
CN112771480B (zh) 2022-04-12
CN112771480A (zh) 2021-05-07

Similar Documents

Publication Publication Date Title
WO2023000119A1 (zh) 手势识别方法、装置、系统及车辆
JP6453449B2 (ja) ボウル型イメージングシステムにおけるオブジェクト追跡
US8773512B1 (en) Portable remote control device enabling three-dimensional user interaction with at least one appliance
KR20210104107A (ko) 주시 영역 검출 방법, 장치 및 전자 디바이스
KR20160012139A (ko) 홀로그램 앵커링 및 동적 포지셔닝 기법
CN111353930B (zh) 数据处理方法及装置、电子设备及存储介质
EP4318186A1 (en) Multi-screen interaction method and apparatus, and terminal device and vehicle
CN104081307A (zh) 图像处理装置、图像处理方法和程序
US11195259B2 (en) Apparatus and method for dynamic multi-camera rectification using depth camera
CN111061363A (zh) 一种虚拟现实系统
KR20190027079A (ko) 전자 장치, 그 제어 방법 및 컴퓨터 판독가능 기록 매체
WO2022000149A1 (zh) 交互方法及电子设备
JP2018528514A (ja) 後方交差交通のクイックルック
CN113840757B (zh) 一种显示屏调整方法及装置
US20220155859A1 (en) Human-machine Interaction method, System, Processing Device and Computer Readable Storage Medium
CN116400878B (zh) 头戴显示设备的显示方法、装置、电子设备以及存储介质
TWI553509B (zh) 手勢控制系統及方法
EP3088991B1 (en) Wearable device and method for enabling user interaction
CN108401452B (zh) 使用虚拟现实头戴式显示器系统来执行真实目标检测和控制的装置和方法
WO2019100547A1 (zh) 投影控制方法、装置、投影交互系统及存储介质
CN110895675B (zh) 用于确定3d空间中的对象的特征点的坐标的方法
US20200167005A1 (en) Recognition device and recognition method
CN104238734A (zh) 三维交互系统及其交互感测方法
US20240129448A1 (en) Method and system for converting single-view image to 2.5d view for extended reality (xr) applications
TWI817540B (zh) 深度圖像獲取方法、電子設備及計算機可讀存儲媒體

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20942766

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020942766

Country of ref document: EP

Effective date: 20220802

NENP Non-entry into the national phase

Ref country code: DE