WO2016070688A1 - 虚拟操作界面遥控控制方法及系统 - Google Patents

虚拟操作界面遥控控制方法及系统 Download PDF

Info

Publication number
WO2016070688A1
WO2016070688A1 PCT/CN2015/090466 CN2015090466W WO2016070688A1 WO 2016070688 A1 WO2016070688 A1 WO 2016070688A1 CN 2015090466 W CN2015090466 W CN 2015090466W WO 2016070688 A1 WO2016070688 A1 WO 2016070688A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
remote control
operation interface
virtual operation
fingertip
Prior art date
Application number
PCT/CN2015/090466
Other languages
English (en)
French (fr)
Inventor
赵波
Original Assignee
深圳Tcl新技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳Tcl新技术有限公司 filed Critical 深圳Tcl新技术有限公司
Priority to EP15857359.2A priority Critical patent/EP3217637A4/en
Priority to US15/125,134 priority patent/US10429936B2/en
Publication of WO2016070688A1 publication Critical patent/WO2016070688A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G06F3/0426—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking fingers with respect to a virtual keyboard projected or printed on the surface
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482—Interaction with lists of selectable items, e.g. menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/113—Recognition of static hand signs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G08—SIGNALLING
    • G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
    • G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
    • G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3141—Constructional details thereof
    • H04N9/3173—Constructional details thereof wherein the projection device is specially adapted for enhanced portability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • G—PHYSICS
    • G08—SIGNALLING
    • G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
    • G08C2201/00—Transmission systems of control signals via wireless link
    • G08C2201/30—User interface
    • G08C2201/31—Voice input
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42203—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] sound input device, e.g. microphone
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/42222—Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/4223—Cameras

Definitions

  • the present invention relates to the field of remote control technologies, and in particular, to a virtual operation interface remote control method and system.
  • the main object of the present invention is to improve the accuracy of remote command input and improve the flexibility of remote command input.
  • the present invention provides a virtual operation interface remote control method, comprising the following steps:
  • the controlled terminal acquires a preset voice command according to the voice information input by the user, and sends the voice command to the remote control terminal; the voice command has a one-to-one correspondence with the preset virtual operation interface;
  • the remote control terminal projects a virtual operation interface corresponding to the voice command according to the voice command;
  • the remote control terminal collects a screen image formed by the virtual operation interface every preset time interval;
  • the remote control terminal analyzes a remote control instruction corresponding to the position of the fingertip on the virtual interface according to the collected screen image, and sends the remote control command to the controlled terminal to control the controlled terminal to perform a corresponding remote operation.
  • the remote control terminal analyzes a remote control instruction corresponding to the position of the fingertip on the virtual interface according to the collected screen image, and sends the remote control command to the controlled terminal to control the controlled terminal to perform the corresponding remote control.
  • Operations include:
  • the remote control terminal After the projection device projects the virtual operation interface, the remote control terminal acquires the first frame image of the virtual interface, and sets the first frame image as a template image;
  • the method further includes:
  • the step of determining the key value input by the user on the virtual operation interface according to the fingertip image is performed.
  • the determining, according to the fingertip image, the key value input by the user on the virtual operation interface comprises:
  • the key value input by the user on the virtual operation interface is analyzed according to the midpoint coordinate value.
  • the method further includes:
  • the remote control terminal detects whether the distance of its own moving distance is greater than a second preset value in real time
  • the present invention also provides a virtual operation interface remote control system, package Including a remote control terminal and a controlled terminal, wherein
  • the controlled terminal is configured to send a preset voice command according to the voice information input by the user to the remote control terminal; the voice command has a one-to-one correspondence with the preset virtual operation interface;
  • the remote control terminal is configured to project a virtual operation interface corresponding to the voice command according to the voice command; collect a screen image formed by the virtual operation interface every preset time interval; analyze finger fingertips according to the collected screen image
  • the remote interface corresponding to the position is stopped on the virtual interface, and the remote control command is sent to the controlled terminal to control the controlled terminal to perform a corresponding remote operation.
  • the remote control terminal is specifically configured to acquire a first frame image of the virtual interface after the projection device projects the virtual operation interface, and set the first frame image as a template image; Presetting the absolute value of the difference between the pixel value of the screen image formed by the virtual operation interface and the pixel value of the template image is greater than the first preset value; if yes, extracting the fingertip image in the currently acquired screen image; The fingertip image determines a key value input by the user on the virtual operation interface; and sends a corresponding remote control command to the controlled terminal according to the key value to control the controlled terminal to perform a corresponding remote operation.
  • the remote control terminal is further configured to: calculate an area of the fingertip image; determine whether an area of the fingertip image is greater than a maximum value of the preset range; if yes, obtain a fingertip image according to a circle-like feature of the fingertip image The arc jump point, the image segmentation is performed at the arc jump point, the target image having the continuous arc shape is set as the fingertip image; and the key value input by the user on the virtual operation interface is determined according to the fingertip image The operation; if not, executing the key value operation input by the user on the virtual operation interface according to the fingertip image.
  • the remote control terminal is specifically configured to calculate a midpoint coordinate value of the fingertip image; and analyze a key value input by the user on the virtual operation interface according to the midpoint coordinate value.
  • the remote control terminal is further configured to detect, in real time, whether the distance of the self-moving distance is greater than a second preset value; when the distance of the remote terminal moving distance is greater than the second preset value, projecting and speaking according to the voice command The virtual operation interface corresponding to the voice command.
  • the virtual operation interface is output by using the remote control terminal, and the screen image formed by the virtual operation interface is collected by the remote control terminal every preset time interval, thereby determining the remote control instruction corresponding to the position of the fingertip of the user on the virtual operation interface. And outputting the remote control command to the controlled terminal to implement the input of the remote control command.
  • the invention realizes the remote command input of the virtual operation interface Compared with the prior art, the mechanical button is used to realize the input of the remote control command, which can effectively prevent the button from malfunctioning and improve the accuracy of the remote command input.
  • the smart wear device is used to input the remote control command, the user is convenient to operate, and the flexibility of the remote operation command input is improved.
  • FIG. 1 is a schematic flow chart of a preferred embodiment of a virtual operation interface remote control method according to the present invention
  • FIG. 2 is a schematic flow chart of an embodiment of performing fingertip image recognition in FIG. 1;
  • FIG. 3 is a schematic diagram showing the refinement of another embodiment of fingertip image recognition in FIG. 1.
  • FIG. 3 is a schematic diagram showing the refinement of another embodiment of fingertip image recognition in FIG. 1.
  • the present invention provides a virtual operation interface remote control method.
  • the virtual operation interface remote control method includes:
  • Step S10 The controlled terminal acquires a preset voice command according to the voice information input by the user, and sends the preset voice command to the remote control terminal; the voice command has a one-to-one correspondence with the preset virtual operation interface;
  • the virtual operation interface remote control method provided in this embodiment is mainly applied to a remote control operating system to implement input of a remote control command based on a virtual remote operation interface.
  • a television remote control system may be taken as an example for detailed description.
  • the remote control terminal may be a smart wearable device (such as glasses, etc.)
  • the controlled terminal may be a smart television, wherein the smart television can be established with the smart wearable device.
  • the voice input function can be set on the smart TV and the smart wearable device. When the user turns on the voice function, the user can input the voice information of the user through the smart wearable device, and transmit the voice information to the smart TV, or directly open the smart TV to implement the voice information. input of.
  • the virtual operation interface includes a digital remote control projection interface, a conventional remote control manipulation interface, and an alphabet keyboard projection interface.
  • the type of the virtual operation interface is not further limited herein. It should be noted that the above voice commands may include: “open digital remote projection interface”, “open traditional remote control interface”, “open letter keyboard projection interface” and “invalid command”.
  • voice/audio data pre-processing may be added when voice information is received.
  • the voice/audio data preprocessing refers to processing the original voice information, filtering out unimportant information and background noise, and processing such as endpoint detection, voice framing, and pre-emphasis, thereby reducing the difficulty of subsequent voice recognition.
  • the speech recognition can be implemented by using the existing mature speech recognition technology, and will not be described herein.
  • HMM Hidden Markov Model
  • DTW Dynamic Time Warping Technology
  • VQ Vector Quantization Technology
  • ANN Artificial Neural Network Technology
  • SVN Small Vector Machine Technology
  • Step S20 the remote control terminal projects a virtual operation interface corresponding to the voice command according to the voice command;
  • Step S30 the remote control terminal collects a screen image formed by the virtual operation interface every preset time interval
  • Step S40 the remote control terminal analyzes a remote control instruction corresponding to the position of the fingertip on the virtual interface according to the collected screen image, and sends the remote control command to the controlled terminal to control the controlled terminal to execute the corresponding remote control. operating.
  • the remote control terminal has a built-in projection device for projecting a virtual operation interface, and the remote control terminal further includes an image acquisition device (camera) for collecting a screen image formed by the virtual operation interface.
  • the preset time interval system defaults to an initial experience value, and the user can also adjust the settings by himself.
  • the virtual operation interface corresponding to the voice command is projected.
  • the projection displays the digital remote projection interface; when the voice command is “open the traditional remote control interface”, the projection displays the traditional remote control interface; when the voice command is “open the letter keyboard projection” In the interface, the projection displays the alphabet keyboard projection interface; when the voice command is “invalid command”, the projection display is not performed, or the virtual display interface is displayed.
  • the camera preset on the remote control terminal collects the image formed by the virtual operation interface every preset time. Specifically, the camera collects the image.
  • the size can be set according to actual needs. For example, the image size captured by the camera is the same as the size of the virtual operation interface projection. Alternatively, the image captured by the camera is larger than the virtual operation interface projection.
  • the virtual operation interface projection is at least all included in the camera. Within the area of the collection.
  • the screen image corresponding to the user's finger staying may be first determined according to the collected screen image, and then the coordinate information of the user's fingertip is analyzed according to the screen image.
  • the input remote control command is determined according to the coordinate information, and then the remote control terminal outputs the remote control command to the controlled terminal, so that the controlled terminal performs a corresponding remote control operation, thereby implementing the input of the remote control command.
  • the virtual operation interface is output by using the remote control terminal, and the screen image formed by the virtual operation interface is collected by the remote control terminal every preset time interval, thereby determining the remote control instruction corresponding to the position of the fingertip of the user on the virtual operation interface. And outputting the remote control command to the controlled terminal to implement the input of the remote control command. Since the invention realizes the remote command input of the virtual operation interface, the mechanical button is used to realize the input of the remote control command compared with the prior art, which can effectively prevent the button from malfunctioning and improve the accuracy of the remote command input. In addition, since the smart wear device is used to input the remote control command, the user is convenient to operate, and the flexibility of the remote operation command input is improved.
  • the foregoing step S40 includes:
  • Step S41 after the projection device projects the virtual operation interface, the remote control terminal acquires the first frame image of the virtual interface, and sets the first frame image as a template image;
  • Step S42 determining whether the absolute value of the difference between the pixel value of the screen image formed by the virtual operation interface and the template image pixel value is greater than the first preset value; and if yes, executing step S43; Continue to return to step S42;
  • Step S43 extracting a fingertip image in the currently acquired screen image
  • Step S44 determining, according to the fingertip image, a key value input by the user on the virtual operation interface
  • Step S45 Send a corresponding remote control command to the controlled terminal according to the key value to control the controlled terminal to perform a corresponding remote control operation.
  • the built-in camera of the remote control terminal acquires the first frame image of the virtual interface and sets it as a template image, for example, the pixel value of the template image is i 1 (x, y), the screen image formed by the virtual operation interface is acquired every preset time interval, for example, the pixel value of each frame image obtained is i n (x, y), and n represents the number of frames of the image, for example, the second The pixel value of the frame image is i 2 (x, y).
  • the image pixel value is A 1 (x, y)
  • the image pixel value obtained after the second frame is optimized is A n (x, y).
  • the optimization process includes illumination correction, image noise reduction filtering, and image grayscale processing.
  • the illumination correction is: using the image illumination correction algorithm to process some variation information (such as illumination change, angle change, etc.) in the original specific information map, and the available methods are (not limited to): Gamma correction, image brightness Nonlinear transformation, histogram equalization and highest brightness point illumination compensation algorithm.
  • Gamma correction and image brightness nonlinear transformation can remove some irrelevant change information in the original information map by nonlinear adjustment of image pixel points; histogram equalization and highest brightness point illumination compensation algorithm is a kind of image contrast adjustment
  • the method can enhance the local contrast of the image.
  • the image denoising algorithm mainly filters out the noise information in the original image.
  • the typical noise in the image includes Gaussian noise, salt and pepper noise, additive noise, etc.
  • Common methods include mean filtering, median filtering and Gaussian filtering.
  • the above image acquired by Camera is a color image, and the color image is usually represented by the value of RGB three components, but there is a high correlation between the three components of RGB, and it is generally difficult to obtain the desired effect by directly using these components. At the same time, the computational complexity will also be increased. Since the color information is not concerned in the subsequent algorithm, the color image is grayed out in the preprocessing.
  • the morphological method of the image is further optimized.
  • the algorithms available for morphology include expansion, erosion, opening and closing operations, etc., wherein the expansion is For the treatment of discontinuous points, the corrosion is for the processing of isolated points, and the opening and closing operations are the combination of the two.
  • a fingertip image is extracted from the image subjected to the above morphological processing, and the key value input by the user on the virtual operation interface is determined based on the position of the fingertip image.
  • the method further includes:
  • Step S46 calculating an area of the fingertip image
  • Step S47 it is determined whether the area of the fingertip image is greater than the maximum value of the preset range; if yes, step S48 is performed, otherwise step S44 is performed;
  • step S48 the arc jump point of the fingertip image is acquired according to the circle-like feature of the fingertip image, the image segmentation is performed at the arc jump point, and the target image having the continuous arc shape is set as the fingertip image; then step S44 is performed.
  • the number of pixels of the fingertip image of the human is different.
  • a specific range is generally used to determine whether the currently extracted fingertip image is a simple finger.
  • Sharp image Specifically, calculating an area of the fingertip image is specifically calculating a number of pixel points of the extracted fingertip image, and when the number satisfies a preset range, indicating that the extracted fingertip image is a simple fingertip image, when When the number exceeds the maximum value of the preset range, it means that the extracted fingertip image is not a simple fingertip image, and includes an image such as a projection of a finger.
  • the arc jump point of the fingertip image is acquired according to the circle-like feature of the fingertip image, the image segmentation is performed at the arc jump point, and the target image having the continuous arc shape is set as the fingertip image.
  • the accuracy of the remote command input can be further improved.
  • the manner of determining the key value input by the user on the virtual operation interface according to the fingertip image may be set according to actual needs, for example, may be determined according to the center coordinates of the fingertip image, or may be based on the entire fingertip image. The area covered by the area is judged.
  • the remote control terminal may first calculate a midpoint coordinate value of the fingertip image; and then analyze a key value input by the user on the virtual operation interface according to the midpoint coordinate value.
  • the image of the fingertip image center is Where i n (x) and i n (y) represent the x-coordinate and y-coordinate values of the fingertip image, respectively, and R is a set of coordinates representing all coordinate values (x, y) on the fingertip image, where N represents The number of pixels of the fingertip image. Then determine The range corresponding to the virtual button that falls in, thereby determining the key value input by the user.
  • the method further includes:
  • the remote control terminal detects in real time whether its own moving distance is greater than a second preset value
  • the remote control terminal after the projection device projects the virtual operation interface, if the remote control terminal moves, the remote control terminal will turn off the camera and control the projection device to reset.
  • the remote terminal movement is set to an experience threshold (ie, the second preset value), and the resetting process is started only when the moving distance of the user wearing the remote control terminal exceeds the experience threshold, so as to avoid the user's slight The sway is misidentified as moving.
  • connection communication can be set according to actual needs.
  • wireless communication methods such as WiFi/Bluetooth/2.4G can be used for data interaction.
  • the connection is preferably established through NFC communication.
  • the current wireless interaction mode needs to be determined first.
  • the following is a smart terminal for the remote control terminal, and the controlled terminal is a smart TV. The two are described in detail by Bluetooth and WiFi respectively:
  • the remote terminal If the data is exchanged by means of Bluetooth, the connection is established, and the remote terminal first obtains the Bluetooth MAC address (BMacAddr) of the machine, and after obtaining the BMacAddr, it is cached by the storage module, and when the user performs the touch operation, That is, when the remote control terminal and the controlled terminal perform P2P interaction, the BMacAddr is transmitted to the controlled terminal through the NFC (in the process of transmission, if the received result feedback of the controlled terminal is not received, the transmission is continued until the receiving feedback of the controlled terminal is received. Result information), the controlled terminal automatically adapts to the remote control terminal after acquiring BMacAddr through NFC, completes the establishment of the Bluetooth connection, and realizes the code matching process.
  • BMacAddr Bluetooth MAC address
  • the WiFi connection cannot be manually performed, that is, the WiFi cannot be connected, and the SSID (identifier) and KEY required for WiFi networking can only be obtained from the controlled terminal. (secure key), so in the process of establishing the connection, the controlled terminal first needs to open a service to obtain the SSID (identifier) and KEY (secure key) of the WiFi currently connected to the machine, of course, if the terminal is controlled If the network is not currently connected, the user will be prompted to perform networking operations in the initialization step of the controlled terminal. After obtaining the SSID and KEY, the user will be cached through the storage module.
  • the remote control When the user performs the operation of the remote control terminal to touch the controlled terminal, the remote control is performed.
  • the terminal interacts with the controlled terminal for P2P, the SSID and the KEY are transmitted to the remote control terminal through the NFC (transmitted In the process, if the receiving result feedback of the remote terminal is not received, the transmission is continued until the received feedback result of the controlled and total segment is received.
  • the remote terminal acquires the SSID and the KEY through the NFC, the network is first completed, and then Controlled to adapt, complete the establishment of the WiFi connection, and achieve the code process.
  • the present invention also provides a virtual operation interface remote control system, and the virtual operation interface remote control system provided in this embodiment includes a remote control terminal and a controlled terminal, wherein
  • the controlled terminal is configured to send a preset voice command according to the voice information input by the user to the remote control terminal; the voice command has a one-to-one correspondence with the preset virtual operation interface;
  • the virtual operation interface remote control method provided in this embodiment is mainly applied to a remote control operating system to implement input of a remote control command based on a virtual remote operation interface.
  • a television remote control system may be taken as an example for detailed description.
  • the remote control terminal may be a smart wearable device (such as glasses, etc.)
  • the controlled terminal may be a smart television, wherein the smart television can be established with the smart wearable device.
  • the voice input function can be set on the smart TV and the smart wearable device. When the user turns on the voice function, the user can input the voice information of the user through the smart wearable device, and transmit the voice information to the smart TV, or directly open the smart TV to implement the voice information. input of.
  • the virtual operation interface includes a digital remote control projection interface, a conventional remote control manipulation interface, and an alphabet keyboard projection interface.
  • the type of the virtual operation interface is not further limited herein. It should be noted that the above voice commands may include: “open digital remote projection interface”, “open traditional remote control interface”, “open letter keyboard projection interface” and “invalid command”.
  • voice/audio data pre-processing may be added when voice information is received.
  • the voice/audio data preprocessing refers to processing the original voice information, filtering out unimportant information and background noise, and processing such as endpoint detection, voice framing, and pre-emphasis, thereby reducing the difficulty of subsequent voice recognition.
  • the speech recognition can be implemented by using the existing mature speech recognition technology, and will not be described herein.
  • HMM Hidden Markov Model
  • DTW Dynamic Time Warping Technology
  • VQ Vector Quantization Technology
  • ANN Artificial Neural Network Technology
  • SVN Small Vector Machine Technology
  • the remote control terminal is configured to project a virtual operation interface corresponding to the voice command according to the voice command; collect a screen image formed by the virtual operation interface every preset time interval; analyze finger fingertips according to the collected screen image
  • the remote interface corresponding to the position is stopped on the virtual interface, and the remote control command is sent to the controlled terminal to control the controlled terminal to perform a corresponding remote operation.
  • the remote control terminal has a built-in projection device for projecting a virtual operation interface, and the remote control terminal further includes an image acquisition device (camera) for collecting a screen image formed by the virtual operation interface.
  • the preset time interval system defaults to an initial experience value, and the user can also adjust the settings by himself.
  • the virtual operation interface corresponding to the voice command is projected.
  • the projection displays the digital remote projection interface; when the voice command is “open the traditional remote control interface”, the projection displays the traditional remote control interface; when the voice command is “open the letter keyboard projection” In the interface, the projection displays the alphabet keyboard projection interface; when the voice command is “invalid command”, the projection display is not performed, or the virtual display interface is displayed.
  • the camera preset on the remote control terminal will collect the image formed by the virtual operation interface every preset time.
  • the size of the image captured by the camera can be set according to actual needs, for example, The size of the image captured by the camera is the same as the size of the virtual operation interface projection. Alternatively, the image captured by the camera is larger than the virtual operation interface projection.
  • the virtual operation interface projection is at least all included in the area collected by the camera.
  • the screen image corresponding to the user's finger staying may be first determined according to the collected screen image, and then the coordinate information of the user's fingertip is analyzed according to the screen image.
  • the input remote control command is determined according to the coordinate information, and then the remote control terminal outputs the remote control command to the controlled terminal, so that the controlled terminal performs a corresponding remote control operation, thereby implementing the input of the remote control command.
  • the virtual operation interface is output by using the remote control terminal, and the screen image formed by the virtual operation interface is collected by the remote control terminal every preset time interval, thereby determining the remote control instruction corresponding to the position of the fingertip of the user on the virtual operation interface. And outputting the remote control command to the controlled terminal to implement the input of the remote control command. Since the invention realizes the remote command input of the virtual operation interface, the mechanical button is used to realize the input of the remote control command compared with the prior art, which can effectively prevent the button from malfunctioning and improve the accuracy of the remote command input. In addition, due to the use of smart wearable devices for remote control The input of the instruction is convenient for the user to operate, and the flexibility of the remote operation command input is improved.
  • the remote control terminal is specifically configured to: after the projection device projects the virtual operation interface, the remote control terminal acquires the first frame image of the virtual interface, and the first frame is Setting an image as a template image; determining whether an absolute value of a difference between a pixel value of the screen image formed by the virtual operation interface and the pixel value of the template image is greater than a first preset value; and if so, extracting a fingertip image in the currently acquired screen image; determining a key value input by the user on the virtual operation interface according to the fingertip image; and transmitting a corresponding remote control command to the controlled terminal according to the key value to control the controlled terminal to execute Corresponding remote operation.
  • the built-in camera of the remote control terminal acquires the first frame image of the virtual interface and sets it as a template image, for example, the pixel value of the template image is i 1 (x, y), the screen image formed by the virtual operation interface is acquired every preset time interval, for example, the pixel value of each frame image obtained is i n (x, y), and n represents the number of frames of the image, for example, the second The pixel value of the frame image is i 2 (x, y).
  • the image pixel value is A 1 (x, y)
  • the image pixel value obtained after the second frame is optimized is A n (x, y).
  • the optimization process includes illumination correction, image noise reduction filtering, and image grayscale processing.
  • the illumination correction is: using the image illumination correction algorithm to process some variation information (such as illumination change, angle change, etc.) in the original specific information map, and the available methods are (not limited to): Gamma correction, image brightness Nonlinear transformation, histogram equalization and highest brightness point illumination compensation algorithm.
  • Gamma correction and image brightness nonlinear transformation can remove some irrelevant change information in the original information map by nonlinear adjustment of image pixel points; histogram equalization and highest brightness point illumination compensation algorithm is a kind of image contrast adjustment
  • the method can enhance the local contrast of the image.
  • the image denoising algorithm mainly filters out the noise information in the original image.
  • the typical noise in the image includes Gaussian noise, salt and pepper noise, additive noise, etc.
  • Common methods include mean filtering, median filtering and Gaussian filtering.
  • the above image acquired by Camera is a color image, and the color image is usually represented by the value of RGB three components, but there is a high correlation between the three components of RGB, and it is generally difficult to obtain the desired effect by directly using these components. At the same time, the computational complexity will also be increased. Since the color information is not concerned in the subsequent algorithm, the color image is grayed out in the preprocessing.
  • the morphological method of the image is further optimized.
  • the algorithms available for morphology include expansion, erosion, opening and closing operations, etc., wherein the expansion is For the treatment of discontinuous points, the corrosion is for the processing of isolated points, and the opening and closing operations are the combination of the two.
  • the fingertip image is extracted from the image after the morphological processing described above, and the key value input by the user on the virtual operation interface is determined according to the position of the fingertip image.
  • the remote control terminal is further configured to calculate an area of the fingertip image; determine whether an area of the fingertip image is greater than a maximum value of a preset range; if yes, according to a circle-like feature of the fingertip image acquires a radians jumping point of the fingertip image, performs image segmentation at the arc jumping point, sets a target image having a continuous arc shape as a fingertip image; and performs the determining the user according to the fingertip image a key value operation input on the virtual operation interface; if not, performing the key value operation input by the user on the virtual operation interface according to the fingertip image.
  • the number of pixels of the fingertip image of the human is different.
  • a specific range is generally used to determine whether the currently extracted fingertip image is a simple finger.
  • Sharp image Specifically, calculating an area of the fingertip image is specifically calculating a number of pixel points of the extracted fingertip image, and when the number satisfies a preset range, indicating that the extracted fingertip image is a simple fingertip image, when When the number exceeds the maximum value of the preset range, it means that the extracted fingertip image is not a simple fingertip image, and includes an image such as a projection of a finger.
  • the arc jump point of the fingertip image is acquired according to the circle-like feature of the fingertip image, the image segmentation is performed at the arc jump point, and the target image having the continuous arc shape is set as the fingertip image.
  • the accuracy of the remote command input can be further improved.
  • the manner of determining the key value input by the user on the virtual operation interface according to the fingertip image may be set according to actual needs, for example, may be determined according to the center coordinates of the fingertip image, or may be based on the entire fingertip image. The area covered by the area is judged.
  • the remote control terminal is specifically configured to calculate a midpoint coordinate value of the fingertip image; and analyze a key value input by the user on the virtual operation interface according to the midpoint coordinate value.
  • the image of the fingertip image center is Where i n (x) and i n (y) represent the x-coordinate and y-coordinate values of the fingertip image, respectively, and R is a set of coordinates representing all coordinate values (x, y) on the fingertip image, where N represents The number of pixels of the fingertip image. Then determine The range corresponding to the virtual button that falls in, thereby determining the key value input by the user.
  • the remote control terminal is further configured to detect, in real time, whether the distance of the self-moving distance is greater than a second preset value; when the distance of the remote terminal moving distance is greater than the second preset value, A virtual operation interface corresponding to the voice command is projected according to the voice command.
  • the remote control terminal after the projection device projects the virtual operation interface, if the remote control terminal moves, the remote control terminal will turn off the camera and control the projection device to reset.
  • the remote terminal movement is set to an experience threshold (ie, the second preset value), and the resetting process is started only when the moving distance of the user wearing the remote control terminal exceeds the experience threshold, so as to avoid the user's slight The sway is misidentified as moving.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • a storage medium such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Signal Processing (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本发明公开了一种虚拟操作界面遥控控制方法，包括：被控终端根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；遥控终端根据语音命令投影与语音命令对应的虚拟操作界面；遥控终端每间隔预置时间采集虚拟操作界面形成的画面图像；遥控终端根据采集的画面图像分析手指指尖在虚拟界面上停留位置对应的遥控指令，并将遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。本发明还提供一种虚拟操作界面遥控控制系统。本发明提高了遥控指令输入的准确率以及遥控操作指令输入的灵活性。

Description

虚拟操作界面遥控控制方法及系统 技术领域
本发明涉及遥控技术领域，尤其涉及虚拟操作界面遥控控制方法及系统。
背景技术
随着技术的发展，特别是智能云技术和智能OS的普及和成熟，以及越来越多的可穿戴设备的兴起，人机交互已经不再像以往那样只能通过单一模式进行，例如人类和电视之间的交互目前就不再局限于遥控，其它如手势/语音/体感等等，越来越多的交互方式已经越来越普及，在以遥控为中心的传统交互模式上，通常遥控器进行按键遥控操作，但是由于机械按键容易损坏，从而导致按键失灵，影响遥控指令输入的准确率。
上述内容仅用于辅助理解本发明的技术方案，并不代表承认上述内容是现有技术。
发明内容
本发明的主要目的在于提高遥控指令输入的准确率，提高遥控操作指令输入的灵活性。
为了实现发明目的，本发明提供一种虚拟操作界面遥控控制方法，包括以下步骤：
被控终端根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；
所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面；
所述遥控终端每间隔预置时间采集所述虚拟操作界面形成的画面图像；
所述遥控终端根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。
优选地，所述遥控终端根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作包括：
当投影装置将虚拟操作界面进行投影后，遥控终端获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；
判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；
若是，则提取当前采集的画面图像中指尖图像；
根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；
根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
优选地，所述提取当前采集的画面图像中指尖图像之后还包括：
计算所述指尖图像的面积；
判断所述指尖图像的面积是否大于预置范围的最大值；
若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并返回执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤；
若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤。
优选地，所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值包括：
计算所述指尖图像的中点坐标值；
根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
优选地，所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面之后还包括：
所述遥控终端实时检测其自身移动距离的距离是否大于第二预设值；
若是，则返回执行所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面的步骤。
为了实现上述目的，本发明还提供一种虚拟操作界面遥控控制系统，包 括遥控终端和被控终端，其中
被控终端，用于根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；
所述遥控终端，用于根据所述语音命令投影与所述语音命令对应的虚拟操作界面；每间隔预置时间采集所述虚拟操作界面形成的画面图像；根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。
优选地，所述遥控终端具体用于当投影装置将虚拟操作界面进行投影后，获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；若是，则提取当前采集的画面图像中指尖图像；根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
优选地，遥控终端还用于，计算所述指尖图像的面积；判断所述指尖图像的面积是否大于预置范围的最大值；若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值操作；若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值操作。
优选地，所述遥控终端具体用于计算所述指尖图像的中点坐标值；根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
优选地，所述遥控终端还用于实时检测其自身移动距离的距离是否大于第二预设值；当遥控终端移动距离的距离大于第二预设值时，根据所述语音命令投影与所述语音命令对应的虚拟操作界面。
本发明实施例通过采用遥控终端输出虚拟操作界面，并由遥控终端每间隔预置时间采集虚拟操作界面形成的画面图像，从而确定用户手指指尖在虚拟操作界面上点击的位置所对应的遥控指令，并将该遥控指令输出至被控终端，以实现遥控指令的输入。由于本发明实现了虚拟操作界面的遥控指令输 入，相对于现有技术采用机械按键实现遥控指令的输入，可以有效防止按键失灵，提高遥控指令输入的准确率。此外，由于采用智能穿戴设备实现遥控指令的输入，方便用户操作，提高了遥控操作指令输入的灵活性。
附图说明
图1为本发明虚拟操作界面遥控控制方法较佳实施例的流程示意图；
图2为图1中进行指尖图像识别一实施例的细化流程示意图；
图3为图1中进行指尖图像识别另一实施例的细化流程示意图。
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
本发明提供一种虚拟操作界面遥控控制方法，参照图1，在一实施例中，该虚拟操作界面遥控控制方法包括：
步骤S10，被控终端根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；
本实施例提供的虚拟操作界面遥控控制方法主要应用于遥控操作系统当中，以实现基于虚拟遥控操作界面的遥控指令的输入。以下实施例中可以以电视遥控系统为例作出详细说明，具体地，上述遥控终端可以为智能穿戴设备(如眼镜等)，上述被控终端可以为智能电视，其中智能电视可以与智能穿戴设备建立通讯连接，实现数据传输。上述智能电视和智能穿戴设备上均可设置语音输入功能，当用户打开语音功能后，可输入通过智能穿戴设备接收用户的语音信息，并传送给智能电视，也可以直接开启智能电视，实现语音信息的输入。上述虚拟操作界面包括数字遥控投影界面、传统遥控操控界面和字母键盘投影界面，具体地虚拟操作界面的类型形式在此不作进一步地限定。应当说明的是，上述语音命令可以包括：“打开数字遥控投影界面”、“打开传统遥控操控界面”、“打开字母键盘投影界面”和“无效命令”。
例如，在用户开启遥控终端上的语音输入功能后，由遥控终端将用户输入的语音信息发送给被控终端，被控终端在接收到用户输入的语音信息时，将对该语音信息进行识别，并将识别的结果与语音命令进行匹配，得到用户输入的语音命令，然后将该语音命令输出至遥控终端。应当说明的是，在接收到语音信息时，可加入语音/音频数据预处理。该语音/音频数据预处理是指对原始语音信息进行处理，滤除掉其中不重要的信息和背景噪声，以及端点检测、语音分帧和预加重等处理，从而降低后续语音识别的难度。可以理解的是，语音识别可以采用现有成熟的语音识别技术实现，在此不作赘述。例如可以采用HMM(隐马尔可夫模型)、DTW(动态时间规整技术)、VQ(矢量量化技术)、ANN(人工神经网络技术)、SVN(支持向量机技术)，这些技术都是现有技术，可直接采用或采用其它成熟的语音识别技术进行语音识别。
步骤S20，所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面；
步骤S30，所述遥控终端每间隔预置时间采集所述虚拟操作界面形成的画面图像；
步骤S40，所述遥控终端根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。
本实施例中，上述遥控终端内置有投影装置，用于投影虚拟操作界面；遥控终端还内置有图像获取装置(摄像头)，用于采集虚拟操作界面形成的画面图像。上述预置时间间隔系统默认会给一个初始经验值，用户也可以自行调整设定。
具体地，在遥控终端接收到被控终端发送的语音命令时，将投影语音命令对应的虚拟操作界面。例如当语音命令为“打开数字遥控投影界面”时，投影显示数字遥控投影界面；当语音命令为“打开传统遥控操控界面”时，投影显示传统遥控操控界面；当语音命令为“打开字母键盘投影界面”时，投影显示字母键盘投影界面；当语音命令为“无效命令”，不进行投影显示，或者投影显示空的虚拟操作界面。
当投影显示相应的虚拟操作界面后，遥控终端上预置的摄像头将每隔预置时间采集所述虚拟操作界面形成的画面图像，具体地，摄像头采集图像的 大小可根据实际需要进行设置，例如，摄像头采集的图像大小与虚拟操作界面投影的大小一致，也可以是，摄像头采集的图像大于虚拟操作界面投影，使用时，虚拟操作界面投影至少全部包含在摄像头采集的区域范围内。
当用户手指在投影形成的虚拟操作菜单界面上进行遥控指令输入时，可根据采集的画面图像首先确定用户手指停留对应的画面图像，然后跟据该画面图像分析用户手指指尖停留的坐标信息。根据该坐标信息确定输入的遥控指令，然后由遥控终端将该遥控指令输出至被控终端，以使被控终端执行相应的遥控操作，从而实现遥控指令的输入。
本发明实施例通过采用遥控终端输出虚拟操作界面，并由遥控终端每间隔预置时间采集虚拟操作界面形成的画面图像，从而确定用户手指指尖在虚拟操作界面上点击的位置所对应的遥控指令，并将该遥控指令输出至被控终端，以实现遥控指令的输入。由于本发明实现了虚拟操作界面的遥控指令输入，相对于现有技术采用机械按键实现遥控指令的输入，可以有效防止按键失灵，提高遥控指令输入的准确率。此外，由于采用智能穿戴设备实现遥控指令的输入，方便用户操作，提高了遥控操作指令输入的灵活性。
进一步地，参照图2，提供了本发明虚拟操作界面遥控控制方法第二实施例，基于上述实施例，本实施例中，上述步骤S40包括：
步骤S41，当投影装置将虚拟操作界面进行投影后，遥控终端获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；
步骤S42，判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；若是，则执行步骤S43；若否继续返回执行步骤S42；
步骤S43，提取当前采集的画面图像中指尖图像；
步骤S44，根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；
步骤S45，根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
应当说明的是，在采集画面图像后，可对画面图像进行优化处理，可以提高图像的信噪比，抑制背景噪声，最真实的还原原始图像信息。本实施例 中，当投影装置将虚拟操作界面进行投影后，遥控终端的内置摄像头获取所述虚拟界面的第一帧图像，并将其设为模板图像，例如该模板图像的像素值为i1(x，y)，后续每间隔预置时间采集虚拟操作界面形成的画面图像，如获取的每一帧图像的像素值为in(x，y)，n表示图像的帧数，例如第二帧图像的像素值为i2(x，y)。模板图像进行优化处理后得到图像像素值为A1(x，y)，第二帧以后的图像进行优化后得到的图像像素值为An(x，y)。
具体地，优化处理包括光照校正、图像降噪滤波、图像灰度化处理。其中，光照校正为：利用图像光照校正算法对原始特定信息图中夹杂的一些变化信息(如光照变化、角度变化等)进行处理，可采用的方法有(不局限于)：Gamma校正、图像亮度非线性变换、直方图均衡化和最高亮度点光照补偿算法。Gamma校正和图像亮度非线性变换通过对图像像素点的非线性调整，能去除原始信息图中的一些无关的变化信息；直方图均衡化和最高亮度点光照补偿算法是一种针对图像对比度调整的方法，能增强图像局部对比度。图像降噪算法主要是对原始图像中的噪声信息进行滤除，图像中典型的噪声有高斯噪声、椒盐噪声、加性噪声等，常用的方法有均值滤波、中值滤波和高斯滤波等。上述通过Camera获取的图像是彩色图像，彩色图像通常是用RGB三分量的值来表示的，但RGB三分量之间有很高的相关性，直接利用这些分量一般很难得到所需的效果，同时也将增加计算的复杂度，由于后续算法中不关心色彩信息，所以在预处理中对彩色图像进行灰度化，灰度化的算法如下公式所示：A′n(x，y)＝(AnR(x，y)+AnG(x，y)+AnB(x，y))/3  (1-1)，采用平均值法，其中AnR，AnG，AnB分别表示RGB分量。
当获取到优化处理后的图像像素值A1(x，y)和An(x，y)时，通过
Figure PCTCN2015090466-appb-000001
计算当前获取的帧图像与模板图像之间的像素差值Bn(x，y)，其中T1为一个经验阈值(即上述第一预设值)，该公式表示当第n帧图像与模版图像的每个像素点很相似/相近时，将输出图像对应坐标点的像素值置为0，反之置为255。
当得到Bn(x，y)为255对应的图像时，对该图像进行形态学的方法进一步优化，形态学可采用的算法包括膨胀，腐蚀，开运算和闭运算等，其中膨胀针对的是不连续点的处理，腐蚀针对的是孤立点的处理，开和闭运算是两者的综合。然后在针对上述形态学处理后的图像中提取指尖图像，根据指尖图 像的位置，从而确定用户在所述虚拟操作界面上输入的键值。
进一步地，参照图3，提供了本发明虚拟操作界面遥控控制方法第三实施例，基于上述实施例，本实施例中，上述步骤S43之后还包括：
步骤S46，计算所述指尖图像的面积；
步骤S47，判断所述指尖图像的面积是否大于预置范围的最大值；若是则执行步骤S48，若否则执行步骤S44；
步骤S48，根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；接着执行步骤S44。
本实施例中，由于人手指尖的大小不同，因此人手指尖图像的像素点的个数不同，现有技术中通常采用一个特定的范围确定当前提取的指尖图像中是否为单纯的手指指尖图像。具体地，计算上述指尖图像的面积具体为计算提取的指尖图像的像素点个数，当该个数满足预置范围时，则表示提取的指尖图像为单纯的手指指尖图像，当该个数超过预置范围的最大值时，则表示提取的指尖图像并非单纯的手指指尖图像，还包括手指的投影等图像。此时，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，并将具有连续弧形的目标图像设定为指尖图像。本实施例中，由于对指尖图像的有效性进行判断，从而可以进一步提高遥控指令输入的准确性。
可以理解的是，上述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的方式可根据实际需要进行设置，例如可根据指尖图像中心坐标进行确定，也可以根据整个指尖图像区域覆盖的面积进行判断。本实施例中，优选地，遥控终端可首先计算所述指尖图像的中点坐标值；然后根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。例如该指尖图像中心图像为
Figure PCTCN2015090466-appb-000002
Figure PCTCN2015090466-appb-000003
其中in(x)和in(y)分别表示指尖图像的x坐标值和y坐标值，R是一坐标集合，表示指尖图像上的所有坐标值(x，y)，N表示所述指尖图像的像素个数。然后确定
Figure PCTCN2015090466-appb-000004
所落入的虚拟按键对应的范围，从而确定用户输入的键值。
进一步地，基于上述实施例，本实施例中，在上述步骤S20之后还包括：
所述遥控终端实时检测其自身移动距离是否大于第二预设值；
若是，则返回执行所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面的步骤。
本实施例中，当投影装置投影出虚拟操作界面后，如若遥控终端发生了移动，此时遥控终端将将关闭摄像头，并控制投影装置进行重设。此处的遥控终端移动会设定一个经验阈值(即上述第二预设值)，只有用户穿戴遥控终端发生的移动距离超过了该经验阈值才开始上述重设过程，这样是为了避免用户轻微的晃动误识别为移动。
应当说明的是，遥控终端和被控终端进行连接通讯的方式可根据实际需要进行设置。例如可以采用WiFi/蓝牙/2.4G等无线通讯方式进行数据交互。本实施例中优选通过NFC通讯建立连接，在NFC建立连接过程中，首先需要确定当前的无线交互方式。以下以遥控终端为智能穿戴，被控终端为智能电视，两者采用蓝牙和WiFi两种方式分别进行详细介绍：
如若通过蓝牙的方式进行数据交互，则连接建立中，遥控终端首先会去获取本机的蓝牙MAC地址(BMacAddr)，获取到BMacAddr后，将其通过存储模块进行缓存，当用户执行触碰操作，即遥控终端与被控终端进行P2P交互时，将BMacAddr通过NFC传递给被控终端(在传递过程中，若未接收到被控终端的接收结果反馈则持续传送直至接收到被控终端的接收反馈结果信息)，被控终端通过NFC获取BMacAddr后自动与遥控终端适配，完成蓝牙连接的建立，实现对码过程。
如若通过WiFi的方式进行数据交互，由于遥控终端无人机交互界面，无法手动进行WiFi的连接，也即无法连接WiFi，只能从被控终端中获取WiFi联网需要的SSID(标示符)和KEY(安全密匙)，所以在连接的建立过程中，被控终端首先需要开启一个服务去获取本机当前连上的WiFi的SSID(标示符)和KEY(安全密匙)，当然如若被控终端当前未联网，则在被控终端初始化步骤中会提示用户进行联网操作，获取到SSID和KEY后，将其通过存储模块进行缓存，当用户执行遥控终端触碰被控终端的操作时，即遥控终端与被控终端进行P2P交互时，将SSID和KEY通过NFC传递给遥控终端(在传递过 程中，若未接收到遥控终端的接收结果反馈则持续传送直至接收到被控和总段的接收反馈结果信息)，遥控终端通过NFC获取SSID和KEY后，首先完成自身的联网，然后再与被控适配，完成WiFi连接的建立，实现对码过程。
本发明还提供一种虚拟操作界面遥控控制系统，本实施例提供的虚拟操作界面遥控控制系统包括遥控终端和被控终端，其中
被控终端，用于根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；
本实施例提供的虚拟操作界面遥控控制方法主要应用于遥控操作系统当中，以实现基于虚拟遥控操作界面的遥控指令的输入。以下实施例中可以以电视遥控系统为例作出详细说明，具体地，上述遥控终端可以为智能穿戴设备(如眼镜等)，上述被控终端可以为智能电视，其中智能电视可以与智能穿戴设备建立通讯连接，实现数据传输。上述智能电视和智能穿戴设备上均可设置语音输入功能，当用户打开语音功能后，可输入通过智能穿戴设备接收用户的语音信息，并传送给智能电视，也可以直接开启智能电视，实现语音信息的输入。上述虚拟操作界面包括数字遥控投影界面、传统遥控操控界面和字母键盘投影界面，具体地虚拟操作界面的类型形式在此不作进一步地限定。应当说明的是，上述语音命令可以包括：“打开数字遥控投影界面”、“打开传统遥控操控界面”、“打开字母键盘投影界面”和“无效命令”。
例如，在用户开启遥控终端上的语音输入功能后，由遥控终端将用户输入的语音信息发送给被控终端，被控终端在接收到用户输入的语音信息时，将对该语音信息进行识别，并将识别的结果与语音命令进行匹配，得到用户输入的语音命令，然后将该语音命令输出至遥控终端。应当说明的是，在接收到语音信息时，可加入语音/音频数据预处理。该语音/音频数据预处理是指对原始语音信息进行处理，滤除掉其中不重要的信息和背景噪声，以及端点检测、语音分帧和预加重等处理，从而降低后续语音识别的难度。可以理解的是，语音识别可以采用现有成熟的语音识别技术实现，在此不作赘述。例如可以采用HMM(隐马尔可夫模型)、DTW(动态时间规整技术)、VQ(矢量量化技术)、ANN(人工神经网络技术)、SVN(支持向量机技术)，这些技术都是现有技术，可直接采用或采用其它成熟的语音识别技术进行语音识别。
所述遥控终端，用于根据所述语音命令投影与所述语音命令对应的虚拟操作界面；每间隔预置时间采集所述虚拟操作界面形成的画面图像；根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。
本实施例中，上述遥控终端内置有投影装置，用于投影虚拟操作界面；遥控终端还内置有图像获取装置(摄像头)，用于采集虚拟操作界面形成的画面图像。上述预置时间间隔系统默认会给一个初始经验值，用户也可以自行调整设定。
具体地，在遥控终端接收到被控终端发送的语音命令时，将投影语音命令对应的虚拟操作界面。例如当语音命令为“打开数字遥控投影界面”时，投影显示数字遥控投影界面；当语音命令为“打开传统遥控操控界面”时，投影显示传统遥控操控界面；当语音命令为“打开字母键盘投影界面”时，投影显示字母键盘投影界面；当语音命令为“无效命令”，不进行投影显示，或者投影显示空的虚拟操作界面。
当投影显示相应的虚拟操作界面后，遥控终端上预置的摄像头将每隔预置时间采集所述虚拟操作界面形成的画面图像，具体地，摄像头采集图像的大小可根据实际需要进行设置，例如，摄像头采集的图像大小与虚拟操作界面投影的大小一致，也可以是，摄像头采集的图像大于虚拟操作界面投影，使用时，虚拟操作界面投影至少全部包含在摄像头采集的区域范围内。
当用户手指在投影形成的虚拟操作菜单界面上进行遥控指令输入时，可根据采集的画面图像首先确定用户手指停留对应的画面图像，然后跟据该画面图像分析用户手指指尖停留的坐标信息。根据该坐标信息确定输入的遥控指令，然后由遥控终端将该遥控指令输出至被控终端，以使被控终端执行相应的遥控操作，从而实现遥控指令的输入。
本发明实施例通过采用遥控终端输出虚拟操作界面，并由遥控终端每间隔预置时间采集虚拟操作界面形成的画面图像，从而确定用户手指指尖在虚拟操作界面上点击的位置所对应的遥控指令，并将该遥控指令输出至被控终端，以实现遥控指令的输入。由于本发明实现了虚拟操作界面的遥控指令输入，相对于现有技术采用机械按键实现遥控指令的输入，可以有效防止按键失灵，提高遥控指令输入的准确率。此外，由于采用智能穿戴设备实现遥控 指令的输入，方便用户操作，提高了遥控操作指令输入的灵活性。
进一步地，基于上述实施例，本实施例中，上述遥控终端具体用于当投影装置将虚拟操作界面进行投影后，遥控终端获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；若是，则提取当前采集的画面图像中指尖图像；根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
应当说明的是，在采集画面图像后，可对画面图像进行优化处理，可以提高图像的信噪比，抑制背景噪声，最真实的还原原始图像信息。本实施例中，当投影装置将虚拟操作界面进行投影后，遥控终端的内置摄像头获取所述虚拟界面的第一帧图像，并将其设为模板图像，例如该模板图像的像素值为i1(x，y)，后续每间隔预置时间采集虚拟操作界面形成的画面图像，如获取的每一帧图像的像素值为in(x，y)，n表示图像的帧数，例如第二帧图像的像素值为i2(x，y)。模板图像进行优化处理后得到图像像素值为A1(x，y)，第二帧以后的图像进行优化后得到的图像像素值为An(x，y)。
具体地，优化处理包括光照校正、图像降噪滤波、图像灰度化处理。其中，光照校正为：利用图像光照校正算法对原始特定信息图中夹杂的一些变化信息(如光照变化、角度变化等)进行处理，可采用的方法有(不局限于)：Gamma校正、图像亮度非线性变换、直方图均衡化和最高亮度点光照补偿算法。Gamma校正和图像亮度非线性变换通过对图像像素点的非线性调整，能去除原始信息图中的一些无关的变化信息；直方图均衡化和最高亮度点光照补偿算法是一种针对图像对比度调整的方法，能增强图像局部对比度。图像降噪算法主要是对原始图像中的噪声信息进行滤除，图像中典型的噪声有高斯噪声、椒盐噪声、加性噪声等，常用的方法有均值滤波、中值滤波和高斯滤波等。上述通过Camera获取的图像是彩色图像，彩色图像通常是用RGB三分量的值来表示的，但RGB三分量之间有很高的相关性，直接利用这些分量一般很难得到所需的效果，同时也将增加计算的复杂度，由于后续算法中不关心色彩信息，所以在预处理中对彩色图像进行灰度化，灰度化的算法如 下公式所示：A′n(x，y)＝(AnR(x，y)+AnG(x，y)+AnB(x，y))/3  (1-1)，采用平均值法，其中AnR，AnG，AnB分别表示RGB分量。
当获取到优化处理后的图像像素值A1(x，y)和An(x，y)时，通过
Figure PCTCN2015090466-appb-000005
计算当前获取的帧图像与模板图像之间的像素差值Bn(x，y)，其中T1为一个经验阈值(即上述第一预设值)，该公式表示为，表示当第n帧图像与模版图像的每个像素点很相似/相近时，将输出图像对应坐标点的像素值置为0，反之置为255。
当得到Bn(x，y)为255对应的图像时，对该图像进行形态学的方法进一步优化，形态学可采用的算法包括膨胀，腐蚀，开运算和闭运算等，其中膨胀针对的是不连续点的处理，腐蚀针对的是孤立点的处理，开和闭运算是两者的综合。然后在针对上述形态学处理后的图像中提取指尖图像，根据指尖图像的位置，从而确定用户在所述虚拟操作界面上输入的键值。
进一步地，基于上述实施例，本实施例中，上述遥控终端还用于，计算所述指尖图像的面积；判断所述指尖图像的面积是否大于预置范围的最大值；若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值操作；若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值操作。
本实施例中，由于人手指尖的大小不同，因此人手指尖图像的像素点的个数不同，现有技术中通常采用一个特定的范围确定当前提取的指尖图像中是否为单纯的手指指尖图像。具体地，计算上述指尖图像的面积具体为计算提取的指尖图像的像素点个数，当该个数满足预置范围时，则表示提取的指尖图像为单纯的手指指尖图像，当该个数超过预置范围的最大值时，则表示提取的指尖图像并非单纯的手指指尖图像，还包括手指的投影等图像。此时，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，并将具有连续弧形的目标图像设定为指尖图像。本实施例中，由于对指尖图像的有效性进行判断，从而可以进一步提高遥控指令输入的准确性。
可以理解的是，上述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的方式可根据实际需要进行设置，例如可根据指尖图像中心坐标进行确定，也可以根据整个指尖图像区域覆盖的面积进行判断。本实施例中，优选地，遥控终端具体用于计算所述指尖图像的中点坐标值；根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
例如该指尖图像中心图像为
Figure PCTCN2015090466-appb-000006
Figure PCTCN2015090466-appb-000007
其中in(x)和in(y)分别表示指尖图像的x坐标值和y坐标值，R是一坐标集合，表示指尖图像上的所有坐标值(x，y)，N表示所述指尖图像的像素个数。然后确定
Figure PCTCN2015090466-appb-000008
所落入的虚拟按键对应的范围，从而确定用户输入的键值。
进一步地，基于上述实施例，本实施例中，上述遥控终端还用于实时检测其自身移动距离的距离是否大于第二预设值；当遥控终端移动距离的距离大于第二预设值时，根据所述语音命令投影与所述语音命令对应的虚拟操作界面。
本实施例中，当投影装置投影出虚拟操作界面后，如若遥控终端发生了移动，此时遥控终端将将关闭摄像头，并控制投影装置进行重设。此处的遥控终端移动会设定一个经验阈值(即上述第二预设值)，只有用户穿戴遥控终端发生的移动距离超过了该经验阈值才开始上述重设过程，这样是为了避免用户轻微的晃动误识别为移动。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。
以上仅为本发明的优选实施例，并非因此限制本发明的专利范围，凡是 利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。

Claims (16)

  1. 一种虚拟操作界面遥控控制方法，其特征在于，包括以下步骤：
    被控终端根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；
    所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面：
    所述遥控终端每间隔预置时间采集所述虚拟操作界面形成的画面图像；
    所述遥控终端根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。
  2. 如权利要求1所述的虚拟操作界面遥控控制方法，其特征在于，所述遥控终端根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作包括：
    当投影装置将虚拟操作界面进行投影后，遥控终端获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；
    判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；
    若是，则提取当前采集的画面图像中指尖图像；
    根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；
    根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
  3. 如权利要求2所述的虚拟操作界面遥控控制方法，其特征在于，所述提取当前采集的画面图像中指尖图像之后还包括：
    计算所述指尖图像的面积；
    判断所述指尖图像的面积是否大于预置范围的最大值；
    若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧 跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并返回执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤；
    若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤。
  4. 如权利要求3所述的虚拟操作界面遥控控制方法，其特征在于，所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值包括：
    计算所述指尖图像的中点坐标值；
    根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
  5. 如权利要求1所述的虚拟操作界面遥控控制方法，其特征在于，所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面之后还包括：
    所述遥控终端实时检测其自身移动距离的距离是否大于第二预设值；
    若是，则返回执行所述遥控终端根据所述语音命令投影与所述语音命令对应的虚拟操作界面的步骤。
  6. 如权利要求5所述的虚拟操作界面遥控控制方法，其特征在于，所述遥控终端根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作包括：
    当投影装置将虚拟操作界面进行投影后，遥控终端获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；
    判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；
    若是，则提取当前采集的画面图像中指尖图像；
    根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；
    根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
  7. 如权利要求6所述的虚拟操作界面遥控控制方法，其特征在于，所述提取当前采集的画面图像中指尖图像之后还包括：
    计算所述指尖图像的面积；
    判断所述指尖图像的面积是否大于预置范围的最大值；
    若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并返回执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤；
    若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤。
  8. 如权利要求7所述的虚拟操作界面遥控控制方法，其特征在于，所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值包括：
    计算所述指尖图像的中点坐标值；
    根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
  9. 一种虚拟操作界面遥控控制系统，其特征在于，包括遥控终端和被控终端，其中
    被控终端，用于根据用户输入的语音信息获取预置的语音命令发送至遥控终端；所述语音命令与预置的虚拟操作界面一一对应；
    所述遥控终端，用于根据所述语音命令投影与所述语音命令对应的虚拟操作界面；每间隔预置时间采集所述虚拟操作界面形成的画面图像；根据采集的画面图像分析手指指尖在所述虚拟界面上停留位置对应的遥控指令，并将所述遥控指令发送至被控终端，以控制被控终端执行相应的遥控操作。
  10. 如权利要求9所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端具体用于当投影装置将虚拟操作界面进行投影后，获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；若是，则提取当前采集的画面图像中指尖图像；根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值； 根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
  11. 如权利要求10所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端还用于，计算所述指尖图像的面积；判断所述指尖图像的面积是否大于预置范围的最大值；若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值操作；若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值操作。
  12. 如权利要求11所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端具体用于计算所述指尖图像的中点坐标值；根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
  13. 如权利要求9所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端还用于实时检测其自身移动距离的距离是否大于第二预设值；当遥控终端移动距离的距离大于第二预设值时，根据所述语音命令投影与所述语音命令对应的虚拟操作界面。
  14. 如权利要求13所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端具体用于，
    当投影装置将虚拟操作界面进行投影后，遥控终端获取所述虚拟界面的第一帧图像，并将所述第一帧图像设定为模板图像；
    判断后续每间隔预置时间采集虚拟操作界面形成的画面图像的像素值与所述模板图像像素值的差值的绝对值是否大于第一预设值；
    若是，则提取当前采集的画面图像中指尖图像；
    根据所述指尖图像确定用户在所述虚拟操作界面上输入的键值；
    根据所述键值发送相应的遥控指令至被控终端，以控制被控终端执行相应的遥控操作。
  15. 如权利要求14所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端还用于，
    计算所述指尖图像的面积；
    判断所述指尖图像的面积是否大于预置范围的最大值；
    若是，则根据指尖图像的类圆特征获取指尖图像的弧度跳跃点，在圆弧跳跃点进行图像分割，将具有连续弧形的目标图像设定为指尖图像；并返回执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤；
    若否，则执行所述根据指尖图像确定用户在所述虚拟操作界面上输入的键值的步骤。
  16. 如权利要求15所述的虚拟操作界面遥控控制系统，其特征在于，所述遥控终端具体用于，
    计算所述指尖图像的中点坐标值；
    根据所述中点坐标值分析用户在所述虚拟操作界面上输入的键值。
PCT/CN2015/090466 2014-11-06 2015-09-23 虚拟操作界面遥控控制方法及系统 WO2016070688A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP15857359.2A EP3217637A4 (en) 2014-11-06 2015-09-23 Remote control method and system for virtual operation interface
US15/125,134 US10429936B2 (en) 2014-11-06 2015-09-23 Remote control method and system for virtual operating interface

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410620941.5A CN105635776B (zh) 2014-11-06 2014-11-06 虚拟操作界面遥控控制方法及系统
CN201410620941.5 2014-11-06

Publications (1)

Publication Number Publication Date
WO2016070688A1 true WO2016070688A1 (zh) 2016-05-12

Family

ID=55908540

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/090466 WO2016070688A1 (zh) 2014-11-06 2015-09-23 虚拟操作界面遥控控制方法及系统

Country Status (4)

Country Link
US (1) US10429936B2 (zh)
EP (1) EP3217637A4 (zh)
CN (1) CN105635776B (zh)
WO (1) WO2016070688A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105898256A (zh) * 2016-05-30 2016-08-24 佛山市章扬科技有限公司 一种动作识别无屏电视
CN110427148A (zh) * 2019-07-26 2019-11-08 苏州蜗牛数字科技股份有限公司 一种语音触发屏幕的方法
CN114399896A (zh) * 2020-09-25 2022-04-26 汉桑（南京）科技有限公司 一种根据遥控设备图像配置遥控功能数据的方法和系统

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105635776B (zh) * 2014-11-06 2019-03-01 深圳Tcl新技术有限公司 虚拟操作界面遥控控制方法及系统
CN106293346A (zh) * 2016-08-11 2017-01-04 深圳市金立通信设备有限公司 一种虚拟现实视景的切换方法及终端
WO2018112856A1 (zh) * 2016-12-22 2018-06-28 深圳前海达闼云端智能科技有限公司 基于语音控制的位置定位方法、装置、用户设备及计算机程序产品
WO2018120768A1 (zh) * 2016-12-30 2018-07-05 华为技术有限公司 一种遥控方法和终端
CN109640064A (zh) * 2019-01-11 2019-04-16 郑州嘉晨电器有限公司 一种带语音识别功能的激光投影系统
CN112073958A (zh) * 2020-09-15 2020-12-11 苏州臻迪智能科技有限公司 连接建立方法、装置、移动终端及计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110138285A1 (en) * 2009-12-09 2011-06-09 Industrial Technology Research Institute Portable virtual human-machine interaction device and operation method thereof
CN102780864A (zh) * 2012-07-03 2012-11-14 深圳创维－Rgb电子有限公司 一种基于投影菜单的电视遥控方法、装置及电视
CN102819315A (zh) * 2012-07-23 2012-12-12 中兴通讯股份有限公司 一种3d人机交互方法及系统
CN103092437A (zh) * 2012-12-13 2013-05-08 同济大学 一种基于图像处理技术的便捷式触控交互系统
CN103428339A (zh) * 2012-05-17 2013-12-04 西安闻泰电子科技有限公司 通过语音输入控制手机进行操作的方法

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6204852B1 (en) * 1998-12-09 2001-03-20 Lucent Technologies Inc. Video hand image three-dimensional computer interface
US8472665B2 (en) * 2007-05-04 2013-06-25 Qualcomm Incorporated Camera-based user input for compact devices
US7916126B2 (en) * 2007-06-13 2011-03-29 Apple Inc. Bottom-up watershed dataflow method and region-specific segmentation based on historic data to identify patches on a touch sensor panel
JP4840620B2 (ja) * 2009-04-30 2011-12-21 株式会社デンソー 車載用電子機器操作装置
US20110074675A1 (en) * 2009-09-29 2011-03-31 Nokia Corporation Method and apparatus for initiating a feature based at least in part on the tracked movement
JP5743416B2 (ja) * 2010-03-29 2015-07-01 ソニー株式会社 情報処理装置、情報処理方法、およびプログラム
US9122307B2 (en) * 2010-09-20 2015-09-01 Kopin Corporation Advanced remote control of host application using motion and voice commands
US9135503B2 (en) * 2010-11-09 2015-09-15 Qualcomm Incorporated Fingertip tracking for touchless user interface
KR101646616B1 (ko) * 2010-11-30 2016-08-12 삼성전자주식회사 객체 제어 장치 및 방법
US8928589B2 (en) * 2011-04-20 2015-01-06 Qualcomm Incorporated Virtual keyboards and methods of providing the same
CN102200834B (zh) * 2011-05-26 2012-10-31 华南理工大学 面向电视控制的指尖鼠标交互方法
US9317130B2 (en) * 2011-06-16 2016-04-19 Rafal Jan Krepec Visual feedback by identifying anatomical features of a hand
US8228315B1 (en) * 2011-07-12 2012-07-24 Google Inc. Methods and systems for a virtual input device
WO2013012107A1 (ko) * 2011-07-19 2013-01-24 엘지전자 주식회사 전자 기기 및 그 제어 방법
US10140011B2 (en) * 2011-08-12 2018-11-27 Microsoft Technology Licensing, Llc Touch intelligent targeting
US8941560B2 (en) * 2011-09-21 2015-01-27 Google Inc. Wearable computer with superimposed controls and instructions for external device
US9459457B2 (en) * 2011-12-01 2016-10-04 Seebright Inc. Head mounted display with remote control
US20130147686A1 (en) * 2011-12-12 2013-06-13 John Clavin Connecting Head Mounted Displays To External Displays And Other Communication Networks
US9773345B2 (en) * 2012-02-15 2017-09-26 Nokia Technologies Oy Method and apparatus for generating a virtual environment for controlling one or more electronic devices
US9389420B2 (en) * 2012-06-14 2016-07-12 Qualcomm Incorporated User interface interaction for transparent head-mounted displays
US9213436B2 (en) * 2012-06-20 2015-12-15 Amazon Technologies, Inc. Fingertip location for gesture input
KR101958778B1 (ko) * 2012-08-31 2019-03-15 엘지전자 주식회사 헤드 마운트 디스플레이 및 이를 이용한 디지털 디바이스 제어 방법
KR101991133B1 (ko) * 2012-11-20 2019-06-19 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 헤드 마운트 디스플레이 및 그 제어 방법
US20140160162A1 (en) * 2012-12-12 2014-06-12 Dhanushan Balachandreswaran Surface projection device for augmented reality
CN103067766A (zh) * 2012-12-30 2013-04-24 深圳市龙视传媒有限公司 数字电视应用业务语音控制方法、系统及终端
US20140282239A1 (en) * 2013-03-15 2014-09-18 Lenovo (Singapore) Pte, Ltd. Selecting a touch screen hot spot
US10846942B1 (en) * 2013-08-29 2020-11-24 Ultrahaptics IP Two Limited Predictive information for free space gesture control and communication
KR101465894B1 (ko) * 2013-09-13 2014-11-26 성균관대학교산학협력단 손가락에 부착한 마커를 이용하여 제어 명령을 생성하는 이동 단말 및 손가락에 부착한 마커를 이용하여 단말에서 제어 명령을 생성하는 방법
CN105635776B (zh) * 2014-11-06 2019-03-01 深圳Tcl新技术有限公司 虚拟操作界面遥控控制方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110138285A1 (en) * 2009-12-09 2011-06-09 Industrial Technology Research Institute Portable virtual human-machine interaction device and operation method thereof
CN103428339A (zh) * 2012-05-17 2013-12-04 西安闻泰电子科技有限公司 通过语音输入控制手机进行操作的方法
CN102780864A (zh) * 2012-07-03 2012-11-14 深圳创维－Rgb电子有限公司 一种基于投影菜单的电视遥控方法、装置及电视
CN102819315A (zh) * 2012-07-23 2012-12-12 中兴通讯股份有限公司 一种3d人机交互方法及系统
CN103092437A (zh) * 2012-12-13 2013-05-08 同济大学 一种基于图像处理技术的便捷式触控交互系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3217637A4 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105898256A (zh) * 2016-05-30 2016-08-24 佛山市章扬科技有限公司 一种动作识别无屏电视
CN110427148A (zh) * 2019-07-26 2019-11-08 苏州蜗牛数字科技股份有限公司 一种语音触发屏幕的方法
CN114399896A (zh) * 2020-09-25 2022-04-26 汉桑（南京）科技有限公司 一种根据遥控设备图像配置遥控功能数据的方法和系统
CN114399896B (zh) * 2020-09-25 2023-04-07 汉桑(南京)科技股份有限公司 一种根据遥控设备图像配置遥控功能数据的方法和系统

Also Published As

Publication number Publication date
EP3217637A4 (en) 2018-06-06
CN105635776B (zh) 2019-03-01
EP3217637A1 (en) 2017-09-13
US20160378194A1 (en) 2016-12-29
CN105635776A (zh) 2016-06-01
US10429936B2 (en) 2019-10-01

Similar Documents

Publication Publication Date Title
WO2016070688A1 (zh) 虚拟操作界面遥控控制方法及系统
CA3016921C (en) System and method for deep learning based hand gesture recognition in first person view
EP3965003A1 (en) Image processing method and device
WO2020216054A1 (zh) 视线追踪模型训练的方法、视线追踪的方法及装置
CN103353935B (zh) 一种用于智能家居系统的3d动态手势识别方法
WO2017054605A1 (zh) 一种图片的处理方法和装置
WO2018121428A1 (zh) 一种活体检测方法、装置及存储介质
WO2019120029A1 (zh) 智能调节屏幕亮度的方法、装置、存储介质及移动终端
WO2017181769A1 (zh) 一种人脸识别方法、装置和系统、设备、存储介质
CN106774850B (zh) 一种移动终端及其交互控制方法
JP2019532543A (ja) 制御システムならびに制御処理方法および装置
CN108712603B (zh) 一种图像处理方法及移动终端
WO2019052329A1 (zh) 人脸识别方法及相关产品
CN107231470B (zh) 图像处理方法、移动终端及计算机可读存储介质
CN104410883A (zh) 一种移动可穿戴非接触式交互系统与方法
CN105787884A (zh) 一种图像处理方法及电子设备
WO2019011098A1 (zh) 解锁控制方法及相关产品
WO2020078319A1 (zh) 基于手势的操控方法及终端设备
WO2016019673A1 (zh) 虚拟键盘系统及其键入方法
WO2017152592A1 (zh) 一种移动终端应用操作方法及移动终端
WO2019061466A1 (zh) 一种飞行控制方法、遥控装置、遥控系统
CN104065949A (zh) 一种电视虚拟触控方法及系统
CN112700568B (zh) 一种身份认证的方法、设备及计算机可读存储介质
WO2024040861A1 (zh) 操作权限的控制方法、控制系统、电子设备及存储介质
CN110443752B (zh) 一种图像处理方法和移动终端

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15857359

Country of ref document: EP

Kind code of ref document: A1

REEP Request for entry into the european phase

Ref document number: 2015857359

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 15125134

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE