WO2020132831A1 - Systems and methods for pairing devices using visual recognition - Google Patents
Systems and methods for pairing devices using visual recognition Download PDFInfo
- Publication number
- WO2020132831A1 WO2020132831A1 PCT/CN2018/123202 CN2018123202W WO2020132831A1 WO 2020132831 A1 WO2020132831 A1 WO 2020132831A1 CN 2018123202 W CN2018123202 W CN 2018123202W WO 2020132831 A1 WO2020132831 A1 WO 2020132831A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- peripheral device
- peripheral
- host device
- host
- processor
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/80—Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
- G06F13/22—Handling requests for interconnection or transfer for access to input/output bus using successive scanning, e.g. polling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B17/00—Monitoring; Testing
- H04B17/30—Monitoring; Testing of propagation channels
- H04B17/309—Measuring or estimating channel quality parameters
- H04B17/318—Received signal strength
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B7/00—Radio transmission systems, i.e. using radiation field
- H04B7/24—Radio transmission systems, i.e. using radiation field for communication between two or more posts
- H04B7/26—Radio transmission systems, i.e. using radiation field for communication between two or more posts at least one of which is mobile
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/52—Network services specially adapted for the location of the user terminal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W12/00—Security arrangements; Authentication; Protecting privacy or anonymity
- H04W12/50—Secure pairing of devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W12/00—Security arrangements; Authentication; Protecting privacy or anonymity
- H04W12/60—Context-dependent security
- H04W12/63—Location-dependent; Proximity-dependent
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/023—Services making use of location information using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Definitions
- This disclosure relates generally to pairing a peripheral electronic device to a host electronic device.
- peripheral devices have gained popularity and spread all across user homes. These peripheral devices may include small devices to bigger mainstream home appliances. Besides earphones, various other devices like virtual reality peripherals, mousing devices, keyboards, smart home devices, and many more are part of our daily life. However, despite such ubiquitous popularity, the process of pairing the peripheral devices, for example, a smartphone with wireless headphones, is not standardized and often requires pressing certain buttons for a certain time in a certain order, and may require an unnecessary look into the user manual.
- the present disclosure describes systems and methods for pairing a peripheral electronic device to a host electronic device using visual recognition and/or machine learning (e.g., deep learning) .
- the host device may receive an indication of a peripheral device via a sensor (e.g., a camera) or as a result of searching for the peripheral device (e.g., due startup of a related application or periodic scanning) .
- the host device may also receive an image of the peripheral device (e.g., captured via the sensor) , and determine a visual distance to the peripheral device based on the image.
- the host device may also determine a signal strength of the peripheral device, and determine a signal distance to the peripheral device based on the signal strength.
- the host device may pair the peripheral device to the host device if the visual distance and the signal distance are approximately equal.
- FIG. 1 is a block diagram of a peripheral device pairing system that may use visual recognition and/or machine learning, according to embodiments of the present disclosure
- FIG. 2 is a flowchart illustrating a process for pairing a host device of FIG. 1 with a peripheral device, according to embodiments of the present disclosure
- FIG. 3 is a perspective diagram of the host device of FIG. 1 displaying a prompt asking a user whether to pair with a peripheral device, according to embodiments of the present disclosure
- FIG. 4 is a perspective diagram of the host device of FIG. 1 determining visual and signal distances to a peripheral device, according to embodiments of the present disclosure
- FIG. 5 is a perspective diagram of the host device of FIG. 1 completing a pairing process with a peripheral device, according to embodiments of the present disclosure
- FIG. 6 is a perspective diagram of the host device of FIG. 1 focusing on a speaker peripheral device instead of a keyboard peripheral device, according to embodiments of the present disclosure
- FIG. 7 is a perspective diagram of the host device of FIG. 1 capturing an image of both the speaker peripheral device and the keyboard peripheral device of FIG. 6, according to embodiments of the present disclosure
- FIG. 8 is a perspective diagram of the host device of FIG. 1 capturing an image of a user wearing a peripheral device, according to embodiments of the present disclosure.
- FIG. 9 is a perspective diagram of an example host device capturing an image of a peripheral device to be paired with, according to embodiments of the present disclosure.
- a wireless peripheral electronic device may be paired to a host electronic device (e.g., a smartphone, wearable device, virtual reality or mixed reality headset, tablet, desktop computer, laptop computer, or other computing device) to enable the host device to control and use the peripheral device.
- a host electronic device e.g., a smartphone, wearable device, virtual reality or mixed reality headset, tablet, desktop computer, laptop computer, or other computing device
- To pair the peripheral device a user often manually places both the peripheral device and the host device in respective pairing modes. For example, the user may navigate through a configuration menu on the host device to place the host device in a pairing mode. The host device may then search for pairable devices. The user may also depress a button of the peripheral device for three seconds to place the peripheral device in a pairing mode.
- the host device discovers the peripheral device, the user may often manually select the peripheral device from a list of devices, and then confirm that the peripheral device is the device the user desires to be paired to the host device. Once confirmed, the host device may finally pair to the peripheral
- the pairing may be performed by placing the host device in a passcode entry mode to enter a passcode associated with the peripheral device, in a scanning mode to scan a QR code on the peripheral device with a camera of the host device, or in a network discovery mode to discover the peripheral device via connectivity of the peripheral device to a network.
- a user may perform additional actions to complete the pairing.
- the present disclosure describes systems and methods for pairing a peripheral electronic device to a host electronic device using visual recognition and/or machine learning (e.g., deep learning) .
- the host device may receive an indication of a peripheral device via a sensor (e.g., a camera) or as a result of searching for the peripheral device (e.g., due startup of a related application or periodic scanning) .
- the host device may also receive an image of the peripheral device (e.g., captured via the sensor) , and determine a visual distance to the peripheral device based on the image.
- the host device may also determine a signal strength of the peripheral device, and determine a signal distance to the peripheral device based on the signal strength.
- the host device may pair the peripheral device to the host device if the visual distance and the signal distance are approximately equal.
- the host device may determine that it is not a user’s intention to pair these other peripheral devices as the signal distances of these other devices do not approximately match the visual distance of the peripheral device.
- the image of the peripheral device also includes other peripheral devices.
- the host device may employ any suitable techniques, including deep learning, to determine the peripheral device that the user intends to pair with. For example, the host device may determine whether any of the peripheral devices have been paired with before, and pair with the peripheral device that has been paired with before.
- the host device may additionally or alternatively determine or receive indications of how the peripheral device may be used (e.g., via open software applications or a most recently used software application on the host device or via the image showing the peripheral devices) , and pair with the peripheral device based on the indications.
- the host device may use a weighting or confidence factor system to determine which peripheral device to pair with based on, for example, the distance to the peripheral device, history and/or frequency of pairing with the peripheral device, indications of how the peripheral device may be used, or any other suitable factor.
- FIG. 1 is a block diagram of a peripheral device pairing system 10 that may use visual recognition and deep learning, according to embodiments of the present disclosure.
- the system 10 includes a host device 12 that may be paired with a peripheral device 14.
- the host device 12 may include any suitable electronic device that may be paired with the peripheral device 14, such as a cell phone, smartphone, wearable device, virtual reality or mixed reality headset, tablet, desktop computer, laptop computer, or any other suitable computing device.
- the peripheral device 14 may include any suitable electronic device that may be paired with the host device 12, such as a set of earphones, headphones, speakers, a keyboard, mousing device, car stereo, printer, webcam, smart home device, or any other suitable electronic device.
- the host device 12 may include a controller 16 that facilitates pairing with the peripheral device 14.
- the controller 16 may include one or more processors 18 (e.g., processing circuitry) and one or more memory devices 20 (which may include one or more storage devices) .
- the processor 18 may execute software programs and/or instructions to facilitate pairing with the peripheral device 14.
- the processor 18 may include multiple microprocessors, one or more application specific integrated circuits (ASICS) , and/or one or more reduced instruction set (RISC) processors.
- the memory device 20 may store machine-readable and/or processor-executable instructions (e.g., firmware or software) for the processor 18 to execute, such as instructions that facilitate pairing with the peripheral device 14.
- the memory device 20 may store, for example, control software, look up tables, configuration data, and so forth, to facilitate pairing with the peripheral device 14.
- the processor 18 and the memory device 20 may be external to the controller 16.
- the memory device 20 may include a tangible, non-transitory, machine-readable-medium, such as a volatile memory (e.g., a random access memory (RAM) ) and/or a nonvolatile memory (e.g., a read-only memory (ROM) , flash memory, hard drive, and/or any other suitable optical, magnetic, or solid-state storage medium) .
- a volatile memory e.g., a random access memory (RAM)
- a nonvolatile memory e.g., a read-only memory (ROM) , flash memory, hard drive, and/or any other suitable optical, magnetic, or solid-state storage medium
- the controller 16 may receive image data from a sensor 22 of the host device 12.
- the sensor 22 may be a visual sensor that detects and/or captures images or videos of the peripheral device 14, and send the images or videos to the controller 16 as image data.
- the visual sensor 22 may be a camera, such as a monocular, stereoscopic, or depth camera, or other suitable image capture device.
- the system 10 may include an object recognition engine 24 that identifies the peripheral device 14 in the image data using image/object recognition techniques.
- the object recognition engine 24 may use object recognition techniques to determine that the peripheral device 14 in the image data is a set of earphones, headphones, speakers, a keyboard, mousing device, car stereo, printer, webcam, or smart home device.
- the object recognition engine 24 may also use object recognition techniques to determine that other objects in the image data are not peripheral devices that may be paired with the host device 12 (such as coffee cups, pencils, paper, or the like) .
- the object recognition engine 24 may send indications of objects recognized in the image data to the controller 16, which may then associate the peripheral device 14 with an identified object.
- the controller 16 may then determine a distance to (or depth of) the peripheral device 14 in the image data using image distance determination techniques.
- the controller 16 may also include a receiver 26 and antenna 28 that enable receiving information from the peripheral device 14.
- the receiver 26 and the antenna 28 may operate using any suitable wireless communication protocol used to pair devices, such as Bluetooth, Bluetooth Low Energy (BLE) , WiFi, WiMax, or ZigBee.
- the antenna 28 may receive information related to a distance to the peripheral device 14 or information that may be used to determine the distance to the peripheral device 14. That is, the antenna 28 may receive a radio signal sent from the peripheral device 14, and the receiver 26 and/or the controller 16 may determine an indication of power present in the radio signal, such as a measurement or value of the power present in the radio signal.
- the receiver 26 may then send the measurement or value of the power present in the radio signal to the controller 16.
- the measurement or value of the power present in the radio signal may be a Received Signal Strength Indicator (RSSI) value.
- RSSI Received Signal Strength Indicator
- the controller 16 may then convert the RSSI value of the peripheral device 14 to a distance value to determine a distance from the host device 12 to the peripheral device 14. For example, the controller 16 may use the following equation that relates distance (d) and RSSI:
- the host device 12 may also include a deep learning engine 30 that facilitates determining the peripheral device 14 to pair with.
- the deep learning engine 30 may use any suitable machine learning techniques or architecture, such as deep neural networks, deep belief networks or recurrent neural networks.
- the host device 12 may determine there are multiple peripheral devices 14 that may be paired with (e.g., that are at approximately the same distance away from the host device 12) .
- the deep learning engine 30 may determine whether each peripheral device 14 has been paired before (e.g., based on a stored list of devices that have been paired with the host device 12) , and determine that the peripheral device 14 that has been paired with more frequently and/or more recently is more likely the peripheral device 14 a user desires to pair with the host device 12.
- the deep learning engine 30 may alternatively or additionally determine or receive one or more indications of how the peripheral device may be used (e.g., via open software applications or a most recently used software application on the host device 12 or via the image showing the peripheral devices) , and determine that the peripheral device 14 that is associated with such indications is more likely the peripheral device 14 that a user desires to pair with the host device 12.
- the deep learning engine 30 may determine that a word processing, email, messaging, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that a peripheral device associated with such a software application (e.g., a keyboard) is more likely the peripheral device 14 that a user desires to pair with the host device 12 than a peripheral device not associated with such an application (e.g., a set of earphones) .
- a peripheral device associated with such a software application e.g., a keyboard
- a peripheral device not associated with such an application e.g., a set of earphones
- the deep learning engine 30 may determine that an audio, video, music, movie, television, streaming, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that a peripheral device associated with such a software application (e.g., a set of earphones) is more likely the peripheral device 14 that a user desires to pair with the host device 12 than a peripheral device not associated with such an application (e.g., a set of earphones keyboard)
- a peripheral device associated with such a software application e.g., a set of earphones
- the deep learning engine 30 may send an indication of the peripheral device 14 that the deep learning engine 30 determines a user desires to pair with the host device 12 or that has the highest likelihood of being the peripheral device 14 that the user desires to pair with the host device 12.
- the deep learning engine 30 and/or the controller 16 may use a weighting or confidence factor system to determine which peripheral device to pair with based on, for example, the distance to the peripheral device, history and/or frequency of pairing with the peripheral device, indications of how the peripheral device may be used, or any other suitable factor.
- the host device 12 may also include a pairing engine 32 that facilitates pairing the host device 12 with the peripheral device 14.
- the pairing engine 32 may use the antenna 28, the receiver 26, and/or any other components of the host device 12 (e.g., such as a transmitter) , as well as components of the peripheral device (e.g., an antenna, a receiver, and/or a transmitter) to pair the host device 12 with the peripheral device 14.
- engine may include and/or be implemented in hardware (e.g., circuitry) , software (e.g., instructions for execution by a processor) , or a combination of the two.
- each or any of the object recognition engine 24, the deep learning engine 30, and the pairing engine 32 may be part of and internal to the controller 16 (e.g., in the form of circuitry of the processor 18 and/or software instructions stored in the memory device 20) .
- FIG. 2 is a flowchart illustrating a process 40 for pairing the host device 12 of FIG. 1 with the peripheral device 14, according to embodiments of the present disclosure. While the process 40 is described as being performed by the processor 18, it should be understood that the process 40 may be performed by any suitable device that may pair with the peripheral device 14. Furthermore, while the process 40 is described using steps in a specific sequence, it should be understood that the present disclosure contemplates that the described steps may be performed in different sequences than the sequence illustrated, and certain described steps may be skipped or not performed altogether. In some embodiments, the process 40 may be implemented by executing instructions stored in a tangible, non-transitory, computer-readable medium, such as the memory device 20, using any suitable processing circuitry, such as the processor 18.
- the processor 18 receives an indication of a peripheral device 14.
- the visual sensor 22 e.g., camera
- the visual sensor 22 may be activated.
- a camera or image capture software application on the host device 12 may be open.
- the visual sensor 22 may capture an image of the peripheral device 14, and send it to the processor 18, which may identify the peripheral device 14 using the object recognition engine 24.
- a software application on the host device 12 associated with a peripheral device 14 may be opened, which may cause the visual sensor 22 to search for the peripheral device 14.
- a user may open a word processing, email, messaging, or other software application associated with a keyboard or mousing device, and, as a result, the processor 18 may automatically search for the keyboard or mousing device.
- the user may open an audio, video, music, movie, television, streaming, or other software application associated with a set of earphones, headphones, speakers, or car stereo, and, as a result, the processor 18 may automatically search for the set of earphones, headphones, speakers, or car stereo.
- the processor 18 may also or alternatively periodically (e.g., every 30 seconds, every minute, or any other suitable period of time) search for peripheral devices 14.
- the processor 18 may automatically search for peripheral devices 14 when a user places the host device 12 in a pairing mode and/or opens a configuration menu associated with pairing peripheral devices 14.
- the peripheral device 14 may be activated and/or in a pairing mode.
- an indicator 72 e.g., a light or light-emitting diode (LED)
- LED light or light-emitting diode
- the processor 18 may display a prompt that asks a user whether to pair with the peripheral device 14.
- FIG. 3 is a perspective diagram of the host device 12 of FIG. 1 displaying a prompt 70 asking a user whether to pair with the peripheral device 14 (e.g., a headset) , according to embodiments of the present disclosure.
- the prompt 70 may be displayed on a display 68 of the host device 12 as a result of the processor 18 receiving an indication of a peripheral device 14.
- the processor 18 receives an image of the peripheral device 14. That is, the visual sensor 22 may automatically capture an image of the peripheral device 14 in response to receiving the indication of the peripheral device 14 from process block 42.
- the processor 18 determines a visual distance to the peripheral device 14 based on the image. For example, the processor 18 may determine a distance to (or depth of) the peripheral device 14 in the image data from the host device 12 using image distance determination techniques. It should be understood that the processor 18 may, in some embodiments, use any suitable software application or software applications, such as a third party distance-determining software application, to determine the distance to the peripheral device 14. For example, the processor 18 may execute instructions of a third party software application (e.g., stored in the memory device 20) to capture an image of the peripheral device 14 and/or determine a distance to the peripheral device 14 capture in the image.
- a third party software application e.g., stored in the memory device 20
- non-visual sensors may be used to determine the distance to the peripheral device 14.
- the non-visual sensors may include, for example, audio sensors, infrared sensors, sonar sensors, laser sensors, ultrasonic sensors, and the like.
- an ultrasonic sensor may be used to determine the distance to a peripheral device 14 by using a transducer to send an ultrasonic pulse to the peripheral device 14 and determine the distance to the peripheral device 14 based on receiving a returning ultrasonic pulse.
- the processor 18 determines or receives a signal strength of the peripheral device 14.
- the receiver 26 and antenna 28 of the host device 12 may receive and/or determine an indication of power present in a radio signal sent from the peripheral device 14, and the receiver 26 and/or the processor 18 may determine a measurement or value of the power present in the radio signal.
- the measurement or value of the power present in the radio signal may be a Received Signal Strength Indicator (RSSI) value.
- RSSI Received Signal Strength Indicator
- the processor 18 determines a signal distance to the peripheral device 14 based on the signal strength.
- the processor 18 may convert a measurement or value of the power present in the radio signal to a distance associated with the peripheral device 14. For example, when the measurement or value of the power present in the radio signal is an RSSI value, the processor 18 may use Equation 1 above to convert the RSSI value to the signal distance.
- the processor 18 determines whether the visual distance and the signal distance are approximately equal. For instance, the processor 18 may determine the visual distance and the signal distance are approximately equal when the visual distance and the signal distance are within a suitable margin of error (e.g., 2%, 5%, 10%, or the like) .
- a suitable margin of error e.g., 2%, 5%, 10%, or the like
- FIG. 4 is a perspective diagram of the host device 12 of FIG. 1 determining visual and signal distances to the peripheral device 14, according to embodiments of the present disclosure.
- a user directs or focuses the visual sensor 22 (e.g., camera) of the host device 12 at the peripheral device 14, such that the peripheral device 14 is displayed on a display 68 of the host device 12.
- the processor 18 may determine the visual distance 80 to the peripheral device 14 (e.g., from the image data using image distance determination techniques) and the signal distance 82 to the peripheral device 14 (e.g., based on an RSSI value 84 of the peripheral device 14) .
- the processor 18 determines that the visual distance and the signal distance are approximately equal, in process block 54, the processor 18 pairs the peripheral device 14. In some embodiments, the processor 18 may confirm with the user that the peripheral device 14 should be paired with the host device 12 (e.g., for security or privacy reasons) .
- FIG. 5 is a perspective diagram of the host device 12 of FIG. 1 completing the pairing process with the peripheral device 14, according to embodiments of the present disclosure.
- the processor 18 may display a prompt 92 to indicate to the user that the peripheral device 14 is paired. In this manner, the process 40 enables pairing the host device 12 with the peripheral device 14.
- FIG. 4 illustrates a keyboard peripheral device 86 that may be pairable with the host device 12. That is, the keyboard peripheral device 86 may, like the headset peripheral device 14, emit a radio signal with an RSSI value 88. As such, the processor 18 may also determine a signal distance 90 to the keyboard peripheral device 86 based on the RSSI value 88 of the keyboard peripheral device 86.
- the visual distance 80 to the headset peripheral device 14 may not approximately equal the signal distance 90 to the keyboard peripheral device 86, and thus the processor 18 may not pair the keyboard peripheral device 86.
- FIG. 6 is a perspective diagram of the host device 12 of FIG. 1 focusing on a speaker peripheral device 100 instead of a keyboard peripheral device 102, according to embodiments of the present disclosure.
- the peripheral devices 100, 102 may both be approximately the same visual distance (e.g., 104) from the host device 12 (as perceived by the visual sensor 22 of the host device 12) .
- the peripheral devices 100, 102 may both be approximately the same signal distance (e.g., 106, 108, respectively) from the host device 12 (based on respective RSSI values 110, 112 of the peripheral devices 100, 102) .
- the processor 18 may identify the peripheral device that the user desires to pair as the speaker peripheral device 100. As such, the processor 18 may pair with the speaker peripheral device 100 (e.g., in process block 54) .
- FIG. 7 is a perspective diagram of the host device 12 of FIG. 1 capturing an image of both the speaker peripheral device 100 and the keyboard peripheral device 102 of FIG. 6, according to embodiments of the present disclosure.
- the deep learning engine 30 may determine whether each of the peripheral devices 100, 102 has been paired before (e.g., based on a stored list of devices that have been paired with the host device 12) , and determine that the peripheral device that has been paired with more frequently and/or more recently is more likely the peripheral device a user desires to pair with the host device 12.
- the processor 18 may use the deep learning engine 30 to determine or receive indications of how the peripheral device desired to be paired with may be used (e.g., via open software applications or a most recently used software application on the host device 12 or via the image showing the peripheral devices) , and determine that the peripheral device that is associated with such indications is more likely the peripheral device that a user desires to pair with the host device 12.
- the deep learning engine 30 may determine that a word processing, email, messaging, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that the keyboard peripheral device 102 is associated with such a software application, and thus is more likely the peripheral device that the user desires to pair with the host device 12 than the speaker peripheral device 100.
- the deep learning engine 30 may determine that an audio, video, music, movie, television, streaming, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that the speaker peripheral device 100 is associated with such a software application, and thus is more likely the peripheral device that the user desires to pair with the host device 12 than the keyboard peripheral device 102.
- the indications of how the peripheral device may be used may be provided in the image data captured by the visual sensor 22.
- one indication may be a proximity of the peripheral device to the user, such as if the user is wearing or holding the peripheral device.
- FIG. 8 is a perspective diagram of the host device 12 of FIG. 1 capturing an image of a user wearing a peripheral device 120 (e.g., a headset) , according to embodiments of the present disclosure.
- the host device 12 uses a rear-facing or user-facing visual sensor 22 (e.g., a camera) .
- the processor 18 may determine that the user is wearing the peripheral device 120 using the object recognition engine 24.
- the processor 18 may determine that, because the user is wearing the peripheral device 120, even if there are other pairable peripheral devices nearby, if the peripheral device 120 is unpaired, the user desires to pair the host device 12 with the peripheral device 120. As such, the processor 18 may pair the peripheral device 120 with the host device 12.
- FIG. 9 is a perspective diagram of an example host device 130 capturing an image of a peripheral device 132 (e.g., a wrist controller) to be paired with, according to embodiments of the present disclosure.
- the host device 130 is a virtual reality or mixed reality headset.
- the user directs his focus (and the focus of a visual sensor 22 or camera) to the peripheral device 132.
- the processor 18 of the host device 130 may determine that the user is holding the peripheral device 132 using the object recognition engine 24. Using the deep learning engine 30, the processor 18 may determine that, because the user is holding the peripheral device 132, even if there are other pairable peripheral devices nearby, if the peripheral device 132 is unpaired, the user desires to pair the host device 130 with the peripheral device 132.
- the processor 18 may pair the peripheral device 132 with the host device 130.
- the peripheral device 132 may include any suitable peripheral devices, including any suitable virtual reality or mixed reality peripheral devices, such as virtual reality or mixed reality input controllers, handheld units, and/or gloves (e.g., any of which may include input devices such as track pads, buttons, triggers, analog sticks, and/or the like) .
- the processor 18 may also or alternatively enable a user to provide an indication of the peripheral device that the user desires to pair with the host device 12.
- the processor 18 may enable the user to point at, snap his or her fingers at, draw a circle around, or perform any other suitable indicating gesture at the peripheral device the user desires to pair with the host device 12.
- the user points his or her fingers at the speaker peripheral device 100, indicating that the speaker peripheral device 100 is the peripheral device he or she desires to be paired with the host device 12.
- the processor 18 may pair the speaker peripheral device 100 with the host device 12.
- the processor 18 may receive an identifier of the peripheral device 14.
- the identifier may include device specific characteristics, such as a device manufacturer, a device type, a device model, or the like.
- the processor 18 may determine the device specific characteristics, which may be used to, for example, identify the peripheral device 14 among additional peripheral devices in the image of the peripheral device 14.
- the deep learning engine 30 and/or the processor 18 may use a weighting or confidence factor system to determine which peripheral device to pair with based on, for example, the distance to the peripheral device, history and/or frequency of pairing with the peripheral device, indications of how the peripheral device may be used, or any other suitable factor.
- the processor 18 may receive the indication of the peripheral device 14, but the peripheral device 14 may not be in view. That is, the visual sensor 22 (e.g., camera) of the host device 12 at the peripheral device 14 may not detect the peripheral device 14, and as such may not display the peripheral device 14 on the display 68 of the host device 12.
- the processor 18 may determine an approximate location of the peripheral device 14 based on receiving the indication of power present in the radio signal sent from the peripheral device 14 and/or determining the measurement or value of the power present in the radio signal. The processor 18 may then indicate the approximate location of the peripheral device 14. For example, the processor 18 may display an arrow directed at or a box surrounding the approximate location of the peripheral device 14 on the display 68.
- the host device 12 may include a device usable by multiple users, such as a music, video, or other media streaming device provided by a service, and the peripheral device 14 may include a smartphone, wearable device, laptop computer, or other computing device of a user.
- the host device 12 may be a public streaming device (e.g., in an airport, library, museum, or any other suitable public facility) .
- the host device 12 may include or be coupled to a visual sensor (e.g., camera) that may receive the indication of the peripheral device 14 (e.g., by having the user hold and/or point to the peripheral device 14) .
- a visual sensor e.g., camera
- the visual sensor may be associated with a sign, graphic, or other indication that indicates that users interested in the streaming service should show hold their peripheral device 14 and/or point to the peripheral device 14.
- the processor 18 of the host device 12 may then pair the peripheral device 14 to the host device 12, thus enabling users to use the public streaming service.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
- Mobile Radio Communication Systems (AREA)
- Telephone Function (AREA)
- Image Analysis (AREA)
Abstract
The present disclosure is directed at pairing a host electronic device (12) with a peripheral electronic device (14) using visual recognition and deep learning techniques. In particular, the host device (12) may receive an indication of a peripheral device (14) via a camera (22) or as a result of searching for the peripheral device (14) (e.g., due startup of a related application or periodic scanning). The host device (12) may also receive an image of the peripheral device (14) (e.g., captured via the camera (22) ), and determine a visual distance (80) to the peripheral device (14) based on the image. The host device (12) may also determine a signal strength of the peripheral device (14), and determine a signal distance (82) to the peripheral device (14) based on the signal strength. The host device (12) may pair with the peripheral device (14) if the visual distance (80) and the signal distance (82) are approximately equal.
Description
This disclosure relates generally to pairing a peripheral electronic device to a host electronic device.
This section is intended to introduce the reader to various aspects of art that may be related to aspects of the present disclosure, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it may be understood that these statements are to be read in this light, and not as admissions of prior art.
Over the last few years, wireless peripheral devices have gained popularity and spread all across user homes. These peripheral devices may include small devices to bigger mainstream home appliances. Besides earphones, various other devices like virtual reality peripherals, mousing devices, keyboards, smart home devices, and many more are part of our daily life. However, despite such ubiquitous popularity, the process of pairing the peripheral devices, for example, a smartphone with wireless headphones, is not standardized and often requires pressing certain buttons for a certain time in a certain order, and may require an unnecessary look into the user manual.
SUMMARY
A summary of certain embodiments disclosed herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be set forth below.
The present disclosure describes systems and methods for pairing a peripheral electronic device to a host electronic device using visual recognition and/or machine learning (e.g., deep learning) . In particular, the host device may receive an indication of a peripheral device via a sensor (e.g., a camera) or as a result of searching for the peripheral device (e.g., due startup of a related application or periodic scanning) . The host device may also receive an image of the peripheral device (e.g., captured via the sensor) , and determine a visual distance to the peripheral device based on the image. The host device may also determine a signal strength of the peripheral device, and determine a signal distance to the peripheral device based on the signal strength. The host device may pair the peripheral device to the host device if the visual distance and the signal distance are approximately equal.
Various refinements of the features noted above may exist in relation to various aspects of the present disclosure. Further features may also be incorporated in these various aspects as well. These refinements and additional features may exist individually or in any combination. For instance, various features discussed below in relation to one or more of the illustrated embodiments may be incorporated into any of the above-described aspects of the present disclosure alone or in any combination. Again, the brief summary presented above is intended only to familiarize the reader with certain aspects and contexts of embodiments of the present disclosure without limitation to the claimed subject matter.
Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:
FIG. 1 is a block diagram of a peripheral device pairing system that may use visual recognition and/or machine learning, according to embodiments of the present disclosure;
FIG. 2 is a flowchart illustrating a process for pairing a host device of FIG. 1 with a peripheral device, according to embodiments of the present disclosure;
FIG. 3 is a perspective diagram of the host device of FIG. 1 displaying a prompt asking a user whether to pair with a peripheral device, according to embodiments of the present disclosure;
FIG. 4 is a perspective diagram of the host device of FIG. 1 determining visual and signal distances to a peripheral device, according to embodiments of the present disclosure;
FIG. 5 is a perspective diagram of the host device of FIG. 1 completing a pairing process with a peripheral device, according to embodiments of the present disclosure;
FIG. 6 is a perspective diagram of the host device of FIG. 1 focusing on a speaker peripheral device instead of a keyboard peripheral device, according to embodiments of the present disclosure;
FIG. 7 is a perspective diagram of the host device of FIG. 1 capturing an image of both the speaker peripheral device and the keyboard peripheral device of FIG. 6, according to embodiments of the present disclosure;
FIG. 8 is a perspective diagram of the host device of FIG. 1 capturing an image of a user wearing a peripheral device, according to embodiments of the present disclosure; and
FIG. 9 is a perspective diagram of an example host device capturing an image of a peripheral device to be paired with, according to embodiments of the present disclosure.
DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS
One or more specific embodiments will be described below. In an effort to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It may be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers’specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it may be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.
A wireless peripheral electronic device (e.g., a set of earphones, headphones, speakers, a keyboard, mousing device, car stereo, printer, webcam, or smart home device) may be paired to a host electronic device (e.g., a smartphone, wearable device, virtual reality or mixed reality headset, tablet, desktop computer, laptop computer, or other computing device) to enable the host device to control and use the peripheral device. To pair the peripheral device, a user often manually places both the peripheral device and the host device in respective pairing modes. For example, the user may navigate through a configuration menu on the host device to place the host device in a pairing mode. The host device may then search for pairable devices. The user may also depress a button of the peripheral device for three seconds to place the peripheral device in a pairing mode. Once the host device discovers the peripheral device, the user may often manually select the peripheral device from a list of devices, and then confirm that the peripheral device is the device the user desires to be paired to the host device. Once confirmed, the host device may finally pair to the peripheral device.
In some cases, the pairing may be performed by placing the host device in a passcode entry mode to enter a passcode associated with the peripheral device, in a scanning mode to scan a QR code on the peripheral device with a camera of the host device, or in a network discovery mode to discover the peripheral device via connectivity of the peripheral device to a network. However, in each case, a user may perform additional actions to complete the pairing.
The present disclosure describes systems and methods for pairing a peripheral electronic device to a host electronic device using visual recognition and/or machine learning (e.g., deep learning) . In particular, the host device may receive an indication of a peripheral device via a sensor (e.g., a camera) or as a result of searching for the peripheral device (e.g., due startup of a related application or periodic scanning) . The host device may also receive an image of the peripheral device (e.g., captured via the sensor) , and determine a visual distance to the peripheral device based on the image. The host device may also determine a signal strength of the peripheral device, and determine a signal distance to the peripheral device based on the signal strength. The host device may pair the peripheral device to the host device if the visual distance and the signal distance are approximately equal.
In some cases, there may be other peripheral devices within a searching radius of the host device. As such, the host device may determine that it is not a user’s intention to pair these other peripheral devices as the signal distances of these other devices do not approximately match the visual distance of the peripheral device. Moreover, there may be times that the image of the peripheral device also includes other peripheral devices. The host device may employ any suitable techniques, including deep learning, to determine the peripheral device that the user intends to pair with. For example, the host device may determine whether any of the peripheral devices have been paired with before, and pair with the peripheral device that has been paired with before. The host device may additionally or alternatively determine or receive indications of how the peripheral device may be used (e.g., via open software applications or a most recently used software application on the host device or via the image showing the peripheral devices) , and pair with the peripheral device based on the indications. In some embodiments, the host device may use a weighting or confidence factor system to determine which peripheral device to pair with based on, for example, the distance to the peripheral device, history and/or frequency of pairing with the peripheral device, indications of how the peripheral device may be used, or any other suitable factor.
With the foregoing in mind, FIG. 1 is a block diagram of a peripheral device pairing system 10 that may use visual recognition and deep learning, according to embodiments of the present disclosure. The system 10 includes a host device 12 that may be paired with a peripheral device 14. The host device 12 may include any suitable electronic device that may be paired with the peripheral device 14, such as a cell phone, smartphone, wearable device, virtual reality or mixed reality headset, tablet, desktop computer, laptop computer, or any other suitable computing device. The peripheral device 14 may include any suitable electronic device that may be paired with the host device 12, such as a set of earphones, headphones, speakers, a keyboard, mousing device, car stereo, printer, webcam, smart home device, or any other suitable electronic device.
The host device 12 may include a controller 16 that facilitates pairing with the peripheral device 14. The controller 16 may include one or more processors 18 (e.g., processing circuitry) and one or more memory devices 20 (which may include one or more storage devices) . The processor 18 may execute software programs and/or instructions to facilitate pairing with the peripheral device 14. Moreover, the processor 18 may include multiple microprocessors, one or more application specific integrated circuits (ASICS) , and/or one or more reduced instruction set (RISC) processors. The memory device 20 may store machine-readable and/or processor-executable instructions (e.g., firmware or software) for the processor 18 to execute, such as instructions that facilitate pairing with the peripheral device 14. As such, the memory device 20 may store, for example, control software, look up tables, configuration data, and so forth, to facilitate pairing with the peripheral device 14. In one embodiment, the processor 18 and the memory device 20 may be external to the controller 16. The memory device 20 may include a tangible, non-transitory, machine-readable-medium, such as a volatile memory (e.g., a random access memory (RAM) ) and/or a nonvolatile memory (e.g., a read-only memory (ROM) , flash memory, hard drive, and/or any other suitable optical, magnetic, or solid-state storage medium) .
The controller 16 may receive image data from a sensor 22 of the host device 12. The sensor 22 may be a visual sensor that detects and/or captures images or videos of the peripheral device 14, and send the images or videos to the controller 16 as image data. In some embodiments, the visual sensor 22 may be a camera, such as a monocular, stereoscopic, or depth camera, or other suitable image capture device.
The system 10 may include an object recognition engine 24 that identifies the peripheral device 14 in the image data using image/object recognition techniques. For example, the object recognition engine 24 may use object recognition techniques to determine that the peripheral device 14 in the image data is a set of earphones, headphones, speakers, a keyboard, mousing device, car stereo, printer, webcam, or smart home device. The object recognition engine 24 may also use object recognition techniques to determine that other objects in the image data are not peripheral devices that may be paired with the host device 12 (such as coffee cups, pencils, paper, or the like) . The object recognition engine 24 may send indications of objects recognized in the image data to the controller 16, which may then associate the peripheral device 14 with an identified object. The controller 16 may then determine a distance to (or depth of) the peripheral device 14 in the image data using image distance determination techniques.
The controller 16 may also include a receiver 26 and antenna 28 that enable receiving information from the peripheral device 14. The receiver 26 and the antenna 28 may operate using any suitable wireless communication protocol used to pair devices, such as Bluetooth, Bluetooth Low Energy (BLE) , WiFi, WiMax, or ZigBee. For example, the antenna 28 may receive information related to a distance to the peripheral device 14 or information that may be used to determine the distance to the peripheral device 14. That is, the antenna 28 may receive a radio signal sent from the peripheral device 14, and the receiver 26 and/or the controller 16 may determine an indication of power present in the radio signal, such as a measurement or value of the power present in the radio signal. The receiver 26 may then send the measurement or value of the power present in the radio signal to the controller 16. In some embodiments, the measurement or value of the power present in the radio signal may be a Received Signal Strength Indicator (RSSI) value.
The controller 16 may then convert the RSSI value of the peripheral device 14 to a distance value to determine a distance from the host device 12 to the peripheral device 14. For example, the controller 16 may use the following equation that relates distance (d) and RSSI:
where:
F
m=Fade margin
n=Path-Loss Exponent
P
0=Signal power (dBm) at zero distance
P
r=Signal power (dBm) at distance
f=signal frequency in MHz
The host device 12 may also include a deep learning engine 30 that facilitates determining the peripheral device 14 to pair with. The deep learning engine 30 may use any suitable machine learning techniques or architecture, such as deep neural networks, deep belief networks or recurrent neural networks. For example, the host device 12 may determine there are multiple peripheral devices 14 that may be paired with (e.g., that are at approximately the same distance away from the host device 12) . The deep learning engine 30 may determine whether each peripheral device 14 has been paired before (e.g., based on a stored list of devices that have been paired with the host device 12) , and determine that the peripheral device 14 that has been paired with more frequently and/or more recently is more likely the peripheral device 14 a user desires to pair with the host device 12.
The deep learning engine 30 may alternatively or additionally determine or receive one or more indications of how the peripheral device may be used (e.g., via open software applications or a most recently used software application on the host device 12 or via the image showing the peripheral devices) , and determine that the peripheral device 14 that is associated with such indications is more likely the peripheral device 14 that a user desires to pair with the host device 12. For example, the deep learning engine 30 may determine that a word processing, email, messaging, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that a peripheral device associated with such a software application (e.g., a keyboard) is more likely the peripheral device 14 that a user desires to pair with the host device 12 than a peripheral device not associated with such an application (e.g., a set of earphones) . As another example, the deep learning engine 30 may determine that an audio, video, music, movie, television, streaming, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that a peripheral device associated with such a software application (e.g., a set of earphones) is more likely the peripheral device 14 that a user desires to pair with the host device 12 than a peripheral device not associated with such an application (e.g., a set of earphones keyboard)
In some embodiments, the deep learning engine 30 may send an indication of the peripheral device 14 that the deep learning engine 30 determines a user desires to pair with the host device 12 or that has the highest likelihood of being the peripheral device 14 that the user desires to pair with the host device 12. In some embodiments, the deep learning engine 30 and/or the controller 16 may use a weighting or confidence factor system to determine which peripheral device to pair with based on, for example, the distance to the peripheral device, history and/or frequency of pairing with the peripheral device, indications of how the peripheral device may be used, or any other suitable factor.
The host device 12 may also include a pairing engine 32 that facilitates pairing the host device 12 with the peripheral device 14. In particular, the pairing engine 32 may use the antenna 28, the receiver 26, and/or any other components of the host device 12 (e.g., such as a transmitter) , as well as components of the peripheral device (e.g., an antenna, a receiver, and/or a transmitter) to pair the host device 12 with the peripheral device 14.
It should be understood that the term “engine” as used herein may include and/or be implemented in hardware (e.g., circuitry) , software (e.g., instructions for execution by a processor) , or a combination of the two. Moreover, in additional or alternative embodiments, each or any of the object recognition engine 24, the deep learning engine 30, and the pairing engine 32 may be part of and internal to the controller 16 (e.g., in the form of circuitry of the processor 18 and/or software instructions stored in the memory device 20) .
FIG. 2 is a flowchart illustrating a process 40 for pairing the host device 12 of FIG. 1 with the peripheral device 14, according to embodiments of the present disclosure. While the process 40 is described as being performed by the processor 18, it should be understood that the process 40 may be performed by any suitable device that may pair with the peripheral device 14. Furthermore, while the process 40 is described using steps in a specific sequence, it should be understood that the present disclosure contemplates that the described steps may be performed in different sequences than the sequence illustrated, and certain described steps may be skipped or not performed altogether. In some embodiments, the process 40 may be implemented by executing instructions stored in a tangible, non-transitory, computer-readable medium, such as the memory device 20, using any suitable processing circuitry, such as the processor 18.
As illustrated, in process block 42, the processor 18 receives an indication of a peripheral device 14. In some cases, the visual sensor 22 (e.g., camera) of the host device 12 may be activated. For example, a camera or image capture software application on the host device 12 may be open. The visual sensor 22 may capture an image of the peripheral device 14, and send it to the processor 18, which may identify the peripheral device 14 using the object recognition engine 24.
In additional or alternative embodiments, a software application on the host device 12 associated with a peripheral device 14 may be opened, which may cause the visual sensor 22 to search for the peripheral device 14. For example, a user may open a word processing, email, messaging, or other software application associated with a keyboard or mousing device, and, as a result, the processor 18 may automatically search for the keyboard or mousing device. Similarly, the user may open an audio, video, music, movie, television, streaming, or other software application associated with a set of earphones, headphones, speakers, or car stereo, and, as a result, the processor 18 may automatically search for the set of earphones, headphones, speakers, or car stereo.
The processor 18 may also or alternatively periodically (e.g., every 30 seconds, every minute, or any other suitable period of time) search for peripheral devices 14. In one embodiment, the processor 18 may automatically search for peripheral devices 14 when a user places the host device 12 in a pairing mode and/or opens a configuration menu associated with pairing peripheral devices 14. For the processor 18 to receive the indication of the peripheral device 14, the peripheral device 14 may be activated and/or in a pairing mode. In such cases, an indicator 72 (e.g., a light or light-emitting diode (LED) ) may indicate that the peripheral device 14 is in the pairing mode (e.g., the indicator 72 may blink) .
In some embodiments, the processor 18 may display a prompt that asks a user whether to pair with the peripheral device 14. For example, FIG. 3 is a perspective diagram of the host device 12 of FIG. 1 displaying a prompt 70 asking a user whether to pair with the peripheral device 14 (e.g., a headset) , according to embodiments of the present disclosure. The prompt 70 may be displayed on a display 68 of the host device 12 as a result of the processor 18 receiving an indication of a peripheral device 14.
In process block 44, the processor 18 receives an image of the peripheral device 14. That is, the visual sensor 22 may automatically capture an image of the peripheral device 14 in response to receiving the indication of the peripheral device 14 from process block 42.
In process block 46, the processor 18 determines a visual distance to the peripheral device 14 based on the image. For example, the processor 18 may determine a distance to (or depth of) the peripheral device 14 in the image data from the host device 12 using image distance determination techniques. It should be understood that the processor 18 may, in some embodiments, use any suitable software application or software applications, such as a third party distance-determining software application, to determine the distance to the peripheral device 14. For example, the processor 18 may execute instructions of a third party software application (e.g., stored in the memory device 20) to capture an image of the peripheral device 14 and/or determine a distance to the peripheral device 14 capture in the image.
While the present disclosure discusses using a visual sensor 22 to capture the image of the peripheral device 14, and the processor 28 determining a visual distance to the peripheral device 14, it should be understood that any suitable distance-determining technique may be used to determine the distance (e.g., a visual distance) to the peripheral device 14. That is, in some embodiments, non-visual sensors may be used to determine the distance to the peripheral device 14. The non-visual sensors may include, for example, audio sensors, infrared sensors, sonar sensors, laser sensors, ultrasonic sensors, and the like. As an example, an ultrasonic sensor may be used to determine the distance to a peripheral device 14 by using a transducer to send an ultrasonic pulse to the peripheral device 14 and determine the distance to the peripheral device 14 based on receiving a returning ultrasonic pulse.
In process block 48, the processor 18 determines or receives a signal strength of the peripheral device 14. In particular, the receiver 26 and antenna 28 of the host device 12 may receive and/or determine an indication of power present in a radio signal sent from the peripheral device 14, and the receiver 26 and/or the processor 18 may determine a measurement or value of the power present in the radio signal. In some embodiments, the measurement or value of the power present in the radio signal may be a Received Signal Strength Indicator (RSSI) value.
In process block 50, the processor 18 determines a signal distance to the peripheral device 14 based on the signal strength. In particular, the processor 18 may convert a measurement or value of the power present in the radio signal to a distance associated with the peripheral device 14. For example, when the measurement or value of the power present in the radio signal is an RSSI value, the processor 18 may use Equation 1 above to convert the RSSI value to the signal distance.
In decision block 52, the processor 18 determines whether the visual distance and the signal distance are approximately equal. For instance, the processor 18 may determine the visual distance and the signal distance are approximately equal when the visual distance and the signal distance are within a suitable margin of error (e.g., 2%, 5%, 10%, or the like) .
For example, FIG. 4 is a perspective diagram of the host device 12 of FIG. 1 determining visual and signal distances to the peripheral device 14, according to embodiments of the present disclosure. As illustrated, a user directs or focuses the visual sensor 22 (e.g., camera) of the host device 12 at the peripheral device 14, such that the peripheral device 14 is displayed on a display 68 of the host device 12. The processor 18 may determine the visual distance 80 to the peripheral device 14 (e.g., from the image data using image distance determination techniques) and the signal distance 82 to the peripheral device 14 (e.g., based on an RSSI value 84 of the peripheral device 14) .
If the processor 18 determines that the visual distance and the signal distance are approximately equal, in process block 54, the processor 18 pairs the peripheral device 14. In some embodiments, the processor 18 may confirm with the user that the peripheral device 14 should be paired with the host device 12 (e.g., for security or privacy reasons) .
If the processor 18 determines that the visual distance and the signal distance are not approximately equal, in process block 56, the processor 18 does not pair the peripheral device 14. In some embodiments, if the processor 18 does not pair the peripheral device 14, the user may still pair the peripheral device 14 to the host device 12 conventionally (e.g., by navigating through a configuration menu on the host device 12) . Keeping the foregoing in mind, FIG. 5 is a perspective diagram of the host device 12 of FIG. 1 completing the pairing process with the peripheral device 14, according to embodiments of the present disclosure. As illustrated, the processor 18 may display a prompt 92 to indicate to the user that the peripheral device 14 is paired. In this manner, the process 40 enables pairing the host device 12 with the peripheral device 14.
In some embodiments, there may be additional peripheral devices within a search radius of the host device 12, such that searching for pairable peripheral devices may result in finding these additional peripheral devices. For example, FIG. 4 illustrates a keyboard peripheral device 86 that may be pairable with the host device 12. That is, the keyboard peripheral device 86 may, like the headset peripheral device 14, emit a radio signal with an RSSI value 88. As such, the processor 18 may also determine a signal distance 90 to the keyboard peripheral device 86 based on the RSSI value 88 of the keyboard peripheral device 86. However, because the user directs or focuses the visual sensor 22 (e.g., camera) of the host device 12 at the headset peripheral device 14, the visual distance 80 to the headset peripheral device 14 may not approximately equal the signal distance 90 to the keyboard peripheral device 86, and thus the processor 18 may not pair the keyboard peripheral device 86.
In some cases, there may be multiple peripheral devices at approximately the same distance from the host device 12. As such, the processor 18 may determine the identity of the peripheral device 14 (e.g., using the object recognition engine 24) to determine the peripheral device 14 to pair with. For example, FIG. 6 is a perspective diagram of the host device 12 of FIG. 1 focusing on a speaker peripheral device 100 instead of a keyboard peripheral device 102, according to embodiments of the present disclosure. As illustrated, the peripheral devices 100, 102 may both be approximately the same visual distance (e.g., 104) from the host device 12 (as perceived by the visual sensor 22 of the host device 12) . Moreover, the peripheral devices 100, 102 may both be approximately the same signal distance (e.g., 106, 108, respectively) from the host device 12 (based on respective RSSI values 110, 112 of the peripheral devices 100, 102) . However, because the user directs or focuses the visual sensor 22 of the host device 12 at the speaker peripheral device 100 as shown in FIG. 6, the processor 18 may identify the peripheral device that the user desires to pair as the speaker peripheral device 100. As such, the processor 18 may pair with the speaker peripheral device 100 (e.g., in process block 54) .
There may be circumstances where more than one pairable device is shown in the image captured by the visual sensor 22. For example, in FIG. 7 is a perspective diagram of the host device 12 of FIG. 1 capturing an image of both the speaker peripheral device 100 and the keyboard peripheral device 102 of FIG. 6, according to embodiments of the present disclosure. In such a case, the deep learning engine 30 may determine whether each of the peripheral devices 100, 102 has been paired before (e.g., based on a stored list of devices that have been paired with the host device 12) , and determine that the peripheral device that has been paired with more frequently and/or more recently is more likely the peripheral device a user desires to pair with the host device 12.
Additionally or alternatively, the processor 18 may use the deep learning engine 30 to determine or receive indications of how the peripheral device desired to be paired with may be used (e.g., via open software applications or a most recently used software application on the host device 12 or via the image showing the peripheral devices) , and determine that the peripheral device that is associated with such indications is more likely the peripheral device that a user desires to pair with the host device 12. For example, the deep learning engine 30 may determine that a word processing, email, messaging, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that the keyboard peripheral device 102 is associated with such a software application, and thus is more likely the peripheral device that the user desires to pair with the host device 12 than the speaker peripheral device 100. As another example, the deep learning engine 30 may determine that an audio, video, music, movie, television, streaming, or other software application is an open, recently opened, recently used, or most recently used software application on the host device 12, and determine that the speaker peripheral device 100 is associated with such a software application, and thus is more likely the peripheral device that the user desires to pair with the host device 12 than the keyboard peripheral device 102.
In some embodiments, the indications of how the peripheral device may be used may be provided in the image data captured by the visual sensor 22. For example, one indication may be a proximity of the peripheral device to the user, such as if the user is wearing or holding the peripheral device. As an example, FIG. 8 is a perspective diagram of the host device 12 of FIG. 1 capturing an image of a user wearing a peripheral device 120 (e.g., a headset) , according to embodiments of the present disclosure. In this case, the host device 12 uses a rear-facing or user-facing visual sensor 22 (e.g., a camera) . The processor 18 may determine that the user is wearing the peripheral device 120 using the object recognition engine 24. Using the deep learning engine 30, the processor 18 may determine that, because the user is wearing the peripheral device 120, even if there are other pairable peripheral devices nearby, if the peripheral device 120 is unpaired, the user desires to pair the host device 12 with the peripheral device 120. As such, the processor 18 may pair the peripheral device 120 with the host device 12.
Similarly, FIG. 9 is a perspective diagram of an example host device 130 capturing an image of a peripheral device 132 (e.g., a wrist controller) to be paired with, according to embodiments of the present disclosure. In this case, the host device 130 is a virtual reality or mixed reality headset. As illustrated, the user directs his focus (and the focus of a visual sensor 22 or camera) to the peripheral device 132. The processor 18 of the host device 130 may determine that the user is holding the peripheral device 132 using the object recognition engine 24. Using the deep learning engine 30, the processor 18 may determine that, because the user is holding the peripheral device 132, even if there are other pairable peripheral devices nearby, if the peripheral device 132 is unpaired, the user desires to pair the host device 130 with the peripheral device 132. As such, the processor 18 may pair the peripheral device 132 with the host device 130. It should be understood that the peripheral device 132 may include any suitable peripheral devices, including any suitable virtual reality or mixed reality peripheral devices, such as virtual reality or mixed reality input controllers, handheld units, and/or gloves (e.g., any of which may include input devices such as track pads, buttons, triggers, analog sticks, and/or the like) .
The processor 18 may also or alternatively enable a user to provide an indication of the peripheral device that the user desires to pair with the host device 12. For example, the processor 18 may enable the user to point at, snap his or her fingers at, draw a circle around, or perform any other suitable indicating gesture at the peripheral device the user desires to pair with the host device 12. As illustrated in FIG. 7, the user points his or her fingers at the speaker peripheral device 100, indicating that the speaker peripheral device 100 is the peripheral device he or she desires to be paired with the host device 12. As such, the processor 18 may pair the speaker peripheral device 100 with the host device 12.
In some cases, where the indication of the peripheral device 14 is received based on activating and/or placing the peripheral device 14 in a pairing mode, the processor 18 may receive an identifier of the peripheral device 14. The identifier may include device specific characteristics, such as a device manufacturer, a device type, a device model, or the like. The processor 18 may determine the device specific characteristics, which may be used to, for example, identify the peripheral device 14 among additional peripheral devices in the image of the peripheral device 14.
In some embodiments, the deep learning engine 30 and/or the processor 18 may use a weighting or confidence factor system to determine which peripheral device to pair with based on, for example, the distance to the peripheral device, history and/or frequency of pairing with the peripheral device, indications of how the peripheral device may be used, or any other suitable factor.
In some circumstances, the processor 18 may receive the indication of the peripheral device 14, but the peripheral device 14 may not be in view. That is, the visual sensor 22 (e.g., camera) of the host device 12 at the peripheral device 14 may not detect the peripheral device 14, and as such may not display the peripheral device 14 on the display 68 of the host device 12. In some embodiments, the processor 18 may determine an approximate location of the peripheral device 14 based on receiving the indication of power present in the radio signal sent from the peripheral device 14 and/or determining the measurement or value of the power present in the radio signal. The processor 18 may then indicate the approximate location of the peripheral device 14. For example, the processor 18 may display an arrow directed at or a box surrounding the approximate location of the peripheral device 14 on the display 68.
In some embodiments, the host device 12 may include a device usable by multiple users, such as a music, video, or other media streaming device provided by a service, and the peripheral device 14 may include a smartphone, wearable device, laptop computer, or other computing device of a user. For example, the host device 12 may be a public streaming device (e.g., in an airport, library, museum, or any other suitable public facility) . The host device 12 may include or be coupled to a visual sensor (e.g., camera) that may receive the indication of the peripheral device 14 (e.g., by having the user hold and/or point to the peripheral device 14) . For example, the visual sensor may be associated with a sign, graphic, or other indication that indicates that users interested in the streaming service should show hold their peripheral device 14 and/or point to the peripheral device 14. Based on performing the process 40, for example, the processor 18 of the host device 12 may then pair the peripheral device 14 to the host device 12, thus enabling users to use the public streaming service.
While the embodiments set forth in the present disclosure may be susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and have been described in detail herein. However, it may be understood that the disclosure is not intended to be limited to the particular forms disclosed. The disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the following appended claims.
The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform] ing [a function] …” or “step for [perform] ing [a function] …” , it is intended that such elements are to be interpreted under 35 U.S.C. 112 (f) . However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. 112 (f) .
Claims (20)
- A device pairing system comprising:a peripheral device; anda host device comprising:an antenna configured to receive a signal from the peripheral device;a receiver configured to determine a signal strength from the signal;a camera configured to capture an image of the peripheral device; anda controller comprising a processor and at least one memory, wherein the at least one memory is configured to store instructions for pairing the host device with the peripheral device, wherein the processor, in executing the instructions, is configured to pair the host device with the peripheral device based on the image of the peripheral device and the signal strength of the peripheral device.
- The device pairing system of claim 1, wherein the processor, in executing the instructions, is configured to:identify the peripheral device based on the image of the peripheral device; anddetermine a visual distance from the host device to the peripheral device based on the identity of the peripheral device.
- The device pairing system of claim 2, wherein the processor, in executing the instructions, is configured to determine a signal distance to the peripheral device based on the signal strength of the peripheral device.
- The device pairing system of claim 3, wherein the processor, in executing the instructions, is configured to pair the host device with the peripheral device in response to the visual distance being approximately equal to the signal distance of the peripheral device.
- The device pairing system of any of claims 1-4, wherein the peripheral device comprises a set of earphones, headphones, speakers, a keyboard, mousing device, car stereo, printer, webcam, or smart home device.
- The device pairing system of any of claims 1-4, wherein the host device comprises a cell phone, smartphone, wearable device, virtual reality or mixed reality headset, tablet, desktop computer, or laptop computer.
- The device pairing system of any of claims 1-4, wherein the signal strength comprises a Received Signal Strength Indicator (RSSI) value.
- A host device comprising:an antenna configured to receive a signal from a peripheral device;a receiver configured to determine a signal strength from the signal;a camera configured to capture an image of the peripheral device; anda controller comprising a processor and at least one memory, wherein the at least one memory is configured to store instructions for pairing the host device with the peripheral device, wherein the processor, in executing the instructions, is configured to:determine a visual distance to the peripheral device based on the image of the peripheral device;determine a signal distance to the peripheral device based on the signal strength of the peripheral device; andpair the host device with the peripheral device in response to the visual distance being approximately equal to the signal distance of the peripheral device.
- The host device of claim 8, wherein the processor, in executing the instructions, is configured to determine to pair the host device with the peripheral device based on a pairing history between the host device and the peripheral device.
- The host device of claim 8, comprising a deep learning engine configured to facilitate determining to pair the host device with the peripheral device.
- The host device of claim 10, wherein the deep learning engine is configured to determine one or more indications of how the peripheral device is used.
- The host device of claim 11, wherein the one or more indications are based on an identity of the peripheral device.
- The host device of claim 11, wherein the one or more indications are based on an open software application or a most recently used software application of the host device.
- The host device of claim 11, wherein the one or more indications are based on a proximity to a user.
- The host device of claim 11, wherein the one or more indications are based on a user wearing or holding the peripheral device.
- The host device of claim 7, wherein the processor, in executing the instructions, is configured to determine to pair the host device with the peripheral device based on a user gesture indicating the peripheral device.
- A method for pairing a host device with a peripheral device comprising:receiving an image of the peripheral device;determining a visual distance to the peripheral device based on the image of the peripheral device;determining a signal strength of the peripheral device;determining a signal distance to the peripheral device based on the signal strength of the peripheral device; andpairing the host device with the peripheral device in response to the visual distance being approximately equal to the signal distance of the peripheral device
- The method of claim 17, comprising periodically searching for the peripheral device, wherein receiving the image of the peripheral device occurs in response to finding the peripheral device.
- The method of claim 17, comprising searching for the peripheral device in response to receiving an indication that a software application associated with the peripheral device was opened on the host device, wherein receiving the image of the peripheral device occurs in response to finding the peripheral device.
- The method of claim 17, comprising searching for the peripheral device in response to receiving the image of the peripheral device.
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP18944439.1A EP3903168A4 (en) | 2018-12-24 | 2018-12-24 | Systems and methods for pairing devices using visual recognition |
KR1020207035959A KR20210108302A (en) | 2018-12-24 | 2018-12-24 | Systems and Methods for Pairing Devices Using Visual Recognition |
PCT/CN2018/123202 WO2020132831A1 (en) | 2018-12-24 | 2018-12-24 | Systems and methods for pairing devices using visual recognition |
CN201880094433.4A CN113168217A (en) | 2018-12-24 | 2018-12-24 | System and method for pairing devices using visual recognition |
US17/253,998 US20210311893A1 (en) | 2018-12-24 | 2018-12-24 | Systems and methods for pairing devices using visual recognition |
JP2020568333A JP7295147B2 (en) | 2018-12-24 | 2018-12-24 | Systems and methods for pairing devices using visual recognition |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2018/123202 WO2020132831A1 (en) | 2018-12-24 | 2018-12-24 | Systems and methods for pairing devices using visual recognition |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020132831A1 true WO2020132831A1 (en) | 2020-07-02 |
Family
ID=71129465
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/123202 WO2020132831A1 (en) | 2018-12-24 | 2018-12-24 | Systems and methods for pairing devices using visual recognition |
Country Status (6)
Country | Link |
---|---|
US (1) | US20210311893A1 (en) |
EP (1) | EP3903168A4 (en) |
JP (1) | JP7295147B2 (en) |
KR (1) | KR20210108302A (en) |
CN (1) | CN113168217A (en) |
WO (1) | WO2020132831A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112668589A (en) * | 2020-12-31 | 2021-04-16 | 闪耀现实(无锡)科技有限公司 | Pairing method and apparatus for head-mounted electronic device |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11882192B2 (en) * | 2022-05-25 | 2024-01-23 | Microsoft Technology Licensing, Llc | Intelligent near-field advertisement with optimization |
US12086393B2 (en) * | 2022-08-12 | 2024-09-10 | Samsung Electronics Co., Ltd. | Communication method and device using avatar in virtual space |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017030584A1 (en) * | 2015-08-20 | 2017-02-23 | Hewlett-Packard Development Company, L.P. | Peripheral device pairing |
US20170242480A1 (en) * | 2014-10-06 | 2017-08-24 | Koninklijke Philips N.V. | Docking system |
CN107534653A (en) * | 2015-04-06 | 2018-01-02 | 微软技术许可有限责任公司 | The digital pen pairing of striding equipment based on cloud |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9436805B1 (en) * | 2008-07-03 | 2016-09-06 | Prime Research Alliance E, Inc. | Method and system for transfer of subscription media |
US8750799B2 (en) * | 2010-09-30 | 2014-06-10 | Apple Inc. | Wireless accessory device pairing determination for multiple host devices |
EP2837167B1 (en) * | 2012-04-12 | 2018-12-19 | Telefonaktiebolaget LM Ericsson (publ) | Pairing a mobile terminal with a wireless device |
US9154904B2 (en) * | 2012-11-12 | 2015-10-06 | Tata Consultancy Services Limited | Method and system for accurate straight line distance estimation between two communication devices |
US8706162B1 (en) * | 2013-03-05 | 2014-04-22 | Sony Corporation | Automatic routing of call audio at incoming call |
US9453904B2 (en) * | 2013-07-18 | 2016-09-27 | Golba Llc | Hybrid multi-camera based positioning |
WO2016088185A1 (en) * | 2014-12-01 | 2016-06-09 | 富士通株式会社 | Pairing management method, pairing management program, and terminal device |
US20160285299A1 (en) * | 2015-03-26 | 2016-09-29 | Intel Corporation | Method and apparatus for low power wireless docking discovery |
CN106507240A (en) * | 2015-09-04 | 2017-03-15 | 音乐集团公司 | A kind of physical location of the speaker by speaker system is associated to the method for speaker identifier |
US10042595B2 (en) * | 2016-09-06 | 2018-08-07 | Apple Inc. | Devices, methods, and graphical user interfaces for wireless pairing with peripheral devices and displaying status information concerning the peripheral devices |
US10574665B2 (en) * | 2016-09-29 | 2020-02-25 | AVAST Software s.r.o. | System and method for determining device proximity |
US10820360B1 (en) * | 2017-08-30 | 2020-10-27 | Socket Mobile, Inc. | Application-assisted friendly pairing |
US10737781B2 (en) * | 2017-09-14 | 2020-08-11 | Drone Racing League, Inc. | Three-dimensional pathway tracking system |
US20190079510A1 (en) * | 2017-09-14 | 2019-03-14 | Drone Racing League, Inc. | End gate structure with automatic power down |
US11394930B2 (en) * | 2019-10-11 | 2022-07-19 | At&T Intellectual Property I, L.P. | Filtering and correlating radio signals based on a field of view of a camera |
US11868294B2 (en) * | 2021-06-28 | 2024-01-09 | Learning Squared, Inc. | Image based pairing system |
-
2018
- 2018-12-24 EP EP18944439.1A patent/EP3903168A4/en active Pending
- 2018-12-24 KR KR1020207035959A patent/KR20210108302A/en not_active Application Discontinuation
- 2018-12-24 CN CN201880094433.4A patent/CN113168217A/en active Pending
- 2018-12-24 US US17/253,998 patent/US20210311893A1/en active Pending
- 2018-12-24 WO PCT/CN2018/123202 patent/WO2020132831A1/en unknown
- 2018-12-24 JP JP2020568333A patent/JP7295147B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170242480A1 (en) * | 2014-10-06 | 2017-08-24 | Koninklijke Philips N.V. | Docking system |
CN107534653A (en) * | 2015-04-06 | 2018-01-02 | 微软技术许可有限责任公司 | The digital pen pairing of striding equipment based on cloud |
WO2017030584A1 (en) * | 2015-08-20 | 2017-02-23 | Hewlett-Packard Development Company, L.P. | Peripheral device pairing |
Non-Patent Citations (1)
Title |
---|
See also references of EP3903168A4 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112668589A (en) * | 2020-12-31 | 2021-04-16 | 闪耀现实(无锡)科技有限公司 | Pairing method and apparatus for head-mounted electronic device |
Also Published As
Publication number | Publication date |
---|---|
EP3903168A4 (en) | 2022-08-03 |
JP7295147B2 (en) | 2023-06-20 |
EP3903168A1 (en) | 2021-11-03 |
KR20210108302A (en) | 2021-09-02 |
US20210311893A1 (en) | 2021-10-07 |
JP2022522912A (en) | 2022-04-21 |
CN113168217A (en) | 2021-07-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10390213B2 (en) | Social reminders | |
EP3163569B1 (en) | Method and device for controlling a smart device by voice, computer program and recording medium | |
RU2670786C2 (en) | System, method and apparatus for device group control | |
US9348412B2 (en) | Method and apparatus for operating notification function in user device | |
US11350264B2 (en) | Method and apparatus for establishing device connection | |
US11705120B2 (en) | Electronic device for providing graphic data based on voice and operating method thereof | |
US20220004349A1 (en) | Controlling visual indicators in an audio responsive electronic device, and capturing and providing audio using an api, by native and non-native computing devices and services | |
KR20180068127A (en) | Mobile terminal and method for controlling the same | |
CA2939001A1 (en) | Performing actions associated with individual presence | |
JP2015055718A (en) | Information processing device, information processing method, and program | |
CN103632165A (en) | Picture processing method, device and terminal equipment | |
WO2020132831A1 (en) | Systems and methods for pairing devices using visual recognition | |
US9703577B2 (en) | Automatically executing application using short run indicator on terminal device | |
CN106203235A (en) | Live body discrimination method and device | |
US20180160286A1 (en) | Method of controlling device and device thereof | |
CN113138560A (en) | Terminal control method, device, equipment and readable storage medium | |
US20140176689A1 (en) | Apparatus and method for assisting the visually impaired in object recognition | |
US20190052745A1 (en) | Method For Presenting An Interface Of A Remote Controller In A Mobile Device | |
CN106951243B (en) | Screen control method and terminal equipment | |
US9858025B2 (en) | Method for controlling combined display and electronic device thereof | |
CN109408245A (en) | A kind of recognition methods of multi-medium data, device and electronic equipment | |
CN104793729A (en) | Method and electronic equipment for processing information | |
US11900929B2 (en) | Electronic apparatus providing voice-based interface and method for controlling the same | |
AU2017100235A4 (en) | Social reminders |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18944439 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2020568333 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2018944439 Country of ref document: EP Effective date: 20210726 |