JP2016502137A - Unified framework for device configuration, interaction and control, and related methods, devices and systems - Google Patents

Abstract

A method of operating a device includes: buffering sensor information from an environment surrounding the device as buffered sensor information; detecting a person's line of sight in the environment; and buffering based on the detected line of sight. Starting recognition of some sensor information including ringed sensor information, wherein the recognition determines at least one instruction in the sensor information including buffered sensor information; and at least one instruction Operating the device based on

Description

[Copyright Statement]
This patent document contains content that is subject to copyright protection. The copyright holder does not object to the reproduction of any relevant material in this patent document or US Patent and Trademark Office file, but otherwise retains all copyrights for any reason. To do.

[Related applications]
The present application is filed on November 16, 2012, "Unified Framework For Device Configuration, Interaction And Control, And Associated Methods, Patents and Seventeenth Patent Application No. 27 / Applications and Systems 27". The entire contents of both are hereby fully incorporated by reference for all purposes.

  The present invention relates to a unified framework for device configuration, interaction and control, and related methods, devices, and systems.

  Computer-based devices, particularly consumer computer-based devices, are ubiquitous. These devices are generally designed and manufactured independently of each other and tend to attempt to use ad hoc or standardized techniques to do so as long as they can interact with each other or use each other is there. As a result, if consumers attempt to use multiple different devices together, even if the same vendor creates the multiple devices, they will be complex for the multiple devices they acquire. Often it is forced to perform a complicated setup procedure. Even for multiple single, stand-alone devices, the setup procedure is generally complex. Some companies have attempted to simplify the use of their own devices, but not for devices from other companies.

  It is desirable and an object of the present invention to provide a system / framework in which multiple devices can be easily provisioned, configured and fused.

  It is further desirable and an object of the present invention to provide a system / framework that allows multiple different types of devices to interact with multiple different users.

  It is further desirable and object of the present invention to provide a core system in multiple devices to support their interaction with each other and with a common framework.

  Other objectives, features, and characteristics of the present invention, as well as multiple functions of related elements of multiple methods and structures of operation, and combinations of parts and efficiency of manufacture, All of which will become apparent upon consideration of the following description and appended claims, taken in conjunction with the accompanying drawings, which form a part hereof.

2 illustrates a framework for an exemplary system, according to embodiments herein.

2 illustrates aspects of the configuration of a user's device within the framework of FIG. 1 in accordance with embodiments herein.

2 shows details of the database of the framework of FIG. 1 according to embodiments herein. 2 shows details of the database of the framework of FIG. 1 according to embodiments herein. 2 shows details of the database of the framework of FIG. 1 according to embodiments herein.

FIG. 2 illustrates a general device aspect for use within the framework of FIG. 1 in accordance with embodiments herein. FIG. 2 illustrates a general device aspect for use within the framework of FIG. 1 in accordance with embodiments herein. FIG. 2 illustrates a general device aspect for use within the framework of FIG. 1 in accordance with embodiments herein. FIG. 2 illustrates a general device aspect for use within the framework of FIG. 1 in accordance with embodiments herein. FIG. 2 illustrates a general device aspect for use within the framework of FIG. 1 in accordance with embodiments herein.

2 illustrates an exemplary mechanism of multiple corpora in a system, according to embodiments herein. 2 illustrates an exemplary mechanism of multiple corpora in a system, according to embodiments herein. 2 illustrates an exemplary mechanism of multiple corpora in a system, according to embodiments herein. 2 illustrates an exemplary mechanism of multiple corpora in a system, according to embodiments herein. 2 illustrates an exemplary mechanism of multiple corpora in a system, according to embodiments herein.

2 is a flowchart illustrating an exemplary operation of a device within the framework of FIG. 1 according to embodiments herein. 2 is a flowchart illustrating an exemplary operation of a device within the framework of FIG. 1 according to embodiments herein. 2 is a flowchart illustrating an exemplary operation of a device within the framework of FIG. 1 according to embodiments herein. 2 is a flowchart illustrating an exemplary operation of a device within the framework of FIG. 1 according to embodiments herein. 2 is a flowchart illustrating an exemplary operation of a device within the framework of FIG. 1 according to embodiments herein. 2 is a flowchart illustrating an exemplary operation of a device within the framework of FIG. 1 according to embodiments herein.

1 illustrates aspects of a general computer system in which multiple embodiments of the present disclosure may be implemented and executed. 1 illustrates aspects of a general computer system in which multiple embodiments of the present disclosure may be implemented and executed. 1 illustrates aspects of a general computer system in which multiple embodiments of the present disclosure may be implemented and executed. 1 illustrates aspects of a general computer system in which multiple embodiments of the present disclosure may be implemented and executed. 1 illustrates aspects of a general computer system in which multiple embodiments of the present disclosure may be implemented and executed.

2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein. 2 illustrates exemplary aspects of device provisioning and configuration within the framework of FIG. 1, in accordance with embodiments herein.

2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein. 2 illustrates aspects of multiple device interaction within the framework of FIG. 1 in accordance with embodiments herein.

FIG. 4 illustrates an exemplary particular device aspect for sound rendering, according to embodiments herein. FIG. FIG. 4 illustrates an exemplary particular device aspect for sound rendering, according to embodiments herein. FIG. FIG. 4 illustrates an exemplary particular device aspect for sound rendering, according to embodiments herein. FIG. FIG. 4 illustrates an exemplary particular device aspect for sound rendering, according to embodiments herein. FIG.

6 illustrates aspects of cooperation between multiple sound rendering devices, according to embodiments herein. 6 illustrates aspects of cooperation between multiple sound rendering devices, according to embodiments herein. 6 illustrates aspects of cooperation between multiple sound rendering devices, according to embodiments herein.

  The term “M” is used herein as the name of the described system, multiple devices, multiple processes, multiple interactions, etc. It should be understood that this name is only used to aid the description and is not intended to limit the scope of the system at all.

  FIG. 1 shows a framework / system for an exemplary M system 100 in which multiple M-enabled devices (102-1, 102-2,... 102-n, collectively 102) are used. Show. M-enabled device (ie, M device) 102 can be any device and any type of device, can be a stand-alone device, or can be integrated into or combined with any type of device or system. Can be done. For example, the M-enabled device 102 may generate and / or generate devices and / or content (including again digital and / or analog content) that capture and / or create content (including digital or analog content), without limitation. Device to render. The M-compatible device can be (or can be incorporated), for example, a camera, speaker, computer, telephone, set-top box, television, appliance, and the like. A combination of these devices is also conceivable, for example, a set top box including a camera and speakers and a television monitor, and the computer can be one or more M-enabled devices. It should be understood that the examples given herein for device 102 are merely exemplary and are not intended to be limiting in any way. Multiple M-enabled devices (ie, M devices) 102 are described in more detail below.

  System 100 preferably includes one or more certificate authorities (CA) 106 that may be part of a larger collection or hierarchy of certificate authorities in a public key infrastructure (PKI) scheme (not shown). Or access. Each CA 106 is an entity that issues digital certificates for the system 100 or on behalf of the system 100.

  The plurality of M-enabled devices 102 are preferably manufactured (at least in part) by one or more authorized device manufacturers 108. It should be understood that the plurality of devices 102 in FIG. 1 can be a plurality of different types of devices manufactured by a plurality of different manufacturers.

  Multiple M-enabled devices 102 can interact with each other and with the back-end system 104. It should be understood that multiple different types of devices can interact with each other (eg, an M device embodied in a mobile phone can interact with another M device embodied in a speaker). .

  Each of the plurality of M-compatible devices 102 is associated with the user 110. A particular user 110 may have multiple devices 102 associated with it. In some implementations, each device 102 may be associated with only one user 110. It should be understood that the term “user” is an internal entity within system 100 and is used to define a connection between multiple devices within system 100. User 110 may be considered an entity having a specific relationship within system 100. The entity may correspond to a person, or a group of people, or any other type of entity (eg, company, school, club, etc.). In a preferred implementation, user 110 is an entity registered with the system. A particular person (or entity) may correspond to more than one user 110. For example, a person may choose to have two users 110 within the system 100.

  The backend 104 includes one or more backend applications 112, at least some of which interact with one or more databases 114 that store and maintain information about the plurality of devices 102 and the plurality of users 110. The back end 104 also includes a plurality of social networking services 116 (such as Facebook®, LinkedIn®, and the like) and a plurality of content providers 118 (such as RDIO and the like). Can interact with various other entities. Although multiple social networking services 116 are shown separate from multiple content providers 118, it should be understood that there may be some overlap between these entities. The backend 104 may interact with multiple entities 120 that can provide additional functionality to multiple users 110 (or to multiple devices 102). Additional functions may include, for example, voice or command recognition functions, face recognition, and the like. The back end 104 can also interact with various other external components 122.

  Additional functionality may provide, enhance, or enhance functionality provided on some or all devices 102 to provide functionality in addition to that provided on multiple devices. The additional function 120 can be used, for example, to provide a function that exceeds the hardware capability of the device 120. Thus, for example, as described in more detail below, additional functionality 120 exceeds what is provided on a particular device 102 (or beyond what is possible using the hardware of a particular device 102). ) Can provide voice recognition. In practice, additional functionality 120 may be used, at least in part, to extend the functionality of any particular M-enabled device 102.

  It should be understood that some or all of the functionality provided by multiple entities 120 may be integrated into backend 104 (or into multiple backend applications 112).

It should also be understood that the separation of multiple components in multiple drawings implies no (or lack of) ownership or control or control. Thus, for example, the CA 104 need not be owned or managed (or not owned) by the same entity that manages the back end 104. Similarly, multiple social networking services 116 and / or multiple content providers 118 and / or multiple entities providing additional functionality 120 need to be separately owned and managed (or they are owned and managed together). There is no. Some or all of the system 100 may be integrated into the social networking service 116 and / or the content provider 118.
Multiple system and component interactions

Various interactions between various components can occur in the system 100. In FIG. 1, these interactions are indicated by numbered arcs or lines (designated 1 to 15). In the system 100 depicted in FIG.
Arc line # 1 refers to multiple interactions between the two devices 102.
Arc line # 2 refers to multiple interactions between device maker 108 and multiple devices 102.
Arc line # 3 refers to multiple interactions between CA 106 and multiple devices 102.
Arc line # 4 refers to multiple interactions between CA 106 and multiple users 110.
Arc line # 5 refers to multiple interactions between multiple users 110 and backend 104.
Arc # 6 points to multiple interactions between CA 106 and back end 104.
Arc line # 7 refers to multiple interactions between multiple devices 102 and backend 104.
Arc line # 8 points to multiple interactions between the backend and database 114.
Arc line # 9 refers to multiple interactions between the backend and multiple social networking services 116.
Arc line # 10 points to multiple interactions between the backend and content provider 118.
Arc # 11 points to multiple interactions between backend 104 and multiple entities 120 that provide additional functionality.
Arc line # 12 refers to multiple interactions between backend 104 and various other external components 122.
Arc line # 13 refers to multiple interactions between device maker 108 and backend 104.
Arc line # 14 refers to multiple interactions between multiple devices 102 and multiple users 110.
Arc line # 15 points to multiple interactions between CA 106 and device manufacturer 108.

  The various interactions described herein can be performed using any known method and protocol and can be wired, wireless, or any combination thereof. Some interactions may occur at least in part via the network 101 (eg, a packet-based network such as the Internet). Network 101 can be a public network or a private network, or some combination thereof. The network may include one or more cellular and / or satellite-based components. It should be understood that the communication medium or communication protocols through which the various components interact is not system limited.

  Back end 104 and various other components or entities (eg, multiple CAs 106, multiple social networking services 116, content providers 118, device manufacturers 108, multiple entities 120 that provide additional functionality, and various other external components. 122) may use multiple application program interfaces (APIs), or other multiple interfaces of those components or entities. In some embodiments, the back end 104 can be integrated into some or all of the other components. It should be understood that the manner in which the back end communicates with any other components is not intended to limit the system 100 in any way, and different modes and manners of communication are contemplated herein. . Further, the degree of integration (or lack thereof) between the backend 104 and any other components is not intended to limit the system 100 in any way, different modes of integration, different ways, and multiples. Are considered herein.

  As described above, various interactions and communications can be performed using any known method and protocol, but multiple device interactions (arc # 1 in FIG. 1) are preferably the fastest and cheapest multiple Communication technology-if available, generally using local (eg, Bluetooth® and the like, or Wi-Fi on local network), preferably via other networks Avoid communication.

As will be appreciated and understood by those skilled in the art upon reading this description, different and / or other interactions are possible in the system and are not excluded by omission from the drawings. Thus, various entities have multiple other interactions (not shown in the drawing), and further, not all implementations of the system 100 need to support all the interactions shown in the drawing. Should be understood.
Multiple user IDs

  Each user 110 must have at least one unique user identification information (ID) within the system 100. A user's user ID may be based on or derived from the user's social network ID (eg, their Facebook ID). Referring to FIG. 2, the user ID of the user can be asserted in the form of a user certificate 175 issued by the CA 106. It should be understood that the device CA 124 and the user CA 126 are the same entity, but the CA 106 is one or more devices CA 124 (shown in FIG. 6B) and one or more users CA 126 (shown in FIG. 2). May be included). If the CA 106 includes multiple user CAs 126, the user's user certificate 175 is preferably provided by the back end during the registration process or the like.

The user certificate 175 is stored in a location 176 for such multiple certificates and user identification information on the user's device 174 (eg, smart phone, smart device enabled by M technology, or the like). Can be done.
User's multiple friends

  Within system 100, user 110 may have one or more friends in system 100. A user's friend in system 100 is another user 110 with whom the first user has established a relationship through system 100. The relationship can be established in various ways, for example, by sharing or interacting with each other's devices 102. A user may establish multiple permissions with the user's friends. Multiple permissions may depend, for example, on the type of device 102 associated with the user and may vary from friend to friend.

  A user's “friend” relationship with another user within the system 100 can be in a limited range and can be determined, for example, by multiple permissions. The range of friendship relationships can be used to limit various aspects of friendship relationships, such as the duration of friendships in the system, multiple rights of friends in the system, etc.

  The user's friends in the system 100 may be distinguished from the user's social network friends (eg, the user's Facebook friends), although there may be overlap. For example, if a user's user ID is based on or derived from the user's social network ID (eg, their Facebook ID), multiple friends of the user's social network Can overlap with multiple friends.

It should be understood that the system is in no way limited by the manner in which multiple users can establish “friend” relationships with other multiple users. The friend system concept need not have any relationship with the actual friendship relationships or other relationships outside the system 100, and friendships within the system can be associated and linked within the system 100. It should also be understood that it is used to do.
Multiple device IDs

Each device 102 has device identification information (ID) that must be unique within the system 100. The creation and storage of multiple device IDs is discussed in more detail below.
Multiple databases

  Referring to FIG. 3A, the backend 104 includes a backend application 112 that includes one or more applications that interact with one or more databases 114. Database 114 includes a device database 128 and a user database 130 for storing and maintaining information about multiple devices 102 and multiple users 110, respectively.

It should be understood that the multiple databases may use any type of database technology, the particular database implementation is not described or required here, and the system 100 is not at all limited by the database implementation. It is. In some cases, some or all of the database implementations may use multiple third party databases. It should also be understood that the user database 130 and device database 128 may be integrated or may be composed of a plurality of different, possibly distributed databases.
Device database 128

  As shown in FIGS. 3A and 3B, the database 114 includes device owners (users 110), device capabilities, information about device history, and, for example, the last known location of the device, device Information about each device 102 in the system 100 may be maintained (in the device database 128), including other information such as the manufacturer of the device. Preferably, the device ID is used as a key (or index) to the device database 128. When device 102 is manufactured, it has no owner (unless specifically provisioned in manufacturing). The functions of the device can include its type (eg, speaker, etc.). The terms “owned” and “owner” are not used herein to imply or require any legal concept of ownership of a device by a user, as they are multiple devices within system 100. It should be understood that it refers to an association or association between 102 and multiple users 110.

Device database 128 may include multiple corpora of devices for each device. As described in more detail below, multiple corpora of a device may correspond to multiple corpora stored on the device and may be used by the device or multiple users of the device.
User database 130

Preferably, the user ID acts as the primary key for the user database 130 database. The user database 130 preferably associates with each user 110 specific information including (see FIG. 3C) one or more of the following:
A user profile that describes information about the user. The user profile may be linked to the user's social network data (eg, the user's Facebook® profile) or may obtain information from the user's social network data. The user profile may also include information about multiple content providers with which the user is associated (ie, having multiple accounts), such as, for example, RDIO.
The user's devices 102 and, optionally, information about those devices (eg, device capabilities). As those skilled in the art will appreciate upon reading this description, some information in the user database 130 may be obtained from the device database 128 and vice versa. For example, if the user database 130 stores multiple device identifiers for multiple devices for each user, the device identifiers can be used to obtain device specific information for the corresponding multiple devices from the device database 128. .
Multiple "friends" of the user within the system 100 and possibly multiple permissions associated with those friends. In preferred implementations, the user's "friends" in the system are other users 110, and the user is listed as a list of user IDs with zero or more information about each of the user's multiple friends. It can be stored in the database 130. Multiple users can set multiple customized permissions for each friend. In some embodiments, multiple users may be able to define or use multiple classes or multiple groups or multiple types of friends and assign multiple permissions based on classification or group or type . In this way, a user may be able to use a template or the like to quickly set or change multiple permissions for one or more users. It should be understood that the system is not limited by how multiple users select their friends or set multiple permissions for their friends. Multiple users may also exclude other multiple users as multiple friends.
Preferably stored by one or more of device 102, friend (ie, using one or more devices or friends as keys for history search), time, or Searchable user history. The device history may be a sequence of multiple time stamped events for each of the user's multiple devices. For example, if a particular device 102 is a speaker, the history of that device may be a time-stamped list of what was played on that speaker. The user's history for multiple friends may include multiple times and multiples when the user's device interacted with the user's friend's device (or used by the user's friend's device) (and vice versa) Can include location. The examples given here for the details of the types of searchable history are not intended to limit the scope of this description at all, as those skilled in the art will realize upon reading this description Different and / or other information and / or multiple factors may be used to search the user's history.
User IDs and certificates (including those issued by user CA 126).
• Multiple corpora of users for various device interface mechanisms (described in more detail below). The user's plurality of corpora preferably includes a plurality of local corpora and a plurality of extended corpora. As described in more detail below, a user's multiple corpora may correspond to that user's multiple corpora on one or more devices, and multiple devices if under the user's control Can be used. A user's multiple corpora may extend beyond the capacity of some or all of the devices 102 and may correspond to the user's multiple corpora that may be used by additional functionality 120.
Preferably, configuration information including configuration details for the user's devices and information that can be used to configure other (eg, new) devices. The configuration information includes information about multiple settings of the user's wireless network at various locations, which information includes multiple passwords and other information that may be required for the device to connect to those networks. .

As those skilled in the art will appreciate upon reading this description, different and / or other information, associations, and relationships may be stored in the device database 128 and the user database 130.
Multiple devices

  Each device 102 has a specific device-specific function / components associated with it. For example, if the device 102 is a speaker (or camera or phone or the like), the device specific function / components may be the function / multiple used to operate the device as a speaker (or camera or phone etc.). Will contain components. Referring now to FIG. 4A, device 102 includes a plurality of features 132 that support device-specific functions / components of the device.

  As used herein, the term “mechanism” means hardware in combination with a single piece of hardware or software and includes firmware.

  Device 102 also includes various M system features / data 134 that are used to support and implement functions within system 100. Multiple system features / data 134 are preferably separate from device-specific functions 132, but multiple system features / data 134 may interact (at 136) with device-specific features 132 of the device.

  Device 102 preferably includes one or more sensors 138 that can be used by multiple system mechanisms 134 to implement multiple aspects of system functionality. As used herein, a “sensor” is any mechanism that can sense and / or measure a physical property or stimulus (eg, heat, light, sound, pressure, magnetism, movement, touch, capacitance). Or mean device. The sensor preferably provides an indication of the characteristic or stimulus it senses / measures (eg, as an electrical signal). A sensor can also record, display, or respond to physical properties or stimuli it detects / measures. The sensor may be implemented in hardware, software, or a combination of multiple thereof. The sensors can be provided as stand-alone devices or chips, or can be integrated into other devices (or sensors) or chips. The particular plurality of sensors 138 may be implemented, at least in part, using a plurality of dedicated chips or circuits, or the like. As one skilled in the art will appreciate upon reading this description, the system is not limited in any way by the manner in which multiple sensors are implemented or integrated. It should also be understood that a particular sensor can sense and / or measure more than one type of physical property or stimulus.

  4A-1, the plurality of sensors 138 may include, for example, one or more cameras, microphones, motion sensors (external motion and / or device motion), multiple accelerometers, compass, and multiple positioning systems. (LPS), and the like. As used herein, LPS generally refers to any position positioning system that can be used to determine the position of a device, including the United States Global Positioning System (GPS) and Russian Global Navigation. Includes the satellite system (GLONASS), the Galileo positioning system of the European Union, the Chinese compass navigation system, and the regional navigation satellite system of India. LPS also includes Wi-Fi and / or cell phone systems that can provide location data, as well as assisted and augmented multiple positioning systems using, for example, Wi-Fi and / or cell phone systems. Although only one camera can be used for face and gesture processing, two or more cameras allow three-dimensional (3D) processing if necessary.

  Multiple sensors 138 may interact with multiple system features 134 via multiple standard interfaces provided to each sensor (at 140). Not all devices 102 need to have all types of sensors 138, and multiple different types of devices (or multiple different implementations of the same type of device) can have multiple different sensors or multiple types of sensors. It should be understood.

  Device 102 preferably includes one or more communication mechanisms 142 that may be used by a plurality of system mechanisms 134 (at 144) to implement aspects of the system functionality. 4A-2, the plurality of communication mechanisms 142 may include, for example, a plurality of mechanisms for local communication (for example, Bluetooth (registered trademark) including Bluetooth Low Energy (BLE), ZigBee, etc.), Wi-Fi, and the like. Multiple mechanisms for communication (e.g., 802.11, etc.), multiple mechanisms for cellular communication (e.g., multiple devices using multiple modems or cellular network), and wired communication A plurality of mechanisms (eg, Ethernet® or the like) and one or more of the following. The plurality of communication mechanisms 142 may be implemented on a plurality of protocol specific chips or the like. Not all devices 102 need to have all types of communication mechanisms 142, and multiple different types of devices (or multiple different implementations of the same type of device) have multiple different sensors or multiple types of communication mechanisms. It should be understood that it is possible. However, at a minimum, each device 102 will be backend 104 at least sometime, in some manner (whether by cellular / telephone network, by Wi-Fi, by wire, or in some other manner). (Arc line # 7 in FIG. 1). Also, preferably each device 102 has at least one communication mechanism that allows it to communicate with other devices 102 of the system 100 for at least some time. Each device also includes a plurality of required / appropriate connectors and / or antennas for its various communication mechanisms. These connectors and / or antennas can be incorporated into various communication mechanisms, particularly when multiple communication mechanisms are provided on multiple dedicated chips or multiple circuits.

  Multiple system features 134 (including any required sensors 138) can be integrated into device 102 as separate boards or chipsets, or they can be used to implement device specific functions. Can share multiple components (as indicated by the dotted oval connecting them). For example, if a device-specific function requires a microphone, that microphone may be shared with (or used by) multiple system features 134 as a sensor 138. Similarly, at least some of the plurality of communication mechanisms may be shared between the plurality of system mechanisms 134 and the device specific function 132 (as indicated by the dotted ellipse connecting them). However, multiple system mechanisms 134, multiple sensors 138, and multiple communication mechanisms 142 need to be able to be operated and controlled independently of multiple device specific mechanisms, and this need is It should be understood that in some implementations, it may override or prevent sharing of multiple components.

  Device 102 may include a computer system 146 (described in more detail below). Computer system 146 may interact with a plurality of system features 134 (at 148) and may implement aspects of those features. Although shown as multiple separate components in the drawing of FIG. 4A, some or all of computer system 146 is shared with multiple system features / data 134 and one of multiple system features / data 134. It should be understood that they are parts (as shown by the ellipses drawn by the dots connecting them). Similarly, at least some of the computer systems 146 may overlap with device-specific functions 132 of the device 102 (as indicated by the ellipses drawn by the dots connecting them).

  Device identification information (ID) and other identification information of the device, as well as certificate information, are stored (at 150) in multiple system features / data 134.

  Each device 102 preferably includes a plurality of bootstrapping / provisioning mechanisms 152 and various operating mechanisms 154 (both described in more detail below). Various operation mechanisms 154 have a corresponding operation storage 155 on the device. The operational storage 155 can store data used by various operational mechanisms and can be used for permanent and / or temporary storage. As one skilled in the art will appreciate upon reading this description, some or all of the operational storage 155 may be integrated with other storage on the device, and the operational storage 155 is separate to assist in this description. It is shown in the drawing as a component.

Each device 102 includes at least one power system 157 that can provide power to a device including a system mechanism 134, a computer system 146, a plurality of sensors 138, a plurality of communications 142, and device-specific functions 132. The power system 157 may include multiple systems that are separate for some or all of the multiple components, and may include multiple battery power sources alone or in combination with multiple external power sources. Preferably, the plurality of system features have a power source (eg, a battery) that is separate from device-specific functions. When an external power source is used (eg, A / C power through an adapter), all components of the system have multiple separate internal power systems for use when they are not connected to an external source. Even if you have one, you should use an external power source. However, it should be understood that the system is not limited by the manner in which power is supplied to multiple components.
Multiple operating mechanisms

4A-4C, the plurality of operating mechanisms 154 of the device may include some or all of the following plurality of mechanisms.
A plurality of mechanisms 156 that support device-to-device interaction and communication (this corresponds to the interaction indicated by arc # 1 in FIG. 1). Multiple device-to-device interaction mechanisms 156 may have and use corresponding device-to-device storage 169 (FIG. 4C).
Multiple mechanisms 158 for command and / or control of the device 102; The plurality of command and / or control mechanisms 158 may have and use corresponding command and / or control storage 159 (FIG. 4C).
Multiple mechanisms 160 that support device-to-backend interaction and communication (this corresponds to the interaction indicated by arc # 7 in FIG. 1). The plurality of device-to-back-end mechanisms 160 have a corresponding device-to-back-end storage 161 (FIG. 4C) and can use it.
A plurality of interface mechanisms 162 that are used to operate the device within the system 100 and to support interaction with other mechanisms and functions of the device, preferably including human interaction. The plurality of interface mechanisms 162 have a corresponding plurality of interface mechanism storages 163 (FIG. 4C) and may use it.
Other multiple operating mechanisms 164 used to operate devices within system 100 that may have and use corresponding storage 165 (FIG. 4C).

The plurality of interface mechanisms 162 include one or more of the following.
O Multiple gesture mechanisms 166 that can be used by multiple movement mechanisms 154 to implement multiple operational and / or functional features using multiple user gestures (e.g., multiple gesture commands, and the like). . The plurality of gesture mechanisms 166 may have a gesture storage 167 and may use it. The plurality of gesture mechanisms 166 preferably includes one or more gesture detection mechanisms 168 and one or more gesture recognition mechanisms 170, and the gesture storage 167 is preferably associated with a plurality of associated gestures for use by various gesture mechanisms. A gesture corpus 172 is included. A gesture corpus (plural “corpora”) refers to a plurality of gestures or a collection of gesture samples for which a gesture mechanism can be used to detect and / or recognize a plurality of gestures. In preferred systems, multiple gesture detection mechanisms 168 and / or gesture recognition mechanisms 170 are trained and adapted to detect and / or recognize multiple gestures of one or more people (which may be multiple users). A plurality of associated gesture corpora can be changed based on this training. The gesture mechanism 166 may use one or more sensors 138 including a camera sensor, for example.
Multiple operating mechanisms to implement multiple operational and / or functional features that use multiple human voices (eg, multiple users) (eg, multiple voice commands, and the like) A plurality of voice / speech mechanisms 174 that may be used by 154. The plurality of voice / speech mechanisms 174 can have and use voice / speech storage 175. The plurality of speech / speech mechanisms 174 preferably includes one or more speech / speech detection mechanisms 176 and / or speech / speech recognition mechanisms 178, and the speech / speech storage 175 is preferably used by various speech / speech mechanisms. A plurality of associated corpora 180 to be included. A speech / speech corpus refers to a collection of multiple word or phrase samples that a speech / speech mechanism can utilize to detect and / or recognize speech / speech. In preferred systems, multiple speech / speech recognition mechanisms can be trained and adapted to recognize the speech / speech of one or more people (eg, multiple users), and the associated speech corpora is Can change based on training. The plurality of voice / speech mechanisms 174 may use one or more sensors 138 including, for example, microphone sensors.
A face / line-of-sight mechanism 182 that can be used by multiple movement mechanisms 154 to implement multiple operational and / or functional features that use multiple faces and / or multiple lines of sight of people. The face / gaze mechanism 182 may use a face / gaze storage 183. The face / gaze mechanism 182 may include a face / gaze detection mechanism 184 and / or a face / gaze recognition mechanism 186 and / or a face motion detection mechanism 187, and the face / gaze storage 183 is preferably face / gaze / recognition / detection. A plurality of associated face / line-of-sight corpus 188 for use by mechanism 182 is included. A face / line-of-sight corpus refers to a collection of multiple samples of faces and / or lines of sight that the face / line-of-sight mechanism can use to detect and / or recognize multiple faces and / or multiple lines of sight. In preferred systems, multiple face / gaze / recognition / detection mechanisms 182 can be trained and adapted to recognize one or more faces, and the associated multiple face / gaze corpus 182 can be modified based on this training. Can be done. The face movement detection mechanism 187 can detect movements of a plurality of parts of the face, for example, movements of the mouth, both eyes, etc., and can be used, for example, to try to confirm that a person is speaking. The face / line-of-sight mechanism 182 may use a plurality of sensors 138 including camera sensors, for example.
O Multiple others that can be used by multiple motion mechanisms 154 to implement multiple motional and / or functional features using other types of user interaction (eg, touch, typing, device movement, etc.) Interface mechanism 190 of FIG. The plurality of other interface mechanisms 190 can use the storage 191 of the plurality of other interface mechanisms. Other interface mechanisms 190 may use multiple sensors 138.

  While multiple corpora for speech, gesture, and face / gaze recognition / detection have been described above, it will be appreciated by those skilled in the art upon reading this description that other mechanisms, particularly multiple The interface mechanism 162 may have multiple corpora associated with it, and may also have multiple interactions with people, multiple interactions with other devices 102, multiple interactions with the backend 104, and multiple users. Learning and adaptation may also be based on multiple interactions that the device has internally (or external to system 100), including multiple interactions with 110. In general, a corpus for a particular interface mechanism refers to a collection of samples that can be used by that particular gaze mechanism to function and execute. Thus, for example, as shown in FIG. 4C, a plurality of other interface mechanisms 190 may include a plurality of corpus 192 associated therewith.

  Unless stated otherwise, as used herein, the term “multiple corpora” refers to one “corpus” and / or multiple corpora. Thus, for example, if a device is described as having a plurality of corpora of a feature, it may be that the device has at least one corpus of that feature, or the device has one corpus of that feature. Or it should be read as having multiple corpora of that feature.

  In some preferred embodiments, the user can train their devices to recognize specific words and / or gesture patterns. One or more of these characteristic words / gesture patterns may be used (and required), for example, to trigger some or all of the commands for the device. It should be understood that devices can be learned (and therefore trained) without any specific user intervention or requirement, preferably each device is not specifically required to do so Learn to recognize multiple user interactions.

  Device 102 also preferably includes a heartbeat (HB) mechanism 194 (described in more detail below). The HB mechanism may interact with a number of other operating mechanisms 154 including a device to device mechanism 156 and a device to backend mechanism 158.

  As those skilled in the art will appreciate upon reading this description, a particular device may use only some of the plurality of interface mechanisms 162. It should also be understood that various interface mechanisms 162 may be used in combination, as will be described in more detail below.

  Preferably, each operating mechanism 154 on the device 102 can operate on the device without any external interaction (ie, without any interaction with any other devices 102 or backend 104). However, various operating mechanisms 154 may operate or may use (or be enhanced by) additional functionality 120 provided, for example, via back end 104 or one or more other devices 102. Should be understood. Thus, for example, the various voice / speech mechanisms 174 may support limited use of multiple voice commands and the like when used without any external interaction. These voice commands may be limited, for example, by multiple functions of various mechanisms and by device capacity (such as memory and computational capacity). The system 100 may provide multiple voice commands and multiple interactions extended to the device when enhanced with additional functionality 120. Limited local voice / speech interactions will be reflected in voice / speech corpora 180 stored in voice / speech storage 175 on the device. In the case of multiple voice interactions, for example, the device 102 can recognize a corpus of words (eg, voice / speech storage) that can be recognized and used to control multiple aspects of the device. 175, stored as a plurality of voice / speech corpora 180). Using the additional function 120 provides the device with access to a potentially very large corpus as well as the ability to parse more complex instructions / queries. For example, if the device 102 is a speaker, a corpus on the device 102 with multiple audio / speech mechanisms 174 may support multiple commands / commands such as “play at higher volume”, “more gently”, etc. On the other hand, the external corpus provided by the additional function 120 may support more complex instructions such as, for example, “play the song that I listened to last night”. This latter requirement is grammatically more complex (and therefore probably requires higher speech recognition capabilities than what the device can provide), and may require access to external data (eg, in database 114).

  The interface mechanism 162 may be provided with or include multiple learning mechanisms so that they can learn about multiple users of the device and how they interact with the device. . For example, multiple voice / speech mechanisms 174 may learn to recognize multiple voices of a particular person (preferably including the person corresponding to user 110 associated with the device).

  The device is preferably initially configured with multiple generic corpora for its various interface mechanisms 162. As the device's multiple interface features 162 learn, they may update various corpora to reflect what they have learned. For example, when a device learns to recognize a particular person's voice, it may update multiple voice / speech corpuses for that person. Preferably, the device keeps a copy of the first plurality of common corpora and / or the plurality of original corpora can be stored somewhere separate from the device (eg, in a “cloud”) .

A device may have multiple different corpora of different users or people. However, it should be understood that when each device is associated with a user, each interface mechanism on the device preferably has at least one corpus associated with that user. Other people using the device may have their own corpora associated with it.
Monitoring multiple interactions

  As shown in FIG. 4I, device 102 monitors a number of possible interactions (at S402). The device can be continuously monitored (at S402) or it can be monitored at multiple specific times and / or under multiple specific conditions. This monitoring preferably uses one or more of a plurality of device sensors 138 (eg, device camera, microphone, etc.). To avoid missing any interaction, the device preferably buffers or stores possible interactions (at S404). These buffered interactions can be stored by various interface mechanisms 162 in any known and appropriate manner for later use, if desired. For example, sound detected by the microphone of the device can be buffered by the voice / speech mechanism 174, if necessary, in a manner suitable for later use. Similarly, external motion detected by the device's camera can be buffered by the device's gesture mechanism 166, if necessary, in a manner suitable for later use, and the plurality of images detected by the device can be The face / line-of-sight mechanism 182 and / or other mechanism 190 can be stored in a manner suitable for use, if desired. It should be appreciated and understood that not all sensor inputs can be buffered for all devices or all types of devices. However, even if the device only understands that the interaction will occur some time after the interaction has begun, the device can reconstruct the interaction, so buffering sensor information allows the device to It should also be understood that more precise multiple interactions may be provided via the flexible interface mechanism 162.

  As will be appreciated and understood by those skilled in the art upon reading this description, the amount of information buffered (at S404) depends on the type of information (eg, audio, multiple images, motion history, etc.). Similarly, different information may be buffered in different time periods, as will be appreciated and understood by those skilled in the art upon reading this description. For example, multiple sounds detected by the device's microphone can be buffered for 30 seconds, while multiple images detected by the camera can be buffered for 15 seconds. Buffering may use any technique, such as circular or wraparound buffering, and the system is not limited by the type of buffering used or the buffering implementation. Different buffering techniques can be used for different types of information. Device 102 should have sufficient memory to store the required buffered information.

  Buffering a possible interaction (at S404), the device attempts to determine whether the possible (buffered) interaction is an actual interaction (at S406). If the device determines (at S406) that the possible interaction is (or may be) an actual interaction, the device may continue processing the actual interaction (at S408). The actual dialog processing (in S408) may use the multiple corpus selection processing described above with reference to FIG. 4I.

  Determining (at S406) whether actual interaction with the device is (or may have occurred) may use one or more triggers. For example, if a person begins to give multiple voice commands to a device, the device may not know that the sound it is detecting corresponds to those voice commands. However, if a person is also looking at the device while speaking (as can be detected using gaze detection), the device will combine with the detected speech as a trigger to handle the actual interaction, Rely on the detected line of sight.

  Device 102 may operate in various types of environments, and multiple environmental factors and associated noise may affect interaction detection and processing. As used herein, the term “noise” for any type of information or signal can reduce the ability of information or signal processing (eg, corresponding detection and / or recognition ability of information in a signal). Information) and / or multiple signals. For example, the background sound of an air conditioner or fan can interfere with voice / speech recognition or reduce its ability, or a continuously flashing light can impair or be capable of multiple faces or gesture mechanisms. Can be reduced. It is therefore useful for the device to attempt to filter or remove noise prior to processing. In this regard, other mechanisms 164 may include one or more denoising filtering / cleaning mechanisms to remove noise from multiple inputs detected by various sensors. One such mechanism is noise cancellation that removes multiple ambient sounds (eg, from multiple air conditioners and multiple fans) from the sound detected (and buffered) by the microphone of the device. Different and / or other denoising filtering can be used, and these filtering mechanisms can adapt to and learn from the environment in which the device is located.

  In multiple cases where multiple filtering / cleaning mechanisms are used, preferably all buffered information is filtered / cleaned.

  The flowchart in FIG. 4J describes an exemplary process that the device 102 may use to determine (at S406 in FIG. 4I) whether an interaction is taking place. Recall that the various sensors 138 are monitored (at S402 in FIG. 4I) and the outputs of multiple sensors are buffered (at S404 in FIG. 4I) if actual interaction is detected and sensor outputs are required. I want to be. Outputs from various sensors 138 are provided to various interface mechanisms 162 (directly or via buffering), each of which attempts to determine whether an interaction is taking place. For example, gesture detection mechanism 168 attempts to determine if any movement they are detecting via camera sensors corresponds to a gesture (at S410). The face / gaze detection mechanism 184 attempts to determine if any images they detect via the camera sensor correspond to at least one face (at S412). The voice / speech detection mechanism 176 attempts to determine whether any sound they detect via the microphone sensor corresponds to speech (at S414). Other mechanisms 190 attempt to determine whether what they are detecting via sensor 138 (eg, touch, movement, proximity of other user devices) corresponds to interaction with the device (at S416). ). The user detection mechanism attempts to detect whether the user is interacting with the device (at S418).

  If a face (or more than one face) is detected (at S412), a gaze detection mechanism can be used to detect whether the detected face is looking at the device (at S420). A face motion detection mechanism 187 may be used to determine whether the face in motion is moving in response to a gesture and / or speech (at S422).

  The face motion detection mechanism 187 is also used alone or in conjunction with the face / gaze detection mechanism 184 to determine whether any detected face is moving in response to speech and / or multiple gestures. Can be.

  Some of the multiple mechanisms (eg, gaze detection and face motion detection) may depend on multiple determinations of other mechanisms, but the various interaction detection mechanisms may operate simultaneously (in the flowchart depicted in FIG. 4J). As shown).

In some implementations, various detection mechanisms may generate Boolean values (true or false) that reflect their detection decisions. In those cases, the final decision as to whether an interaction was detected is determined by the logical OR of those values. For example, if there are N detection mechanisms and the i-th detection mechanism produces a Boolean value b _i , the final determination as to whether an interaction has been detected can be calculated using:

  In this implementation, any true value (ie, any detection by any detection mechanism) results in a determination that an interaction is taking place. Referring to the flowchart in FIG. 4J, any “yes” value by any mechanism results in an affirmative decision that a dialogue is taking place.

  The label “Yes” on multiple lines in the flowchart of FIG. 4J represents a determination or determination by each of each mechanism that they are likely to have detected some feature (eg, gesture, face, speech, etc.). It should be understood. Similarly, the label “No” on multiple lines in the flow chart indicates a determination or determination by each of each mechanism that they are most likely not detecting any feature (eg, gesture, face, speech, etc.). Represent. Thus, the various labels on these lines should not be construed to mean that a feature has occurred or has not occurred, and it has been detected or not detected with some degree of accuracy. It should be understood that it means only. Thus, as one of ordinary skill in the art will realize upon reading this description, multiple “yes” labels on multiple flow lines in FIG. 4J are “maybe” or “maybe”. Or, it may be read as “somewhat so” and multiple “No” labels on multiple flow lines in the diagram may be read as “probably not”. Thus, if the value of Interaction Detected (determined using Equation 1 above) is “true”, it means “highly likely true” or “somehow more true than false”. It should be interpreted, and “false” should be interpreted to mean “high possibility of false”.

式２における複数の重みｗ_ｉは、Ｉｎｔｅｒａｃｔｉｏｎ Ｓｃｏｒｅ≦１．０になるようなものであることが理解されるべきである。Ｉｎｔｅｒａｃｔｉｏｎ Ｄｅｔｅｃｔｅｄの値は、例えば、Ｉｎｔｅｒａｃｔｉｏｎ Ｓｃｏｒｅ（例えば、数式２において決定されるような）を静的または動的閾値と比較することによって決定されうる。例えば、
Ｉｎｔｅｒａｃｔｉｏｎ Ｄｅｔｅｃｔｅｄ＝Ｉｎｔｅｒａｃｔｉｏｎ Ｓｃｏｒｅ＞閾値（Ｔ） （３） The device may use the probabilistic nature of interaction detection. Thus, in some implementations, each detection mechanism may generate a value or number that reflects the degree of accuracy of the detection result (eg, a real number from 0.0 to 1.0, where 0.0 is a dialog Means that no interaction was detected, 1.0 means that an interaction has been detected reliably, or an integer from 0 to 100, where 0 means no interaction, and 100 means a reliable interaction. means). In those cases, a final determination as to whether an interaction has been detected is determined using at least in part a score determined as a function of a plurality of values generated by various detection mechanisms. For example, if there are N detection mechanisms, the i-th detection mechanism generates a real value r _i in the range of 0.0 to 1.0, and the score of the i-th detection mechanism has a weight of w _i For a certain function F,

It should be understood that the weights w _i in Equation 2 are such that Interaction Score ≦ 1.0. The value of Interaction Detected can be determined, for example, by comparing Interaction Score (eg, as determined in Equation 2) with a static or dynamic threshold. For example,
Interaction Detected = Interaction Score> Threshold (T) (3)

上述の通り(式２を参照して)、複数の重みｗ_ｉはＩｎｔｅｒａｃｔｉｏｎ Ｓｃｏｒｅ≦１．０になるようなものである。 In some implementations, function F (Equation 2) generates a weighted sum or an average of multiple values generated by various detection mechanisms. Here, different weights may be given to different detection mechanisms depending on, for example, their known or recognized or previous accuracy. For example, if there are N detection mechanisms, the i-th detection mechanism generates a real value r _i in the range of 0.0 to 1.0, and the score of the i-th detection mechanism has a weight of w _i The final determination as to whether an interaction was detected can be calculated using:

As described above (see Equation 2), the plurality of weights w _i are such that Interaction Score ≦ 1.0.

Although a weighted sum is used in this example, different and / or other functions may be used to determine whether an interaction is taking place, as those skilled in the art will notice and understand upon reading this description. Can be used. For example, a weighted average score can be used to determine the Interaction Score, as shown in Equation 2 ″ below.

The flowchart depicted in FIG. 4K illustrates the interaction detection stage (S406 ′) in an implementation where each detection mechanism generates a score (eg, a real value R _{i in} the range of 0.0 to 1.0), The final decision as to whether interaction is detected is determined at least in part as a function of their scores (eg, according to Equation 2 or 2 ′ above). In this example, gesture detection mechanism 168 attempts to determine (at S410 ′) whether any movement they detect via camera sensors corresponds to the gesture, and scores (R ₁ ε [0) indicating this determination. .. 1]). The face / gaze detection mechanism 184 attempts to determine (at S412 ′) whether some of the images they detect via the camera sensor correspond to at least one face, and a score (R ₂ ε [ 0.1]). The voice / speech detection mechanism 176 attempts to determine (at S414 ′) whether some of the sounds they detect via the microphone sensor correspond to speech and scores (R ₃ ε [0. 1]). Other interface mechanisms 190 attempt to determine if what they detect via sensor 138 (eg, touch, movement, proximity of other user devices) corresponds to interaction with the device (at S416 '). ), Generating a score (R ₄ ε [0 ... 1]) indicating these decisions. The user detection mechanism attempts to detect whether the user is interacting with the device (at S418 ') and to generate a score (R ₇ ε [0 ... 1]) indicating these decisions.

As in the previous example (FIG. 4J), when a face is detected (in S412 ′), a gaze detection mechanism can be used (in S420 ′) to determine whether the detected face is looking at the device. ). In this case, the gaze detection mechanism, if used, generates a score (R ₅ ε [0 ... 1]) that indicates those decisions. Similarly, the face motion detection mechanism 187 determines whether the detected face is moving to correspond to gestures and / or speech (in S422 ′) and scores (R ₆ ∈ [0) indicating those determinations. .. 1]). The determination as to whether to start or use multiple gaze detection and / or face motion detection mechanisms may be based, for example, on the score (R ₂ ) generated by the face detection mechanism (at S412 ′). Here, a plurality of predetermined thresholds (denoted by T _G and T _M in FIG. 4K) can be used to initiate a plurality of gaze and / or face motion detections. These threshold values (T _G and T _M ) may be the same. These thresholds can be preset and fixed, or they can be dynamic, for example based on information that the system learns about its detection success.

As described above, each detection mechanism may have a corresponding detection weight associated with it. For example, it is summarized as shown in the following table.

The system preferably generates a running value (in S407) corresponding to a plurality of latest scores (in this example implementation, R _1... R ₇ ) generated by each of the various detection mechanisms. Each detection mechanism may have a corresponding detection weight associated with it, and the running value may be generated using various score weight functions (eg, using equations 2, 2 ′ or 2 ″ above). The determination as to whether an interaction has been detected is based at least in part on a comparison between this score (calculated in S407) and another threshold (indicated by the T _interaction in the drawing of FIG. 4K). sell. The threshold T _interaction can be preset and fixed, or it can be dynamic, for example, based on information the system learns about its detection success.

Various interaction detection mechanisms / processes may proceed in parallel and independently of each other. However, it is to be understood that various mechanisms / processes may access and use scores generated by other mechanisms in some cases. The multiple scores of these other mechanisms can be used, for example, to trigger other detections (the score (R ₂ ) generated by face detection in S412 ′ is used as a trigger for gaze and face motion detection. As in the exemplary case of FIG. 4K).

The weights for the various detection mechanisms can change dynamically based on information that the device has learned from previous detections. Each weight is a value in a range of 0 to max_weight with respect to a certain value of max_weight. Any value can be used as long as the score threshold (T- _interaction in the above example) is set accordingly, but for the purposes of this description, it is assumed that max_weight is 1.0. Initially all mechanisms can be given the same weight (eg, max_weight), and then the weights assigned to specific mechanisms can be adjusted based on the accuracy or usefulness of those mechanisms. For example, in a dark room, the gesture and face detection mechanism can be given a reduced weight, or in a noisy room (the device cannot filter out enough noise), the speech detection mechanism is small. Weights can be given. As the light in the room changes or the noise in the room is reduced, the weights of the corresponding features can be adjusted upwards. In addition, when the device learns from its previous multiple decisions, multiple weights can be adjusted.

  It should be understood that it is generally preferable to have a false positive detection of a possible interaction rather than missing an actual interaction. If the device does not provide a sufficient (eg, near real time) response to multiple interactions, multiple users will cease to use multiple interactions and possibly the device. Thus, in order to detect multiple possible interactions with the device, preferably the weights should be set conservatively high and the corresponding thresholds should be conservatively low.

  Each of the multiple detection mechanisms consumes power, and the constant use of these mechanisms by the device can cause it to use excessive power. The need for accurate dialog detection must therefore be balanced with the need to save power on the device. However, when the device is connected to an external power source, there is no need to impose restrictions on users of multiple detection mechanisms. Multiple triggers for various mechanisms can be used to save power. For example, multiple speech detection mechanisms can be triggered by a simpler sound detection mechanism (which consumes less power) and multiple face and gesture detection mechanisms (which consume less power) by a camera sensor It can be triggered by a simpler mechanism that detects multiple changes in multiple captured images.

  In some implementations, multiple devices can be put into a mode in which they perform minimal interaction detection (eg, sleep mode or ignore mode). This mode may be entered based on, for example, time, location, user command, or some other factor. For example, the device may be set to perform minimal interaction detection between midnight and 7:00 am on weekdays. In some cases where the device is used in a crowded or noisy location (eg, a speaker at a party or a telephone at a busy airport), the device ignores certain interactions or triggers multiple interactions Or it can be set to require multiple confirmations. In some cases, certain multiple interactions may be disabled based on what the device is currently doing. For example, speech input can be disabled while the music playback device is actively playing music, and can be reactivated between songs. This type of setting can be made by adjusting multiple weights associated with some of the multiple detection mechanisms (eg, gesture detection weights are set very low in dance parties). Setting a low weight for a particular detection mechanism does not disable that mechanism, so it still operates and consumes power. Further, even with a low weight value, the mechanism can detect gestures that the device can interpret as a possible interaction. In preferred implementations, certain dialog detection mechanisms can be turned off or disabled so that they do not operate or consume no power. In some cases this may be achieved by setting the corresponding weight of the detection mechanism to zero (0.0).

  In preferred implementations, each detection mechanism resets its score to zero (or false) after a period of time (eg, 5-10 seconds) or after the detected interaction is actually performed.

  The processing of ongoing checks for multiple possible dialogs (eg, S406 in FIG. 4I) is typically performed without knowing who has started the dialog. Thus, as long as this process requires (or can benefit from) the use of multiple corpora, the device preferably uses multiple corpora of the user associated with the device to perform multiple recognition processes. As described below, once a possible interaction is detected, the device may use (or attempt to use) different corpora to recognize and process the actual interaction.

Upon detecting a possible interaction (S406 in FIG. 4I), the device preferably maintains an indication of what type of interaction is occurring. This can be accomplished by having multiple possible dialog decision processes (S406 in FIGS. 4J, 4K) generate and provide information for use by the next process. In the case of implementation using Boolean operators Figure 4J, the process, a plurality of values used to determine the value of Interaction Detected In the above formula _1, b i, bits corresponding to i = 1 ... N A vector can be set. In the case of multiple real values used in the implementation in FIG. 4K, the process may be an array or vector of multiple values corresponding to the score generated by each detection mechanism for use by the next process. Can be set. Other approaches can be used for the device to maintain information about multiple possible interactions, and the system is not limited by the manner in which this information is maintained or communicated.

  While attempting to determine if an interaction is taking place (at S406 of FIG. 4I), the device may not know (or need not know) which corpora to use for the various interface mechanisms 162. . In preferred implementations, if interaction detection requires a corpus, the device 102 uses multiple corpora associated with the user 110 with which the device is associated. In some cases, the device may use other corpora (e.g., multiple common corpuses or multiple corpora associated with other authorized users) at this stage of processing. .

  Upon detecting a possible interaction (S406 in FIG. 4I), the device proceeds to process the interaction. The actual interaction processing preferably uses a plurality of interface mechanisms 162 along with other operation mechanisms 154 to process the interaction. To handle the interaction, the device may need to use multiple corpora for various interface mechanisms 162. Therefore, as shown in the flowchart in FIG. 4L, the device first determines which multiple corpora to use for the various interface mechanisms (S409).

  Multiple corpora (both on the device and in the backend) associate a particular person with that person if they recognize that person by their face, multiple gestures, voice, or something else Other corpora to be determined can be determined and organized, for example, as shown in FIGS. 4D-4H, so that they can be used by corresponding mechanisms. For example, if the device first recognizes a person's face, the device will access and use other corpus (FIG. 4D) associated with that person, if necessary (or it may be the user's smart Based on device (eg, phone or tablet)). Similarly, if a device first recognizes a person's multiple gestures (FIG. 4E), voice / speech (FIG. 4F), or some other aspect of a person (FIG. 4G), the device may, if necessary, May access and use other corpora associated with the. Further, if the device can determine the user with which it is interacting, the device may access multiple corpora of that user (FIG. 4H) if necessary. It should be recalled that within system 100, “user” 110 is a concept that is used to form some kind of association or association of one or more devices in the system. User 110 typically corresponds to a person. However, not all people who interact with or attempt to interact with a device are necessarily users within the system 100.

  An exemplary flow process used to select multiple corpora (S409 in FIG. 4L) is shown in FIG. 4M. For this exemplary implementation, it is assumed that the device includes at least some common corpora for various interface mechanisms. These common corpora may be included in the device during manufacture or during later provisioning or configuration. To reach this process (selecting multiple corpora), the device has detected some kind of interaction (at S406 in FIG. 4I). The device may have detected one or more of sound, external motion, touch, device movement, etc. This detection may use some or all of the plurality of device sensors 138, for example, the device microphone detects sound, the device camera detects external motion, the device accelerometer Detecting movement, multiple touch sensors on the device may detect that the device is being touched by a person or another device, the device may detect user interaction, and so on. At this point, the device uses which possible type of interaction was detected (eg, a bit vector (for the implementation of FIG. 4J), or a vector of multiple score values (for the implementation of FIG. 4K)). Recall that you can have indicators about The device may use this information to attempt to determine which interaction is and which corpora to use.

  Depending on which possible interaction the device has detected (at S406 in FIG. 4J, S406 ′ in FIG. 4K), then the device may determine whether it can recognize a particular user or person (FIG. 4M). In S426). If the device recognizes a user or person, the device selects the corresponding corpus of the user or person (at S428 in FIG. 4M). On the other hand, if the device does not recognize (at S426 in FIG. 4M) that the possible interaction corresponds to any user or person known to the device, the device selects multiple general corpora of the device (S430 in FIG. 4M). In).

  Depending on the type of interaction that may be detected by the device (at S424), the device uses an appropriate interface mechanism (at S426) to attempt to recognize a person or user known to the device. If a device has at least one corpus stored for a person or user, that person or user is considered known to the device. A person can become known to the device based on previous interactions with the device. Because the person is a user associated with the device, or the person is associated with a user associated with the device (eg, his friend) and given permission to access the device in certain multiple ways Because. In preferred implementations, the user associated with the device should always be known to the device.

  Thus, for example, as shown in FIG. 4I, it is detected that some interaction is taking place (at S424) and a determination is made as to whether the detected gestures correspond to those of a known user / person. To attempt, the device may use gesture detection mechanism 168 and gesture recognition mechanism 170 (at S432). If the device recognizes multiple gestures as being of a known user / person (at S432), the device selects the corresponding corpus (at S434). The device may use the mapping shown in FIG. 4E to determine multiple corpora based on recognized gestures.

  The device is also (or alternatively) a plurality of sensors 138 (eg, cameras) to attempt to determine whether the interaction is for a known user / person with a face known to the device or to the system. The face / line-of-sight detection mechanism 184 and the face / line-of-sight recognition mechanism 186 can be used in combination with a plurality of sensors) (in S436). If the device determines that a nearby or visible face is known to the device (ie, corresponds to a user / person known to the device), the device may select the corresponding corpora based on that face ( In S438). The device may use the mapping shown in FIG. 4D to determine multiple corpora based on the recognized face.

  The device may also (or alternatively) use the sound detected (using a plurality of sensors 138 that are microphones) and the sound / speech detection mechanism 176 and the sound / speech recognition mechanism 178 to detect the detected sound. It may attempt to determine (at S440) whether it is speech and, if so, whether it corresponds to a person / user speech known to the device. If it is determined that the detected sound is a person / user speech known to the device (at S440), the device may select a plurality of appropriate corpora based on the speech (at S442). The device may use the mapping shown in FIG. 4F to determine multiple corpora based on the recognized speech.

  The device may also (or alternatively) recognize the person / user using some other interface mechanism 190 (at S444), in which case the device may, for example, use the mapping shown in FIG. 4G. Based on that person / user, an appropriate corresponding corpus may be selected (at S446).

  The device may also (or alternatively) recognize that the interaction is with a known user (at S448), in which case the device may be based on, for example, using the mapping shown in FIG. 4H. A corresponding plurality of corresponding corpora may be selected (at S450).

  Although FIG. 4M shows various recognition attempts (at S426), not all of these steps are performed (or even available) on all devices or all types of devices. It should be understood. Further, it should be understood that in some devices or some types of devices, multiple stages may be performed in parallel, sequentially, or some combination thereof. In some cases, multiple tests may be used (at S426) to acknowledge and / or confirm recognition before multiple corpora are selected. Thus, for example, a particular device or type of device may first use facial recognition and then use some other technique only if it fails. In another example, a particular device or type of device may simultaneously recognize multiple faces (S412 in FIG. 4J; in S412 ′ in FIG. 4K) and recognize multiple users (S418 in FIG. 4J; FIG. 4K). In S418 ') and then optionally continue with other recognition approaches.

  As one skilled in the art will appreciate upon reading this description, the system is limited by the manner or order in which the various user / person recognition steps are performed (including parallel, serial, or combinations thereof). In addition, the system is not limited in any way by the multiple mechanisms available or used to attempt user / person recognition.

  If more than one recognition mechanism is used (in S426), the device needs to be able to handle multiple conflicts. For example, a conflict may occur if one mechanism identifies one user / person and another recognition mechanism identifies different users / persons. This can occur when the device cannot determine enough information about the person / user interacting with the device because the device does not have enough information about the person and / or interacts with the device This may be because there are two or more people who are likely to do. For example, a new device may not yet have enough information about its potential users to make accurate multiple recognition decisions, so multiple different mechanisms will make multiple different recognition decisions. sell. The device can also be confused by the presence of more than one person in the vicinity, so the sensors capture details from different people. For example, if there are multiple people in the vicinity of the device, the microphone may capture the voice of one person while the camera captures the face of another person.

  It is also possible for a particular sensor to recognize more than one person. For example, a camera in the device may recognize more than one face or find multiple faces.

  Device 102 preferably has at least one conflict resolution strategy. The conflict resolution strategy may be said to be adaptive as the device learns based on multiple previous interactions and multiple recognition decisions. In some cases, the device may use multiple recognition decision weight functions. For example, give the largest weight for user recognition (at S448), a smaller weight for face recognition (at S436), and an even smaller weight for speech / speech recognition (at S440). Etc. As will be appreciated and understood by those skilled in the art upon reading this description, different and / or other functions may be used to provide different and / or other conflict resolution strategies. For example, the conflict resolution strategy can be dynamic while changing over time (eg, based on learning).

  In addition to multiple conflict resolution strategies, device 102 may also include various optimizations to improve awareness of whether a person is attempting to interact with the device and if so. One exemplary optimization is to use gaze detection (using the face / gaze mechanism 182) to determine if someone is actually looking at the device. Gaze detection can be used to trigger corpus selection, for example, as a trigger for other recognitions and multiple interactions and / or as part of a conflict resolution strategy. For example, if the device detects sound (using its own microphone), the device may not know whether the sound corresponds to multiple commands or multiple queries for the device. If the device can also detect that someone is looking at the device, that line of sight can be used to trigger speech recognition, alone or with the detected sound. It should be understood that gaze detection does not require or rely on face recognition.

  As an additional optimization, in some implementations, the face detected by the device (using the face / gaze detection mechanism 184) is moving so that it can be used to confirm (or reject) the detection. To determine whether the device can use a face motion detection mechanism 187. For example, if the system finds multiple faces (using the face / gaze detection mechanism 184) and also detects speech (using the voice / speech mechanism 174), the movement corresponding to the speech (eg, moving the mouth) Any face that indicates that it is) is a suitable candidate for selection. It should be understood that the face motion detection mechanism 187 need not interpret speech, and they only need to detect mouth or chin motion or some other type of motion that can accommodate speech. .

  In some implementations, the face motion detection mechanism 187 is used to detect movement of other parts of the face, eg, eyebrows, eyes, chin, mouth, etc., as used by the gesture mechanism 166. Can be. The multiple facial movements are multiple gestures that are themselves recognized by the device, or they can be used to identify other multiple detected gestures.

  Similarly, multiple gestures (as determined by gesture mechanism 166) can be alone or to confirm other detected information (eg, multiple faces, multiple sounds, multiple movements, etc.) Can be used.

  As those skilled in the art will appreciate upon reading this description, in general, each interface mechanism 162 may be used alone or in combination with multiple other interface mechanisms. When used together, the various interface mechanisms can be used to confirm each other and / or as multiple triggers for each other.

  It is understood that not all devices or all types of devices need to have all of the multiple interface mechanisms, and not all of these mechanisms can be used on all devices or all types of devices. It should be. In addition, it should be understood that some devices may use different and / or other techniques to determine which corpora to use.

  Interaction detection (eg, S406 in FIG. 4J, S406 ′ in FIG. 4K) and corpus / multiple corpus selection (FIG. 4L) are shown as separate processes, but those skilled in the art will be aware upon reading this description. As would be the case, some of the decisions made by dialog detection are used in subsequent processing, eg, in determining whether a person / user is recognized (at S426 in FIG. 4M). Can be. Therefore, in some cases, in the process of detecting a possible dialog (S406 in FIG. 4J, S406 ′ in FIG. 4K), the user / person recognition process (S426) may be skipped (or simplified) The device may have determined sufficient information about the person / user interacting with the device.

  Referring again to FIG. 4L, it is determined which corpus is to be used for the actual interaction (at S409) and the determined corpus is used (at S409) and the various interface mechanisms 162. The device can then proceed to determine the actual interaction (at S411). As will become apparent to those skilled in the art upon reading this description, the actual interaction may be determined (completely or partially) by multiple processes of human recognition (S426) and corpus selection (S428). If these two processes do not result in a determination of the actual interaction, the device proceeds to determine the actual interaction (at S411).

An alternative implementation / approach that handles the actual embodiment assumes that the corpus to be used is that of the device user 110, unless someone else is specifically recognized or identified. . In this exemplary implementation, multiple scores generated by multiple interaction detection mechanisms are used to control which multiple recognition mechanisms are used. Referring to the flowchart of FIG. 4N, the device determines which corpus to use for the actual interaction (at S409 ″). If the user detection process (eg, S418 ″ in FIG. 4K) generates a score that is greater than a certain threshold (indicated by T _user in FIG. 4N), the device attempts to recognize the user (in S418 ″) and succeeds. For example, a plurality of recognized corpus of the user is selected (in S450 ″). If the device fails to recognize a user (preferably an authorized user), or if the score (R ₇ ) generated by the user detection process / mechanism does not exceed a threshold (T _user ), the device will A plurality of corpora associated with the user of the device). Having a plurality of selected corpora (in S409 ″), the device proceeds to the actual dialog decision (in S411 ″). In the example implementation of FIG. 4N, the device may use multiple scores generated by various interface detection mechanisms to determine whether to activate the corresponding multiple recognition mechanisms. For example, as shown in FIG. 4N, if the score (R ₁ ) generated by gesture detection mechanism 168 (eg, at S410 ′ in FIG. 4K) exceeds a threshold (indicated by T _gesture in FIG. 4N), the device Can activate the gesture recognition mechanism 170 (in S410 ″). Similarly, if the score (R ₂ ) generated by the face detection mechanism 168 (eg, at S412 ′ in FIG. 4K) exceeds a threshold (indicated by a T _face in FIG. 4N), the device activates the face recognition mechanism 184. Yes (in S412 ″). And similarly, if the score (R ₃ ) generated by the speech / speech detection mechanism 176 (eg, at S414 ′ in FIG. 4K) exceeds a threshold (indicated by T _speech in FIG. 4N), the device and give start recognition mechanism 178 (in S414''), it is produced by other interface detection mechanism 190 (e.g., in S416' in FIG 4K) score _{(R 4)} represented by T _other in the threshold (Fig. 4N The device may activate another interface mechanism 190 (in S416 ″).

  It should be understood and understood that the various mechanisms used to determine the interaction can be based on a simultaneous or some predetermined order (eg, scores generated by multiple detection mechanisms or weighted scores). It is possible to go on. As one skilled in the art will appreciate upon reading this description, the system is not limited in any way by the order in which the various recognition mechanisms are activated.

  It should also be understood and appreciated that some of the plurality of recognition mechanisms may operate with or be integrated with a plurality of other recognition mechanisms.

  The thresholds described with reference to exemplary implementations (eg, in FIG. 4N) can be static or dynamic, and activation of any particular recognition mechanism is different and / or Or it can be based on several other factors. In some implementations, some or all of the plurality of thresholds may be changed based on information that the device learns from multiple previous interactions. As will be appreciated and understood by those skilled in the art upon reading this description, the system is not limited in any way by multiple values or multiple triggers used to activate the various recognition mechanisms.

As is apparent from the diagram in FIG. 4N, some of the mechanisms used to detect multiple possible dialogs (eg, gaze detection and face motion detection) are required from the multiple dialog recognition mechanisms. Cannot be done or cannot be used. Thus, for example, when a line of sight is detected, that information can be used to trigger dialogue recognition, but it can no longer be needed from actual recognition. It should be understood that some of the information used during the detection process (in addition to multiple scores generated by various mechanisms) can be provided by multiple recognition mechanisms. So, for example, the face recognition mechanism (activated in S412 ″ in FIG. 4N) may already have information from the line of sight detection mechanism to know which face to try to recognize. Alternatively, the face recognition mechanism and / or gesture recognition mechanism may themselves activate the gaze detection and / or face movement detection mechanism.
Processing the actual conversation

  Having determined the actual interaction (at S411), the device proceeds to execute the instructions or commands or queries associated with the actual interaction (at S413). The actual interaction determination and / or execution (at S411, S413) can be done locally on the device, or it can be from the backend 104 as well as processing or assistance from multiple additional functional entities 120. Information (eg, from database 114), interaction with some other entity (eg, social networking service 116 or content provider 118, etc.) via the backend may be required (or benefited from). The actual execution of the interaction may also involve an interaction with another device 102. The multiple interactions between the device and the back end correspond to arc # 7 in FIG. 1, the multiple interactions with the back end database 114 correspond to arc # 8, and the social networking service 116 or content provider 118 The multiple dialogs correspond to arc lines # 9 and # 10, respectively, and the multiple dialogs with multiple additional function entities 120 correspond to arc line # 11. Device interactions with other devices correspond to arc # 1 in FIG.

  The actual interaction execution (in S413) may require the device to use previously buffered information (S404 in FIG. 4I). For example, if the device detects and buffers a sound that may be a speech dialogue, the device also detects (or does not detect) a person who is looking (looking at) the device (eg, S420 in FIG. 4J). , S420 ′ of FIG. 4K), the device cannot start speech recognition. Therefore, the line of sight acts as a trigger for subsequent speech recognition (eg, S414 ″ in FIG. 4N). This approach allows the device to capture speech or other information (eg, multiple gestures) that it begins to give while the person is not yet looking at the device and continues to provide while looking at the device. Other conversations (eg, multiple gestures, multiple facial movements, etc.) can be used as multiple triggers for later recognition of multiple conversations, including multiple conversations that could already be initiated It should be understood. Since the system has a buffered possible interaction, the buffered information can be used when one or more trigger events occur.

As one of ordinary skill in the art will appreciate upon reading this description, any type of information that can be sensed and / or measured by multiple sensors 138 can be buffered, from which one or more trigger events are detected. Can be used after
Learning

  As noted above, the various mechanisms used for detecting and recognizing multiple interactions can include multiple learning functions so that they can become more accurate over time and use. In some cases, when a mechanism learns about a particular user, that mechanism may update that user's corresponding corpora. For example, when a speech recognition mechanism learns a particular user's speech, it may update multiple speech corpuses associated with that particular user.

  In addition to individual mechanisms learning from their interactions with multiple users, the device itself can learn how a particular user interacts with the device and based on that learning Its processing can be optimized. For example, if a particular user always uses the same gesture (eg, pointing to the device) in combination with multiple speech commands, the device can learn the pattern. In that example, the pointing gesture can be given a higher weight as a trigger for speech recognition. As another example, if a particular user always uses a specific hand gesture with multiple facial movements (eg, raising eyebrows) and a specific plurality of words, those features are combined and more depending on the device High weight can be given.

Although various operating mechanisms are shown in the diagram in FIG. 4B as separate mechanisms, these separations are provided by way of example only and to assist in this description, and the scope of this description There is no intention to limit this. As one of ordinary skill in the art will appreciate and understand upon reading this description, multiple separations in FIG. 4B are not required and some or all of the multiple mechanisms may differ and / or other multiple Can be combined into one functional unit, which includes a combination into one functional operating mechanism. For example, gesture detection mechanism 168 and gesture recognition mechanism 170 can be part of one gesture mechanism. Similarly, voice / speech detection mechanism 176 and voice / speech recognition mechanism 178 may be part of one voice / speech mechanism. Similarly, the face / gaze detection mechanism 184 and the face / gaze recognition mechanism 186 may be part of one face / gaze detection and recognition mechanism. Similarly, the logical depiction of multiple operating mechanism storage 155 in FIG. 4C is provided to aid in this description and is in no way intended to limit the scope of this description. As will be appreciated and understood by those skilled in the art upon reading this description, different and / or other storage mechanisms are possible and contemplated herein.
Multiple device states, multiple device provisioning and configuration

  As used herein, with reference to device 102 within framework 100, the term “provisioning” refers to the process of installing (or updating) various system features used by the device. Provisioning can include installing and / or updating firmware or software on the device. As used herein, with reference to device 102 within framework 100, the term “configuring” refers to operational options and / or multiple configurations for various mechanisms used by the device. Refers to the process of establishing or setting parameters. For example, configuring a device may include setting a plurality of passwords for the device, establishing a plurality of network parameters for the device, and the like.

  Referring now to FIG. 6A, the device 102 may be considered in various provisioning and configuration states. A device 102 is in a pre-provisioned state if it has not yet been provisioned with multiple system features. A device 102 is in a generic provisioned state if it has been provisioned with multiple system features but has not yet been associated with a user 110. Device 102 is in an unconfigured state if it has not been configured yet. A device 102 is in a generic configured state when it is configured for use on the system 100 but is not configured for a particular user 110. A device 102 is said to be in a user-provisioned state when it is provisioned for a particular user 110. Device 102 is said to be in a user-configured state if it is configured for a particular user 110. Various possible state transitions of device 102 (shown as T1, T2 ... T6) are shown in FIG. 6A. These states and transitions are merely descriptive and exemplary and are used here only to assist in this description. It should be understood that these various states, what they are called, or the multiple state transitions described herein, do not limit the system. It should also be understood that these states and transitions may be independent of device specific functions.

  There are two aspects to provisioning and configuration of the device 102 within the framework 100. The first aspect is essentially independent of any user, and makes the device 102 subject to multiple, up-to-date, and / or localized versions of all system features. This aspect corresponds to device state transition T1 from pre-provisioning to general provisioning and may also correspond to transition T2 from unconfigured to general configuration.

  A second aspect of provisioning and configuring the device 102 within the framework 100 causes the device to comply with the configuration / requests of a particular user 110 currently associated with the device. For previously commonly provisioned or commonly configured devices, this aspect corresponds to the general provisioning to user provisioning state transition T3, and from general configuration to user configuration. There is also a possibility of corresponding to the transition T4. The association of a device with a particular user can probably change only temporarily or under limited conditions (eg, location, duration, usage, etc.). Thus, this second aspect of provisioning and configuration is also in state transition T5 where the user provisioning device is provisioned for different users and / or state transition T6 where the user configuration device is set for different users. It can correspond to.

  Thus, the process of provisioning a device may include installing multiple latest versions of all features (eg, software, firmware, etc.) on the device. When device 102 is manufactured, manufacturer 108 typically installs a version of multiple system features 134 on the device (along with multiple versions of other features such as for device-specific features). However, these versions are often outdated even at the time of manufacture. Thus, in the provisioning process, the bootstrap / provisioning mechanism 152 may include some or all of the mechanisms (eg, software, firmware, etc.) either at initial power up or even before the device is delivered or sold. ) Can be updated. In some cases, the device 102 will power up after a period of time (eg, 36 hours) to attempt to update all mechanisms and then in a low power mode with a timer setting that causes the bootstrap / provisioning mechanism 152 to operate. Can be delivered. When the device 102 is powered on, it will look for known wireless (Wi-Fi) networks or other ways to connect to the Internet. Preferably the device is pre-configured (as part of its general settings) with information about multiple known and trusted networks, and the device is known and reliable where multiple updates can be obtained These networks are used to connect to the source location via the network 101. Once the network connection is discovered and established, the device 102 can initiate updates on its own from a trusted source. Thus, if a device reaches the user (eg after it has been sold to the user), the device should have the most recent version of all features (ie, the latest version) and the user Should be fully (or substantially fully) provisioned against.

  For those mechanisms that may require multiple corpora (eg, speech recognition, gesture recognition, etc.), multiple generic corpora are initially installed on the device.

  Preferably, each device 102 is configured with sufficient information to allow the device to establish a secure contact with the backend 104 (eg, at the time of manufacture). To achieve this goal, the first general configuration for a device is multiple for various wireless networks to support secure wireless (eg, cellular or WiFi) communication between the device and the backend 104. Name and multiple passwords, and / or multiple general certificates. The term secure is used herein to refer to a plurality of communication channels that are reliable and preferably not spoofed. The degree of security depends on the type of device, and multiple different types of devices may require multiple different degrees of security for device-to-device and device-to-backend communications.

  The process by the manufacturer until the device is associated with any user corresponds to the state transition T1 (possibly T2) in FIG. 6A.

  Each device 102 is preferably associated with a user 110 to operate to use the system 100. A device may not be associated with a user (eg, when first manufactured), but a device may not be associated with more than one owner (and preferably more than one user at a time). Cannot be associated). As described herein, the device 102 can be used by more than one person or user 110, but within the system 100, the device is only associated with one user.

  In some cases, as part of the provisioning phase of the manufacturing and / or provisioning process, the device may be associated with the user (e.g., the user orders or purchases the device and the user's identification information before the device 102 is manufactured). (When providing a user ID).

  The first provisioning process (described above) may be performed without the device being associated with the user. Preferably, the second level of provisioning and configuration occurs when device 102 is associated with user 110. In particular, when the device 102 is associated with a user, the device 102 can obtain configuration information (eg, wireless network information such as multiple network IDs and multiple passwords) from the user. If a device is configured for a particular user, the device may obtain information about that user from the user database 130. For example, the device 102 may obtain user profile information, multiple user local corpora, and / or configuration information about the user. This information can be stored by the device, for example, to be used by multiple operating mechanisms 155. For example, the device 102 may obtain a plurality of user local corpora from the user database 130 and store the corpora in the corresponding appropriate interface mechanism storage 163 on the device. Thus, if a user has already used a specific device or type of device, it can be said that the newly acquired device does not need to be retrained to detect and recognize various interactions with the user.

  Some information from the user database 130 (e.g., a list of multiple devices of the user—a list of multiple device IDs of multiple devices owned by the user, and a list of multiple users of the user—multiple of the user A list of friends' multiple user IDs) may be encoded with a one-way encoding on the device (eg, using a cryptographic hash such as MD5). In this way, multiple device IDs and multiple user IDs are not disclosed, but (as will be explained later) information in multiple lists can be used (eg, possible multiples between two devices). To evaluate the relationship).

  The device and multiple user databases are preferably updated (preferably in near real time) to reflect the provisioning and configuration status of each device, so preferably the device database is current for the status and configuration of each device in the system. Provide a display of It should be understood that multiple devices that are not connected to network 101, cannot connect, or use some other method (eg, a cellular system) may postpone providing information to the backend. . However, preferably each device updates its backend with its state information while connected to the network 101 and when reconnecting. Similarly, if the device 102 connects to the back end 104 in any way, the device preferably obtains multiple update data from the back end, including any update data regarding the user of the device.

  In particular, each device 102 should periodically update multiple corpora in the device database 128 to reflect the current multiple corpora on the device. These updates are made periodically (eg, once a day) or whenever the device can connect to the database (via the backend 104) and determine that the device database needs to be updated. Can be done.

  In addition, the information on the corpora in the user database 130 should reflect the current state of the corpora on each user's device. Accordingly, the user database 130 should be updated to reflect the user's latest corpus (eg, similar to the device database). Recall that the user database 130 may include multiple user local corpora and multiple user extended corpora. Multiple user local corpora correspond to multiple corpora on multiple devices of the user. Multiple user corpus corresponds to multiple corpuses used by the backend or other external systems (eg, additional function 120) to handle multiple user interactions. For example, a plurality of user local corpora may include a plurality of limited speech recognition corpora that the device 102 may use, while a plurality of user extended corpora may be used, for example, by the additional function 120. May include multiple extended speech recognition corpora of the user.

  Multiple corpora for each user may be organized or stored on the user database 130 based on the user's multiple device types or multiple capabilities. This allows the system to support multiple types of devices. When a user updates the user database 130 with multiple corpora from a particular device, the database preferably stores those corpora based on device type and multiple capabilities.

  Preferably, device database 128 and user database 130 maintain previous versions of multiple corpora.

  In some cases, a user may have multiple devices of the same type (eg, multiple speakers). Preferably, the corpora on each of the user's devices (especially devices of the same type) have the latest version of the user's corpora for that type of device. Therefore, each device should contact the back end periodically to determine if it has the latest version of multiple corpora. Otherwise, the device may obtain the latest versions of the corpora from the user database 130.

Device provisioning and configuration, if initially associated with a user, corresponds to state transition T3 (and possibly T4) of FIG. 6A.
Device manufacturing and provisioning

  FIGS. 6B-6I illustrate multiple examples of manufacturing multiple devices 102. Device manufacturer 108 is preferably authorized to manufacture a plurality of system capable devices 102 from system 100.

  In the example in FIG. 6B, each device has a unique serial number provided by the manufacturer, and each device has a device ID corresponding to the unique serial number of the device. In the example in FIG. 6C, each device has a unique serial number provided by the M system, as described below. It should be understood that the unique serial number used by the M system may be different from other serial numbers used by the device manufacturer.

  6B and 6C, in one exemplary embodiment, device manufacturer 108 assigns multiple device serial numbers (individually) to device certificate generator 172 (which may be part of back end 104). Or in batch) (in S601). The device certificate generator 172 uses the device information (eg, serial number) provided by the manufacturer to create a unique device ID for that device (ie, for the device associated with or associated with that serial number). (In S602). The unique device ID is transmitted to the device CA 124 (at S603), and the device CA inserts it into the certificate signed by the device CA 124. The device CA 124 then sends the signed certificate back to the manufacturer (in S604, S605) and is inserted into the device. The signed certificate having the unique device ID is stored in the device certificate 150.

  As one skilled in the art will appreciate upon reading this description, different and / or other information (other than or in addition to that provided by manufacturer 108 in S601) may generate multiple device certificates. Can be used by the device certificate generator (at S602).

  Preferably, the information provided by the manufacturer (at S602) also includes information about multiple functions and / or multiple components of the device.

  Once the device certificate is generated and signed, information about the certificate (which may include a copy of the certificate) is sent to the backend 112 (at S606). The device ID associated with the certificate can be used by the device database 128 as a key or index into the database. Since the backend 104 is provided with information that is used to generate multiple device IDs, information about the device (eg, its capabilities) can already be known by the backend, so they are in the device database 128. Can be associated with the device ID.

  Preferably the information in the certificate is encrypted so that it can only be read by the device with an appropriate decryption key.

  Once the device certificate is generated by backend 104 (eg, via device certificate generator 172), the certificate can be associated with one device (with a unique device ID associated with the certificate). It should be understood that only. If for some reason a device becomes inactive or lost (or never actually manufactured), its device ID and certificate should not be reused.

  A plurality of dialogs (through the device certificate generator 172) (in S601, S605, and S606) between the manufacturer 108 and the backend 104 correspond to the dialog # 13 in FIG. The multiple interactions between device CA 124 and backend 104 (via device certificate generator 172) correspond to interaction # 6 in FIG.

  In some cases, the device certificate generator 172 may provide a large number of certificates to the manufacturer based on a list of multiple serial numbers from the manufacturer.

  In an alternative and presently preferred exemplary embodiment, the system generates (or obtains) a plurality of chunks of a plurality of serial numbers and a plurality of associated certificates, and the plurality of chunks of the serial numbers and certificates. To multiple manufacturers. Accordingly, referring to the diagrams in FIGS. 6D and 6E, the back end 104 generates a plurality of serial numbers and provides them to the device certificate generator 172 ′ (in S651). As in the previous example, the device certificate generator 172 ′ can be part of the back end 104 and / or can be co-located with the back end 104.

  The device certificate generator 172 ′ uses information provided by the backend (eg, serial number) to create a unique device ID for the device associated with or associated with the serial number. The unique device ID is transmitted to the device CA 124 (at S653), and the device CA 124 inserts it into the certificate signed by the device CA 124. The device CA 124 then sends the signed certificate back to the device certificate generator 172 ′ (in S654), and the device certificate generator 172 ′ sends it back to the back end (in S655). The signed certificate with the unique device ID is stored in the device certificate 150.

  Information in multiple certificates is preferably encrypted so that it can only be read by authorized devices. The device certificate generator 172 ′ may encrypt the serial number and unique device ID before providing the certificate to the device CA 124.

  As those skilled in the art will appreciate upon reading this description, since the backend 104 provides multiple serial numbers, these numbers are used at least in part to form multiple device IDs. Can be.

  The back end 104 provides a plurality of serial numbers and a corresponding plurality of certificates to the device manufacturer 108, preferably in the form of such a plurality of numbers and a plurality of certificates (at S656). The manufacturer can use some or all of those certificates in multiple devices and provides back end 104 with a list of serial numbers / multiple certificates that it uses (in S657).

  Since the backend has a list of all serial numbers and multiple copies of all certificates, it can track multiple numbers used, thereby verifying information provided by multiple manufacturers. Should be understood. For example, if a manufacturer fails to report the use of a particular serial number on a particular device, the back end will detect the use of that serial number when that particular device connects to the system. Similarly, if a manufacturer uses the same serial number / certificate combination for multiple devices, the backend will detect the duplication.

  The flow chart in FIG. 6E illustrates several aspects of the manufacturing embodiment described in FIG. 6D. The vertical dashed lines in FIG. 6E are provided to indicate which aspects are performed by the system and which aspects are performed by the manufacturer. The system generates multiple serial numbers and multiple certificates (at S661) and provides them to the backend (at S662), preferably in multiple chunks of multiple numbers / multiple certificates. The manufacturer acquires a group of a plurality of numbers / a plurality of certificates (in S663), and uses some of them for a plurality of devices (in S664). The manufacturer reports which serial number / multiple certificates have been used (in S665), and the back end obtains a list of the serial numbers / multiple certificates used (in S666).

In some implementations, the manufacturer may report additional information such as device type, function, etc. (in S665).
Provisioning, configuration, and association to users

  An exemplary initial provisioning and configuration of device 102 (corresponding to state changes T1 and T2 in FIG. 6A, respectively) is shown in the flow diagram of FIG. 6F. At this stage, it is assumed that the device has a device ID associated with it (as described above). The manufacturer provides the device 102 with a plurality of required mechanisms and other components (for example, a plurality of system mechanisms 134, a plurality of sensors 138, a plurality of communications 142) as necessary (in S608). ). These features and other components can be provided in the form of a single substrate or kit with multiple connections to multiple device specific components, or they can be completely combined with multiple device specific components. Or it can be partially integrated. When provided as a kit or substrate, not all components can be active or activated. For example, a plurality of specific components may be deactivated based on the location of the device, the device type, or the reason that users are charged an additional fee for their subsequent activation.

  Multiple features are installed and the device is provided with initial settings (at S610). The initial settings can include multiple generic corpora, as needed, for various mechanisms. The initial settings may also include information (eg, information about known or trusted networks) that supports connecting the device to the back end.

  Preferably, the device maintains a list or manifest of the current state of all features and their settings, including version information and details of multiple corpora. When a device is initially provisioned and configured (eg, during manufacturing), its current manifest is provided to the back end for storage in a database entry associated with the device (at S612).

As described above, a provisioned / configured device may update its multiple features and settings before being associated with a user (at S614). For example, the device may use a known / reliable Wi-Fi connection to update its firmware during delivery. All such updates should be reflected in the device's manifest, both on the device and in the device database (at S616). The device database may maintain a history of updates to devices. Some mechanisms used by the device (eg, voice / speech recognition, gesture recognition, etc.) may be provided by multiple third parties. In those cases, the mechanism may have firmware and / or multiple corpora included therein, and multiple updates to those components must be obtained from multiple third parties.
user registration

  Recall that each user 110 must have at least one unique user identification information (ID) within the system 100. Preferably, each user 110 acquires their user ID by registering with the system 100. User registration may be performed by offline processing, by a web interface to the back end, or by the device 102. The user may register before having any device 102. As part of the user registration (as described above), each user has a user ID that must be unique within the system 100. When a user registers, an entry for that user is created in a user database 130 that is preferably keyed or indexed to the primary by the user's user ID. The user's database entries (corresponding to their user IDs) are populated during the registration process and information provided by the user (eg, via a form or survey) and / or the user (directly or indirectly) Include information that the system can obtain or derive based on the information provided. For example, if a user uses a social network ID or registers via a social network, information about the user from that social network may be included in the database.

  The user database 130 is preferably indexed using a user ID, but different from this and / or others to access data in the user database, as those skilled in the art will recognize and understand upon reading this description. A plurality of keys can be used.

  As described above, multiple devices 102 may be associated with each user 110. While the plurality of devices preferably do not have a hierarchical status, it is useful to describe the process of setting the first device that the user obtains.

  As shown in FIG. 2, each user preferably has at least one user device 174 (eg, a smartphone or the like) that has a stored user ID of the user and an associated user certificate 175. Yes. User device 174 may also have one or more system features 178 (eg, in the form of an application (app) running on user device 174). The system mechanism / user application 178 provides the user with a way to configure the user device 174 within the system 100, as well as other aspects of the user device. In particular, the user device 174 allows the user to configure Wi-Fi and other network information (such as identification data for multiple local Wi-Fi networks, multiple service set identifiers (SSID), and multiple passwords). Can be provided. This information can be stored as setting information 180 on the user device 174 (FIG. 2), and can also be stored as setting information associated with the user in the user database 130 (FIG. 3A). As will be appreciated and understood by those skilled in the art upon reading this description, this configuration information should be treated with caution and kept secret (eg, by encryption).

  It should be understood that the user device 174 can be an example of the device 102.

Exemplary user registration is described with reference to FIG. 6G. First, the user obtains a user ID from the system (in S620) (eg, as described above). The system then creates a user database entry for the user (preferably keyed or indexed with the user ID) (at S622). The system then populates the database fields for the user (at S624).
Associating devices with users

  Each device 102 needs to be associated with a user 110 in order for the device to operate fully within the system 100. It should be understood that a device that is not associated with any user may still provide some or all of the device-specific functionality.

  When a user acquires a device 102 (eg, a new device), that device needs to be associated with that user. An exemplary device association with a user is described with reference to FIG. 6H. First (at S626), the device is associated with the user in the device database 128 and the user database 130. In the device database 128, the owner of the device is set to the user ID of the user. In the user database 130, the unique device ID of the device is added to a plurality of devices associated with the user ID of the user. Information about the device may be added to other fields in the user database 130 and the history updated to reflect the user's association with this device.

  In some embodiments, a device can become associated with a user by having the user touch (or tap) the device with another device of the user. In preferred implementations, if a particular device is not already owned by any user, the first time that particular device is tapped by another device, that particular device will be associated with the other device's user. Be able to. The specific device may obtain user setting information from the backend database and / or from the device of the user tapping it (S628 in FIG. 6H). If a particular device is already associated with the user, the next touch (or tap) from another device can be used to provide multiple temporary permissions to that particular device. Thus, for example, it allows multiple devices to be combined in some way, or that particular device takes over (preferably temporarily) some configuration information from the device that touched it Enable. Once multiple devices are paired (eg, by touch), they can then share information via a Bluetooth® signal or WiFi signal, or the like. It should be understood that the sharing of information can take several forms. For example, content (eg, music or video content) can be shared via WiFi, while metadata can be shared via Bluetooth. Furthermore, if previously paired, the two devices can detect each other's presence to continue or re-establish cooperation (eg, via Bluetooth signals within their range). .

  Preferably, each device maintains some information about the user associated with that device (eg, system features / data 134 of FIG. 4A). Information on the device may be updated to reflect information about the user (at S628). This information may include at least some of the information associated with the user ID of the user in the user database 130. For example, the information stored on the device 102 may include the user's user ID, information from the user's profile, information about multiple other devices associated with the user, information about the user's multiple friends (eg, their Each user ID), multiple user certificates, multiple corpus of users, and user configuration information. It should be understood that at least some of the user information on the device should be stored and maintained, preferably in a secure manner, eg, in encrypted form.

  Once the device has been updated to include user information (at S628), the device database 128 may need to be updated to reflect multiple changes made to the device (at S630). For example, if multiple local corpora of a user are stored on a device (for example, multiple generic corpora instead of whatever corpora already exists there), the device database 128 is ( It should be updated to reflect this information (in the device's multiple corpora and device history).

  The exemplary process described here for associating a device with a particular user 110 in the system 100 includes a general provisioning to user provisioning state change T3 in FIG. 6A, and a general setting to user setting. This corresponds to the state change T4.

If a user obtains a device that was previously used in the system (and hence is associated with another user), the device may first need to be restored to a state where it does not have user information associated with it. . The device can be restored by any technique that causes its factory settings (or multiple previous user settings) to be restored. This kind of reset corresponds to the state change T3 ′ for returning from user provisioning to general provisioning and the state change T4 ′ for returning from user settings to general settings in FIG. 6A.
Associating configuration information with devices

  As described above, the device may obtain user configuration information from the user database 130 (or from another device) when the device is first associated with the user (S628 in FIG. 6H). However, user information can change (e.g., the user gets a new friend in the system, or the user has updated or new wireless network information, or the user has new cellular communication information, Etc.), the information should propagate from the database to the device (and vice versa). Accordingly, as shown in FIG. 6H, the provision of user setting information to the device (in S628) and the updating of the user database 130 and device database 128 (in S630) are repeated as necessary (possible). If). For example, the device may check multiple databases when possible (eg, if it can connect to the backend 104) to determine if it has the latest version of the user's information. In some cases, the back end may attempt to push multiple notifications to alert the device about user information updates. Since the device may make changes (or cause changes) to the user's information (eg, the device may have updated multiple corpora or network configuration information), these changes will also propagate and the device database and user Need to be returned to the database. This process is reflected in FIG. 6I, which is repeated (necessary) by providing information from the device to the backend 104 (at S632) and then updating the user database and device database accordingly (at S634). Depending on or where possible. It is understood that any update of a user database 130 for a particular user may require a corresponding plurality of updates to be sent to the user's devices and a corresponding plurality of updates to the device database 128. Should. For example, if a user has multiple devices and multiple corpora and / or configuration information is changed on one of the devices, the multiple corpora and / or configuration information may be Should be propagated to other devices. In this way, the user does not have to train or configure all of their devices independently or separately, and each device benefits from the training and settings applied to its other devices. It is possible to receive.

  It should be appreciated and understood that multiple devices of a user may not be synchronized with each other and / or with information in the user database 130 and device database 128. This can occur, for example, when multiple devices are unable to connect to the backend for some period of time. The system preferably applies conflict resolution techniques to synchronize multiple devices and multiple databases. In an exemplary conflict resolution approach, multiple timestamps may be used to select the latest multiple version settings and multiple corpus information. Another exemplary conflict resolution approach may always assume that multiple versions of the settings in the user database and multiple corpus information are correct.

In order to allow the user to select specific settings and / or corpora, the user may be provided with an interface to the backend and / or to the user's devices (eg, telephone 174). Preferably, any conflict resolution techniques can be performed without user intervention, such as by an application on a user device or by a web interface. In some implementations, the user may be able to force (eg, push) multiple updates on their devices.
A note about multiple corpora

  Multiple corpora are used, for example, by various interface mechanisms 162 in device 102. For example, the voice / speech recognition mechanism 178 may use a plurality of local speech corpora (stored in the device 102). However, as those skilled in the art will appreciate, speech / speech recognition can be affected by several factors, even for the same speech / speech recognition mechanism 178. For example, multiple different qualities or types of input sensors (eg, multiple microphones) may provide multiple different corpora, even for the same voice / speech recognition mechanism 178. To achieve this goal, if necessary, multiple corpora in user database 128 may be organized based on hardware details of multiple devices of multiple users. In these cases, if a particular plurality of corpora for a particular user changes (eg, based on a learning process on a particular device associated with that particular user), those corpora are updated in the database. But only to the user's equivalent devices (ie, to the user's devices that have equivalent hardware and sensor configurations for the recognition mechanism associated with their corpus) .

  For example, assuming that a particular user 110 has multiple devices, some of them have a first hardware configuration and / or multiple sensors 138 for their system features 134 and other multiple These devices have a second hardware configuration for their system features 134 and / or a plurality of sensors 138. Devices with a first hardware configuration use a first set of corpora for their corresponding operating mechanisms, and devices with a second hardware configuration have their corresponding A second set of corpora (separate from the first set of corpora) is used for the operating mechanism. In this example, if a first device with a first hardware configuration updates its multiple corpora (eg, speech recognition corpus or gesture recognition corpus), the update is in user database 130 and device database 128. It should only propagate to other devices of that particular user that have the first hardware configuration.

  As will be appreciated and understood by those skilled in the art upon reading this description, multiple different types of devices (ie, having different basic device-specific functions) have the same hardware for their system mechanisms. Hardware configuration. Similarly, multiple devices of the same type (eg, multiple speakers) may have different hardware configurations for their system features.

The inventors have recognized that it can be useful to have the device learn various setup and configuration information with minimal user intervention and action. Therefore, they recognized that it would be advantageous and preferred to have multiple devices learn from each other. Thus, in some aspects, device 102-A may obtain configuration information from another device 102-B. In some cases, two devices can touch each other so that the device can obtain information from another device. These interactions correspond to the device to device interaction # 1 depicted in FIG. 1 and may be implemented at least in part by the device to device mechanism 156. In other cases, a device may obtain information from another device upon being instructed by a user command to obtain the information. The user's device may also obtain settings and other information from the user database.
Device heartbeat and multiple interactions

  Recall that each device 102 preferably includes a heartbeat (HB) mechanism 194 (FIG. 4B). The heartbeat mechanism 194 on the device 102 has (1) a function for generating a plurality of heartbeat messages (a plurality of heartbeats), and (2) a monitoring of a plurality of heartbeats from a plurality of other devices. It has two main functions.

  Accordingly, the heartbeat mechanism 194 on a particular device 102 may communicate with the system (eg, backend 104) about the status or presence or presence of the particular device 102, and / or other devices. 102 can be used to provide various signals. The device heartbeat (HB) mechanism 194 includes a plurality of device mechanisms for local communications (eg, Bluetooth Low Energy, including Bluetooth Low Energy (BLE), Zigbee, etc.) and Wi-Fi communications (eg, Device mechanisms for cellular communications (e.g., multiple modems or other devices using a cellular phone network, etc.) and wired communications (e.g., , Ethernet®, or the like) the device's communication mechanisms 142 to broadcast the device's heartbeat (and associated information) by one or more of the device's mechanisms. Can be used. Each heartbeat message allows other components of the system 100 (eg, backend 104, other devices 102) to recognize (probably confirm) that it is a heartbeat message. Information and information that identifies the device such that other components of the system 100 can recognize (and possibly verify the device identification information).

  Different heartbeat messages (having different formats, having different information and at different frequencies) can be broadcast by different communication mechanisms. For example, heartbeat messages sent over the network 101 or cellular network for the backend 104 may be sent out daily or if some historical information is to be provided. On the other hand, broadcast over the device's local communication mechanisms (eg, Bluetooth®, and the like) for other devices, or transmitted on the local network to which the device is connected. Heartbeat messages are sent out every minute (or at some other fixed and short time interval).

  The heartbeat signal should contain some information about the device, preferably at least the device ID of the device. For example, as shown in FIG. 7A, a heartbeat signal 700 from a device includes a corresponding device ID encoding, and optionally a user ID for the device owner. . The heartbeat signal may include additional information (shown as a dashed line in the drawing in FIG. 7A). The signal transmitted to the backend may include additional information such as device location, history, etc., for example. The local heartbeat signal can include only the device ID. Information in the heartbeat signal is preferably protected (eg, by encryption). The device ID and user ID can also be encoded with a one-way encoding (eg, a cryptographic hash such as MD5) to prevent them from being exposed.

  Each device 102 should also regularly (preferably continue) monitoring multiple heartbeats from other multiple devices 102 that may be nearby or on the same network (e.g., continuously). , Using multiple mechanisms of the device for local communication and multiple mechanisms of the device for Wi-Fi and wired communication). If a particular device can capture the heartbeat of other devices via a particular device's local communication mechanism, it may be said that another device is near the particular device. Several other ideas of proximity are used and may be considered herein.

  FIG. 7B illustrates a device A (102-A) that broadcasts a heartbeat signal via a device heartbeat (HB) mechanism 194-A using a plurality of device communication mechanisms 142-A (eg, local communication mechanisms). An example is shown. As shown in the drawing, the second device (device B, 102-B) detects the heartbeat signal of device A via the communication mechanism 142-B of device B. Although not shown in the drawings, it should be understood that device B is also broadcasting its heartbeat signal, and device A can detect device B's heartbeat signal. Furthermore, although this is also not shown in the drawing, it can be said that each of the plurality of devices is also transmitting a plurality of other heartbeat signals via a plurality of different communication mechanisms.

FIG. 7C illustrates an exemplary process by each device 102 (eg, by the device heartbeat mechanism 194) for monitoring (at S702) a plurality of heartbeats from other devices. If a heartbeat is detected from another device (at S704), the heartbeat is processed (at S706), otherwise the system continues to monitor multiple heartbeats (at S702). Once the detected heartbeat is processed (or when the device begins processing the detected heartbeat), the device continues to monitor multiple heartbeats from other multiple devices.
Device collaboration / cooperation / combination

  Some devices 102 may operate alone and in combination with one or more other devices. It should be understood that the devices need not be the same or even the same type to work together. For example, a plurality of devices that are a plurality of speakers may operate together (as described below). As another example, a device that is a speaker may operate with a device that is a video display.

  A plurality of devices 102 operating together in some way are said to cooperate or cooperate. For the remainder of this document, the terms “cooperate”, “cooperation” and “cooperating” are respectively “cooperating and / or cooperating”, “cooperation and / or cooperating”, “cooperating”. Are and / or linked ”.

  As used herein, a plurality of devices 102 are said to be combined when they are combined and cooperate for the purpose of at least some aspects of their operation. Multiple devices can be combined in various ways. In some cases, a device may couple one or more other devices simply by being placed near the other devices. In some cases, multiple devices may be coupled by multiple specific instructions via a user interface. In some cases, multiple devices may be combined by touching one of them to the other.

It should be understood that multiple devices can cooperate without changing their ownership. That is, one user's device collaborates with another user's device, neither of which changes ownership (ie, neither device is associated with a different user in system 100). Can be combined.
Heartbeat processing

The processing of a heartbeat of a device detected from another device (S704) may depend on a number of factors including, for example, at least some of the following.
Whether the first device knows the second device (eg from a previous interaction).
Whether the first device and the second device are both owned (by the same user).
• Whether those devices are owned by multiple friends.
Whether these devices can cooperate in any way (this depends, at least in part, on the type of device, or the specific function of each device, the proximity of the device, and / or Depending on what you are doing). For example, a smartphone device and a speaker device can cooperate to play music from a smartphone device on the speaker device if they are in the same room. Or two speaker devices may cooperate to play both the same music that one of them is already playing. On the other hand, two headphone devices cannot cooperate if they are both already playing (ie, rendering) sound.

  The factors and examples given here are provided as examples only. Multiple different factors can be used to determine how to handle multiple detected heartbeats.

  FIG. 7D illustrates an exemplary process (at S706) by the device (referred to herein as device A for purposes of this discussion) of the heartbeat of another device (referred to herein as device B). . First, device A (which is executing this process) determines the device ID of device B from the received heartbeat message (for example, heartbeat signal 700-A in FIG. 7B). The device ID can be encoded in the signal in such a way that it can be extracted by other devices. In some cases, the device heartbeat may include a cryptographic hash (eg, MD5 hash) of the device ID of the device. In these cases, other devices can use the hash of the device ID as a basis for their determination, and the device ID itself is not disclosed.

  Once the other device's device ID (or its hash) has been determined (at S708), then device A determines whether device A and device B are owned by the same user (ie, whether they are shared). Determine (in S710). Recall that each device stores and maintains information from the user database 130 of the user of that device. This information preferably includes a list of the user's multiple devices, eg, by device ID. Device A can determine whether the device ID in the heartbeat message matches the device ID in the list of multiple devices for the user. If the device ID is hashed or otherwise unidirectionally encoded, device A may store a list of devices that are shared using the same encoding.

  If it is determined that the devices are shared (at S710), device A evaluates the possible cooperation with device B (at S712). Possible cooperation may depend on several factors, as described above. In addition to cooperation on their basic functions (eg, as multiple speakers, etc.), if multiple shared devices discover each other (eg, by heartbeat), those devices are Configuration information can be shared and / or updated.

  Regardless of whether they actually cooperate (as determined in S712), device A preferably updates its history to reflect its encounter with device B (in S714). . Device A may then notify the backend of the encounter (at S716). Note that processing of detected heartbeats, preferably by the device, is done without the device having to contact the backend. Thus, notification of the encounter to the backend need not occur until another periodic connection is made to the backend. Note also that if any device updates its configuration as a result of an encounter, the update should eventually propagate and be returned to the backend.

  If the devices are shared (as determined at S710), device A attempts to determine whether device B is owned by the friend of the owner of device A (at S718). Recall that the heartbeat message may include an encoded version of the user ID of the user of device B, with each device storing information from the user database 130, including a list of friends. The user ID in the heartbeat message can be compared to multiple user IDs in a list of multiple friends to determine if there is a match. It should be understood that if the user ID in the heartbeat message is unidirectionally encoded, multiple user IDs in the list of friends should be encoded as well.

  If it is determined that device B is owned by the friend of the owner of device A (at S718), some cooperation between those devices may still occur. Therefore, device A then evaluates the possible cooperation between those devices (as cooperation between multiple friends' multiple devices) (at S720). This type of cooperation may include multiple types of cooperation that are the same among multiple shared devices (at S712). However, it can depend on multiple permissions associated with friends. Multiple devices of multiple friends may even share some settings and multiple corpus information. However, this is preferably done on a limited and temporary basis.

  Device A preferably has its history regardless of whether multiple friends' devices actually cooperate (as determined in S720), such as cooperation between multiple shared devices (in S712). To reflect the encounter with device B (at S714), and then again device A can notify the backend of the encounter (at S716).

  If it is determined that the devices are not owned by multiple friends (at S718), device A determines whether it has previously encountered device B (at S722). Device A can then record information about device B (at S724), then update its history (at S714), and ultimately notify the backend of the encounter (at S716). ,move on.

  As will be appreciated, cooperation between multiple devices (shared or multiple friends' devices) may require additional communication between those devices. For example, multiple shared devices may need to communicate with each other to synchronize their configuration information. Thus, each device 102 preferably has at least one running mechanism or process waiting for multiple communications from other multiple devices via various communication mechanisms. The two devices that encounter each other (eg, via one or both of their heartbeats) can then interact further as needed. Communication between the two devices (eg, via a local radio mechanism or a local wired network) preferably uses secure communication over that channel no matter what channel is used. In some cases, two devices may first encounter each other via a heartbeat on one communication mechanism (eg, Bluetooth®) and then use a different communication mechanism (eg, Wi-Fi). Can have subsequent communications.

  An exemplary process of possible cooperation (in S712) between multiple shared devices is shown in FIG. 7E. In this example, device A (the one that detected the heartbeat of device B) first contacts device B and attempts to establish a connection with device B (in S726). Once connected, the devices can update / synchronize (if necessary) their configuration information (at S728). (Update / synchronization (in S728) corresponds to state transition T6 in FIG. 6A.) The device also determines itself and the other device to determine if some cooperation is possible and desirable. Also determine information. The device determines the device-specific function of the other device (at S730), determines what the other device is executing (at S732), and determines what the device itself is executing ( In S734). This information can be used to determine (at S736) possible cooperation between those devices. Multiple protocols may be established for various devices or device types to support their cooperation. For example, in some implementations, one device is touched to another device to establish or indicate a desired cooperation between them. Hence, device A also has any indication of the desired cooperation (eg, if one device touches the other or if discovered, a person cooperates with a particular device) (In S738) can be determined. Based on the determined information (at S730, S732, S734, S736, S738), the device may select and initiate possible cooperation (at S740). If multiple collaborations are possible (as determined in S736), selection of one of them may be prioritized by an indication of the desired collaboration (as determined in S738).

  Note that because device A has detected device B's heartbeat, it is possible that device B has also detected device A's heartbeat. If two devices value each other's cooperation, a provision may need to be established as to which device makes a particular plurality of decisions. One exemplary rule is that the device that first initiates contact with the other device takes the initiative as needed in making the decision. Another possible approach is to have the device with the highest device ID take the initiative as needed in making decisions. As one skilled in the art will appreciate upon reading this description, different and / or other conflict resolution approaches may be used as needed.

  It should be understood that just because device A initiates cooperation with device B (at S740), device B does not follow that cooperation. In some cases, devices A and B negotiate and agree to cooperate and start it, in which case device B follows as agreed.

  As one of ordinary skill in the art will appreciate upon reading this description, just because a particular collaboration is possible among multiple devices that are shared does not mean that it will. Furthermore, it should be understood that different and / or other factors may be used to determine and possibly initiate cooperation between shared devices.

  An exemplary process (at S720 in FIG. 7D) for possible cooperation between multiple devices of multiple friends is shown in FIG. 7F. This process is similar to that for multiple shared devices (described above with reference to FIG. 7E), but (i) those devices must have permission to cooperate, (Ii) Preferably only the configuration information required to support any selected cooperation is updated. As shown in FIG. 7F, if authorized (based on a friend's multiple permissions associated with the owner of device B), device A establishes a connection with device B (at S742). If so, device A determines the unique function of device B (at S744) and determines what device B is doing (at S746). Device A determines what it is currently doing (at S748). Based at least in part on some of the information that it has determined, and based on multiple permissions of the friend associated with the owner of device B, device A then determines possible cooperation with device B ( In S750). Device A also determines whether there are any other desired indicators of cooperation between the devices (at S752). Based at least in part on these decisions, device A selects possible allowed cooperation with device B (at S754). If multiple collaborations are possible (as determined at S736), the selection of one of them may be prioritized by the desired collaboration indicator (as determined at S738). Devices A and B update their configuration information (at S756) (if necessary) to support the selected / authorized cooperation. The permitted collaboration is then initiated (at S758).

  It should be understood that just like device shared, just because device A initiates cooperation with device B (at S758), device B does not follow that cooperation. In some cases, devices A and B negotiate and agree to cooperate and start it, in which case device B follows as agreed.

  As one of ordinary skill in the art will realize upon reading this description, just because a particular collaboration is possible between multiple devices of a friend does not mean that it will be performed. In addition, it should be understood that different and / or other factors may be used to determine and possibly initiate cooperation between a friend's devices.

  Although device cooperation has been described above based on heartbeat detection (with other factors that may be used to confirm or select multiple possible interactions of the device) As those skilled in the art will realize and understand, a number of other factors (as described above) can be used to initiate device collaboration. For example, as noted above, in some cases a device may be coupled to one or more other devices by simply being placed near other devices, and in some cases The multiple devices are coupled by specific instructions via the user interface, and in some cases, the multiple devices can be coupled by touching one of them to the other. These other factors (eg, touch, proximity, specific commands, etc.) are in some cases more than other factors, including which of those devices are running that time have priority.

  Thus, for example, as shown in FIG. 7G, each device 102 may periodically monitor some type of contact from other devices (at S760). The type of contact detected may depend on one or more factors such as, for example, the type of device (i.e., on its basic device-specific functionality). Some devices may attempt to initiate contact with other devices based on physical touch, multiple user commands, and the like. Accordingly, detection of a contact attempt from another device (at S762) may involve interpretation of multiple voice and / or gesture commands (voice / speech mechanism 174 and / or) from sensor input (from one or more sensors 138). Or using a gesture mechanism 166). Upon detecting a possible contact attempt (at S762), the device proceeds to process a contact with the other device (at S764).

  The possible contact attempt process (at S764) may be similar to the heartbeat process described above (see FIGS. 7D-7F). Especially when the first device detects a contact attempt from the second device, the first device still determines the device ID of the second device (and vice versa) and the devices It will be necessary to determine if it is shared or owned by a friend and handle contact attempts as appropriate. However, in the case of a contact attempt, the device may assume that the other device (the device that initiated the contact) is attempting to establish some form of cooperation between those devices.

  Thus, in the case of device-driven contacts, a collaborative evaluation between shared devices (in S712 in FIGS. 7D and 7E) and a collaborative evaluation between multiple friends devices (S720 in FIGS. 7D and 7F). )) Can be modified as described herein with reference to FIGS. 7H-7J. Note that the main difference between the processes is that the desired cooperation is given priority in the choice of possible cooperation. Thus, in the evaluation of device-initiated collaboration among multiple shared devices (in S712 'of FIG. 7I), the desired device-initiated collaboration is selected if it is determined that it is possible cooperation And started (S740 ′ in FIG. 7I), in the evaluation of device-led collaboration among multiple devices of multiple friends (in S720 ′ of FIG. 7J), the desired device-led collaboration is If possible and selected, it is selected and started (in S754 ', S758').

  Multiple devices in cooperation must also be able to terminate their cooperation. Cooperation includes, but is not limited to, multiple permission changes (from updated configuration information received by the device), by explicit user instruction, due to one or more of those devices being powered off. Various, such as by separating those devices in a way that they can no longer cooperate (eg, one device is moved to a different room in the house) Can be terminated in a simple way. As will be appreciated and understood by those skilled in the art upon reading this description, different and / or other ways of terminating device cooperation may be used.

  The interaction between multiple devices corresponds to arc # 1 in FIG.

The processing described above for device-to-device interaction is provided merely as an example and is in no way intended to limit the scope of the system. As one of ordinary skill in the art will appreciate upon reading this description, different and / or other multiple interactions may be performed and are included within the scope of the system.
Computing

  As one of ordinary skill in the art will readily recognize and understand upon reading this description, the various processes described herein may be performed by, for example, a plurality of suitably programmed general purpose computers of any size or complexity. May be implemented by multiple dedicated computers and multiple computing devices. One or more such computers or computing devices may be referred to as a computer system.

  The services, mechanisms, operations, and actions shown and described above are implemented, at least in part, by software running on one or more computers of system 100. For example, functionality associated with the back end 104 may be implemented by software running on one or more computers.

  Multiple programs (as well as other types of data) that implement such multiple methods may be stored and transmitted using various media (eg, computer readable media) in several ways. Hardwired circuitry or custom hardware may be used in place of or in combination with some or all of the plurality of software instructions that can implement the processes of the various embodiments. Thus, various combinations of hardware and software can be used instead of software alone.

  As will be readily appreciated and understood by those skilled in the art upon reading this description, the various processes described herein may include, for example, a plurality of suitably programmed general purpose computers, a plurality of dedicated computers, and It can be implemented by multiple computing devices. One or more such computers or computing devices may be referred to as a computer system.

  FIG. 5A is a schematic diagram of a computer system 500 upon which multiple embodiments of the present disclosure may be implemented and executed.

  According to this example, computer system 500 includes a bus 502 (ie, an interconnect), one or more processors 504, one or more communication ports 514, a main memory 506, a read only memory 508, a removable storage medium 510, and a large Capacity storage 512 may be included.

  As used herein, “processor” means one or more microprocessors, multiple central processing units (CPUs), multiple computing devices, multiple microcontrollers, multiple digital signal processors, or the like Means multiple devices, or any combination thereof, regardless of their architecture. An apparatus that performs processing may include, for example, a processor and a plurality of devices such as a plurality of input devices and output devices that are suitable to perform the processing.

  The processor 504 may include, but is not limited to, an Intel® Itanium® or Itanium2® processor, an AMD® Opteron® or Athlon MP® processor, or a plurality of Motorola. It can be a custom processor or any known processor, such as a family of processors, a plurality of ARM-based processors, and the like. The communication port 514 is an RS-232 port, 10/100 Ethernet (registered trademark) port, a gigabit port using copper wire or optical fiber, or a USB port, and the like. It can be arbitrary. Communication port 514 may be selected according to a network, such as a local area network (LAN), a wide area network (WAN), or any network to which computer system 500 is connected. Computer system 500 may be in communication with a plurality of peripheral devices (eg, display screen 516, input device 518) via input / output (I / O) port 520.

  Although referred to herein as multiple peripherals, it is to be understood that such multiple devices may be integrated into the form of a device comprising a computer system 500. For example, a computer system used in a mobile phone has a display screen and an input device as part of the phone. It should also be understood that if provided, multiple peripherals may be combined (eg, in the case of a touch screen or the like).

  As one skilled in the art will appreciate upon reading this description, not all computer systems 500 need include all of the components. For example, not all computer systems 500 require removable storage media 510 or mass storage 512. Similarly, not all computer systems 500 have a display screen 516.

  Main memory 506 may be random access memory (RAM) or any other dynamic storage device commonly known in the art. Read only memory 508 may be any static storage device, such as a plurality of programmable read only memory (PROM) chips, for storing static information such as a plurality of instructions to processor 504. Mass storage 512 may be used to store information and multiple instructions. For example, a series of hard disks and optical disks such as a plurality of Adaptec (registered trademark) family small computer serial interface (SCSI) drives, a RAID such as a plurality of Adaptec (registered trademark) family RAID (Redundant Array of Independent Disks) drives, etc. Disks, or any other plurality of mass storage devices may be used.

  Bus 502 communicatively couples processor 504 with the other memory, storage, and communication blocks. The bus 502 may be PCI / PCI-X, SCSI, a universal serial bus (USB) based system bus (or other) depending on the storage devices used, and the like. The removable storage medium 510 can be any type of external hard drive, floppy (registered trademark) drive, IOMEGA (registered trademark) zip drive, compact disk-read only memory (CD-ROM), compact disk-rewritable (CD-RW). ), A digital video disk-read only memory (DVD-ROM) or the like.

  Embodiments herein may include a machine-readable medium having a plurality of stored instructions that may be used to program a computer (or other electronic device) to perform processing. It can be provided as a plurality of computer program products. As used herein, the term “machine-readable medium” refers to any that provides for the provision of data (eg, multiple instructions, multiple data structures) that can be read by a computer, processor or similar device. Media, multiple media, or a combination of different media. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, and other persistent memory. Volatile media include dynamic random access memory, which typically constitutes the computer's main memory. Transmission media includes coaxial cables, copper wire, and optical fiber, including a plurality of wires with a system bus coupled to a processor. Transmission media can include or transmit sound waves, light waves and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications.

  Machine-readable media include, but are not limited to, floppy diskette, optical disk, CD-ROM, magneto-optical disk, ROM, RAM, erase and programmable read only memory (EPROM), electrical erase and programmable read. It may include only memory (EEPROM), magnetic or optical card, flash memory, or other types of media / machine-readable media suitable for storing multiple electronic instructions. Further, the embodiments herein can also be downloaded as a computer program product. The program can be transmitted from the remote computer to the requesting computer as a plurality of data signals embodied in a carrier wave or other propagation medium via a communication link (eg, a modem or network connection).

  Various forms of computer readable media may be involved in carrying data (eg, a sequence of instructions) to a processor. For example, data is (i) distributed from a RAM to a processor, (ii) propagated through a wireless transmission medium, (iii) formatted and / or transmitted according to a number of formats, standards or protocols, and / or ( iv) Can be encrypted in any of a variety of ways well known in the art.

  A computer-readable medium may store (in any suitable format) those program elements that are suitable for performing the methods.

  As shown, the main memory 506 is encoded with an application 522-1 that supports functionality as discussed herein (application 522-1 is a multi-service described herein). It can be an application that provides some or all of the functionality, eg, backend processing). Application 522-1 (and / or other resources as described herein) may support data and / or multiples that support processing functions according to different embodiments described herein. It may be embodied as software code, such as logic instructions (eg, code stored in memory or on another computer readable medium such as a disk).

  For example, as shown in FIGS. 5B-5C, if the computer system 500 is used to implement the functionality of the back end 104, the application 522-1 may include multiple back end applications 524-1, and the computer system If 500 is used to implement device functionality, multiple applications 522-1 may include multiple device applications 526-1.

  During operation of one embodiment, the processor 504 may use a bus to launch, run, execute, interpret, or perform a plurality of logic instructions of the application 522-1. The main memory 506 is accessed using 502. The execution of the application 522-1 generates a processing function of a service related to the application. In other words, process 522-2 represents one or more portions of application 522-1 executing in or on processor 504 in computer system 500.

  For example, as shown in FIGS. 5D-5E, if computer system 500 is used to implement the functions of backend 104, process 522-2 may include backend process 524-2, and computer system 500 may be If used to implement device functionality, process 522-2 may include device process 526-2.

  In addition to the process 522-2 for performing multiple operations as discussed herein, other embodiments herein may include the application 522-1 itself (ie, not being executed or disabled). It should be noted that it includes multiple logic instructions and / or data for execution). Application 522-1 may be stored on a computer-readable medium (eg, repository) such as a disk, or in an optical medium. According to other embodiments, application 522-1 may also be read-only memory in memory-type systems, such as in firmware, as code executable in main memory 506 (eg, in random access memory or RAM). (ROM), or as in this example. For example, application 522-1 may also be stored in removable storage medium 510, read only memory 508, and / or mass storage device 512.

  As those skilled in the art will appreciate, computer system 500 may include other processes and / or software and hardware components, such as an operating system that controls the allocation and use of hardware resources. For example, a plurality of operating system (OS) programs including a kernel may be a plurality of processes on a computer system.

  As discussed herein, embodiments of the present invention include various stages or operations. Various of these stages may be performed by multiple hardware components or embodied in multiple machine-executable instructions that may be used to cause a general purpose processor or dedicated processor programmed with multiple instructions to perform multiple operations. Can be realized. Alternatively, the steps may be performed with a combination of hardware, software, and / or firmware. The term “module” refers to a self-contained functional component that can include hardware, software, firmware, or any combination thereof.

  As will be readily appreciated and understood by those skilled in the art upon reading this description, embodiments of the apparatus are operable to perform some (but not necessarily all) of the described processes. Various computer / computing devices.

  Embodiments of a computer readable medium for storing a program or data structure may store a program that, when executed, may cause a processor to perform some (but not necessarily all) of the processing described. Includes readable media.

  The process is described herein and one of ordinary skill in the art will understand that the process can operate without any user intervention. In another embodiment, the process includes some human intervention (eg, a stage is performed with or with human assistance).

Referring back to FIGS. 4A-4B, recall that device 102 includes a computer system 146. As should be appreciated, the computer system 146 may not include all of the multiple components shown in FIG. 5A, and the computer system 146 may be required by multiple additional components (eg, device 102). In some cases, the computer system 146 alone or in combination with system features / data 134 is described above (see FIGS. 5A-5C). As described above, the computer system 500 can be supported. For example, the computer system 146 may include a plurality of processors, a plurality of memories, and the like. It should also be understood that the computer system 146 can be formed of multiple computer systems 500. In addition, the computer system 146 may implement some or all of the device specific functions 132.
Multiple control devices

The system-enabled device 102 may be controlled by one or more of voice control, gesture control, contact control (eg, using multiple buttons and the like). In addition, a particular type of system-enabled device 102 may be selected by one or more of the other devices or of other devices when there are other similar devices. It can be completely or partially controlled by a plurality of instructions given to one or more. For example, if the multiple devices 102 are multiple speakers, the multiple devices can be combined to operate together. In such cases, the specific commands (eg, increase volume) may be provided by the user to one of the devices (speakers), but the cooperating devices. Should be followed by all of them.
Voice control

  The device audio mechanism 166 may be used to support device audio control. The voice mechanism 166 preferably includes a plurality of voice recognition mechanisms for a plurality of basic commands suitable for such devices. For example, for a device that is primarily a speaker, multiple voice commands can be used to power off the device (or power on from low power mode), play at a higher volume, play at a more gentle volume, etc. Can be included. The audio mechanism 166 can be implemented using specialized hardware or circuitry and multiple DSPs (Digital Signal Processors).

  Each device preferably maintains a corpus of recognized words from a plurality of users. In some cases, the device may maintain multiple corpora of multiple words, one for each of several users. A device can be controlled by more than one person (and depending on the permissions set on the device, the person who controls the device may not be a known user of the system) The command needs to be able to be associated with multiple appropriate users. In this way, the device can determine which multiple word corpus to use for voice / command recognition.

  Device 102 uses a face recognition mechanism 168 in combination with one or more cameras (multiple sensors 138) to associate audio with a particular user to select the appropriate corpus.

In some cases, the device may not be able to process voice commands / requests. This can be attributed to any number of factors, including command / request complexity, multiple environmental causes (eg, noise), speaker accents, and the like. In such cases, the device may send voice commands / requests to the backend for processing (eg, voice) if possible (eg, connected to a network) and if allowed. By an additional function 120 provided with recognition). The audio can be transmitted as it is or in some preprocessed form. The result of such processing can be a command / request to the backend (eg, a database query) or a command to the device itself. Multiple device commands that are processed remotely by the back end are not fast enough to control certain aspects of the device (eg, “play at louder volume” for speakers), It should be understood that it is useful for more complex multiple commands, particularly those involving multiple database queries.
Gesture control

The device gesture mechanism 164 may be used alone or in combination with the audio mechanism 166 to support device gesture control. Gesture mechanism 164 preferably includes a plurality of gesture recognition mechanisms for a plurality of basic commands suitable for such devices. Gesture mechanism 164 may use one or more of a plurality of sensors 138 including one or more cameras, for example. Special purpose gesture detection / recognition hardware and circuitry may be used.
Face and gaze detection

  In some cases, there may be multiple people speaking simultaneously near the device. Some of what they say is not intended for multiple commands to the device. Thus, in some cases, the device may use a combination of gaze detection (determined by the face / gaze mechanism 168) to determine if the voice command is for the device. The face / gaze mechanism 168 may use one or more sensors (eg, one or more cameras) to determine whether the person who is speaking is actually looking at the device 102. Since people can start speaking (towards the device) before they are completely confronted with the device, preferably each device will always buffer the sound for a period of time, and if a line of sight is detected, the device will be buffered To start speech recognition of the stream.

  In some cases, mouth movement detection may be used in combination with gaze detection to confirm that the person watching the device is the person talking to the device.

As those skilled in the art will appreciate upon reading this description, speech recognition, gesture detection and recognition, and face and / or gaze detection can be used in various combinations to control the device.
Return report to backend

  In operation, each device reports information to the backend 102 (corresponding to arc # 7 in FIG. 1). The information preferably includes the unique device ID of the reporting device and, if the device is associated with a user, the unique user ID associated with the reporting device owner. Some or all of the information reported by each device may be stored in device database 128 and / or user database 130, for example, as device history and / or user history, respectively. Since each device has a unique device ID and each user has a unique user ID, information from the device can be stored in a database keyed with the device ID and user ID.

  In some cases, device 102 may include information about its location when reporting. In those cases, location information can be stored in the device database (both as a recent device location and device history). Similarly, location information can be stored in the user database as per device history. Thus, multiple queries to the database may include multiple queries for device locations.

  When users register with the system, they can provide location identification information associated with their current location at the time of registration. The user also stores multiple locations within the system, each having a different identification provided by the user. For example, users may store GPS location information about their homes, their work, their friends' homes, and so on. In this way, the system can support multiple database queries based on named locations (eg, the system response to “Where is my device?” Is “Joe's home”. sell). Preferably, the user need not specifically request storage of location information. This is because location (eg, GPS) data is preferably stored automatically as part of historical data or context metadata.

  The device 102 may also report information that is specific to that type of device (eg, the device's unique capabilities). For example, a device that is primarily a speaker may report information to the backend about what it is playing and where it is located. In some cases, the information may include information about multiple device settings and what other multiple devices are involved (eg, coupled). In this way, the database supports multiple types of queries such as "What I was playing at Joe's house last night at 10:00?", Against which the system could provide a list of songs.

The device 102 also reports information about nearby devices or users.
Device Example-Sound Rendering Device

  FIGS. 8A-8D illustrate aspects of the architecture of an exemplary device 800 (device 102 embodiment) where the inherent function of the device is sound rendering. The device 800 can be used as a speaker, for example. As shown in FIGS. 8A-8B, the sound rendering device 800 includes a plurality of components 832 that support device-specific functions. These components 832 include one or more speaker drivers 860, one or more signal processors 862, one or more processors 864, a memory / storage 866, and a plurality of controls 868.

  As shown in FIGS. 8A and 8C, the device 800 includes a plurality of Bluetooth (registered trademark) mechanisms, a plurality of Ethernet (registered trademark) mechanisms, a plurality of ZigBee mechanisms, a plurality of cellular mechanisms, and a plurality of Wi-Fi mechanisms. A plurality of communication mechanisms may be included. In one implementation, the plurality of communication mechanisms of the device 800 includes a plurality of Bluetooth® mechanisms and does not include an Ethernet® mechanism, a ZigBee mechanism, or a cellular mechanism.

  As shown in FIGS. 8A and 8D, the sound rendering device 800 may also include one or more cameras 870, one or more microphones 872, device motion sensors, location / position sensors, external motion sensors, touch / contact sensors, A plurality of sensors 838 may be included, including light sensors, temperature sensors, and other sensors. In one implementation, the plurality of sensors of device 800 does not include a camera or a temperature sensor.

In one exemplary implementation, the following may be used for some of these components.

In another exemplary implementation, the following may be used for some of these components:

  It should be understood that the above list is given as an example only and is not intended to limit the scope of the device at all.

  Any known mechanism can be used for the various interface mechanisms 162. For example, face motion detection may use the CANDIDE system for model-based encoding of multiple human faces. CANDIDE uses a face model with a small number of polygons (approximately 100) that allows fast reconstruction with moderate computational power.

  Sound rendering device 800 may operate as device 102 as described above.

  As those skilled in the art will appreciate upon reading this description, a plurality of different and / or other specific components may be used within the sound rendering device 800, and such other components may be Considered in the specification and within the system scope. It should be understood that the various components are implemented and packaged in multiple ways, and the device is not limited by the manner in which multiple components are implemented or packaged. It should further be understood that the device is not limited by the packaging or the form that the device takes (ie, by the form factor of the device).

The following table includes the syntax of a list of exemplary commands (ie, multiple phrases) that the device can interpret locally using multiple speech recognition mechanisms. As used herein, a phrase means one or more words. In the following corpus table, a plurality of words in bold italic font are in the local corpus, and a plurality of words in square brackets (“[”, “]”) are arbitrary. A vertical line (“|”) between phrases means “or” (ie, one of those phrases). Thus, for example, “A” | “B” means the phrase “A” or the phrase “B”. The phrase indicated by “#n” means a number. A phrase followed by an asterisk ("*") means that the phrase can be repeated. A phrase of a word followed by “(s)” means that the word or words can be used. Thus, for example, item number 2 in the table,
[[Slightly] Slightly (not much) | Much * | #n] Gently | Lower | Higher | Quiet [Quiet]
Is "a little lower", "much higher", "more gentle", "5 times lower", "not a little gentler", "much much gentler", "quiet", "quietly", "a little quiet" This can correspond to “

  For example, item number 3 in the table is “Play next 10 songs”, “Play any song”, “Play next song”, “Play previous song again”, “Any song” It may mean any one of “play a plurality of songs”, “play again”, “play”, and the like. As another example, item number 5 may mean any of “skip”, “next three songs”, “songs”, “seven skips”, “previous song”, and the like. As a further example, item number 8 may mean any of “tone”, “adjust to increase treble”, “lower bus”, and the like.

  The voice / speech recognizer 178 can therefore recognize a particular spoken phrase, and then determine their corresponding semantics (ie, meaning) to actually control the device. Thus, other operating mechanisms (e.g., to command / control mechanism 158) must be provided with corresponding commands.

  It should be understood that this exemplary corpus provides the syntax of the recognized phrases and that not all phrases have meaning (or reasonable meaning) to the device. For example, the number 3 may support the recognition of the phrase “play any three of the next songs again” and the number 5 may support the recognition of the phrase “skip previous song”. Both of these phrases are syntactically correct (according to the corpus syntax), but they cannot correspond to any meaningful command and can be ignored by the device.

  The corpus syntax provided above for the speech / speech recognizer 178 is provided as an example only, and will be different and / or as will be appreciated by those skilled in the art upon reading this description. Other multiple voice commands can be understood by the sound rendering device 800 and are contemplated herein.

  Multiple sound rendering devices 800 may cooperate with each other to render the same sound (eg, to play the same music from the same source simultaneously—preferably synchronized). If two or more sound rendering devices 800 cooperate to render sound from the same source, they need not all render the same sound correctly. For example, multiple sound rendering devices 800 may cooperate to render sound from the same source as a surround sound system. As another example, multiple sound rendering devices 800 cooperate to render sounds from the same source such that some of them render some sound (eg, from several instruments), while , Other sound rendering devices render other sounds (eg, from several other instruments).

  It should be understood that the sound rendering device 800 may also be a source of signals used to generate sound. For example, a smartphone (such as iPhone®, or the like) can have a speaker (although it is small) and can be used to generate sound (by iPhone® itself) , And by other devices 800).

Examples of collaboration between multiple sound rendering devices 800 are given below with reference to FIGS. 9A-9C.
Example of cooperation-stereo

Referring to the drawing of FIG. 9A, two sound rendering devices 800-A and 800-B may cooperate to provide a stereo effect. Multiple DSPs in multiple devices cooperate, for example, to generate a Haas effect. It should be understood that multiple devices can determine their relative position (eg, using echo position or some other mechanism) and they use this relative position information to optimize the cooperative effect. sell.
Example of collaboration-multiple separate instruments

  Referring to the drawing of FIG. 9B, the plurality of sound rendering devices 800-A-800-D may cooperate such that each one of them plays only some of the plurality of instruments in the source signal. . The source signal can provide separate streams for each instrument, or each DSP can be programmed to filter out certain instruments. As multiple devices join this group, each device is assigned to one or more instruments for rendering. For example, assume that device A was initially playing alone and was rendering all sounds in the source signal. If device B joins device A, device A may render a base and a violin, for example, and device B may render a cello and multiple vocals. If device C joins devices A and B, device C can be given violin responsibilities, leaving only the base for device A. When device D joins, it assumes responsibility for multiple vocals from device B (as shown in the drawing of FIG. 9B). If other devices join the group, they can be combined with one or more of the already existing devices, or they can take on some other responsibility. If a device leaves a group, the signal portion it was responsible for should be reassigned to another device that still remains in the group.

Although this example shows the selective assignment of a plurality of different instruments in an audio signal, device 800 may differ in audio streams and / or others, as those skilled in the art will appreciate upon reading this description. Responsibility can be given to a plurality of aspects. It should be understood that device 800 may render (or may not render) any portion or portions of an audio stream that its DSP can filter (can be filtered in or removed). Further, the device 800 may enhance or modify any part or parts of the audio stream that its DSP can filter.
Example of cooperation-arbitrary arrangement

  Referring to the drawing of FIG. 9C, multiple sound rendering devices 800-A-800-E arranged or positioned in any (unplanned) configuration may cooperate. The devices can determine their own relative position (eg, using echo position or some other approach) and they can use this relative position information to optimize the cooperative effect. The devices may also cooperate to produce an optimal or beneficial cooperation effect for the listener (if the listener's location is known or can be determined).

  If those devices assume that the listener is the person giving them multiple commands (speech, gestures, etc.), the devices should be positioned (and follow the listener) Can adjust the sound accordingly. A single camera in a single device can determine the direction in which the listener is located. Some techniques allow those single cameras to determine approximate distances. Multiple cameras (in a single device or multiple devices) can more accurately determine the position of the listener (eg, by tracking the location of the face and / or movement). In addition to (or instead of) using one or more cameras to determine a person's position, position detection can be achieved using speech input and echo detection. Thus, for example, in a device without a camera, voice input and echo detection can be used alone to determine the position. In devices with a camera, voice input and echo detection can be used alone or in combination with a camera to determine position.

  In this example, various options for cooperation are possible. For example, different instruments may be placed along different stereo lines (L1, L2, L3 in the drawing), and some of the devices 800-A to 800-E may be echoes, reverberations, or It can be used to add multiple effects, such as the like, and some of the multiple devices can be used to cancel room noise or multiple echoes.

  For example, as shown in FIG. 9C, the base can be rendered by devices A and C (on stereo line L1), the violin and cello can be rendered by devices A and E (on stereo line L2), Multiple vocals can be rendered by devices D and E (on stereo line L3). Multiple effects (eg, room noise or multiple echo cancellation) may be performed by device B.

  If the audio signal includes multiple channels or other multiple encodings, multiple devices may cooperate to render these channels.

The qualities of multiple sound rendering devices 800, particularly multiple portable devices, add to their cooperation. In some implementations, a user may give a friend multiple guest privileges to share their device 800. In some implementations, the user may grant all other devices multiple temporary (“party mode”) rights to share their sound rendering device 800.
Conflict resolution

As described above (see heartbeat processing), if multiple devices attempt to cooperate, it may be necessary for one device to take control. In the case of the sound rendering device 800, the preferred provision is that the device that first initiated contact with the other device (eg, by touching it, etc.) takes the lead as needed to make the decision. That's it. As one of ordinary skill in the art will appreciate upon reading this description, different and / or other techniques for determining which device has control can be used and are considered herein. It is done.
Genre classification

  Sounds can be categorized into genres (eg, vocals, instruments, jazz, classical, spoken voice, etc.), and these genres can be provided with sound source signals to automatically enable multiple DSPs in the sound rendering device 800. Can be used to set or adjust. In some cases, the pre-configured genre information is combined by multiple preferences of the user (which can be provided via some user interface or learned by the device based on multiple user interactions with the device). Can be ignored or ignored. For example, if the user always adjusts multiple DSP settings to a particular genre of music in preference to multiple preset DSP settings, device 800 may learn the user's desired settings. Thus, for that genre, they can always be used instead of the system's preset settings.

  The genre information can set, for example, offline processing for analyzing the sound of the source in advance. For example, a source library provider may pre-analyze all the music in those libraries to classify the genre of each item of music. The classification can be stored, for example, as a bit vector representing a genre and source data can be provided. However, it should be understood that the processing of genre information in the source signal is independent of the way in which the genre information is obtained or set.

  Multiple devices 800 cooperating may use genre information in the source signal to determine and adjust how they cooperate. Thus, when rendering sounds corresponding to multiple songs, the cooperating devices 800 may change their cooperating manner depending on the genre of each song.

As will be appreciated and understood by those skilled in the art upon reading this description, different and / or other types of cooperation may be used by the plurality of sound rendering devices 800, and such cooperation is described herein. In the scope of the system.
History and learning

  As described above, the system 100 can obtain information from each device 102. In the case of multiple sound rendering devices 800, the multiple devices are preferably what sounds they are rendering (eg, what music, etc. they are playing), and when and where it is The back end 184 is notified if it will be rendered. To accomplish this goal, each device 800 maintains a history of its multiple activities and provides that history to the backend 184 periodically and / or when possible. The history can be provided as a time-stamped, ordered list of activities and device settings that can be used to replicate the activities of the device. If a device collaborates with another device, that information is also included in the history, and both (all) cooperating devices provide their history information to the backend.

  The back end stores device history information in the device database 128 and the user database 130.

This type of device history information supports the following types of subsequent queries (by backend 184 and possibly additional function 120):
1. “Play what I listened to at 4:30 pm on Monday”.
2. “Play what I listened to on Sunday morning with [User] Joe”.
3. “Set the device to multiple settings on July 1, 2012”.
4). "Play what [user and friend] Mary is listening to now."

  Note that query # 2 may require Joe and the user making the query to be friends, and may require Joe's permission. Query # 4 may require that the user who is querying Mary and the user are friends, and may require Mary's permission. Note also that query # 4 assumes that the system has been updated to know what Mary is listening to (in near real time).

These queries are only provided as examples and are not intended to limit the scope of the system at all.
Noise cancellation

  As described above, the device may attempt to filter out ambient noise to more accurately handle multiple voice interactions. Sound rendering device 800 introduces additional problems. This is because the device itself can be a source of environmental noise. In any case, the device should not recognize that the sound rendered by the device is a multiple command for the device itself. Thus, in preferred implementations, the sound rendering device 800 filters out the sound it generates from the sounds acquired by its own multiple sound sensors (multiple microphones).

  As used herein, words such as “first”, “second”, etc., when used as an adjective preceding a term, are merely used to distinguish a plurality of similar terms. Yes, their use does not imply or define any numerical limitation or any ordering (in terms of time or otherwise). Thus, for example, the terms “first device” and “second device” are simply used to refer to and distinguish between different devices.

  As used herein, including the claims, the phrase “based on” means “based at least in part on,” unless specifically stated otherwise. Means. Thus, for example, the phrase “based on XYZ” means “based at least in part on XYZ”.

  Thus, a unified framework for device configuration, interaction and control is described along with multiple systems, multiple devices and multiple mechanisms used in them. Although the invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, the invention is not limited to the disclosed embodiments, and vice versa. It is to be understood that the intention is to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims (51)

A device,
(A) a plurality of first mechanisms that support device-specific functions of the device;
(B) a plurality of second mechanisms that support control of the plurality of first mechanisms, and the plurality of second mechanisms include:
(B) (1) a plurality of sensors for obtaining information on at least one of a plurality of physical and environmental characteristics of the device and an environment surrounding the device;
(B) (2) a plurality of control mechanisms;
(B) (3) A plurality of sensors that obtain sensor inputs from at least some of the plurality of sensors, determine sensor information based on the sensor inputs, and provide the sensor information to the plurality of control mechanisms. A human interface mechanism,
(C) The plurality of control mechanisms include:
(C) (i) determining control information based on the sensor information from the plurality of human interface mechanisms;
(C) (ii) providing at least some of the control information to the plurality of first mechanisms;
(D) the plurality of first mechanisms acquire the control information from the plurality of control mechanisms, and at least partially operate according to the control information acquired from the plurality of control mechanisms;
(E) The plurality of sensors are:
(E) (1) One or more cameras that acquire image information from the environment around the device and provide the image information to the plurality of human interface mechanisms;
(E) (2) One or more microphones that obtain sound information from the environment around the device and provide the sound information to the plurality of human interface mechanisms,
(F) The plurality of human interface mechanisms are:
(F) (1) a plurality of speech mechanisms for recognizing speech in the sound information and providing information about the recognized speech in the sound information to the plurality of control mechanisms as speech information;
(F) (2) a plurality of face mechanisms for detecting at least one of face information and line-of-sight information in the image information,
The plurality of control mechanisms determine the control information based on the speech information,
The plurality of speech mechanisms (F) (1) start speech recognition based on information detected by the plurality of face mechanisms. The device according to claim 1, wherein the plurality of speech mechanisms start speech recognition based on line-of-sight information detected by the plurality of face mechanisms. The plurality of speech mechanisms buffer sound information from the environment around the device and start speech recognition of the buffered sound information based on line-of-sight information detected by the plurality of face mechanisms. The device according to 1 or 2. The plurality of speech mechanisms uses at least one speech corpus, and the plurality of speech mechanisms selects a speech corpus from the at least one speech corpus based on face information provided by the plurality of face mechanisms. Item 3. The device according to Item 1 or 2. The plurality of human interface mechanisms further includes:
(F) (3) A plurality of gestures for detecting and recognizing a plurality of gestures in the image information and providing the plurality of control mechanisms with the information about the recognized plurality of gestures in the image information as gesture information. With a mechanism,
The device according to claim 1, wherein the plurality of control mechanisms determine the control information based on the gesture information. A device,
(A) a plurality of device-specific mechanisms that support device-specific functions of the device;
(B) a plurality of second mechanisms that support control of the plurality of device-specific mechanisms, and the plurality of second mechanisms includes:
(B) (1) a plurality of control mechanisms;
(B) (2) A plurality of sensors including one or more cameras that acquire image information from an environment around the device and one or more microphones that acquire sound information from the environment around the device When,
(B) (3) a plurality of human interface mechanisms, wherein the plurality of human interface mechanisms include:
(B) (3) (1) a plurality of face mechanisms that determine line-of-sight information in image information acquired from the one or more cameras;
(B) (3) (2) Recognizing speech in sound information acquired from the one or more microphones, and making the plurality of control mechanisms recognize information about the recognized speech in the sound information as speech information. A plurality of speech mechanisms to provide,
The plurality of speech mechanisms start speech recognition based on line-of-sight information detected by the plurality of face mechanisms,
The plurality of human interface mechanisms determine interface information based on the information about recognized speech obtained from the plurality of speech mechanisms, and provide the interface information to the plurality of control mechanisms;
The plurality of control mechanisms are:
Determining control information based on the interface information from the plurality of human interface mechanisms;
Providing at least some of the control information to the plurality of device-specific mechanisms;
The plurality of device-specific mechanisms acquire the control information from the plurality of control mechanisms and operate according to the control information acquired at least partially from the plurality of control mechanisms. The device according to claim 1, wherein the device-specific function includes sound rendering. The device according to claim 7, wherein the device is a speaker. A method of operating a device, wherein the method is implemented, at least in part, by hardware including at least one processor and memory, the method comprising:
(A) buffering sensor information from an environment surrounding the device as buffered sensor information in the buffer of the memory;
(B) detecting human eyes in the environment;
(C) starting recognition of some sensor information including the buffered sensor information based on the line of sight detected in (B);
(D) the recognition initiated in (C) determines at least one instruction in the sensor information including the buffered sensor information;
(E) operating the device based on the at least one instruction. The sensor information includes image information, the recognition initiated in (C) includes gesture recognition, and the at least one command is determined based on at least one gesture recognized by the gesture recognition. The method described in 1. The sensor information includes sound information, the recognition initiated in (C) includes speech recognition, and the at least one command is determined based on at least one utterance phrase recognized by the speech recognition. Item 10. The method according to Item 9. A method of operating a device, wherein the method is implemented, at least in part, by hardware including at least one processor and memory, the method comprising:
(A) buffering sound from the environment surrounding the device as buffered sound in the memory;
(B) detecting human eyes in the environment;
(C) starting speech recognition of several sounds including the buffered sound based on the line of sight detected in (B);
(D) the speech recognition determines at least one instruction in the sound including the buffered sound;
(E) operating the device based on the at least one instruction. A method of operating a device, wherein the method is implemented, at least in part, by hardware including at least one processor and memory, the method comprising:
(A) buffering image information from an environment around the device as buffered image information in the memory;
(B) detecting human eyes in the environment;
(C) starting gesture recognition of several pieces of image information including the buffered image information based on the line of sight detected in (B);
(D) the gesture recognition determines at least one instruction in the image information including the buffered image information;
(E) operating the device based on the at least one instruction. The method according to any one of claims 9 to 13, wherein the device is a sound rendering device. The method of claim 14, wherein the device is a speaker. A method usable in a framework, the method comprising:
(A) providing a device, the device having device-specific functions;
(B) associating the device with a user of the framework, wherein the user has user-specific setting information associated with the user, and the user-specific setting information of the user supports recognition of the user's speech Providing at least one speech corpus to:
(C) automatically setting the device with setting information based on the user-specific setting information associated with the user. The user-specific setting information of the user is
Network configuration information,
Password information,
At least one gesture corpus for gesture recognition associated with the user;
The method according to claim 16, further comprising one or more of face information for recognizing the user's face. The method of claim 16 or 17, further comprising: (D) updating at least some of the configuration information on the device based on updated user-specific configuration information associated with the user. The method of claim 18, wherein the device obtains the updated user specific configuration information from another device. The method of claim 19, further comprising: (D) providing at least some user-specific configuration information to another device. The method of claim 18, wherein the device updates some of the user specific configuration information based on one or more human interactions with the device. The method of claim 21, further comprising: (D) providing updated user specific configuration information elsewhere. The method of claim 22, wherein the updated user specific configuration information is stored in the other location. A method usable in a framework, the method comprising:
(A) associating a first device with a user of the framework, the user having user specific configuration information associated with the user, wherein the user specific configuration information is a recognition of a plurality of interactions associated with the user Including at least one corpus for
(B) automatically configuring the first device with at least some of the user-specific configuration information associated with the user. The at least one corpus is
At least one speech corpus for speech recognition associated with the user;
25. The method of claim 24, comprising one or more of at least one gesture corpus for gesture recognition associated with the user. The user-specific setting information is
Network configuration information,
Password information,
The method according to claim 24 or 25, further comprising one or more of face information for face recognition of the user. 27. The method of claim 26, wherein the first device updates some of the user specific configuration information based on one or more human interactions. 28. The method of claim 27, wherein the first device provides updated user specific configuration information elsewhere. 29. The method of claim 28, further comprising: (C) associating updated user specific configuration information received from the first device with the user at the other location. (D) associating a second device with the user;
30. The method of claim 29, further comprising: (E) automatically configuring the second device with the updated user specific configuration information. 31. A method according to any one of claims 24 to 30, wherein the first device is a sound rendering device. The method of claim 30, wherein the first device and the second device are sound rendering devices. A framework that supports operation of a plurality of devices on behalf of a plurality of users, wherein each device of the plurality of devices is mainly associated with one user of the plurality of users in the framework, and the frame Work is
A backend system comprising hardware and software, including at least one processor and memory, the backend system comprising a database system and a plurality of backend applications, wherein the plurality of backend applications are on the hardware; Executed,
(A) connecting to the database system and the plurality of devices through an interface;
(B) In the database system, maintaining device information for each device of the plurality of devices;
(C) In the database system, maintaining user information about each user of the plurality of users,
(D) providing at least some of the user information of the particular user from the database system to at least one device associated with the particular user of the plurality of users;
The device information for each specific device includes information about any user associated with the specific device;
The user information for each particular user includes information that supports at least one human interface control mechanism in a plurality of devices associated with the particular user. The user information of each specific user further includes configuration information associated with the specific user, and the user information of the specific user provided by the backend system in (d) is associated with the specific user. The framework according to claim 33, including setting information. The user information of the particular user provided by the backend system includes the information supporting the at least one human interface control mechanism in a plurality of devices associated with the particular user. Framework. The at least one human interface control mechanism includes a speech recognition mechanism, and for a particular user, the information supporting the at least one human interface control mechanism is associated with the particular user and is by the particular user 36. The framework according to any one of claims 33 to 35, comprising at least one speech corpus usable by a speech recognition mechanism that supports speech recognition. The at least one human interface control mechanism includes a gesture recognition mechanism, and for a particular user, the information supporting the at least one human interface control mechanism is associated with the particular user and is by the particular user 36. The framework according to any one of claims 33 to 35, comprising at least one gesture corpus usable by a gesture recognition mechanism that supports recognition of a plurality of gestures. The plurality of backend applications further includes:
(E) Obtain updated user information from multiple devices,
(F) The framework according to claim 33, wherein the updated user information is associated with a plurality of corresponding users in the database system. 39. The framework of claim 38, wherein the updated user information from a specific device comprises updated information that supports a human interface control mechanism on the specific device. 40. In (f), the plurality of back-end applications associate the updated information supporting the human interface control mechanism on the particular device with a user associated with the particular device. Framework. 40. The framework of claim 39, wherein the human interface control mechanism on the particular device comprises a speech recognition mechanism, and the updated user information comprises an updated speech corpus for the speech recognition mechanism. 42. The frame of claim 41, wherein in (f), the plurality of back-end applications associates the updated speech corpus for the speech recognition mechanism on the specific device with a user associated with the specific device. work. A method that can be used in a framework that supports operation of a plurality of devices on behalf of a plurality of users, wherein each device of the plurality of devices is mainly directed to one user of the plurality of users in the framework. And the framework comprises a backend system comprising hardware and software, including at least one processor and memory, the backend system executing on the hardware and the database system And a plurality of back-end applications that interface with the plurality of devices,
The method
(A) In the database system, maintaining device information for each device of the plurality of devices, wherein the device information for each specific device includes information about an arbitrary user associated with the specific device. Including a stage;
(B) In the database system, maintaining user information for each user of the plurality of users, wherein the user information for each specific user is at least in a plurality of devices associated with the specific user Including information supporting one human interface control mechanism;
(C) providing at least some of the user information of the particular user from the database system to at least one device associated with the particular user of the plurality of users. The user information of each specific user further includes setting information associated with the specific user, and the user information of the specific user provided in (C) includes setting information associated with the specific user. Item 44. The method according to Item 43. 44. The method of claim 43, wherein the user information of the particular user provided in (C) includes the information supporting the at least one human interface control mechanism in a plurality of devices associated with the particular user. . The at least one human interface control mechanism includes a speech recognition mechanism, and for a particular user, the information supporting the at least one human interface control mechanism is associated with the particular user and is by the particular user 46. A method as claimed in any one of claims 43 to 45, comprising at least one speech corpus usable by a speech recognition mechanism that supports speech recognition. The at least one human interface control mechanism includes a gesture recognition mechanism, and for a particular user, the information supporting the at least one human interface control mechanism is associated with the particular user and is by the particular user 46. A method according to any one of claims 43 to 45, comprising at least one gesture corpus usable by a gesture recognition mechanism that supports recognition of a plurality of gestures. By the plurality of backend applications,
(D) obtaining updated user information from a specific device;
44. The method of claim 43, further comprising: (E) associating the updated user information with a corresponding user of the particular device in the database system. 49. The method of claim 48, wherein the updated user information from the specific device comprises updated information that supports a human interface control mechanism on the specific device. 50. The method of claim 49, wherein the human interface control mechanism on the particular device comprises a speech recognition mechanism, and the updated user information comprises an updated speech corpus for the speech recognition mechanism. 44. The method of claim 43, further comprising: (D) associating a new device with a user in the database system.

Images