WO2017034217A1 - Appareil et procédé de surveillance de région interactive configurable par l'utilisateur - Google Patents

Appareil et procédé de surveillance de région interactive configurable par l'utilisateur Download PDF

Info

Publication number
WO2017034217A1
WO2017034217A1 PCT/KR2016/009097 KR2016009097W WO2017034217A1 WO 2017034217 A1 WO2017034217 A1 WO 2017034217A1 KR 2016009097 W KR2016009097 W KR 2016009097W WO 2017034217 A1 WO2017034217 A1 WO 2017034217A1
Authority
WO
WIPO (PCT)
Prior art keywords
interactive region
event
user
predetermined
interactive
Prior art date
Application number
PCT/KR2016/009097
Other languages
English (en)
Inventor
Brian Harms
Pol Pla
Yedan Qian
Olivier Bau
Original Assignee
Samsung Electronics Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/015,040 external-priority patent/US10057078B2/en
Application filed by Samsung Electronics Co., Ltd. filed Critical Samsung Electronics Co., Ltd.
Priority to CN201680048423.8A priority Critical patent/CN108353089B/zh
Priority to EP16839506.9A priority patent/EP3338433B1/fr
Publication of WO2017034217A1 publication Critical patent/WO2017034217A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • H04W4/021—Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences

Definitions

  • This disclosure generally relates to electronic event detection.
  • a depth sensor may generate a depth image that includes an array of pixels corresponding to regions of a physical environment with a depth value for each pixel indicating the depth of the corresponding region.
  • Depth images may be generated by stereo triangulation, sheet of light triangulation, structured light, interferometry, coded aperture, or other means.
  • Connected devices may be part of a communication network such as a local area network, wide area network, cellular network, the Internet, or any other suitable network.
  • a connected device may use a communication network to communicate with other devices, for example, to access remotely-stored data, access remote processing power, access remote displays, provide locally-stored data, provide local processing power, or provide access to local displays.
  • networks may provide communication paths and links to servers, which may host applications, content, and services that may be accessed or utilized by connected devices.
  • the content may include text, video data, audio data, user settings or other types of data.
  • Networks may use any suitable communication protocol or technology to facilitate communication between connected devices, such as, for example, BLUETOOTH, IEEE WI-FI (802.11a/b/g/n/ac), or TCP/IP.
  • An embodiment provides an apparatus comprising: a depth sensor interface, one or more computer-readable storage media embodying instructions, and one or more processors configured to execute the instructions to: monitor, at least in part by output of the depth sensor, a physical environment comprising a user-defined interactive region for a predetermined event, detect, at least in part by the output of the depth sensor, the predetermined event in the interactive region, and in response to detection of the predetermined event, initiate a predetermined action associated with the predetermined event.
  • FIG. 1 illustrates an example network environment associated with an interactive region monitoring system.
  • FIG. 2 illustrates an example interactive region in an example physical environment.
  • FIGS. 3A-D illustrate example embodiments of a user configuring an interactive region.
  • FIG. 4 illustrates multiple example interactive regions in an example physical environment.
  • FIG. 5 illustrates an example user interface for configuring interactive regions, predetermined events, event conditions, and predetermined actions.
  • FIG. 6 illustrates an example method implemented by an example interactive monitoring system.
  • FIG. 7 illustrates an example method for detecting a predetermined event using an image capture by a depth sensor.
  • FIG. 8 illustrates an example computer system.
  • An embodiment provides an apparatus comprising: a depth sensor interface, one or more computer-readable storage media embodying instructions, and one or more processors configured to execute the instructions to: monitor, at least in part by output of the depth sensor, a physical environment comprising a user-defined interactive region for a predetermined event, detect, at least in part by the output of the depth sensor, the predetermined event in the interactive region, and in response to detection of the predetermined event, initiate a predetermined action associated with the predetermined event.
  • the one or more processors are further configured to execute the instructions to identify the interaction region based on user input.
  • the one or more processors are further configured to execute the instructions to: in response to receiving the input identifying the interactive region, allocate, based at least in part on a monitoring load, resources for monitoring the interactive region.
  • the input is based at least in part on a gesture performed in the physical environment by the user.
  • the input is based at least in part on an interaction between the user and a user interface displayed on a client system.
  • one or more of: the predetermined action is defined at least in part by a user, or the association of the predetermined action with the predetermined event is defined at least in part by the user.
  • processors that are configured to execute the instructions to monitor the interactive region for a predetermined event further comprise processors that are configured to execute the instructions to: monitor the interactive region for a threshold event, and in response to detection of the threshold event, monitor the interactive region for the predetermined event.
  • the predetermined event comprises a plurality of event conditions, at least one event condition corresponding to the interactive region.
  • the predetermined event comprises one or more of: a presence of an object or a user in the interactive region, a movement of an object or a user within the interactive region, or contact between a user and at least a portion of an object within the interactive region.
  • the one or more processors are further configured to execute the instructions to periodically monitor, at least in part by output of the depth sensor, a plurality of interactive regions according to a priority associated with each interactive region.
  • a first priority associated with a first interactive region is based at least in part on one or more of: historical event data associated with the first interactive region, an expected frequency of one or more predetermined events in the first interactive region, an expected duration of one or more predetermined events in the first interactive region, or an importance of the first interactive region.
  • the one or more processors are further configured to execute the instructions to monitor, at least in part by the depth sensor, a physical environment comprising a user-defined interactive region for each of a plurality of predetermined events according to a priority associated with each event.
  • the interactive region comprises a first interactive region, at least a portion of the first interactive region comprises at least a portion of a second interactive region associated with one or more second-region predetermined events, and the processors are further configured to execute the instructions to detect, based on data created by monitoring the portion of the first interactive region, one or more second-region predetermined events in the portion of the second interactive region.
  • the physical environment is the user-defined interactive region.
  • An embodiment provides one or more computer-readable storage media embodying instructions that when executed by one or more processors cause the one or more processors to perform operations comprising: monitoring, based at least in part on output from a depth sensor, a physical environment comprising a user-defined interactive region for a predetermined event, detecting, based at least in part on output from the depth sensor, the predetermined event in the interactive region, and in response to detection of the predetermined event, initiating a predetermined action associated with the predetermined event.
  • the operations further comprise identifying the interaction region based on user input.
  • one or more of: the predetermined action is defined at least in part by a user, or the association of the predetermined action with the predetermined event is defined at least in part by the user.
  • An embodiment provides a method comprising: monitoring, at least in part by an output of a depth sensor, a physical environment comprising a user-defined interactive region for a predetermined event, detecting, at least in part by the output of the depth sensor, the predetermined event in the interactive region, and in response to detection of the predetermined event, automatically initiating a predetermined action associated with the predetermined event.
  • the method further comprises identifying the interactive region based on user input.
  • one or more of: the predetermined action is defined at least in part by a user, or the association of the predetermined action with the predetermined event is defined at least in part by the user.
  • the interactive regions may be monitored by an interactive region monitoring system for one or more predetermined events, such as, for example, events that have been defined by a user, automatically defined by a computing device, or both.
  • the interactive region monitoring system may include one or more depth sensors.
  • a predetermined action may be initiated by the interactive region monitoring system.
  • the interactive region monitoring system communicates electronically with another connected device to initiate the action.
  • An interactive region monitoring system may include one or more depth sensors, connected devices, servers, client systems, or any other devices or systems.
  • FIG. 1 illustrates an example network environment 100 for particular embodiments of an interactive region monitoring system.
  • Network environment 100 may include connected devices 130, data store 140, interactive region monitoring server 150, and sensors 160 connected to each other by network 110.
  • FIG. 1 illustrates a particular arrangement of connected devices 130, data store 140, interactive region monitoring server 150, and sensors 160, this disclosure contemplates any suitable arrangement of connected devices 130, data store 140, interactive region monitoring server 150, and sensors 160.
  • two or more of connected devices 130, data store 140, interactive region monitoring server 150, and sensors 160 may be connected to each other directly, bypassing network 110.
  • two or more of connected devices 130, data store 140, interactive region monitoring server 150, and sensors 160 may be physically or logically co-located with each other in whole or in part.
  • data store 140 may be directly connected to or co-located with interactive region monitoring server 150.
  • one or more sensors 160 may be directly connected to or co-located with one or more connected devices 130.
  • FIG. 1 illustrates a particular number of connected devices 130, data stores 140, interactive region monitoring servers 150, sensors 160, and networks 110 this disclosure contemplates any suitable number of connected devices 130, data stores 140, interactive region monitoring servers 150, sensors 160, and networks 110.
  • network environment 100 may include multiple connected devices 130, data stores 140, interactive region monitoring servers 150, sensors 160, and networks 110.
  • network 110 may include any suitable network 110.
  • one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these.
  • Network 110 may include one or more networks 110.
  • Links 120 may connect connected devices 130, data store 140, interactive region monitoring server 150, and sensors 160 to communication network 110 or to each other.
  • This disclosure contemplates any suitable links 120.
  • one or more links 120 include one or more wireline (such as for example Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless (such as for example Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (such as for example Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links.
  • wireline such as for example Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)
  • wireless such as for example Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)
  • optical such as for example Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH) links.
  • SONET Synchronous Optical Network
  • SDH Synchronous Digital Hierarchy
  • one or more links 120 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology-based network, a satellite communications technology-based network, another link 120, or a combination of two or more such links 120.
  • Links 120 need not necessarily be the same throughout network environment 100.
  • One or more first links 120 may differ in one or more respects from one or more second links 120.
  • a link 120 to connected device 130a may differ from a link 120 to connected device 130b and 130c.
  • a link 120 to sensor 160a may differ in one or more respects from a link 120 to sensor 160b and 160c.
  • an interactive region monitoring system may include one or more connected devices 130, data stores 140, interactive region monitoring servers 150, or sensors 160.
  • an interactive region monitoring system may be connected to one or more connected devices 130.
  • Connected devices 130 may include, without limitation, a television set, a mobile device such as a smartphone or tablet, a personal computer, a laptop computer, a gaming console, a thermostat, a media device, or any other appropriate device.
  • Connected devices 130 may, as an example, communicate with network 110 via a wireless communications protocol, such as Wi-Fi or BLUETOOTH.
  • interactive region monitoring server 150 may send an instruction to a connected device, such as an instruction to: turn on, turn off, tune to a particular channel, play audio, play video, or any other instruction.
  • an interactive region monitoring system may be connected to one or more sensors 160.
  • Sensors 160 may include any appropriate sensor, include as an example and not by way of limitation, a depth sensor, an image sensor, a digital camera, a red-green-blue (RGB) camera, an infrared camera, a time-of-flight camera, a structured-light camera, a laser scanner, a radar, a LIDAR, a sonar, a proximity sensor, a motion detector, any other suitable sensor, or any suitable combination thereof.
  • Sensors 160 may send data to interactive region monitoring server 150.
  • data store 140 may include networked storage, as illustrated in FIG. 1, such as cloud storage or other network accessible storage. Additionally or alternatively, data store 140 may include local storage within or directly attached to interactive region monitoring server 150, such as solid state drives (“SSDs”) or hard disk drives (“HDDs”).
  • SSDs solid state drives
  • HDDs hard disk drives
  • data store 140 may store a data structure corresponding to an interactive region.
  • a data structure may include, as an example and not be way of limitation, a point cloud, a wire-frame model, a parametric-based model, a boundary representation, a binary space partitioning tree, a k-dimensional tree, an octree, a constructive solid geometry tree, a grid, a polygon mesh model, a surface model, a solid computer-aided design (CAD) model, or any other suitable data structure.
  • CAD computer-aided design
  • data store 140 may store template data corresponding to an event or event condition.
  • a data structure may include, as an example and not by way of limitation, a template image or template skeletal tracking data.
  • data store 140 may store algorithmic data corresponding to an event condition.
  • data store 140 may store an algorithm that corresponds to an event condition that is a of a user's hand moving to the left at least 13 inches and then moving up at least 5 inches, but only if the time delay between the two movements is less than 1.3 seconds and only if the rest of the user's is relatively stable.
  • data store 140 may store data corresponding to an action.
  • data may include data about one or more connected devices 130.
  • data may include a unique device identifier, information about the user of a connected device 130, device details (e.g., manufacturer, operating system, localized language, audio-visual capabilities, battery capacity, network connectivity specifications, etc.), details about software applications on a connected device (e.g., name, version, maximum message size, etc.), or other appropriate device data.
  • data store 140 may store data about an application programming interface (API) that corresponds to one or more connected devices 130 or one or more software applications configured to run on one or more connected devices 130.
  • API application programming interface
  • data store 140 may store data that indicates an association between interactive regions, predetermined events, and predetermined actions.
  • data store 140 stores historical data. Historical data may include (1) one or more interactive regions that have been created and/or in which user interaction has occurred; (2) a time, date, or duration of user interactions; (3) data from an interactive region monitoring system; (4) information about predetermined events or event conditions; (5) information about predetermined actions; or (6) any other appropriate data, or any suitable combination thereof.
  • interactive region monitoring server 150 communicates with one or more sensors 160. In some embodiments, interactive region monitoring server 150 communicates to one or more sensors 160 instructions to monitor an interactive region. For example, interactive region monitoring server 150 may communicate with data store 140 and retrieve a data structure corresponding to an interactive region. Based on the data structure, interactive region monitoring server 150 may direct one or more sensors 160 to monitor the corresponding interactive region for one or more event or event conditions. In some embodiments, sensors 160 may not monitor portions of a physical environment that do not correspond to an interactive region.
  • an interactive region service 152 of interactive region monitoring server 150 may generate an appropriate form for communication with sensors 160 or data store 140.
  • interactive region service 152 may retrieve data from data store 140 that corresponds to an interactive region.
  • Interactive region service 152 may also retrieve data from one or more sensors 160. Based on a comparison of the data corresponding to an interactive region and the data from one or more sensors 160, interactive region service 152 may instruct one or more sensors 160 to monitor an interactive region.
  • an event detection service 154 of interactive region monitoring server 150 may determine whether an event condition has occurred based on template data. For example, event detection service 154 may retrieve template data corresponding to an event condition from data store 140. Template data may include data from exemplars of an event condition (e.g. , using data based on an exemplar provided by a user). Event detection service 154 may then compare the template data to the data retrieved from one or more sensors 160 to determine whether the event condition is detected. Additionally or alternatively, event detection service 154 may determine whether an event condition has occurred based on template data or algorithmic data corresponding to an event condition, as described above. In some embodiments, event detection service 154 may detect a predetermined event by determining one or more event conditions associated with the predetermined event and based on the detection of the one or more event conditions.
  • interactive region monitoring server 150 communicates with one or more connected devices 130. In some embodiments, interactive region monitoring server 150 communicates to one or more connected devices 130 to initiate a predetermined action. For example, interactive region monitoring server 150 may communicate with data store 140 and retrieve data corresponding to a predetermined action. Based on the data, interactive region monitoring server 150 may send an instruction to one or more connected devices 130 to initiate a predetermined action.
  • an action initiation service 156 of interactive region monitoring server 150 may initiate a predetermined action based on an association between interactive regions, events, and actions.
  • action initiation service 156 may retrieve data from data store 140 that indicates an association between interactive regions, events, and actions.
  • Action initiation service 156 may also receive data from interactive region service 152 and event detection service 154 that indicates that an event in an associated interactive region has been detected. Based on this, action initiation service 156 may use the data indicating an association between interactive regions, predetermined events, and predetermined actions to cross-reference the interactive region and predetermined event with the associated predetermined action.
  • Action initiation service 156 may then initiate the associated predetermined action.
  • an action initiation service 156 may determine an association between interactive regions, events, and actions based on a record for an interactive region.
  • the interactive region monitoring system maintains a data structure that includes a number of records (e.g. , by storing the record in data store 140 and updating the record via instructions from interactive region monitoring server 150).
  • Each record may define one or more interactive regions for monitoring and may be linked to one or more users.
  • each record can include data indicative of the coordinates of the interactive region, an event that triggers an action, the action that occurs in response to the event, and, in some embodiments, the device that performs the action.
  • the interactive region monitoring system may generate a record based on user input ( e.g. , when a user configures an interactive region, defines an event, defines an action, etc.).
  • an action initiation service 156 may initiate an action by generating an appropriate instruction to send to one or more connected devices 130.
  • action initiation service 156 may retrieve data from data store 140 that corresponds to a predetermined action. The action may be performed by one or more of connected devices 130.
  • Action initiation service may also retrieve from data store 140 about the one or more connected devices 130.
  • action initiation service may initiate an action by determining an appropriate instruction and sending that instruction to a connected device. For example, data about a mobile phone may include a phone number.
  • the action initiation service may use the phone number to uniquely identify the mobile phone for delivery of the text message (e.g. , by emailing a short message service (SMS) gateway).
  • SMS short message service
  • FIG. 2 illustrates an example embodiment of an interactive region in an example physical environment.
  • depth sensor 210 generates a depth image as an output.
  • a depth image may be an array of pixels corresponding to regions of a physical environment with a depth value for each pixel indicating the depth of the corresponding region.
  • a depth image may be generated, as an example and not by way of limitation, by: stereo triangulation, sheet of light triangulation, structured light, interferometry, coded aperture, or any other appropriate means of detecting a depth image.
  • depth sensor 210 may include or be a part of an electronic system which includes an image sensor, a digital camera, a red-green-blue (RGB) camera, an infrared camera, a time-of-flight camera, a structured-light camera, a laser scanner, a radar, a LIDAR, a sonar, a proximity sensor, a motion detector, any other suitable sensor, or any suitable combination thereof.
  • depth sensor 210 (or an electronic system of which depth sensor 210 is a part) may also detect sound, chemical particles, electric current, electric potential, magnetic fields, air flow, light, pressure, force, temperature, heat, motion, images, time, date, biometric data, or any other data.
  • depth sensor 210 (or an electronic system of which depth sensor 210 is a part) may detect data about the movement of objects or people or the speed of objects or people.
  • an interactive region monitoring system may monitor ( e.g. , by depth sensor 210) one or more interactive regions, such as interactive region 220, for one or more predetermined events.
  • An interactive region may take any suitable size, shape, or location within physical environment. As illustrated in FIG. 2, in particular embodiments an interactive region may be defined by a three-dimensional shape. In particular embodiments, an interactive region may be (or very nearly be) a surface, such as the surface of a table.
  • an interactive region monitoring system may monitor ( e.g. , by depth sensor 210) one or more interactive regions, such as interactive region 220, for one or more predetermined events.
  • a predetermined event may consist of two or more event conditions that collectively make up the event, i.e. , a predetermined action may be initiated only when multiple events (which collectively are the predetermined event) are detected.
  • a predetermined event may occur within an interactive region when one or more event conditions making up the predetermined event occur within the interactive region.
  • a predetermined event may include two event conditions: 1) that a particular user enter an interactive region around or on a desk chair; and 2) that the temperature in the room be more than a threshold temperature.
  • both event conditions may be required to trigger a predetermined action, such as turning on a fan or air conditioner.
  • a predetermined event may be equivalent to an event condition or that a predetermined event may require the occurrence of multiple event conditions.
  • This disclosure contemplates that, where suitable, disclosure relating to event conditions applies to predetermined events, and vice versa.
  • predetermined events and/or event conditions may include: movement within an interactive region, the presence or absence of an object or person within an interactive region, an object or person entering an interactive region, an object or person leaving an interactive region, or any other suitable event condition.
  • an event condition may be that other event conditions occur a particular order, within a particular time of one another, or any other relational condition.
  • an event condition may include output from any sensor or device, such as the sensors or devices illustrated in Fig. 1.
  • one or more event conditions may be determined by a user, e.g. , from a list of event conditions.
  • an interactive region monitoring system may detect a predetermined event by receiving data from depth sensor 210, a connected device, or by a combination of devices that indicate that event conditions have occurred.
  • a list of event conditions may include event conditions that do not require resource-intensive image processing, such as movement, a level of brightness, an object or user entering or leaving the interactive region, or any other suitable event.
  • an event condition may include recognizing a particular object or object type, recognizing a user (such as the user's face or a client system of the user), recognizing a particular gesture or body position, or any other event condition.
  • a predetermined event or an event condition may not be detected unless a threshold event is detected.
  • Detecting a threshold event may include receiving information (e.g. , from a depth sensor, a connected device, a database, etc.) that a threshold event has been detected.
  • an interactive region monitoring system may monitor interactive region 220 via depth sensor 210 to detect the presence of a person as a threshold event.
  • the interactive region monitoring system may monitor interactive region 220 for particular user 230, for example, by processing the image captured by the depth sensor using facial recognition, which may be more resource-intensive to accomplish than is detection and processing of the triggering event.
  • a relatively low resource-intensive threshold event may be detected and processed before one or more relatively resource-intensive tasks associated with detecting an event, thereby conserving computer resources during periods where the threshold event is not detected.
  • one or more connected devices communicatively coupled to the interactive region monitoring system may be used to detect a predetermined event or an event condition.
  • a particular user may have a connected device which communicates information that uniquely identifies its user.
  • user 230 may have a mobile phone, smart watch, tablet, laptop, or other device that communicates to an interactive region monitoring system (e.g., via BLUETOOTH) a unique identifier (e.g. a username, a phone number, a media access control (MAC) address, etc.) that identifies user 230.
  • the interactive region monitoring system may determine whether a threshold condition has occurred.
  • the interactive region monitoring system may broadcast a request for proximate connected devices to reply with information that uniquely identifies their users or the respective user devices.
  • an interactive region may be associated with one or more particular users.
  • an interactive region might be associated with user 230.
  • the association may be determined from a data store, such as data store 140, that correlates the region and the user, for example through a profile for the user, a profile for the region, or both.
  • the interactive region may not be monitored for a predetermined event unless a particular user's presence is detected in the interactive region.
  • a user's presence may be detected by, for example, facial recognition.
  • interactive region may broadcast a request for proximate connected devices to reply with information that uniquely identifies the user or users of the connected device.
  • the interactive region may be monitored for a predetermined event only if a proximate connected device identifies the particular user associated with the interactive region. Additionally or alternatively, an interactive region may not be associated with any particular user.
  • an interactive region monitoring system may monitor an interactive region only when a threshold event is detected. For example, an interactive region may only be monitored during certain times of day, if an on/off toggle is on, or if any other event condition or predetermined event is satisfied.
  • a predetermined action may be initiated by an interactive region monitoring system when a predetermined event is detected , for example when one or more event conditions for the predetermined event are detected.
  • Event conditions may include detecting a predetermined event within an interactive region and may include one or more other event conditions, such as the time of day; output from a device (such as a smartphone, watch, appliance, etc.); an environmental condition such as light, noise, temperature, etc.; or any other suitable event condition.
  • a predetermined action may be performed by an interactive region monitoring system, by a device connected to an interactive region monitoring system, by a depth sensor, or by any other suitable device.
  • an interactive region monitoring system including depth sensor 210 may monitor interactive region 220 for the predetermined event of a user entering interactive region 220.
  • the interactive region monitoring system may also connect to TV 240.
  • the interactive region monitoring system may initiate the predetermined action of TV 240 turning on by sending an instruction to TV 240.
  • a predetermined action may include delivering a message or reminder to a user.
  • user 230 may have a history of watching the show 'Game of Thrones.' If user 230 enters interactive region 220, an interactive region monitoring system may initiate the playing of an audio reminder that informs user 230 that the season premiere of 'Game of Thrones' will start in 10 minutes.
  • a predetermined event may include multiple event conditions.
  • one event condition may be that other event conditions occur a particular order, within a particular time of one another, or any other relational condition.
  • an interactive region may be monitored for multiple predetermined events, each of which triggers a predetermined action.
  • an interactive region monitoring system may be monitoring interactive region 220 via depth sensor 210 for the predetermined event including the event condition of a user entering interactive region 220. When the event condition is satisfied by the detection of a user entering interactive region 220, the interactive region monitoring system may initiate the predetermined action of turning on TV 240.
  • the interactive region monitoring system may be monitoring interactive region 220 for an additional predetermined event including the event condition that particular user 230 enters interactive region 220. If user 230 enters interactive region 220, then the interactive region monitoring system may initiate the predetermined action of tuning TV 240 to a particular channel particular to user 230. As another example, an interactive region monitoring system may be monitoring interactive region 220 for the predetermined event including the event conditions of a user leaving and TV 240 being on. If the interactive region monitoring system detects that TV 240 is on and that a user left interactive region 220, the interactive region monitoring system may initiate the predetermined action of pausing video content on TV 240.
  • an interactive region monitoring system may enhance a physical object or surface in an environment.
  • an interactive region monitoring system may enhance a retail store display by allowing a user to touch or come near a particular item and displaying the price for the item in response.
  • the interactive region would include all or some of the item or a space near the item, the predetermined event would be contacting or coming near the item, and the predetermined action would be displaying the price of the item.
  • an interactive region monitoring system may be used in a museum to link objects to content on an audio tour.
  • an interactive region monitoring system may be used in a classroom to link different parts of the inside of a computer to different video tutorials or to the display of more detailed information about a particular computer component.
  • an interactive region monitoring system may be used at a corporate event when unveiling a new product to trigger appropriate content in a presentation when interaction with a prototype of the product, such as touching the prototype, occurs.
  • an interactive region may include any suitable space or combination of spaces; a predetermined event may include one or more suitable event conditions; and a predetermined action corresponding to a predetermined event may include any suitable one or more actions.
  • FIG. 3A-D illustrates example embodiments of user 230 configuring an interactive region.
  • configuring an interactive region may include determining an interactive region (e.g., determining the shape, area, location, size, orientation, position, or other properties of an interactive region), moving an existing interactive region, modifying an existing interactive region, or deleting an existing interactive region.
  • an interactive region monitoring system may store data that represents properties of an interactive region. For example, an interactive region monitoring system may store a data structure corresponding to an interactive region ( e.g.
  • a point cloud a wire-frame model, a parametric-based model, a boundary representation, a binary space partitioning tree, a k-dimensional tree, an octree, a constructive solid geometry tree, a grid, a polygon mesh model, a surface model, a solid computer-aided design (CAD) model, or any other suitable data structure).
  • CAD computer-aided design
  • FIG. 3A illustrates an example embodiment where user 230 determines interactive region 320 via a gesture.
  • the interactive region monitoring system may receive from user 230 user input data that indicates a request to define a new interactive region or to redefine an existing interactive region.
  • the interactive region monitoring system may receive location data representative of the location of the interactive region.
  • the interactive region monitoring system may facilitate the user providing location data in a number of ways described below.
  • the interactive region monitoring system may detect and track, using depth sensor 310, a gesture by user 230 to indicate an interactive region.
  • the shape of the interactive region may have a template shape, such as a cuboid (e.g., interactive region 320), a cylinder (e.g., interactive region 350 in FIG. 3B), a polyhedron, a polytope, a sphere, an ellipsoid, a volume bounded by a mesh, a surface area, or any other shape.
  • user 230 may gesture by moving a hand along path 330.
  • the interactive region monitoring system may detect the gesture via depth sensor 310 and may determine the interactive region 320 based on path 330.
  • an interactive region monitoring system may track a gesture via a user device.
  • a user may gesture with a hand that is holding a device, and the device may use an inertial sensor, a GPS sensor, or another suitable sensor to generate and send data to the interactive region monitoring system or other suitable component to determine an interactive region.
  • the shape of an interactive region may be a predetermined type of shape with a minimized volume such that the user's gesture is entirely bounded within the shape.
  • interactive region 320 may be the cube with the smallest possible volume such that path 330 is within interactive region 320.
  • an interactive region may take one of a number of predetermined shapes.
  • an interactive region may take the shape created by the user's gesture. This disclosure contemplates any suitable method for defining an interactive region, including by using one or more voice commands, one or more gestures, one or more instruments (such as, e.g. a stylus), etc.
  • a user may indicate that they desire to define an interactive region.
  • user 230 may indicate the desire to define in interactive environment by causing a connected device to send information indicating the desire to an interactive region monitoring system.
  • user 230 may have a software application installed on a connected device which sends data via a network to an interactive region monitoring system.
  • an interactive region monitoring system may monitor the user's gestures and determine an interactive region based on those gestures.
  • an interactive region monitoring system may detect a voice command signaling a user's desire to define an interactive region.
  • This application contemplates any suitable method for a user to signal a desire to define an interactive region.
  • an interactive region may intersect physical objects (e.g., a stove, walls, floors, furniture, etc.).
  • interactive region 320 may extend downward into stove 340.
  • portions of an interactive region that intersect a physical object may not be monitored for a predetermined event or event condition.
  • the portion of interactive region 320 that includes stove 340 may not be monitored for a predetermined event or event condition.
  • an interactive region, a predetermined event, an action, or any suitable combination thereof may be configured dynamically based on one or more predefined conditions.
  • the shape, area, location, size, orientation, or position of an interactive region may be modified based on a predefined condition.
  • Such conditions may include a time of day, environmental conditions, user preferences, or any other suitable condition.
  • an interactive region around a lamp may become larger as the area near the lamp becomes darker.
  • a predetermined event may include a user's presence in an interactive region and a temperature in the room containing the interactive region that is above a threshold temperature.
  • the corresponding action may include turning on the fan. The speed of the fan may change dynamically based on the temperature in the space when the user is present.
  • an interactive region may be configured to come into existence, be deleted, activate, or deactivate based on conditions.
  • a 24-hour grocery store may use an interactive region monitoring system to monitor two doorways.
  • one of the doorways may be shut and locked during certain times of the day, and the interactive region corresponding to that entry may be configured to be deactivated during the times of day when the door is locked and to reactivate after that time.
  • FIG. 3B illustrates an example embodiment of a physical environment in which multiple interactive regions are present.
  • multiple interactive regions may be configured.
  • the multiple interactive regions may be distinct from one another, partially overlap, or completely overlap.
  • the image data from overlapping interactive regions are processed or monitored in a manner that may reduce duplication.
  • interactive region 320 may be within interactive region 350.
  • An interactive region monitoring system may monitor the portions of the physical environment that are within both interactive region 320 and interactive region 350 using the same data.
  • depth sensor 310 may capture an image of the region of overlap between interactive region 320 and interactive region 350, and that image may be used to monitor both region 320 and region 350.
  • an interactive region monitoring system may be able to monitor a particular region of a physical environment a certain number of times per second. If a particular region of the physical environment is contained within multiple interactive regions, the interactive region monitoring system may monitor the portions of each interactive region that overlaps with another interactive region the same number of times per second as it would if the particular region were only within one of the interactive regions.
  • a predetermined event may include multiple event conditions detected in one or more regions.
  • an interactive region monitoring system may be monitoring both interactive region 320 and interactive region 350 via depth sensor 310.
  • a predetermined event may include event conditions that a pot has been placed on the element in interactive region 320 and that a user has left interactive region 350. If the interactive region monitoring system detects the predetermined event, then the interactive region monitoring system may initiate the predetermined action of turning on the element of stove 340 in interactive region 320.
  • FIG. 3C and 3D illustrate an example embodiment where user 230 determines interactive region 320 using client system 360.
  • a user may use a client system to aid in determining an interactive region.
  • user 230 may wish to determine interactive region 350 via gesture.
  • an interactive region is not visible to a user using only the human eye.
  • client system 360 may display part of the physical environment via a camera.
  • Client system 360 may also receive data that determines an interactive region (e.g., the location, position, size, shape, etc.).
  • Client system 360 may display to user 230 an image that includes a visual representation of the interactive region in the physical environment.
  • a user may be viewing a client system as they are gesturing and the client system may display an interactive region determined by the gesture as the user is gesturing.
  • a client system may display only some interactive regions but not others. For example, FIG. 3D shows only interactive region 350, but not interactive region 320.
  • a user may determine an interactive region via a display of a client system.
  • client system 360 may have software that allows user 230 to determine interactive region 350.
  • the client system may display an image representative of the physical environment (e.g., via a camera).
  • the user may interact with the client system to determine an interactive region.
  • user 230 may use client system 360 to determine interactive region 350 by choosing a shape from a list of shapes and using a touch screen to indicate the position or size of interactive region 350.
  • user 230 may use client system 360 to determine interactive region 350 by drawing the region on a display of client system 360.
  • This disclosure contemplates any suitable method of a user interacting with a client system to determine an interactive region, including voice command, interacting with a user interface (UI), gesture, or any other suitable method.
  • UI user interface
  • FIG. 4 illustrates multiple example interactive regions in an example physical environment.
  • interactive region 420 contains within it several interactive regions 420a-e.
  • interactive region 420 and interactive regions 420a-e may be associated with predetermined events, event conditions, or predetermined actions.
  • an interactive region monitoring system may monitor via depth sensor 410 interactive region 420 and interactive regions 420a-e for the predetermined event including the event condition of the presence of a user's hand.
  • the interactive region monitoring system may initiate the predetermined action of toggling the state of lamp 430 (e.g., turning lamp 430 off or on), which may be connected to the interactive region monitoring system.
  • a user's hand entering interactive region 420 may also be a threshold event after which the interactive region monitoring system may detect a predetermined event including event conditions corresponding to the movement of the user's hand between regions 420a-e. For example, if a user's hand enters interactive region 420 and interactive region 420c, then lamp 430 may turn from the off state to the on state. If the user then moves the hand down from interactive region 420c to interactive region 420e, then the interactive region monitoring system may initiate the predetermined action of dimming lamp 430 (e.g., by varying the voltage lamp 430 uses to control the intensity of light output).
  • each region 420a-e may correspond to a particular level of brightness of lamp 430.
  • FIG. 5 illustrates an example user interface (UI) for associating or configuring interactive regions, predetermined events, event conditions, and predetermined actions.
  • a UI may provide a user with the ability to interact with client system 360 in such a way as to configure interactive regions; configure predetermined events or event conditions; configure predetermined actions; and associate interactive regions, predetermined events, event conditions, and predetermined actions with one another.
  • Client system 360 may communicate data that corresponds to a user's interactions to the interactive region monitoring system ( e.g. , via network 110).
  • a graphical UI e.g., a user interface that uses graphical icons or visual indicators displayed on a client system to provide a user with the ability to interact with a client system
  • this disclosure contemplates using a voice user interface (e.g., a user interface that allows a user to interact with a client system through voice or speech), a gesture interface (e.g., a user interface that provides a user with the ability to interact with a client system through the motion or state of some part of the user's body), or any other suitable type of UI.
  • a UI may include a list of interactive regions.
  • the list dynamically updates as a user configures interactive regions.
  • the interactive regions in the list may be given names.
  • list 510 includes interactive regions with the names "product display,” "store entrance,” “cash register,” and "employee only doorway.”
  • the physical environment may be in a retail store.
  • interactive regions may be named in a manner suggestive of the location of the interactive region in the physical environment.
  • the "product display" interactive region in list 510 may be located at a product display in the physical environment
  • the "store entrance” interactive region in list 510 may be located at the retail store's entrance in the physical environment, and so on.
  • the UI allows a user to configure an interactive region by assigning or altering a name for an interactive region.
  • a UI may present a list of event conditions.
  • a user may configure a predetermined event by selecting one or more event conditions and associate it with one or more regions and one or more predetermined actions.
  • an event condition may have one or more user-configurable fields.
  • FIG. 5 illustrates an example list 520 of event conditions. Some event conditions in list 520 have user-configurable fields 520a-b.
  • the event condition "person enters" has two user-configurable fields (speed and facial recognition).
  • a user may configure via user-configurable field 520a the event condition to be a person enters with a certain speed, greater than a certain speed, less than a certain speed, etc.
  • the event condition "person enters” also has a user-configurable field 520b, which a user may use to configure the event condition to be a person with a particular recognized face enters the interactive region.
  • FIG. 5 illustrates a particular example list 520, this disclosure contemplates any suitable event conditions or predetermined events with any number of suitable user-configurable fields.
  • a user may define a predetermined event, an event condition, or an action via an ostensive definition (e.g. , defining by example via performing or demonstrating an exemplar).
  • an interactive region monitoring system may be configured to recognize user input indicating a user's desire to ostensively define a predetermined event or an event condition in a given interactive region. If a user indicates such a desire, then the interactive region monitoring system may monitor the interactive region. The user may then provide an exemplar of the event condition 'a person enters the interactive region' by entering the interactive region.
  • the interactive region monitoring system may then recognize the exemplar and define based on the exemplar the event condition 'a person enters the interactive region.'
  • the event may be available for selection via a UI (e.g. , an ostensively defined predetermined event may appear in list 520 after it is defined).
  • a UI may include a list of predetermined actions.
  • list 530 includes several example predetermined actions.
  • Each action may be further configurable by a user (e.g., by a user-configurable field).
  • the predetermined action “play a video on a display” may allow a user to select a particular display device and a particular video.
  • predetermined actions may be performed by any connected device, and a UI may allow a user to select which predetermined action is performed by which connected device.
  • a UI may include a list of possible connected devices.
  • a UI may include a different list for each connected device, each list including only those predetermined actions that the particular corresponding connected device is capable of performing.
  • a user may define a predetermined event or a predetermined action.
  • a user may define a predetermined event or a predetermined action via interactions with a connected device or by any other suitable means of defining a predetermined event or predetermined action.
  • data that corresponds to the user-defined predetermined event or predetermined action may be stored in a data store.
  • the user-defined predetermined event or a predetermined action is available in a list, such as list 520 or 530.
  • a UI may allow a user to associate an interactive region, a predetermined event and a predetermined action.
  • a user may select the "product display" interactive region from list 510. The user may then associate that interactive region with the predetermined event that includes the event condition "person enters" from list 520. The user may then associate the interactive region and the predetermined event with the predetermined action "play a video on a display” from list 530. The user may further configure the predetermined action to play a video demonstrating the use of a product in the display on a display device near the interactive region.
  • the interactive region monitoring system may cause the video to play on the display device.
  • an interactive region may move as an object in the physical environment moves.
  • list 510 contains the interactive region "cash register.”
  • This interactive region may correspond to an object cash register in the physical environment of a retail store.
  • the interactive region monitoring system may be monitoring the interactive region for the predetermined event that includes the event condition "cash drawer opened,” and may initiate the predetermined action "use a camera to record and store video" if the predetermined event is detected. In this way, the user may record video of the cash register using a connected camera when the cash drawer of the cash register is opened. An employee may be cleaning the area around the cash register and move the cash register.
  • An interactive region monitoring system may detect that the cash register has moved (e.g., by using an image histogram, Canny edge detection, or other suitable means) and cause the interactive region "cash register" to move along with the cash register.
  • an interactive region monitoring system may have limited processing capabilities, memory, or other computer resources.
  • An interactive region, predetermined event, or event condition may be prioritized by using relatively more or fewer computer resources for the interactive region, predetermined event, or event condition compared to other interactive regions, predetermined events, or event conditions.
  • prioritization may involve using a higher framerate to monitor an interactive region compared to other interactive regions, processing images from an interactive region at a higher rate or sooner than images from other interactive regions, performing more or less resource-intensive types of monitoring, or any other suitable method of prioritization.
  • an interactive region might be monitored for an event condition that involves a high rate of speed.
  • a user may indicate that the event condition involves a high rate of speed or an interactive region monitoring system may determine that an event condition involves a high rate of speed based on machine learning (e.g. , based on an analysis of historical data) or based on an association with a particular type of predetermined event or event condition.
  • the interactive region monitoring system may prioritize interactive regions where an event condition involving a high rate of speed is anticipated by monitoring the interactive regions at a higher framerate than is used for other interactive regions.
  • the event condition "cash register opens" or a predetermined event including that event condition may be prioritized based on the importance of the interactive region or the predetermined event.
  • particular interactive regions, predetermined events, or event conditions may be automatically prioritized by an interactive region monitoring system based on a monitoring load.
  • a monitoring load may be the computer resources (such as the memory, processors, and sensors) required to administer an interactive region monitoring system, such as monitoring interactive regions; detecting events; creating, altering, activating, or deactivating particular interactive regions, events, and actions; etc.
  • the monitoring load may become greater than the computer resources available to an interactive region monitoring system.
  • an interactive region monitoring system may prioritize any suitable aspect of the interactive region monitoring system (such as regions or portions of regions monitored, events monitored, actions initiated, etc.) based on a monitoring load, such as when the monitoring load reaches a threshold monitoring load.
  • prioritizing reduces the monitoring load. Examples of prioritization may include: using a lower framerate to monitor an interactive region, delaying the processing of images from an interactive region, performing less resource-intensive types of resource monitoring, or any other suitable method of prioritization.
  • prioritization based on a monitoring load may be dynamic. For example, an interactive region monitoring system may be automatically prioritized by an interactive region monitoring system based on a monitoring load at a certain time interval or in response to certain triggers ( e.g. , after an interactive region is configured).
  • a predetermined action may occur at any location.
  • a predetermined action may occur within an interactive region.
  • the interactive region "employee only doorway" in list 510 may contain a door within the interactive region.
  • a user may associate the interactive region with the predetermined event including the event condition "person enters" in list 520 and use user-configurable field 520b to indicate that the face must be recognized as an employee.
  • the user may further associate the interactive region and predetermined event with the predetermined action "unlock a door” in list 530 and configure that action to unlock the door within the interactive region.
  • the interactive region monitoring system may detect the employee, recognize the employee's face, and then unlock the door.
  • a predetermined action may occur outside of an interactive region.
  • a user may associate the interactive region "store entrance" in list 510 with the predetermined event including the event conditions "movement" in list 520 and that the retail store is closed (e.g., detected by determining the time of day compared to the retail stores hours of operation).
  • the user may associate the interactive region and predetermined event with the predetermined action "send a text message to a phone number" from list 530 to send a text message to the user's phone indicating that someone may be in the retail store after it has closed.
  • the interactive region monitoring system detects movement in the "store entrance" interactive region and the store is closed, it may cause a text message to be sent to the user, where the text message may be sent or received outside of any interactive region.
  • FIG. 6 illustrates an example method 600 implemented by an example interactive monitoring system.
  • an interactive region monitoring system may receive input identifying an interactive region.
  • the input may be from a user.
  • a user may configure an interactive region by providing input identifying an interactive region via gesture, a connected device, or any other suitable means.
  • a data store of the interactive region monitoring system may store a data structure corresponding to an interactive region, as described above in greater detail in connection with FIG. 1.
  • the interactive region monitoring system may monitor one or more interactive regions for a predetermined event, including one or more event conditions.
  • the interactive region monitoring system may monitor an interactive region by monitoring the physical environment that the interactive region is a part of.
  • the physical environment monitored by the interactive region monitoring system may include regions outside of the interactive regions (e.g. , an entire field of view of the sensor, room, or building).
  • the physical environment monitored by the interactive region monitoring system may include only interactive regions , i.e. , the system or a particular system component may monitor only interactive regions or portions of interactive regions.
  • only portions of regions monitored by the system may be processed by the system, as described more fully herein.
  • the interactive region monitoring system may access the data store 140 to determine the locations of the interactive regions in the physical space.
  • the interactive region monitoring system may selectively process portions of the sensor data that correspond to the determined interactive regions.
  • the interactive region monitoring system may monitor for a predetermined event by receiving data from one or more sensors.
  • data from one or more sensors may be sent to an event detection service of an interactive region monitoring server, which may then use the data received to detect a predetermined event.
  • the interactive region monitoring system may detect a predetermined event in an interactive region.
  • a predetermined event may include one or more event conditions, (e.g. , a change within the interactive region, receiving information, environmental conditions, etc.).
  • an event detection service of an interactive region monitoring server may determine whether a predetermined event has been detected based on the data received in step 620.
  • the interactive region monitoring system may detected a predetermined event based at least in part on a depth image for the interactive region (e.g., via a depth sensor).
  • a depth image may be generated from a depth sensor, as an example and not by way of limitation, by: stereo triangulation, sheet of light triangulation, structured light, interferometry, coded aperture, or any other appropriate means of detecting a depth image.
  • Other data may also be acquired in step 620, including data related to the detection of sound, chemical particles, electric current, electric potential, magnetic fields, air flow, light, pressure, force, temperature, heat, motion, images, time, date, biometric data, or any other data.
  • the depth image may be used, for example, to determine whether an object has entered, is present in, or has left an interactive region.
  • an event detection service of an interactive region monitoring server may retrieve the depth image from one or more sensors.
  • the interactive region monitoring system may initiate a predetermined action if a predetermined event is detected in step 630.
  • a predetermined action may take place within an interactive region, outside an interactive region, or both.
  • a predetermined action may be performed by the interactive region monitoring system, by a connected device, or by any other suitable means.
  • a predetermined action may be initiated by an action initiation service of an interactive region monitoring server.
  • Particular embodiments may repeat one or more steps of the method of FIG. 6, where appropriate.
  • this disclosure describes and illustrates particular steps of the method of FIG. 6 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of FIG. 6 occurring in any suitable order.
  • this disclosure describes and illustrates an example method implemented by an example interactive region including the particular steps of the method of FIG. 6, this disclosure contemplates any suitable method implemented by any suitable interactive region including any suitable steps, which may include all, some, or none of the steps of the method of FIG. 6, where appropriate.
  • this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of FIG. 6, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of FIG. 6.
  • FIG. 7 illustrates an example method 700 for monitoring for a predetermined event or an event condition and detecting the predetermined event based on one or more images captured at least in part by a depth sensor.
  • steps 620, 630 of FIG. 6 may be performed by using method 700.
  • steps 702, 704 of FIG. 7 may correspond to step 620 of FIG. 6
  • steps 706, 708 of FIG. 7 may correspond to step 630 of FIG. 6.
  • Method 700 may begin with step 702 , where a processor of a computing device, such as any suitable computing device described herein, may receive data representing an image of at least a portion of an interactive region.
  • a portion of the image may be characterized as background or other non-event information.
  • the method may refer to previous images of the environment and identify portions that have not changed, or have not changed more than a particular threshold, as background.
  • only background or non-event data for the portion of the image corresponding to the interactive region may be identified.
  • the background of an image may be subtracted from the image prior to processing the image.
  • background subtraction may be accomplished by frame differencing, using a mean filter, running a Gaussian average, using a background mixture model, or any other suitable method.
  • background subtraction may isolate objects in the foreground of an image to aid in the detection of movement, object detection, object localization, object recognition, object tracking, semantic segmentation, or other forms image processing.
  • background subtraction may allow data that indicates change (e.g., movement, change in color, etc.) to be isolated from other data.
  • only background corresponding to the interaction region may be subtracted.
  • a portion of the image not characterized as background or non-event data may be processed to determine whether the portion contains information indicating that an event or event condition has occurred.
  • Image processing may include measuring the level of the background-subtracted information and comparing the level to a threshold. If the level is greater than the threshold, then it may be determined that an event or event condition has occurred. Additionally or alternatively, image processing may also include object recognition, facial recognition, or any other type of image processing. Any suitable techniques may be used to process an image, for example and not by way of limitation: edge matching, detection of a change in lighting or color, greyscale matching, using a histogram, comparing an image to a one or more other images, using an interpretation tree, etc.
  • step 706 may include skeleton tracking (e.g., using skeleton data to infer a body position of a person or a gesture made by a person).
  • skeleton tracking may be accomplished by using a model representing a human target, where the model may include adjustable skeletal points corresponding to the joints on a human.
  • the model may be adjusted into different poses corresponding to poses performed by the human target.
  • skeleton data may be determined based on a comparison of data corresponding to known gestures or poses.
  • machine learning may be used to determine skeleton data.
  • skeleton data may indicate that a person or a part of a person is within, has entered, or has left a particular interactive region.
  • method 700 may end, and step 620 of FIG. 6 may be performed. If an event is detected, method 700 may end, and step 640 of FIG. 6 may be performed.
  • method 700 may include step 708 , where the method determines whether a predetermined event requires any additional event condition(s), and if so, whether those event condition(s) have been satisfied. As described more fully herein, the occurrence of other event conditions may be determined by processing the image captured by the depth sensor or determined from output from any other suitable sensor. If other event conditions are required to detect an event and at least one of those event conditions is not satisfied, then the method may end, and step 620 of FIG. 6 may be performed. If an event does not include additional event conditions, or if all the required additional event conditions are satisfied, then the method may end, and step 640 of FIG. 6 may be performed.
  • Particular embodiments may repeat one or more steps of the method of FIG. 7, where appropriate.
  • this disclosure describes and illustrates particular steps of the method of FIG. 7 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of FIG. 7 occurring in any suitable order.
  • this disclosure describes and illustrates an example method for detecting a predetermined event, or an event condition, including the particular steps of the method of FIG. 7, this disclosure contemplates any suitable method for detecting a predetermined event, or an event condition, which may include all, some, or none of the steps of the method of FIG. 7, where appropriate.
  • this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of FIG. 7, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of FIG. 7.
  • FIG. 8 illustrates an example computer system 800.
  • one or more computer systems 800 perform one or more steps of one or more methods described or illustrated herein.
  • one or more computer systems 800 provide functionality described or illustrated herein.
  • software running on one or more computer systems 800 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein.
  • Particular embodiments include one or more portions of one or more computer systems 800.
  • reference to a computer system may encompass a computing device, and vice versa, where appropriate.
  • reference to a computer system may encompass one or more computer systems, where appropriate.
  • computer system 800 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these.
  • SOC system-on-chip
  • SBC single-board computer system
  • COM computer-on-module
  • SOM system-on-module
  • computer system 800 may include one or more computer systems 800; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks.
  • one or more computer systems 800 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein.
  • one or more computer systems 800 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein.
  • One or more computer systems 800 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
  • computer system 800 includes a processor 802, memory 804, storage 806, an input/output (I/O) interface 808, a communication interface 810, and a bus 812.
  • I/O input/output
  • this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.
  • processor 802 includes hardware for executing instructions, such as those making up a computer program.
  • processor 802 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 804, or storage 806; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 804, or storage 806.
  • processor 802 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 802 including any suitable number of any suitable internal caches, where appropriate.
  • processor 802 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs).
  • TLBs translation lookaside buffers
  • Instructions in the instruction caches may be copies of instructions in memory 804 or storage 806, and the instruction caches may speed up retrieval of those instructions by processor 802.
  • Data in the data caches may be copies of data in memory 804 or storage 806 for instructions executing at processor 802 to operate on; the results of previous instructions executed at processor 802 for access by subsequent instructions executing at processor 802 or for writing to memory 804 or storage 806; or other suitable data.
  • the data caches may speed up read or write operations by processor 802.
  • the TLBs may speed up virtual-address translation for processor 802.
  • processor 802 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 802 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 802 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 802. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.
  • ALUs
  • memory 804 includes main memory for storing instructions for processor 802 to execute or data for processor 802 to operate on.
  • computer system 800 may load instructions from storage 806 or another source (such as, for example, another computer system 800) to memory 804.
  • Processor 802 may then load the instructions from memory 804 to an internal register or internal cache.
  • processor 802 may retrieve the instructions from the internal register or internal cache and decode them.
  • processor 802 may write one or more results (which may be intermediate or final results) to the internal register or internal cache.
  • Processor 802 may then write one or more of those results to memory 804.
  • processor 802 executes only instructions in one or more internal registers or internal caches or in memory 804 (as opposed to storage 806 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 804 (as opposed to storage 806 or elsewhere).
  • One or more memory buses (which may each include an address bus and a data bus) may couple processor 802 to memory 804.
  • Bus 812 may include one or more memory buses, as described below.
  • one or more memory management units reside between processor 802 and memory 804 and facilitate accesses to memory 804 requested by processor 802.
  • memory 804 includes random access memory (RAM).
  • This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM.
  • Memory 804 may include one or more memories 804, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.
  • storage 806 includes mass storage for data or instructions.
  • storage 806 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these.
  • Storage 806 may include removable or non-removable (or fixed) media, where appropriate.
  • Storage 806 may be internal or external to computer system 800, where appropriate.
  • storage 806 is non-volatile, solid-state memory.
  • storage 806 includes read-only memory (ROM).
  • this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these.
  • This disclosure contemplates mass storage 806 taking any suitable physical form.
  • Storage 806 may include one or more storage control units facilitating communication between processor 802 and storage 806, where appropriate. Where appropriate, storage 806 may include one or more storages 806. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.
  • I/O interface 808 includes hardware, software, or both, providing one or more interfaces for communication between computer system 800 and one or more I/O devices.
  • Computer system 800 may include one or more of these I/O devices, where appropriate.
  • One or more of these I/O devices may enable communication between a person and computer system 800.
  • an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these.
  • An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 808 for them.
  • I/O interface 808 may include one or more device or software drivers enabling processor 802 to drive one or more of these I/O devices.
  • I/O interface 808 may include one or more I/O interfaces 808, where appropriate.
  • communication interface 810 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 800 and one or more other computer systems 800 or one or more networks.
  • communication interface 810 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network.
  • NIC network interface controller
  • WNIC wireless NIC
  • WI-FI network wireless network
  • computer system 800 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these.
  • PAN personal area network
  • LAN local area network
  • WAN wide area network
  • MAN metropolitan area network
  • computer system 800 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these.
  • Computer system 800 may include any suitable communication interface 810 for any of these networks, where appropriate.
  • Communication interface 810 may include one or more communication interfaces 810, where appropriate.
  • bus 812 includes hardware, software, or both coupling components of computer system 800 to each other.
  • bus 812 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these.
  • Bus 812 may include one or more buses 812, where appropriate.
  • a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate.
  • ICs such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)
  • HDDs hard disk drives
  • HHDs hybrid hard drives
  • ODDs optical disc drives
  • magneto-optical discs magneto-optical drives
  • references in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.

Abstract

Dans un mode de réalisation de l'invention, un ou plusieurs systèmes peuvent recevoir une entrée, d'un utilisateur, identifiant une région interactive d'un environnement physique. Le ou les systèmes peuvent déterminer un emplacement de la région interactive par rapport à un capteur de profondeur et surveiller, au moins en partie par le capteur de profondeur, la région interactive pour un événement prédéterminé. Le ou les systèmes peuvent détecter, au moins en partie par le capteur de profondeur, l'événement prédéterminé. En réponse à la détection de l'événement prédéterminé, le ou les systèmes peuvent initier une action prédéterminée associée à l'événement prédéterminé.
PCT/KR2016/009097 2015-08-21 2016-08-18 Appareil et procédé de surveillance de région interactive configurable par l'utilisateur WO2017034217A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201680048423.8A CN108353089B (zh) 2015-08-21 2016-08-18 用于用户可配置的交互区域监视的装置和方法
EP16839506.9A EP3338433B1 (fr) 2015-08-21 2016-08-18 Appareil et procédé de surveillance de région interactive configurable par l'utilisateur

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201562208265P 2015-08-21 2015-08-21
US62/208,265 2015-08-21
US15/015,040 US10057078B2 (en) 2015-08-21 2016-02-03 User-configurable interactive region monitoring
US15/015,040 2016-02-03
KR10-2016-0104498 2016-08-17
KR1020160104498A KR20170022912A (ko) 2015-08-21 2016-08-17 사용자 설정가능 인터랙티브 영역 모니터링 장치 및 방법

Publications (1)

Publication Number Publication Date
WO2017034217A1 true WO2017034217A1 (fr) 2017-03-02

Family

ID=58100284

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2016/009097 WO2017034217A1 (fr) 2015-08-21 2016-08-18 Appareil et procédé de surveillance de région interactive configurable par l'utilisateur

Country Status (1)

Country Link
WO (1) WO2017034217A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109946765A (zh) * 2019-04-02 2019-06-28 上海电气风电集团有限公司 风电场的流场的预测方法和系统
WO2020186698A1 (fr) * 2019-03-15 2020-09-24 上海碧虎网络科技有限公司 Système et procédé de suivi mobile multimédia basé sur un positionnement de véhicule

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060071605A1 (en) * 2002-11-22 2006-04-06 Koninklijke Philips Electronics N.V. System for and method of controlling a light source and lighting arrangement
US20060182346A1 (en) * 2001-09-17 2006-08-17 National Inst. Of Adv. Industrial Science & Tech. Interface apparatus
US20060253444A1 (en) * 2005-05-05 2006-11-09 Cisco Technology, Inc. Method and system for dynamically pre-positioning content in a network based detecting or predicting user presence
US20110211110A1 (en) * 2008-03-17 2011-09-01 Antoine Doublet A method and an interactive system for controlling lighting and/or playing back images
US20140106786A1 (en) * 2010-07-30 2014-04-17 Sanyo Electric Co., Ltd. Radio apparatus transmitting or receiving a signal including predetermined information

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060182346A1 (en) * 2001-09-17 2006-08-17 National Inst. Of Adv. Industrial Science & Tech. Interface apparatus
US20060071605A1 (en) * 2002-11-22 2006-04-06 Koninklijke Philips Electronics N.V. System for and method of controlling a light source and lighting arrangement
US20060253444A1 (en) * 2005-05-05 2006-11-09 Cisco Technology, Inc. Method and system for dynamically pre-positioning content in a network based detecting or predicting user presence
US20110211110A1 (en) * 2008-03-17 2011-09-01 Antoine Doublet A method and an interactive system for controlling lighting and/or playing back images
US20140106786A1 (en) * 2010-07-30 2014-04-17 Sanyo Electric Co., Ltd. Radio apparatus transmitting or receiving a signal including predetermined information

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020186698A1 (fr) * 2019-03-15 2020-09-24 上海碧虎网络科技有限公司 Système et procédé de suivi mobile multimédia basé sur un positionnement de véhicule
US11956491B2 (en) 2019-03-15 2024-04-09 Shanghai Bihu Network Technology Co., Ltd Multimedia mobile following system based on vehicle positioning, and method
CN109946765A (zh) * 2019-04-02 2019-06-28 上海电气风电集团有限公司 风电场的流场的预测方法和系统
CN109946765B (zh) * 2019-04-02 2021-05-07 上海电气风电集团股份有限公司 风电场的流场的预测方法和系统

Similar Documents

Publication Publication Date Title
EP3338433A1 (fr) Appareil et procédé de surveillance de région interactive configurable par l'utilisateur
US10536647B2 (en) Using a display as a light source
CN108234918B (zh) 具有隐私意识的室内无人机的勘探和通讯架构方法和系统
US20170094018A1 (en) Facilitating dynamic filtering and local and/or remote processing of data based on privacy policies and/or user preferences
EP3857860B1 (fr) Système et procédé de désambiguïsation de dispositifs internet des objets
EP3090424A1 (fr) Affectation d'une interface utilisateur virtuelle à un objet physique
JP6041515B2 (ja) 画像処理装置および画像処理方法
EP3419020B1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
US20210208673A1 (en) Joint infrared and visible light visual-inertial object tracking
CN111553196A (zh) 检测隐藏摄像头的方法、系统、装置、以及存储介质
US20220253131A1 (en) Systems and methods for object tracking using fused data
WO2017034217A1 (fr) Appareil et procédé de surveillance de région interactive configurable par l'utilisateur
WO2022250300A1 (fr) Procédé et appareil électronique pour acquérir une carte de sol d'un agencement de pièce
US20230132644A1 (en) Tracking a handheld device
CN114365504A (zh) 电子设备及其控制方法
US10838741B2 (en) Information processing device, information processing method, and program
US10545716B2 (en) Information processing device, information processing method, and program
KR20240049534A (ko) 사용자 설정가능 인터랙티브 영역 모니터링 장치 및 방법
EP3417358A1 (fr) Détection, affichage et amélioration de surface d'activité d'une scène virtuelle
JP2021521690A (ja) 画像上へのセンサ及びその検出ゾーンの仮想表現の重畳
TWI837187B (zh) 用於物聯網設備消歧的系統和方法
US20230245404A1 (en) Adaptive Model Updates for Dynamic and Static Scenes
WO2023204422A1 (fr) Systèmes et procédés pour la communication vidéo multi-utilisateurs avec détection de l'engagement et fidélité réglable
EP4298610A1 (fr) Procédé et appareil électronique pour acquérir une carte de sol d'un agencement de pièce

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16839506

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016839506

Country of ref document: EP