WO2015174113A1

WO2015174113A1 - Information-processing device, system, information-processing method, and program

Info

Publication number: WO2015174113A1
Application number: PCT/JP2015/054820
Authority: WO
Inventors: 大夢瀧沢; 丈博萩原; 弘之増田; 玄大近藤
Original assignee: ソニー株式会社
Priority date: 2014-05-15
Filing date: 2015-02-20
Publication date: 2015-11-19

Abstract

[Problem] To enable a condition pertaining to sensor data to be set intuitively and easily by a user. [Solution] The present invention provides an information-processing device equipped with a processing circuit for realizing a sensor data acquisition function for acquiring sensor data supplied by one or a plurality of sensors and a condition-setting function for setting, on the basis of the sensor data acquired while a state to be detected is in practice, a condition pertaining to the sensor data needed for detecting the state to be detected.

Description

Information processing apparatus, system, information processing method, and program

The present disclosure relates to an information processing apparatus, a system, an information processing method, and a program.

Realizing some operation of the device according to the detection value of the sensor is a well-known technique. For example, in Patent Document 1, a detection value of a three-axis acceleration sensor capable of detecting a change in position in the space of an electronic device is acquired, and operation data is generated based on the detection value and a preset threshold value In the information processing apparatus, a technique is described in which the threshold value is changed according to a change in a connection state of a terminal to which an external device can be connected.

JP 2013-54411 A

For example, in the technique described in Patent Document 1, conditions such as a threshold value for the detection value of the sensor are set in advance including changes due to changes in the connection state of the terminals. On the other hand, it is conceivable that various conditions according to the user's needs can be detected by allowing the user to arbitrarily set the conditions. However, there are many cases where the detected value of the sensor is not so simple that the user can intuitively understand, and the relationship between the detected value and the state to be detected is not always intuitively understood by the user. Therefore, it is not always easy for the user to arbitrarily set the conditions.

Therefore, the present disclosure proposes a new and improved information processing apparatus, system, information processing method, and program that allow a user to set conditions regarding sensor data intuitively and easily.

According to the present disclosure, the detection target state is detected based on a sensor data acquisition function for acquiring sensor data provided by one or a plurality of sensors and the sensor data acquired during practice of the detection target state. There is provided an information processing apparatus including a processing circuit that realizes a condition setting function for setting a condition relating to the sensor data.

In addition, according to the present disclosure, one or more sensors and sensor data provided by the one or more sensors are acquired, and the detection is performed based on the sensor data acquired during the practice of the detection target state. A system is provided that includes an information processing device that sets conditions related to the sensor data for detecting a target state, and a terminal device that outputs or presents instructions or information related to the setting of the conditions provided by the information processing device. The

Further, according to the present disclosure, the processing circuit of the information processing apparatus acquires sensor data provided by one or more sensors, and based on the sensor data acquired during the practice of the detection target state, There is provided an information processing method including setting a condition relating to the sensor data for detecting the detection target state.

Further, according to the present disclosure, the detection target state is determined based on a sensor data acquisition function for acquiring sensor data provided by one or more sensors and the sensor data acquired during the practice of the detection target state. There is provided a program for causing a processing circuit to realize a condition setting function for setting conditions relating to the sensor data for detection.

As described above, according to the present disclosure, it is possible for the user to intuitively and easily set conditions regarding sensor data.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

1 is a diagram illustrating a schematic configuration of a system to which a technology according to an embodiment of the present disclosure can be applied. It is a figure which shows the 1st specific structural example of the system shown by FIG. It is a figure which shows the 2nd specific structural example of the system shown by FIG. FIG. 4 is a diagram illustrating a third specific configuration example of the system illustrated in FIG. 1. It is a figure which shows the 4th specific structural example of the system shown by FIG. FIG. 10 is a diagram illustrating a fifth specific configuration example of the system illustrated in FIG. 1. FIG. 8 is a diagram illustrating a sixth specific configuration example of the system illustrated in FIG. 1. FIG. 8 is a diagram illustrating a seventh specific configuration example of the system illustrated in FIG. 1. It is a figure which shows the 8th specific structural example of the system shown by FIG. It is a figure which shows the 1st example of UI provided in the system shown by FIG. FIG. 6 is a diagram illustrating a second example of a UI provided in the system illustrated in FIG. 1. It is a figure for demonstrating the 1st specific example of the cooperation operation | movement of the element in the system shown by FIG. It is a figure for demonstrating the 2nd specific example of the cooperation operation | movement of the element in the system shown by FIG. It is a figure for demonstrating the 3rd specific example of the cooperation operation | movement of the element in the system shown by FIG. It is a figure for demonstrating the 4th specific example of the cooperation operation | movement of the element in the system shown by FIG. It is a figure for demonstrating the 5th specific example of the cooperation operation | movement of the element in the system shown by FIG. It is a figure for demonstrating the 6th specific example of the cooperation operation | movement of the element in the system shown by FIG. 1 is a diagram illustrating a first example of a system configuration according to an embodiment of the present disclosure. FIG. It is a figure which shows the 2nd example of the system configuration | structure in one Embodiment of this indication. It is a figure which shows the 3rd example of the system configuration | structure in one Embodiment of this indication. 10 is a flowchart illustrating an example of processing according to an embodiment of the present disclosure. It is a flowchart which shows the condition production | generation process in FIG. 21 in detail. It is a flowchart which shows the condition verification process in FIG. 21 in detail. In an embodiment of this indication, it is a sequence figure showing an example of processing which acquires a time stamp with sensor data. In the embodiment of this indication, it is a sequence figure showing an example of processing which acquires an image and sound with sensor data. FIG. 3 is a diagram illustrating an example of a user interface for sensor selection according to an embodiment of the present disclosure. 3 is a diagram illustrating an example of a user interface for condition selection according to an embodiment of the present disclosure. FIG. It is a figure showing an example of a user interface for condition proposal in one embodiment of this indication. 3 is a diagram illustrating an example of a user interface for generating a composite condition according to an embodiment of the present disclosure. FIG. It is a figure which shows another example of the user interface for compound condition production | generation in one Embodiment of this indication. It is a figure which shows the example of the user interface for training data input in one Embodiment of this indication. It is a figure which shows another example of the user interface for training data input in one Embodiment of this indication. It is a figure which shows the 1st example of the user interface for training data input containing an upper limit and a lower limit in one Embodiment of this indication. It is a figure showing the 2nd example of the user interface for training data input containing the upper limit and the lower limit in one embodiment of this indication. FIG. 3 is a diagram illustrating an example of a user interface for training data input for classification according to an embodiment of the present disclosure. It is a figure showing the 1st example of a user interface for extracting training data in one embodiment of this indication. FIG. 6 is a diagram illustrating a second example of a user interface for extracting training data according to an embodiment of the present disclosure. It is a figure showing the 3rd example of a user interface for extracting training data in one embodiment of this indication. FIG. 3 is a diagram illustrating an example of a user interface for real-time feedback according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating another example of a user interface for real-time feedback in an embodiment of the present disclosure. FIG. 6 is a diagram illustrating still another example of a user interface for real-time feedback according to an embodiment of the present disclosure. 10 is a diagram for describing a first application example according to an embodiment of the present disclosure. FIG. FIG. 43 is a diagram showing an example of a user interface for creating a program in the example of FIG. 10 is a diagram for describing a second application example according to an embodiment of the present disclosure. FIG. FIG. 3 is a block diagram illustrating a hardware configuration example of an information processing apparatus according to an embodiment of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

The description will be made in the following order.
1. Examples of applicable systems 1-1. Schematic configuration 1-2. Specific configuration example 1-3. Example of user interface 1-4. Example of linkage operation Embodiments of the present disclosure 2-1. System configuration example 2-2. Example of processing flow 2-3. Example of user interface 2-4. 2. Specific application examples Hardware configuration Supplement

(1. Examples of applicable systems)
(1-1. Schematic configuration)
FIG. 1 is a diagram illustrating a schematic configuration of a system to which a technique according to an embodiment of the present disclosure can be applied. Referring to FIG. 1, the system 10 includes an element 100, a manager 200, a server 300, and a UI (User Interface) device 400. These devices are connected to each other via a network NW. The network NW can include, for example, Bluetooth (registered trademark), Wi-Fi, the Internet, and the like. In addition, although the system 10 is demonstrated as an example below, the technique which concerns on embodiment of this indication is applicable also to other various systems.

Element 100 is a device including a communication unit 110, a control unit 120, a function unit 130, and a power supply unit 140. The communication unit 110 includes a communication device that communicates with the manager 200 and / or other elements 100 via the network NW. The control unit 120 is realized by, for example, a microcontroller or a CPU (Central Processing Unit), and controls the functional unit 130. The function unit 130 includes, for example, a sensor and an actuator, and realizes a function unique to each element 100. The power supply unit 140 includes a battery, a power plug, and the like, and supplies power for operating the communication unit 110, the control unit 120, and the function unit 130. In addition, although devices other than the element 100 also have a power supply part, illustration is abbreviate | omitted.

The manager 200 is a device including a communication unit 210, a control unit 220, and a storage unit 230. The communication unit 210 may include a communication device that communicates with the element 100, the server 300, and the UI device 400 via the network NW. The control unit 220 is realized by, for example, a microcontroller or a CPU, and controls communication between the elements 100 and between the element 100 and the server 300 via the communication unit 210. The control unit 220 can include a calculation unit 221. The computation unit 221 performs computations on information received from the element 100, information transmitted to the element 100, information transmitted to the server 300, information received from the server 300, and the like. The storage unit 230 includes a memory or a storage, and stores information exchanged by communication controlled by the control unit 220, information calculated by the calculation unit 221, and the like.

The server 300 includes a communication unit 310, a control unit 320, and a storage unit 330. Communication unit 310 includes a communication device that communicates with manager 200 via network NW. The control unit 320 is realized by, for example, a microcontroller or a CPU, and can include a calculation unit 321 and a UI providing unit 323. The calculation unit 321 performs calculations on information received from the element 100 or the manager 200, information transmitted to the element 100 or the manager 200, information received from other servers, information transmitted to other servers, and the like. . The UI providing unit 323 provides a UI for the user to specify a combination of the element 100 and / or the manager 200 and to confirm various information. The UI is provided via, for example, a display and a touch panel included in the input / output unit 430 of the UI device 400. The storage unit 330 includes a memory or a storage, and stores various information related to the element 100, a program for operating the plurality of elements 100 in combination, software combined with the elements 100, information calculated by the calculation unit 321, and the like. To do.

Here, for example, identification information (ID) of the element 100 in the system 10 is stored in the storage unit 330 of the server 300. The element 100 can be added to the system 10 at any time. The storage unit 330 also stores a program for operating the plurality of elements 100 in combination. The program can be added at any time by, for example, a user, a developer, or a system administrator. Further, the storage unit 330 stores software. For example, the software implements a function by being executed in combination with the element 100. Alternatively, the function may be realized by executing software in combination. Functions realized by the software may include, for example, provision of materials such as voice and images, timers, calendars, analysis functions such as image recognition and voice recognition, information acquisition functions from external services such as weather forecasts and news, etc. . Software can be added at any time by, for example, a user, a developer, or a system administrator.

On the other hand, the UI providing unit 323 of the server 300 provides a UI via, for example, a display and a touch panel included in the input / output unit 430 of the UI device 400. Through this UI, the user selects, for example, a program for operating a plurality of elements 100 in combination from a program preinstalled in the element 100 or the manager 200, a program provided by the server 300, or the like. can do. Also, the user can specify a combination of a plurality of elements 100, a combination of the elements 100 and software, and a combination of a plurality of software via the UI. The specified combination is stored as a program in the storage unit 330 of the server 300, and the user can obtain a desired application by executing the program by the element 100 and the software.

The UI device 400 is a device that includes a communication unit 410, a control unit 420, and an input / output unit 430. Communication unit 410 may include a communication device that communicates with manager 200 and server 300 via network NW. The control unit 420 is realized by, for example, a microcontroller or a CPU, and controls the input / output unit 430 and the exchange of information through the communication unit 410. The input / output unit 430 includes, for example, a display, a speaker, a touch panel, and the like, presents various types of information to the user via the UI, and accepts an operation input from the user.

(1-2. Specific configuration example)
(First example)
FIG. 2 is a diagram showing a first specific configuration example of the system shown in FIG. Referring to FIG. 2, system 10 a includes a first tablet that functions as element 100, a second tablet that functions as element 100, manager 200, and UI device 400, and server 300.

The first and second tablets similarly include a display, a touch panel, a CPU, a sensor, and the like. In the system 10a, one of the two tablets functions as the element 100, the manager 200, and the UI device 400, and the other functions as the element 100. The roles of these tablets may be interchangeable, and the tablets that function as the manager 200 and the UI device 400 may be changed depending on the situation. In the system 10a, for example, the functions desired by the user can be realized by using various sensors of the tablet and operating the two tablets in combination.

Note that the first and second tablets in the example shown in FIG. 2 include a functional unit 130 (such as a sensor) that functions as the element 100 and a control unit 220 (such as a CPU) that functions as the manager 200. It is an example of the terminal device which has. Such a terminal device is not limited to a tablet, and may be another device such as a smartphone. Further, the number of terminal devices included in the system 10a is not limited to the illustrated example, and may be three or more, for example.

(Second example)
FIG. 3 is a diagram illustrating a second specific configuration example of the system illustrated in FIG. 1. Referring to FIG. 3, the system 10b includes elements 100a to 100g, a tablet that functions as the manager 200 and the UI device 400, and a server 300.

In the system 10b, the tablet functions as the manager 200 and the UI device 400. Similar to the first example above, the tablet may be replaced by another device such as a smartphone. In the illustrated example, the element 100 includes an acceleration sensor 100a, a camera 100b, a human sensor 100c, a button 100d, a speaker 100e, an LED (Light Emitting Diode) lamp 100f, and a microphone 100g. Each element 100 communicates with the tablet by wireless communication such as Bluetooth (registered trademark), and operates in association with the control of the manager 200 realized by the tablet.

In the following examples, some or all of the acceleration sensors 100a to 100g are illustrated as examples of the element 100, but this is intended to limit the types of the elements 100 used in each example. is not. In each example, the system 10 may include any of the acceleration sensors 100a to 100g, or any other type of element 100.

(Third example)
FIG. 4 is a diagram showing a third specific configuration example of the system shown in FIG. Referring to FIG. 4, the system 10 c includes an element 100, a manager 200, a tablet that functions as the UI device 400, and a server 300.

In the system 10c, the manager 200 exists separately from the tablet that functions as the UI device 400. The manager 200 may be realized by a dedicated device, for example, or may be realized as one of base station functions such as Wi-Fi. Similar to the first and second examples described above, the tablet functioning as the UI device 400 may be replaced by another device such as a smartphone. The manager 200 and the tablet may be able to communicate with the server 300 independently of each other. In addition, when the setting of the system 10c is changed via the UI provided by the tablet, the tablet may directly transmit the setting information to the manager 200 via Wi-Fi or the like.

In the example of FIG. 4, it is shown that not only Bluetooth (registered trademark) but also Wi-Fi is used for communication between the manager 200 and the element 100. Further, a mesh network repeater NW_m using Wi-Fi or the like is used for connection with the element 100 at a remote location. In addition to the third example shown in the figure, in other examples, various wireless communications such as Bluetooth (registered trademark) and Wi-Fi are used for communication between the element 100 and the manager 200 and / or between the elements 100. Can be used.

(Fourth example)
FIG. 5 is a diagram showing a fourth specific configuration example of the system shown in FIG. Referring to FIG. 5, the system 10 d includes an element 100 that partially functions as the manager 200, a tablet that functions as the UI device 400, and a server 300.

In the system 10d, unlike the first to third examples, at least one of the elements 100 also functions as the manager 200. In the system 10d, the elements 100 form a mesh network using Bluetooth (registered trademark) or the like. With such a configuration, in the system 10d, even when the communication with the server 300 and the UI device 400 (tablet) is temporarily disconnected, the element 100 can operate in an autonomous manner.

(Fifth example)
FIG. 6 is a diagram illustrating a fifth specific configuration example of the system illustrated in FIG. 1. Referring to FIG. 6, the system 10 e includes an element 100 that partially functions as the manager 200, a tablet that functions as the UI device 400, and a server 300. The system 10e is an example in which the manager 200 is incorporated in one of the elements 100 in the system 10c according to the third example.

(Sixth example)
FIG. 7 is a diagram illustrating a sixth specific configuration example of the system illustrated in FIG. 1. Referring to FIG. 7, the system 10 f includes an element 100,

managers

200 a and 200 b, a tablet that functions as the UI device 400, and a server 300. The system 10f is an example in which a plurality of managers 200 are arranged in the system 10c according to the third example. Each element 100 can be connected to, for example, the closer one of the

managers

200a and 200b. In addition, between the plurality of

managers

200a and 200b, the connection state of the element 100 and the program for operating the element 100 in a coordinated manner are synchronized as necessary or periodically.

(Seventh example)
FIG. 8 is a diagram illustrating a seventh specific configuration example of the system illustrated in FIG. 1. Referring to FIG. 8, the system 10g includes an element 100, a manager 200a, a tablet that functions as the manager 200b and the UI device 400, and a server 300. The system 10g is an example in which the function of the manager 200b is integrated into the tablet in the system 10f according to the sixth example. Each element 100 can be connected to, for example, a manager 200a or a tablet closer to the position. Further, between the manager 200a and the tablet, the connection state of the element 100 and the program for operating the element 100 in a coordinated manner are synchronized as necessary or periodically.

(Eighth example)
FIG. 9 is a diagram illustrating an eighth specific configuration example of the system illustrated in FIG. 1. Referring to FIG. 9, the system 10 h includes an element 100, a tablet that functions as the UI device 400, and a server 300 that also functions as the manager 200. The system 10h is an example in which the function of the manager 200 is taken into the server 300 in the system 10b according to the second example. Each element 100 communicates directly with the server 300 via, for example, a mobile communication network.

Heretofore, some specific configuration examples of the system 10 shown in FIG. 1 have been described. The specific configuration example of the system 10 is not limited to the examples described above, and may include various modifications that are clearly understood by those skilled in the art based on these examples.

(1-3. Example of user interface)
FIG. 10 is a diagram showing a first example of a UI provided in the system shown in FIG. Referring to FIG. 10, a screen 4100 displayed on the display of the terminal device functioning as the UI device 400 includes a user profile 4101, a program list tab 4103, and an element list tab 4105. In the illustrated example, since the program list tab 4103 is selected, the program list 4107 is displayed. The program list 4107 includes a program icon 4109, a used element icon 4111, and a description 4113. The screen 4100 can be displayed, for example, as a portal screen when the user uses a service provided by the system 10.

FIG. 11 is a diagram showing a second example of a UI provided in the system shown in FIG. Referring to FIG. 11, a screen 4200 displayed on the display of the terminal device that functions as the UI device 400 includes a toolbar 4201, a canvas 4203, and a tray 4205. The screen 4200 is used, for example, for editing a program for operating the element 100 in association with the system 10. Function buttons such as “Save” and “Redo” are arranged on the toolbar 4201. An element icon 4207, a detailed information icon 4209, and a link 4211 can be arranged or drawn on the canvas 4203. With these icons and links, it is possible to set and confirm the elements 100 used for the linkage operation, the processing executed by each element 100, and the relationship between the elements 100. Further, an element property box 4213 is arranged on the canvas 4203, and for example, the attribute and state of the element 100 arranged as the element icon 4207 are displayed. In the tray 4205, the elements 100, software, and the like that can be incorporated in the program by being arranged on the canvas 4203 are displayed as icons.

Note that the UI described with reference to FIGS. 10 and 11 is merely an example, and various forms of UI can be provided in the present embodiment. For example, in the screen for editing a program described with reference to FIG. 11, various UIs used in the screen for visual programming can be applied.

(1-4. Example of linkage operation)
Hereinafter, a specific example of the element linking operation in the system shown in FIG. 1 will be further described. In the following description, in order to facilitate understanding, reference is made to a diagram visually representing a program for operating the element 100 in a linked manner. Note that these drawings are not necessarily related to the UI provided by the UI device 400 (for example, the UI illustrated as the screen 4200 in FIG. 11). The software described below may be provided as an element (software element) similar to the element 100 (hardware element), or may be provided as part of the function or operation of the element 100.

(First specific example)
FIG. 12 is a diagram for explaining a first specific example of element linking operation in the system shown in FIG. 1. Referring to FIG. 12, in this example, among the elements 100, an acceleration sensor 100a and a camera 100b are used. For example, the acceleration sensor 100a is attached to the door of the refrigerator, and the camera 100b is attached to a position where the interior of the refrigerator can be projected. In the illustrated example, the link 601 from the acceleration sensor 100a to the camera 100b means that “the camera 100b performs shooting based on the detection value of the acceleration sensor 100a”. Further, in the illustrated example, “upload a captured image to the server” is designated as the operation of the camera 100b.

Further, in the illustrated example, software 603a for detecting that the door is opened based on the acceleration is used. The software 603a is executed by the arithmetic unit 221 of the manager 200, for example, and determines that the refrigerator door is opened based on the analysis result of the detection value of the acceleration sensor 100a. By introducing the software 603a, the link 601 performs a process of “when the determination is made that the refrigerator door is opened based on the detection value of the acceleration sensor 100a, the camera 100b performs photographing”.

With the above program settings, in this example, an image captured by the camera 100b when the refrigerator door is opened is uploaded to the server. The user can grasp the change in the contents of the refrigerator and the latest inventory status by browsing the uploaded image.

Here, if it is desired to detect that the refrigerator door is opened, a sensor dedicated to open / close detection using magnetism or the like can also be used. However, in this example, by introducing the software 603a, it is possible to detect that the door is opened by using the acceleration sensor 100a. Therefore, in order to detect that the door of the refrigerator is opened, for example, the acceleration sensor 100a that has been used for another purpose can be used instead of the dedicated sensor. In addition, when it is no longer necessary to detect that the refrigerator door is opened, the acceleration sensor 100a can be diverted to other applications. Thus, in this embodiment, the same element 100 can be utilized for various uses by changing the combination of software and other elements 100.

(Second specific example)
FIG. 13 is a diagram for explaining a second specific example of element linking operation in the system shown in FIG. 1. Referring to FIG. 13, in this example, among the elements 100, an acceleration sensor 100a and a button 100d are used. For example, the acceleration sensor 100a is attached to a toilet door, and the button 100d is attached to a toilet. In the illustrated example, software 603a for detecting that the door is opened based on acceleration and software 603b for recording the provided data are further used. A link 601 from the acceleration sensor 100a to the software 603b through the software 603a means that the software 603b records that it is determined that the door is opened based on the detection value of the acceleration sensor 100a. The link 601 from the button 100d to the software 603b means that the software 603b records that a signal is output from the button 100d.

With the above program settings, in this example, the time when the user presses the button 100d in the bathroom and the time when the toilet door is opened are recorded as a log. For example, if the user decides to press the button 100d when washing his face in the bathroom after getting up, the time when the button 100d was pressed can be recorded as the rising time. From such a log, for example, it is possible to refer to changes in daily wake-up time and the time of going to the toilet in time series, which can be used to improve the user's life rhythm.

(Third specific example)
FIG. 14 is a diagram for explaining a third specific example of the element linking operation in the system shown in FIG. Referring to FIG. 14, in this example, among the elements 100, an acceleration sensor 100a and a human sensor 100c are used. For example, the acceleration sensor 100a is attached to a chair, and the human sensor 100c is attached to a desk in front of the chair. In the illustrated example, software 603c for recording the time that the user has been sitting on the chair based on the detection results of the acceleration sensor 100a and the human sensor 100c is further used. A link 601 from the acceleration sensor 100a to the software 603c means “providing the detection value of the acceleration sensor 100a to the software 603c”. Further, the link 601 from the human sensor 100c to the software 603c means “providing the detection value of the human sensor 100c to the software 603c”.

According to the program settings as described above, in this example, the time that the user has been sitting on the chair is recorded. Based on this record, the user can grasp the sitting time at the workplace and take a break if the sitting time is too long. Alternatively, software that outputs an alert from a smartphone or the like based on the detection result of the software 603c is further combined, and when the user is sitting continuously over a predetermined time, an alert is output from the smartphone or the like to take a break. You may be encouraged to take it.

(Fourth specific example)
FIG. 15 is a diagram for explaining a fourth specific example of the element linking operation in the system shown in FIG. 1. Referring to FIG. 15, in this example, among the elements 100, an acceleration sensor 100a, a speaker 100e, and an LED lamp 100f are used. For example, the acceleration sensor 100a, the speaker 100e, and the LED lamp 100f are each attached to an appropriate part of the body of the user (for example, a child). A plurality of acceleration sensors 100a may be used. In the illustrated example, software 603d that reproduces sound effects according to the acceleration pattern and software 603e that causes the lamp to blink in a predetermined pattern according to the acceleration pattern are also shown. The acceleration pattern handled by these software 603 may be a single acceleration waveform pattern or a combination of a plurality of acceleration waveforms. A link 601 is set from the acceleration sensor 100a to each of the

software

603d and 603e. These links 601 mean “provide the detection values of the acceleration sensor 100a to the

software

603d and 603e, respectively”. Further, a link 601 is set from the software 603d to the speaker 100e, and from the software 603e to the LED lamp 100f. These links 601 mean “the speaker 100e outputs sound according to the audio signal provided by the software 603d” and “the LED lamp 100f emits light according to the signal provided by the software 603e”.

According to the program settings as described above, in this example, when a user (for example, a child) moves his / her limb in a specific pattern, a sound effect is output from the speaker 100e or the LED lamp 100f emits light. As a result, for example, when a child plays as a super hero, it can have an exciting experience by adding sound effects and light.

(Fifth specific example)
FIG. 16 is a diagram for explaining a fifth specific example of element linking operation in the system shown in FIG. 1. Referring to FIG. 16, in this example, among the elements 100, an acceleration sensor 100a, a human sensor 100c, a camera 100b, and an LED lamp 100f are used. For example, the acceleration sensor 100a, the human sensor 100c, and the camera 100b are attached to a bird table, and the LED lamp 100f is attached to a house. In the illustrated example, software 603f for determining that a bird has stopped on the feeding base based on the detection results of the acceleration sensor 100a and the human sensor 100c is further illustrated. A link 601 is set from the acceleration sensor 100a and the human sensor 100c toward the software 603f. The link 601 from the acceleration sensor 100a means “provide the detection value of the acceleration sensor 100a to the software 603f”. Further, the link 601 from the human sensor 100c means “providing the detection value of the human sensor 100c to the software 603f”. In the software 603f, for example, a condition that is established by a combination of sensor data provided by each of the acceleration sensor 100a and the human sensor 100c is provided. Furthermore, a link 601 is set from the software 603f to the camera 100b and the LED lamp 100f. The link 601 to the camera 100b means that “the camera 100b performs shooting based on the output of the software 603f”. The link to the LED lamp 100f means that the LED lamp 100f is caused to emit light based on the output of the software 603f.

According to the program settings as described above, in this example, when a bird stops on the food stand, a still image or a moving image is automatically shot by the camera 100b, and it is possible to know what kind of bird has come. Moreover, when the bird stops on the food stand, the LED lamp 100f in the house emits light, so that it is possible to observe the bird actually coming out of the garden.

(Sixth specific example)
FIG. 17 is a diagram for explaining a sixth specific example of element linking operation in the system shown in FIG. 1. Referring to FIG. 17, in this example, among the elements 100, a button 100d and a speaker 100e are used. For example, the speaker 100e is disposed in a bedroom and the button 100d is disposed in a washroom. In the illustrated example, the alarm alarm software 603g is further shown. A link 601 is set to the speaker 100e from the software 603g. This link 601 means that “the speaker 100e outputs audio according to the audio signal output when the software 603 reaches the set time”. A link 601 is set in the software 603g from the button 100d. This link 601 means “when the button 100d is pressed, the output of the audio signal by the software 603g is stopped”.

By setting the program as described above, in this example, the alarm sound output from the speaker 100e disposed in the bedroom does not stop until the toilet button 100d is pressed. As a result, it is difficult to cause a situation such as stopping the alarm while sleeping, and it is possible to wake up reliably at the scheduled time.

(2. Embodiment of the present disclosure)
The embodiment of the present disclosure relates to a process for a user to set a condition when the system performs a predetermined operation when a condition regarding sensor data provided by a sensor in the system is satisfied. In addition, although the system 10 demonstrated with reference to FIG. 1 etc. is demonstrated below as an example, the system which concerns on embodiment of this indication is not restricted to this.

Sensors in the system include, for example, push buttons, acceleration sensors, image sensors, microphones, color sensors, illuminance sensors, temperature sensors, humidity sensors, human sensors, wireless signal strength detectors, distance sensors, pressure sensors, strain gauges, A gas sensor, an ultraviolet sensor, a gyro sensor, a magnetic sensor, a speed sensor, a tilt sensor, a touch sensor, and the like can be included. The user selects or creates a condition established by sensor data provided by any of these sensors or a combination of sensor data provided by a plurality of types of sensors. Also, the user may be able to verify or adjust the selected or created conditions.

(2-1. System configuration example)
(First example)
FIG. 18 is a diagram illustrating a first example of a system configuration according to an embodiment of the present disclosure. Referring to FIG. 18, the system 10-1 includes an element 100-1, a manager 200-1, and a UI device 400-1.

The element 100-1 includes a communication unit 110, a sensor control unit 120a, an actuator control unit 120b, a sensor 130a, and an actuator 130b. As described above with reference to FIG. 1, the communication unit 110 includes a communication device that communicates with the manager 200-1 and other elements 100-1 via a network. The sensor control unit 120a and the actuator control unit 120b are included in the control unit 120 described with reference to FIG. The control unit 120 is realized by a microcontroller, a CPU, or the like. The sensor control unit 120a controls the sensor 130a, and the actuator control unit 120b controls the actuator 130b. The sensor 130a and the actuator 130b are included in the functional unit 130 described with reference to FIG. The sensor 130a can include, for example, the various sensors described above. The sensor 130a detects a change in the external environment and a user interaction with the element 100-1. The actuator 130b includes, for example, an LED lamp, a motor, a speaker, and the like, and gives some action to the outside world.

The manager 200-1 includes a communication unit 210, an integrated control unit 222, a condition adjustment / generation unit 223, a condition management unit 224, a condition determination unit 225, and a condition handling unit 226. As described above with reference to FIG. 1, the communication unit 210 includes a communication device that communicates with the element 100-1, the UI device 400-1, and the external system 500 via a network. The integrated control unit 222, the condition adjustment / generation unit 223, the condition management unit 224, the condition determination unit 225, and the condition handling unit 226 are included in the control unit 220 described with reference to FIG. The control unit 220 is realized by a microcontroller, a CPU, or the like. Hereinafter, these functional configurations will be further described.

The integrated control unit 222 controls the functional units related to the conditions in the manager 200-1, the element 100, the UI device 400, and the like, and integrates the entire system 10 by executing information input / output and information processing. Control. For example, the integrated control unit 222 acquires sensor data provided by the sensor 130a included in the one or more elements 100-1.

The condition adjustment / generation unit 223 provides a function for the user to adjust the condition or generate a new condition. For example, the condition adjustment / generation unit 223 sets conditions relating to sensor data for detecting the detection target state based on the sensor data acquired during the practice of the detection target state by the user. At this time, the condition adjustment / generation unit 223 instructs the user of the timing of the detection target state practice via the UI device 400, or uses machine data that is acquired during the detection target state practice as training data. Or carry out.

In addition, for example, the condition adjustment / generation unit 223 is a sensor acquired during the practice of the detection target state for a newly generated condition, an existing single condition, a condition that combines a plurality of existing conditions, and the like. Perform verification based on the data. The verification is performed, for example, by determining whether or not the condition is satisfied based on the sensor data acquired during the practice of the detection target state. Here, the condition adjustment / generation unit 223 can change the setting of the condition based on the user operation during the verification of the condition. Therefore, for example, if a condition that should be satisfied is not satisfied during verification, or if it is determined that the reliability is low even if it is satisfied, the user can change the setting of the condition in real time. it can.

The condition management unit 224 manages the conditions that are already generated and stored in the system 10 and the newly generated condition candidates. Conditions and condition candidates are stored in, for example, the storage unit 230 (not shown in FIG. 18) of the manager 200-1. Conditions and candidate conditions are stored by associating identification information (ID) with a mathematical model for determining whether or not the condition is satisfied by sensor data provided by the sensor 130a, for example. The condition management unit 224 stores newly generated conditions (including conditions obtained by combining a plurality of existing conditions) in the storage unit 230 and the like, and reads one or more existing conditions from the storage unit 230.

The condition determination unit 225 determines whether the condition is satisfied based on the sensor data provided by the sensor 130a. The determination is executed by referring to the condition saved in the storage unit 230 of the manager 200-1 by the condition management unit 224 or by copying it into the condition determination unit 225, for example. The condition determining unit 225 determines whether or not the condition is satisfied based on the sensor data, for example, according to a mathematical model defined in the condition.

The condition handling unit 226 realizes a predetermined operation in the system 10 when the condition is satisfied. More specifically, when the condition determination unit 225 determines that the condition is satisfied, the condition handling unit 226 realizes the operation of the system 10 associated with the condition. The operation of the system 10 is performed, for example, by causing the actuator 130b to perform a predetermined operation via the actuator control unit 120b of the element 100-1, requesting a predetermined function from the external system 500, or the control unit 420 of the UI device 400. This is realized by outputting predetermined information from the output unit 430b via the.

The UI device 400-1 includes a communication unit 410, a control unit 420, an input unit 430a, and an output unit 430b. As described above with reference to FIG. 1, the communication unit 410 includes a communication device that communicates with the manager 200-1 and the like via a network. The control unit 420 is realized by a microcontroller, a CPU, or the like, and controls the input unit 430a and the output unit 430b. The input unit 430a and the output unit 430b are included in the input / output unit 430 described with reference to FIG. The input unit 430a is realized by a keyboard, a mouse, a touch panel, and the like, and receives an operation input from a user. The output unit 430b is realized by a display, a speaker, and the like, and transmits the output from the system 10 to the user.

In this embodiment, the UI device 400-1 functions as a terminal device that outputs or presents instructions or information related to setting conditions provided by the condition adjustment / generation unit 223 of the manager 200. In this way, the instruction or information regarding the setting of the condition may be output or presented to the user via a device different from the element 100-1 (the device including the sensor 130a), or the element 100 as in an example described later. -1 may be output or presented via an actuator 130b included in the -1.

The external system 500 is a system that provides a predetermined function without depending on components such as the element 100-1, the manager 200-1, and the UI device 400-1. The external system 500 can be, for example, a system that exists outside the system 10 (including the server 300 not shown). The external system 500 is realized by, for example, one or a plurality of information processing devices that constitute a server. The external system 500 includes a communication unit 510 including a communication device that communicates with the manager 200 (or the server 300) via a network, and a function unit 520 that provides a predetermined function via the communication unit 510. The functional unit 520 can include, for example, a microcontroller or CPU that provides an arithmetic function, a memory or storage that provides a data storage function, and the like.

In the first example of the system configuration as described above, the processing related to the conditions is concentrated in the manager 200-1. Accordingly, the sensor data provided by the sensor 130a included in one or more elements 100-1 is transmitted to the manager 200-1, and in the manager 200-1, the integrated control unit 222 transmits the sensor data to the element 100- 1 is specified, and the condition determination unit 225 determines whether the condition is satisfied based on the sensor data. When the condition is satisfied, the condition handling unit 226 sends a control command for causing the system 10 to perform a predetermined operation to the element 100-1, the UI device 400-1, or the external system 500 via the communication unit 210. Send.

On the other hand, in the UI device 400-1, when the user's operation input for changing the setting of the condition is accepted by the input unit 430a, information indicating the operation input is received via the communication unit 410 of the UI device 400-1. Is transmitted to the manager 200-1. In the manager 200-1, the condition adjustment / generation unit 223 changes the setting of the condition according to the operation input, and the condition management unit 224 updates the information related to the condition and the condition with the changed setting.

In addition, a sensor provided by a sensor 130a included in one or more elements 100-1 when adjusting conditions by real-time feedback for candidate conditions presented by the system 10, or generated or selected by a user The data is transmitted to the manager 200-1, and the condition determination unit 225 determines whether the condition is satisfied based on the sensor data. Further, the determination result as to whether or not the condition is satisfied is transmitted to the UI device 400-1 via the communication unit 210, and is transmitted to the user from the output unit 430b according to the control of the control unit 420. The user can perform readjustment or reselection of conditions while referring to feedback on whether or not the conditions are satisfied based on the transmitted information, that is, sensor data when the detection target state is put into practice.

(Second example)
FIG. 19 is a diagram illustrating a second example of a system configuration according to an embodiment of the present disclosure. Referring to FIG. 19, the system 10-2 includes an element 100-2, a manager 200-2, and a UI device 400-2.

Element 100-2 includes a communication unit 110, a sensor control unit 120a, an actuator control unit 120b, a condition determination unit 121, a sensor 130a, and an actuator 130b. The condition determination unit 121 is also included in the control unit 120 described above with reference to FIG. 1 in the same manner as the sensor control unit 120a and the actuator control unit 120b. The control unit 120 is realized by a microcontroller, a CPU, or the like. The condition determining unit 121 determines whether or not a condition set by, for example, a detection value from the sensor 130a is satisfied, similarly to the condition determining unit 225 of the manager 200-1 in the first example. The conditions used for the determination are stored, for example, in the storage unit 230 of the manager 200-1. In this case, the condition determination unit 121 executes the determination by referring to the condition via the communication unit 210 or by copying the condition into the condition determination unit 121. Alternatively, the conditions used for the determination may be stored in a storage unit (not shown) of the element 100-2. The condition determining unit 121 determines whether or not the condition is satisfied based on the sensor data provided by the sensor 130a, for example, according to a mathematical model defined in the condition. Note that the communication unit 110, the sensor control unit 120a, the actuator control unit 120b, the sensor 130a, and the actuator 130b are the same as in the first example described above, and thus redundant description is omitted.

The manager 200-2 includes a communication unit 210, an integrated control unit 222, a condition adjustment / generation unit 223, a condition management unit 224, and a condition handling unit 226. In the second example, the condition determination unit 121 is included in the element 100-2, while the condition determination unit 121 is not included in the manager 200-2. The integrated control unit 222 transmits the conditions stored in the storage unit 230 of the manager 200-2 by the condition management unit 224 to the element 100-2 via the communication unit 210. The transmission may be executed after the sensor data is acquired, or may be executed in advance prior to acquisition of the sensor data. For example, the integrated control unit 222 may distribute condition information in advance to the condition determination unit 121 of the element 100-2 having the sensor 130a related to the condition. The condition handling unit 226 notifies the determination result of whether or not the condition is satisfied from the condition determination unit 121 of the element 100-2 via the communication unit 210. The communication unit 210, the integrated control unit 222, the condition adjustment / generation unit 223, the condition management unit 224, and the condition handling unit 226 are the same as those in the first example except for the points described above. To do.

Note that the configurations of the UI device 400-2 and the external system 500 are the same as those of the UI device 400-1 and the external system 500 in the first example, and therefore, a duplicate description thereof will be omitted.

In the second example of the system configuration as described above, the condition determination unit 121 is included in the element 100-2 having the sensor 130a that acquires the detection value. Therefore, the detection value of the sensor 130a included in one or more elements 100-2 is used to determine whether or not a condition is satisfied inside the element 100-2. The condition determination unit 121 transmits information indicating the established condition to the manager 200-2 via the communication unit 110. In the manager 200-2, the integrated control unit 222 identifies the element 100-2 to which the information has been transmitted, and the condition handling unit 226 transmits the element 100-2, the UI device 400-2, or the external device via the communication unit 210. A control command for causing the system 10 to realize a predetermined operation is transmitted to the system 500.

On the other hand, in the UI device 400-2, when the user's operation input for changing the setting of the condition is received by the input unit 430a, information indicating the operation input is received via the communication unit 410 of the UI device 400-2. It is transmitted to the manager 200-2. In the manager 200-2, the condition adjustment / generation unit 223 changes the setting of the condition according to the operation input, and the condition management unit 224 updates the condition with the changed setting. Further, the integrated control unit 222 may notify the element 100-2 of the update of the condition, and in the element 100-2, the condition determination unit 121 may change the setting of the condition held internally.

In addition, a sensor provided by the sensor 130a included in the one or more elements 100-2 when the condition is adjusted by real-time feedback for the candidate condition presented by the system 10 or generated or selected by the user. Based on the data, the condition determination unit 121 determines whether or not the condition is satisfied. The determination result as to whether or not the condition is satisfied is transmitted to the manager 200-2 via, for example, the communication unit 110, further transferred to the UI device 400-2 from the communication unit 210, and output unit 430b according to the control of the control unit 420. To the user. The user can perform readjustment or reselection of conditions while referring to feedback on whether or not the conditions are satisfied based on the transmitted information, that is, sensor data provided by the sensors.

(Third example)
FIG. 20 is a diagram illustrating a third example of a system configuration according to an embodiment of the present disclosure. Referring to FIG. 20, the system 10-3 includes an element 100-3, a manager 200-3, and a UI device 400-3.

In the third example, the element 100-3 can have the same configuration as the element 100-2 in the second example. Further, the manager 200-3 can have the same configuration as the manager 200-1 in the first example. That is, in the third example, in addition to at least a part of the element 100-3 including the condition determining unit 121, the manager 200-3 also includes the condition determining unit 225. Note that the configurations of the element 100-3 and the manager 200-3 are the same as those in the first example or the second example described above, and a duplicate description is omitted.

In the third example, for example, the type of detection value of the sensor 130a, the size of the memory or storage of each element 100-3, the bandwidth available for communication between each element 100-3 and the manager 200, etc. Accordingly, it is determined whether to provide the condition determining unit 121 in the element 100-3 or to use the condition determining unit 225 of the manager 200-3. When the condition determination unit 225 of the manager 200-3 is used for the detection value of a certain sensor 130a, the element 100-3 including the sensor 130a may not include the condition determination unit 121.

Note that the configurations of the UI device 400-3 and the external system 500 are the same as those of the UI devices 400-1 and 400-2 and the external system 500 in the first example or the second example described above, and therefore, there is no overlap in these. The explanations made are omitted.

(2-2. Example of processing flow)
FIG. 21 is a flowchart illustrating an example of processing according to an embodiment of the present disclosure. In the following description, the processing subject in the first example of the system configuration will be described as an example, but the same processing can be performed for the second and third examples of the system configuration. In the following description, unless otherwise specified, the system 10-1, the element 100-1, the manager 200-1, and the UI device 400-1 are simply referred to as the system 10, the element 100, the manager 200, and the UI device, respectively. 400.

First, the sensor used for the condition is selected (S101). The sensor is selected by a user operation, for example. In this case, for example, the input unit 430a of the UI device 400 acquires a user operation for selecting a sensor. Information indicating the user operation is transmitted to the manager 200 via the control unit 420 and the communication unit 410. Prior to this, the output unit 430b of the UI device 400 presents a UI screen showing selectable sensors to the user based on the information transmitted from the integrated control unit 222 of the manager 200 via the communication unit 210. Also good. At this point, the integrated control unit 222 may narrow down the selectable sensors from all available sensors, for example, according to the type of condition to be set. Alternatively, the integrated control unit 222 may automatically perform sensor selection itself according to the type of condition to be set.

In the manager 200, the integrated control unit 222 identifies the selected sensor from the sensors 130 a that can be used in the system 10 based on the information indicating the user operation received from the UI device 400 via the communication unit 210. To do. Further, when setting the condition using the sensor data provided by the selected sensor, the integrated control unit 222 displays a UI screen for inquiring whether or not to automatically search for a condition candidate and the output unit 430b of the UI device 400. To the user (S103). Note that the condition candidates here may include conditions that have already been generated and saved in the storage unit 230 of the manager 200 by the condition management unit 224.

In S103 described above, when a user operation indicating that a condition candidate is automatically searched is acquired via the input unit 430a of the UI device 400 (YES), the system 10 enters a detection target state standby state. More specifically, the integrated control unit 222 of the manager 200 notifies the UI device 400 that recording of the detection target state is started via the communication unit 210, and further requests to practice the detection target state. To do. Upon receiving these, the output unit 430b of the UI device 400 notifies the user that recording of the detection target state is started, and then presents a UI screen prompting the user to practice the detection target state. In response to this, the user practices the detection target state (S105).

The practice of the detection target state in S105 described above means that, for example, the user wants to practice a state that the user desires to be detected under the conditions that the user wants to set after operating the sensor 130a. More specifically, for example, when the user wants to detect that the door of the refrigerator is opened using the detection value of the acceleration sensor 100a, in S105, after the acceleration sensor 100a is attached to the refrigerator door, Open the refrigerator door. While the detection target state is put into practice, sensing by one or a plurality of sensors 130a is performed.

Here, the detection target state may be practiced after notifying the user to start recording as described above or requesting the user to practice the detection target state. It may be automatically started without outputting such a notification or request. Moreover, the practice of the detection target state may be performed only once, for example, may be automatically ended after being repeatedly performed, or may be repeatedly performed until an end instruction by the user. When the practice of the detection target state is automatically terminated, the number of implementations reaches a preset number, or the elapsed time after the practice is started can be set as the termination condition. Furthermore, when practicing the detection target state, the time stamp recorded at any timing by the user or the user practicing the detection target state in order to make it easy to specify the section of the detection value indicating the detection target state. An image or sound in which the state of being recorded is recorded may be additionally acquired.

When the practice of the detection target state in S105 ends, the condition determination unit 225 determines whether there is a condition candidate that is satisfied by the detection value of the sensor 130a acquired in the practiced detection target state (S107). As described above, the condition candidates may include conditions that are already generated and stored in the storage unit 230 of the manager 200 by the condition management unit 224. Here, when there is no condition candidate (NO), the integrated control unit 222 may inquire through the communication unit 210 whether or not to retry the practice of the detection target state to the UI device 400. Receiving this, the output unit 430b of the UI device 400 inquires of the user whether or not to retry the practice of the detection target state (S109). When the user selects to retry, the detection target state practice in S105 is re-executed.

On the other hand, if there is a condition candidate in the determination of S107 (YES), the integrated control unit 222 determines to apply the condition candidate (S111). At this time, when there are a plurality of condition candidates, a UI screen showing the condition candidates that can be selected via the output unit 430b of the UI device 400 is presented to the user, and the condition candidates are selected via the input unit 430a. The user operation may be acquired. For the applied condition candidate, a condition verification process (S115) is performed as necessary. Details of the condition verification process will be described later.

On the other hand, when the user operation indicating that the condition candidate is not automatically searched is acquired via the input unit 430a of the UI device 400 in S103 described above (NO), and the user does not retry in S109 described above. Is selected, a condition generation process (S113) is performed. For the conditions generated by the condition generation process, the condition verification process (S115) is performed as necessary.

(Condition generation process)
FIG. 22 is a flowchart showing the condition generation process in FIG. 21 in more detail. Referring to FIG. 22, in the condition generation process, first, the user is inquired whether to use the composite condition via the output unit 430b of the UI device 400 (S121). The compound condition is a condition candidate, for example, a condition generated by combining a plurality of conditions already generated and stored in the storage unit 230 of the manager 200 by the condition management unit 224. For example, in the case of a touch sensor, a combined condition of “double press including long press” can be generated by combining the condition of “long press” and the condition of “double press”.

In S121 described above, when a user operation indicating that the composite condition is used is acquired via the input unit 430a of the UI device 400 (YES), the integrated control unit 222 passes through the output unit 430b of the UI device 400. Then, a UI screen for selecting a condition that becomes a component of the composite condition is presented, and the input unit 430a acquires a user operation for selecting the condition of the component (S123). When the selection of the component conditions is completed, the condition management unit 224 reads the selected plurality of existing conditions, and the condition adjustment / generation unit 223 generates a composite condition using these existing conditions (S125).

On the other hand, in S121, when a user operation indicating that the composite condition is not used is acquired via the input unit 430a of the UI device 400 (NO), the integrated control unit 222 generates a condition using machine learning. Then, it is determined whether or not divertable data exists (S127). The divertable data is provided, for example, in a photograph taken by the camera and stored in the storage, or in the external system 500 if it is desired to generate a condition including identifying the user's face using the camera 100b. It may be an image including the user's face that is already available in the system 10, such as a photo posted to a service such as social media. In S127, when such data exists (YES), training data is diverted (S129). More specifically, the integrated control unit 222 acquires data from the external system 500 and provides the condition adjustment / generation unit 223 with the data.

On the other hand, in S127, when there is no data that can be used for learning (NO), the user practices the detection target state (S131) in the same manner as the process of S105 described above with reference to FIG. Here, the detection target state may be practiced after notifying the user to start recording or requesting the user to practice the detection target state. It may be started automatically without outputting a request. Moreover, the practice of the detection target state may be performed only once, for example, may be automatically ended after being repeatedly performed, or may be repeatedly performed until an end instruction by the user. In practicing the detection target state, the time stamp recorded at any timing by the user or the user practicing the detection target state in order to make it easy to specify the section of the detection value indicating the detection target state. An image or sound in which the state of being recorded is recorded may be additionally acquired. Data including the detection value of the sensor 130a acquired in the practice of the detection target state in S127 is recorded as training data by the integrated control unit 222 (S133).

Further, the condition adjustment / generation unit 223 performs learning using the training data acquired in the above S129 or S133 (S135). The learning here is so-called supervised machine learning, and various known machine learning techniques can be applied. Based on the learning result, the condition adjustment / generation unit 223 generates a new condition (S137). The conditions generated in S125 or S137 are stored in the storage unit 230 of the manager 200 by the condition management unit 224 through the condition verification process (S115) shown in FIG.

(Condition verification process)
FIG. 23 is a flowchart showing the condition verification process in FIG. 21 in more detail. Referring to FIG. 23, in the condition verification process, first, the user is inquired whether to verify the condition via the output unit 430b of the UI device 400 (S141). Here, when a user operation indicating that the condition is not verified is acquired via the input unit 430a (NO), the condition verification process is not performed. That is, the condition candidates applied in S111 in FIG. 21 and the new conditions generated in S113 are determined as conditions as they are.

On the other hand, when a user operation indicating that the condition is to be verified is acquired in S141 described above (YES), the condition is verified (S143). The verification of the condition can be performed by real-time feedback from the system 10 to the user, for example. In this case, for example, after the sensor 130a related to the condition performs sensing, the user practices the detection target state. At this time, the detection value of the sensor 130a is transmitted to the manager 200, and the condition determination unit 225 determines whether or not the condition is satisfied based on the detection value. Further, the determination result as to whether or not the condition is satisfied is transmitted to the UI device 400-1 via the communication unit 210, and is transmitted to the user from the output unit 430b according to the control of the control unit 420.

In response to the verification result of the condition output in S143, the UI device 400 presents the user with a UI screen asking whether or not the condition itself is to be changed by the output unit 430b (S145). Here, when the user operation indicating that the condition itself is changed is acquired by the input unit 430a (YES), the process returns to S103 illustrated in FIG. That is, when it is selected to change the condition itself, the process is re-executed from an inquiry as to whether or not to automatically search for a condition candidate.

On the other hand, if the condition itself is not changed in S145 (NO), the UI device 400 further presents a UI screen asking whether to adjust the condition to the user by the output unit 430b (S147). Here, when a user operation indicating that the condition is adjusted is acquired by the input unit 430a (YES), the condition is adjusted (S149). More specifically, for example, in the UI device 400, a UI screen for adjusting a condition by changing a setting value such as a threshold value or a range of a detection value is presented to the user, and the setting value is set by the input unit 430a. The user operation to change is acquired. When the condition adjustment is completed, the verification of the condition in S143 is performed again.

On the other hand, when a user operation indicating that the condition adjustment is not necessary is acquired in S147 (NO), the condition verification process ends, and the condition candidate applied in S111 in FIG. 21 or the new candidate generated in S113 is displayed. Conditions or conditions obtained by adjusting the conditions in S149 to these condition candidates or conditions are determined.

(Example of acquiring time stamp with sensor data)
FIG. 24 is a sequence diagram illustrating an example of processing for obtaining a time stamp together with sensor data in the embodiment of the present disclosure. As described above, in the practice of the detection target state in S105 of FIG. 21 or S131 of FIG. 22, the time stamp recorded at any timing by the user during the practice of the detection target state by the condition adjustment / generation unit 223. It may additionally be acquired. In the figure, processing related to the practice of the detection target state when the time stamp is acquired is shown.

In the illustrated example, first, in the manager 200, the integrated control unit 222 determines the start of recording (S201). For example, the recording may be started in response to an operation input from the user via the input unit 430a of the UI device 400, or may be automatically started as in the illustrated example. After the start of recording, the sensor 130a performs sensing in the element 100 (S203), and the sensor control unit 120a transmits the detection value to the manager 200 via the communication unit 110. In the manager 200, the integrated control unit 222 stores the sensor detection value received via the communication unit 210 (S205).

On the other hand, after the start of recording, the UI device 400 can perform a time stamp generation operation by the user via the input unit 430a. The time stamp generation operation is executed at an arbitrary timing (S207), and a command instructing generation of a time stamp is transmitted from the UI device 400 to the manager 200. In the manager 200, the integrated control unit 222 records a time stamp based on the command received via the communication unit 210 (S209). Note that the time series of time stamps recorded and the time series of sensor detection values are synchronized. Thereafter, the sensor 130a may further perform sensing (S211), and the sensor detection value may be stored in the manager 200 (S213).

Note that the context of sensing and time stamp recording in the illustrated example is an example, and the time stamp can be generated at an arbitrary timing as described above. For example, time stamp generation according to S207 and S209 may be executed before sensing is performed in S203 and S205. Further, the generation of the time stamp is not limited to once, and may be repeated. In this case, one or more times of sensing may be performed between the time stamps before and after, or generation of time stamps may be continued without performing sensing in between. In addition, after the last time stamp, sensing one or more times may be performed, or sensing may not be performed.

After the sensing and the time stamp are generated at least once, in the manager 200, the integrated control unit 222 determines the end of recording (S215). For example, the recording may be ended in response to a user operation input via the input unit 430a of the UI device 400, or may be automatically ended as in the illustrated example. When recording automatically ends, for example, the number of sensing operations performed or the elapsed time from the start of recording can be a condition for ending. Further, when the time stamp is generated as in the illustrated example, the recording may be automatically terminated on the condition that the time stamp is generated.

After the recording in S215 is completed, the integrated control unit 222 stores, in the UI device 400, data in which the sensor detection values stored in S205, S213, and the like and the time stamps recorded in S209 are associated in time series. It outputs (S217). In the UI device 400, the output unit 430b displays the section setting UI for specifying the section of the detected value indicating the detection target state using the above data (S219). In the section setting UI, for example, a time stamp is displayed together with the time-series waveform of the sensor data, and the user assists the operation of setting the section of the sensor data. A specific example of the section setting UI will be described later. Alternatively, the condition adjustment / generation unit 223 may automatically extract sensor data based on the time stamp.

(Example of acquiring image and sound together with sensor data)
FIG. 25 is a sequence diagram illustrating an example of processing for acquiring an image / sound together with sensor data in the embodiment of the present disclosure. As described above, in the practice of the detection target state in S105 of FIG. 21 and the detection target state in S131 of FIG. 22 and its sensing, in order to make it easy to specify the detection value section indicating the detection target state, the user An image or sound recording a state of practicing the above may be additionally acquired. FIG. 25 shows an example of detection target state recognition in such a case.

In the illustrated example, the recording / recording apparatus 600 records images and sounds. Although not shown in the drawings so far, the recording / recording apparatus 600 is a camera or microphone that operates in conjunction with the manager 200, or an apparatus in which these are incorporated. The recording / recording apparatus 600 may be incorporated in the system 10 or may be outside the system 10, for example. When the recording / recording apparatus 600 is incorporated in the system 10, for example, the element 100 having a recording or recording function, the manager 200, or the UI apparatus 400 may function as the recording / recording apparatus 600.

First, in the manager 200, the integrated control unit 222 determines the start of recording (S231). Similar to the example described above with reference to FIG. 24, recording can be initiated by various triggers. The integrated control unit 222 notifies the recording / recording apparatus 600 of the start of recording. In response to this notification, the recording / recording apparatus 600 starts recording / recording (S233).

On the other hand, after the start of recording, the sensor 130a performs sensing in the element 100 (S235), and the sensor control unit 120a transmits the detection value to the manager 200 via the communication unit 110. In the manager 200, the integrated control unit 222 temporarily stores the sensor detection value received via the communication unit 210 (S237). The sensing by S235 and S237 may be executed only once or may be repeated many times.

Thereafter, in the manager 200, the integrated control unit 222 determines the end of recording (S239). Similar to the example described above with reference to FIG. 24, the recording can be terminated by various triggers. The integrated control unit 222 notifies the recording / recording apparatus 600 of the end of recording. Upon receiving this notification, the recording / recording apparatus 600 ends the recording / recording and outputs the image / sound data to the manager 200 (S241). The integrated control unit 222 of the manager 200 that has received the image / sound data sets data in which the sensor detection values stored in S237 and the like and the image / sound data received from the recording / recording apparatus 600 are associated in time series. The data is output to the UI device 400 (S243). In the UI device 400, the output unit 430b displays the section setting UI for specifying the section of the detection value indicating the detection target state using the above data (S245). Note that a specific example of the section setting UI in this case will be described later in the same manner as the section setting UI in the example of FIG.

(2-3. User interface example)
FIG. 26 is a diagram illustrating an example of a user interface for sensor selection according to an embodiment of the present disclosure. In FIG. 26, sensors (sensors 130 a) realized as functions of the element 100 in the system 10 are displayed as a list 4301. The user can select a sensor for setting a desired condition from the list 4301. As described above, in the present embodiment, the condition adjustment / generation unit 223 of the manager 200 selects the one or more sensors related to the condition from the available sensor group via the UI device 400. An interface can be provided. Alternatively, the condition adjustment / generation unit 223 may automatically select one or more sensors related to the condition from the available sensor group.

For example, in the illustrated example, “Button 1” 4303 in the list is selected. At this time, for example, by expanding the list as shown on the right side of the figure, the condition list 4305 assigned to “Button 1” is displayed. The condition list 4305 in the illustrated example indicates that “LongPress” and “DoublePress” conditions have already been assigned to “Button1”. The user can assign a new condition using “Button 1” by selecting “Assign Condition” displayed below.

FIG. 27 is a diagram illustrating an example of a user interface for condition selection according to an embodiment of the present disclosure. FIG. 27 shows an assignable condition list 4311 displayed when, for example, “Assign Condition” is selected in FIG.

In the condition list 4311, for example, conditions that can be assigned to “Button 1” in the example of FIG. 26 are shown. In the illustrated example, the condition list 4311 shows a default condition 4313 including “Press”, “DoublePress”, and “LongPress”, a composite condition 4315, and a learning generation condition 4317. The predetermined condition 4313 is a condition that has already been generated and saved in the storage unit 230 of the manager 200 or the like. These conditions can be assigned to “Button 1” by specifying them on the condition list 4311, for example. On the other hand, when the composite condition 4315 or the learning generation condition 4317 is selected, for example, a condition generation process as described above with reference to FIG. 22 is started.

FIG. 28 is a diagram illustrating an example of a user interface for condition proposal according to an embodiment of the present disclosure. FIG. 28 shows an example of a UI that is displayed when “automatically search for condition candidates” is designated in S103 in the example described with reference to FIG. In this case, for example, the standby screen 4321 is displayed while waiting for the user to practice the detection target state for searching for condition candidates. In the illustrated example, it is shown that the user is waiting for the detection target state to be put into practice for the sensor “Acceleration 1”.

When a condition candidate is found after the user performs the detection target state (YES in S107 in the example of the flowchart of FIG. 21), a search result presentation screen 4323 is displayed. On the screen 4323, the name of the condition candidate is displayed, and a retry button 4325 is displayed for a case where the user thinks that the condition candidate is not appropriate (this point is retried when there is a condition candidate) This is different from the example shown in Fig. 21. Both examples can be adopted in this embodiment.

On the other hand, if no condition candidate is found (NO in S107 in the example of FIG. 21), a search result presentation screen 4327 is displayed. On the screen 4327, it is displayed that a condition candidate has not been found, and a retry button 4325 is displayed.

FIG. 29 is a diagram illustrating an example of a user interface for generating a composite condition according to an embodiment of the present disclosure. FIG. 29 shows a composite condition list 4331. The composite condition list 4331 includes a rank 4333, a component 4335, and a component candidate 4337. In the composite condition list 4331, each condition included in the composite condition is indicated by the component 4335, and the order of each condition is indicated by the order 4333. The component 4335 can be added or changed by selecting from the component candidates 4337. In addition, the component 4335 can be deleted.

In the illustrated example, “DoublePress” is set in the rank “1”, “LongPress” is set in the rank “2”, and “Press” is set in the rank “3”. In this case, the condition is satisfied when “DoublePress”, “LongPress”, and finally “Press” are executed in this order. The compound condition generated using the compound condition list 4331 can be referred to and selected as the arbitrarily set condition 4319 together with the default condition 4313 in the condition list 4311 described above with reference to FIG.

FIG. 30 is a diagram illustrating another example of a user interface for generating a composite condition according to an embodiment of the present disclosure. FIG. 30 shows a composite condition list 4331 as in the example of FIG. 29 described above. However, in the illustrated example, the same order 4333 “1” is set for the two components 4335 “DetectFace” and “DetectHand”.

In this case, the composite condition is satisfied when “DetectFace” and “DetectHand” occur simultaneously. FIG. 30 shows an example of a condition list 4341 different from that shown in FIG. The compound condition generated using the compound condition list 4311 can be referred to and selected as the arbitrarily set condition 4349 together with the default condition 4343, the (new) combination condition 4345, and the learning generation condition 4347 in the condition list 4341. .

FIG. 31 is a diagram illustrating an example of a user interface for training data input according to an embodiment of the present disclosure. FIG. 31 shows an example of a UI displayed when the detection target state is put into practice in S131 and S133 and training data is recorded in the example described with reference to FIG.

The screen 4401 and the screen 4407 in the illustrated example are received as “single training data corresponding to the detection target state” and “timing to start inputting training data” when the user repeatedly inputs training data. "Time" is displayed to communicate to the user. When a start button 4403 is pressed on the screen 4401, the center circle 4405 starts to be missing as shown by a circle 4411. The time until the circle 4405 is completely missing after the state of the circle 4411 indicates “time accepted as single training data”. Further, the timing at which the circle-missed region goes around and the display of the circle 4411 returns to the circle 4405 in the initial state is “timing to start (restart) input of training data”. When ending the training data input, the user presses a stop button 4409 displayed on the screen 4407.

FIG. 32 is a diagram illustrating another example of a user interface for inputting training data according to an embodiment of the present disclosure. FIG. 32 shows a screen 4413 and a screen 4417 that are partially different from the screen 4401 and the screen 4407 described above with reference to FIG. On the screen 4413, a start button 4403 and a circle 4405 are displayed in the same manner as the screen 4401, and a number setting unit 4415 is displayed. The number setting unit 4415 allows the user to set the number of repetitions of training data input.

On the other hand, on the screen 4417, a circle 4411 is displayed as in the screen 4407, but a stop button is not displayed. Instead, a remaining number display 4419 is displayed on the screen 4417, and the remaining number of times until the input of training data is automatically terminated is displayed.

For example, in the examples of the

screens

4401, 4407, 4413, and 4417 described above, the “time that can be accepted as a single training data” can be automatically specified based on the time stamp and image / sound acquired together with the sensor data. In some cases, the user does not have to measure the input timing of training data from the screen. Therefore, in this case, the user interface may be simplified by omitting the

circles

4405 and 4411.

FIG. 33 is a diagram illustrating a first example of a user interface for inputting training data including an upper limit value and a lower limit value according to an embodiment of the present disclosure. In the illustrated example, the definition of the condition includes a range of detection values included in the sensor data. FIG. 33 shows a screen 4421 for recording and referring to the upper limit value and the lower limit value of the detection value range when generating the condition “Fast & Slow”. Screen 4421 includes a condition name 4423 and a record / reference button 4425. The record / reference button 4425 includes a record button 4425a and a reference button 4425b. When the recording button 4425a is pressed, a training data input user interface as described below is displayed, and training data can be input. On the other hand, when the reference button 4425b is pressed, a user interface for referring to existing data is displayed, and the existing data can be used as training data.

FIG. 34 is a diagram illustrating a second example of a user interface for inputting training data including an upper limit value and a lower limit value according to an embodiment of the present disclosure. FIG. 34 shows a screen 4431 and a screen 4435 that are displayed when the record button 4425a is pressed in the example of FIG. On the screen 4431, it is instructed to input upper limit training data by the text 4433. On the screen 4437, the text 4437 indicates that lower limit training data is input.

Alternatively, the screen 4431 and the screen 4435 described above may be directly displayed without passing through the screen 4421. First, an upper limit value input screen 4431 and then a lower limit value input screen 4435 are automatically and continuously displayed, so that the user can perform

text

4433 and 4437 without any additional operation. The training data input of the upper limit value and the lower limit value of the range can be automatically executed while confirming the above. As described above, in the present embodiment, the condition adjustment / generation unit 223 of the manager 200 includes one or more detection values (not limited to the upper limit value and the lower limit value within the range related to the condition definition, and may include intermediate values). The instruction for practicing the detection target state so as to correspond to “good” may be output via the UI device 400.

FIG. 35 is a diagram illustrating an example of a user interface for inputting training data for classification according to an embodiment of the present disclosure. FIG. 35 shows a screen 4441 for setting a type of condition for classifying sensor data such as identification of a user by a face shown in an image, for example. The screen 4441 includes an index 4443, a label 4445, a record button 4447, a reference button 4449, and an add button 4451.

The label 4445 displays a label given to a class that is classified according to conditions. When the record button 4447 is pressed, a training data input user interface for each class is displayed, and training data can be input. For example, in the case of identification of a user by face, training data can be input by photographing the face of the target user. On the other hand, when a reference button 4449 is pressed, a user interface for referring to existing data is displayed, and the existing data can be used as training data. For example, in the case of identification of a user by face, an image obtained by acquiring an image including the target user's face from a user's local storage or a shared server can be used as training data. When an add button 4451 is pressed, a new class is generated, and a new label 4445, a record button 4447, and a reference button 4449 are displayed.

FIG. 36 is a diagram illustrating a first example of a user interface for extracting training data according to an embodiment of the present disclosure. In the illustrated example, the sensor data acquired during the practice of the detection target state based on the user operation is extracted from the sensor data obtained continuously during and after the detection target state. A user interface is provided. FIG. 36 shows a training data extraction screen 4501. The training data extraction screen 4501 includes an image / sound reproduction unit 4503, a time series waveform 4505 of sensor detection values, and a section designation button 4507.

In the illustrated example, the sensor detection value when the user practices the detection target state is temporarily stored by the integrated control unit 222 of the manager 200. The user specifies the section corresponding to the detection target state from the sensor detection values while referring to the training data extraction screen 4501, for example. More specifically, the user adjusts the section 4505r displayed together with the time-series waveform 4505 of the sensor detection value, and then presses the section designation button 4507, thereby detecting the sensor detection value corresponding to each detection target state. Can be specified.

Here, the image / sound reproduction unit 4503 presents the image or sound recorded during the practice of the detection target state. More specifically, for example, the image / sound reproduction unit 4503 may reproduce the image or sound in the range of the time-series waveform 4505 of the sensor detection value, or the image or sound in the section 4505r selected by the user. May be reproduced, and images or sounds before and after the start point or end point of the section 4505r may be reproduced. In any case, the reproduced image or sound is associated with the sensor data displayed by the time series waveform 4505.

FIG. 37 is a diagram illustrating a second example of a user interface for extracting training data according to an embodiment of the present disclosure. FIG. 37 shows a training data extraction screen 4511. The training data extraction screen 4511 includes an image / sound reproduction unit 4513, a time series waveform 4505 of sensor detection values, and a section designation button 4507.

The example shown is a modification of the example shown in FIG. In the illustrated example, the image / sound reproduction unit 4513 displays an image as a series of still images instead of a moving image. Thereby, for example, the size of the image acquired together with the sensor detection value is suppressed, and the processing load for synchronization with the sensor detection value in the manager 200 and transmission of data to the UI device 400 can be reduced. In addition, since the detection target state is presented as a discrete still image instead of a continuous moving image, it may be easier for the user to set and adjust the section 4505r.

FIG. 38 is a diagram illustrating a third example of a user interface for extracting training data according to an embodiment of the present disclosure. FIG. 38 shows a training data extraction screen 4521. The training data extraction screen 4521 includes a time series waveform 4505 of sensor detection values, a section designation button 4507, and a time stamp 4523.

As described above with reference to FIG. 24, in this embodiment, it is possible to record a time stamp in synchronization with the acquisition of training data. In the training data extraction screen 4521, a time stamp 4523 is displayed at a position on the time series corresponding to the time stamp recorded as described above. By displaying the time stamp 4523, the user can easily set the section 4505r more accurately.

FIG. 39 is a diagram illustrating an example of a user interface for real-time feedback according to an embodiment of the present disclosure. FIG. 39 shows real-

time feedback screens

4531 and 4537. In the illustrated example, the condition definition includes that the volume detected by the microphone 100g exceeds the threshold.

When the condition is verified, if the sensor detection value does not exceed the threshold value (that is, the condition is not satisfied in the above example), a screen 4531 is displayed. On the screen 4531, a gauge 4533 that does not reach the threshold level 4535 and an X mark 4537 indicating that the condition is not satisfied are displayed. For example, when the user who refers to this result wants to adjust the threshold value, the threshold value can be changed in real time by the adjustment button 4539 and the verification can be performed again.

On the other hand, when the sensor detection value exceeds the threshold value (that is, the condition is satisfied in the above example) during the verification of the condition, a screen 4541 is displayed. On the screen 4541, a gauge 4533 exceeding the threshold level 4535 and a ◯ mark 4543 indicating that the condition is satisfied are displayed. Similarly to the screen 4531, the threshold value can be changed in real time using the adjustment button 4539 on the screen 4541.

FIG. 40 is a diagram illustrating another example of a user interface for real-time feedback according to an embodiment of the present disclosure. FIG. 40 shows real-

time feedback screens

4551 and 4557. In the illustrated example, a condition is set for discriminating between a case where the acceleration sensor 100a is moved slowly and a case where the acceleration sensor 100a is moved quickly based on a predetermined range of detection values (corresponding to a case where the acceleration sensor 100a is moved quickly).

When verifying the conditions, if the sensor detection value is outside the predetermined range (that is, in the above example, it is determined that the acceleration sensor 100a is moved slowly), a screen 4551 is displayed. On the screen 4551, a gauge 4553 displaying the range 4555, a detection value display 4557 indicating the gauge 4553 outside the range 4555, and an X mark 4559 indicating that the condition is not satisfied are displayed. For example, when the user who refers to this result wants to adjust the threshold value, the range 4555 may be dragged and moved, and verification may be performed again.

On the other hand, when the sensor detection value is within a predetermined range at the time of verifying the condition (that is, in the above example, it is determined that the acceleration sensor 100a is moved quickly), a screen 4561 is displayed. On the screen 4561, a gauge 4553 displaying the range 4555, a detection value display 4557 indicating the gauge 4553 in the range 4555, and a circle 4559 indicating that the condition is satisfied are displayed. Similarly to the screen 4551, the screen 4561 may be able to change the range in real time by moving the range 4555.

FIG. 41 is a diagram illustrating still another example of a user interface for real-time feedback according to an embodiment of the present disclosure. FIG. 41 shows real-time feedback screens 4601a to 4601c. In the illustrated example, a condition for determining whether or not the user's face is reflected in the image acquired by the camera 100b is set.

The screen 4601a is displayed when the user's face is shown in the image and it is determined that the condition is satisfied. In this case, it is displayed that the condition is satisfied by the circle mark and the text 4603a. The user can input feedback regarding whether or not such a determination result is correct using a button 4605. In the illustrated example, since the determination is correctly performed, the “correct” button 4605 can be pressed.

The screen 4601b is displayed when it is determined that the user's face is not shown on the screen and the condition is not satisfied. In this case, it is displayed that the condition is not satisfied by the x mark and the text 4603b. The user can input feedback using the button 4605 as in the screen 4601a. Also in the example shown in the figure, since the determination is correctly performed, the “correct” button 4605 can be pressed.

On the other hand, the screen 4601c is displayed when it is determined that the condition is satisfied even though the user's face is not shown on the screen. In this case, it is displayed that the condition is satisfied by the circle mark and the text 4603c. In the illustrated example, since the determination is not correctly performed, the user can input feedback for correcting the setting of the condition by pressing a “Wong” button 4605.

In addition to the examples described above, various modifications are possible for the output or presentation of instructions or information in the embodiment of the present disclosure. For example, instead of or together with the output unit 430b of the UI device 400, an instruction or information to the user may be output or presented via the actuator 130b of the element 100. As a result, the user does not necessarily have to carry the UI device 400 when setting or verifying conditions, and in addition, stepwise output or presentation is performed via various types of actuators 130b. Is possible.

For example, when information is presented via an LED lamp included in the actuator 130b, the LED is detected when the detection value of the sensor is close to a value that satisfies the condition (for example, a close value within a predetermined range, a value that is above or below a threshold value). It may be set so that the lamp gradually lights up gradually and the LED lamp blinks when the condition is satisfied.

For example, when information is presented via a speaker included in the actuator 130b, a loud sound is gradually output from the speaker when the detection value of the sensor approaches a value that satisfies the condition, and a predetermined melody is generated when the condition is satisfied. It may be set to flow. In addition, for example, information may be presented in a natural language such as “not likely to react”, “looks to react a little later”, or “reacted”.

(2-4. Specific application examples)
(First example)
FIG. 42 is a diagram for describing a first application example according to an embodiment of the present disclosure. Referring to FIG. 42, in the first application example, acceleration sensor 100a is attached to the door of refrigerator REF. Here, the user uses the acceleration sensor 100a to detect that the door of the refrigerator REF is opened, and in this case, the camera 100b (not shown) wants to acquire an image inside the refrigerator REF, When is closed, it is assumed that the processing is not desired. In such a case, for example, as a practice of the detection target state, the user repeatedly opens and closes the door of the refrigerator REF with sensing performed by the acceleration sensor 100a. At this time, the camera 100b is also photographing the refrigerator REF. At this time, it can be said that the camera 100b functions as the recording / recording apparatus 600 described above with reference to FIG.

In the illustrated example, the integrated control unit 222 of the manager 200 associates the detection value of the acceleration sensor 100a when the user practices the detection target state as described above with the image captured by the camera 100b in time series. Accordingly, the training data extraction screen 4501 described above with reference to FIG. 36 is presented via the UI device 400. On the training data extraction screen 4501, an image / sound reproduction unit 4503 is displayed, and an image of the refrigerator REF captured by the camera 100b can be referred to along with the time-series waveform 4505 of the sensor detection value. Accordingly, the user can adjust the section 4505r indicating the detection target state so as to include a section where the door of the refrigerator REF is opened and not include a section where the door is closed. By pressing the section designation button 4507 in a state where the section 4505r matches the section where the door is opened, the detected value of the acceleration sensor 100a when the door is opened can be specified as training data when the condition is satisfied. it can.

FIG. 43 is a diagram showing an example of a user interface for creating a program in the example of FIG. FIG. 43 shows the list 4301, the element icon 4701, and the condition / processing icon 4703 described above with reference to FIG. In the illustrated example, the condition described above with reference to FIG. 42 is displayed as “Door Open” in the condition list 4305 in the list 4301. For example, when the user selects “Door Open” in the condition list 4305, the condition “Door Open” can be assigned to the acceleration sensor 100a as indicated by the element icon 4701a and the condition / process icon 4703.

Further, in FIG. 43, an element icon 4701b of the camera 100b, a condition / processing icon 4703b indicating that a photograph is to be taken, an icon 4701c indicating a cloud service, and a condition / process indicating that the photographed photograph is to be uploaded. An icon 4703c is shown. The user interface shown in FIG. 43 may implement the first specific example described above with reference to FIG. In FIG. 12, the software 603a used to detect that the door has been opened based on the acceleration is included in the conditions set in the acceleration sensor 100a in the example shown in FIG.

(Second example)
FIG. 44 is a diagram for describing a second application example according to an embodiment of the present disclosure. Referring to FIG. 44, an element icon 4711a and a condition / processing icon 4713a of the acceleration sensor 100a, and an element icon 4711b and a condition / processing icon 4713b of the speaker 100e are shown. The user interface shown in FIG. 44 may realize the fourth specific example described above with reference to FIG. The condition / processing icon 4713a includes three conditions (Motion1, Motion2, and Motion3) that are determined based on the detection value of the acceleration sensor 100a. These conditions are generated, for example, by learning the result of repeating the movement of the limbs in a specific pattern with the acceleration sensor 100a being worn as a practice of the detection target state with the user (for example, a child) wearing the acceleration sensor 100a. On the other hand, the condition / processing icon 4713b indicates three sound effects respectively corresponding to the above three conditions. In FIG. 15, the software 603d used for reproducing the sound effect according to the acceleration pattern is the condition set in the acceleration sensor 100a in the example shown in FIG. 44 and the speaker corresponding to this condition. It is incorporated in the function of 100e.

(3. Hardware configuration)
Next, the hardware configuration of the information processing apparatus according to the embodiment of the present disclosure will be described with reference to FIG. FIG. 45 is a block diagram illustrating a hardware configuration example of the information processing apparatus according to the embodiment of the present disclosure. The illustrated information processing apparatus 900 can realize, for example, the element 100, the manager 200, the server 300, and / or the UI apparatus 400 in the above-described embodiment.

The information processing apparatus 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing apparatus 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Furthermore, the information processing apparatus 900 may include an imaging device 933 and a sensor 935 as necessary. The information processing apparatus 900 may include a processing circuit called DSP (Digital Signal Processor) or ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 901.

The CPU 901 functions as an arithmetic processing device and a control device, and controls all or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 primarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. The CPU 901, the ROM 903, and the RAM 905 are connected to each other by a host bus 907 configured by an internal bus such as a CPU bus. Further, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

The input device 915 is a device operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. The input device 915 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an external connection device 929 such as a mobile phone that supports the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 900.

The output device 917 is a device that can notify the user of the acquired information visually or audibly. The output device 917 can be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro-Luminescence) display, an audio output device such as a speaker and headphones, and a printer device. . The output device 917 outputs the result obtained by the processing of the information processing device 900 as video such as text or an image, or outputs it as audio such as voice or sound.

The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in or externally attached to the information processing apparatus 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. In addition, the drive 921 writes a record in the attached removable recording medium 927.

The connection port 923 is a port for directly connecting a device to the information processing apparatus 900. The connection port 923 can be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like. The connection port 923 may be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like. By connecting the external connection device 929 to the connection port 923, various types of data can be exchanged between the information processing apparatus 900 and the external connection device 929.

The communication device 925 is a communication interface configured with, for example, a communication device for connecting to the communication network 931. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication. The communication device 925 transmits and receives signals and the like using a predetermined protocol such as TCP / IP with the Internet and other communication devices, for example. The communication network 931 connected to the communication device 925 is a wired or wireless network, such as the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

The imaging device 933 uses various members such as an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and a lens for controlling the formation of a subject image on the imaging element. It is an apparatus that images a real space and generates a captured image. The imaging device 933 may capture a still image or may capture a moving image.

The sensor 935 is various sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, and a sound sensor. The sensor 935 acquires information about the state of the information processing apparatus 900 itself, such as the posture of the information processing apparatus 900, and information about the surrounding environment of the information processing apparatus 900, such as brightness and noise around the information processing apparatus 900, for example. To do. The sensor 935 may include a GPS sensor that receives a GPS (Global Positioning System) signal and measures the latitude, longitude, and altitude of the apparatus.

Heretofore, an example of the hardware configuration of the information processing apparatus 900 has been shown. Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Such a configuration can be appropriately changed according to the technical level at the time of implementation.

(4. Supplement)
Embodiments of the present disclosure include, for example, an information processing apparatus (a manager, or an element, a server, or a UI apparatus that also functions as a manager), a system, an information processing apparatus, or an information processing method executed by the system And a program for causing the information processing apparatus to function, and a non-temporary tangible medium on which the program is recorded.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1) a sensor data acquisition function for acquiring sensor data provided by one or more sensors;
An information processing apparatus comprising: a processing circuit that realizes a condition setting function for setting a condition relating to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state.
(2) The condition setting function includes the function of setting the condition by performing machine learning using the sensor data acquired during practice of the detection target state as training data. Information processing device.
(3) The information processing apparatus according to (2), wherein the processing circuit further realizes a function of outputting an instruction of a timing of practice of the detection target state.
(4) The definition of the condition includes a range of detection values included in the sensor data,
The information processing according to (2) or (3), wherein the processing circuit further realizes a function of outputting an instruction to practice the detection target state so as to correspond to one or more detection values within the range. apparatus.
(5) The sensor data acquisition function acquires the sensor data during and before and after the detection target state is put into practice,
The processing circuit further realizes a sensor data extraction function for extracting the sensor data acquired during practice of the detection target state from the acquired sensor data. The information processing apparatus according to claim 1.
(6) The processing circuit further realizes a function of acquiring a time stamp recorded during practice of the detection target state,
The information processing apparatus according to (5), wherein the sensor data extraction function extracts the sensor data based on the time stamp.
(7) The sensor data extraction function extracts the sensor data based on a user operation,
The processing circuit further realizes a function of acquiring an image or sound recorded during practice of the detection target state, and a function of presenting the image or the sound in association with the sensor data to a user, The information processing apparatus according to (5) or (6).
(8) The information processing apparatus according to any one of (2) to (7), wherein the detection target state is practiced only once.
(9) The information processing apparatus according to any one of (2) to (7), wherein the detection target state is repeatedly practiced.
(10) The information processing apparatus according to (1), wherein the condition setting function includes a function of verifying the condition based on the sensor data acquired during practice of the detection target state.
(11) The processing circuit further realizes an existing condition reading function for reading an existing condition,
The information processing apparatus according to (10), wherein the condition includes a single existing condition or a combination of a plurality of existing conditions.
(12) The information processing apparatus according to (10) or (11), wherein the processing circuit further realizes a function of changing the setting of the condition based on a user operation during the verification of the condition.
(13) Any one of (1) to (12), wherein the processing circuit further realizes a sensor selection function for selecting the one or more sensors from a group of available sensors based on a user operation. The information processing apparatus according to item.
(14) The information processing apparatus according to any one of (1) to (13), wherein the definition of the condition includes a threshold value set for a detection value included in the sensor data.
(15) The information processing apparatus according to any one of (1) to (14), wherein the condition includes a condition established by a combination of sensor data provided by a plurality of sensors.
(16) The processing circuit according to (1) to (15), wherein the processing circuit further realizes a function of outputting or presenting instructions or information to a user via a device different from the device including the one or more sensors. The information processing apparatus according to any one of claims.
(17) The processing circuit further realizes a function of operating an actuator when the condition is satisfied, and a function of outputting or presenting an instruction or information to a user via the actuator. The information processing apparatus according to any one of (16).
(18) one or more sensors;
Acquire sensor data provided by the one or more sensors, and set conditions relating to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state An information processing device;
A terminal device that outputs or presents an instruction or information related to the setting of the condition provided by the information processing device.
(19) The processing circuit of the information processing apparatus is
Obtaining sensor data provided by one or more sensors;
An information processing method comprising: setting a condition related to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state.
(20) a sensor data acquisition function for acquiring sensor data provided by one or more sensors;
A program for causing a processing circuit to realize a condition setting function for setting a condition relating to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state.

DESCRIPTION OF SYMBOLS 10 System 100 Element 110 Communication part 120 Control part 121 Condition determination part 130 Function part 140 Power supply part 200 Manager 210 Communication part 220 Control part 222 Integrated control part 223 Condition adjustment / generation part 224 Condition management part 225 Condition determination part 226 Condition handling part 230 Storage Unit 300 Server 310 Communication Unit 320 Control Unit 330 Storage Unit 400 UI Device 410 Communication Unit 420 Control Unit 430 Input / Output Unit

Claims

A sensor data acquisition function for acquiring sensor data provided by one or more sensors;
An information processing apparatus comprising: a processing circuit that realizes a condition setting function for setting a condition relating to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state.
The information processing apparatus according to claim 1, wherein the condition setting function includes a function of setting the condition by performing machine learning using the sensor data acquired during practice of the detection target state as training data. .
The information processing apparatus according to claim 2, wherein the processing circuit further realizes a function of outputting an instruction of a timing of practice of the detection target state.
The definition of the condition includes a range of detection values included in the sensor data,
The information processing apparatus according to claim 2, wherein the processing circuit further realizes a function of outputting an instruction to practice the detection target state so as to correspond to one or a plurality of detection values within the range.
The sensor data acquisition function acquires the sensor data during and before and after the detection target state is put into practice,
The information processing apparatus according to claim 2, wherein the processing circuit further realizes a sensor data extraction function for extracting the sensor data acquired during practice of the detection target state from the acquired sensor data.
The processing circuit further realizes a function of obtaining a time stamp recorded during practice of the detection target state,
The information processing apparatus according to claim 5, wherein the sensor data extraction function extracts the sensor data based on the time stamp.
The sensor data extraction function extracts the sensor data based on a user operation,
The processing circuit further realizes a function of acquiring an image or sound recorded during practice of the detection target state and a function of presenting the image or the sound in association with the sensor data to a user. Item 6. The information processing device according to Item 5.
The information processing apparatus according to claim 2, wherein the detection target state is practiced only once.
The information processing apparatus according to claim 2, wherein the detection target state is repeatedly practiced.
The information processing apparatus according to claim 1, wherein the condition setting function includes a function of verifying the condition based on the sensor data acquired during practice of the detection target state.
The processing circuit further realizes an existing condition reading function for reading an existing condition,
The information processing apparatus according to claim 10, wherein the condition includes a single existing condition or a combination of a plurality of the existing conditions.
The information processing apparatus according to claim 10, wherein the processing circuit further realizes a function of changing the setting of the condition based on a user operation during the verification of the condition.
The information processing apparatus according to claim 1, wherein the processing circuit further realizes a sensor selection function of selecting the one or more sensors from a group of available sensors based on a user operation.
The information processing apparatus according to claim 1, wherein the definition of the condition includes a threshold value set for a detection value included in the sensor data.
The information processing apparatus according to claim 1, wherein the condition includes a condition established by a combination of sensor data provided by a plurality of sensors.
The information processing apparatus according to claim 1, wherein the processing circuit further realizes a function of outputting or presenting an instruction or information to a user via a device different from the device including the one or more sensors.
The information processing according to claim 1, wherein the processing circuit further realizes a function of operating an actuator when the condition is satisfied, and a function of outputting or presenting an instruction or information to a user via the actuator. apparatus.
One or more sensors;
Acquire sensor data provided by the one or more sensors, and set conditions relating to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state An information processing device;
A terminal device that outputs or presents an instruction or information regarding the setting of the condition provided by the information processing device.
The processing circuit of the information processing device
Obtaining sensor data provided by one or more sensors;
An information processing method comprising: setting a condition related to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state.
A sensor data acquisition function for acquiring sensor data provided by one or more sensors;
A program for causing a processing circuit to realize a condition setting function for setting a condition relating to the sensor data for detecting the detection target state based on the sensor data acquired during practice of the detection target state.