KR20170059346A - Server, electronic device and method for performing internet of thing in electronic device - Google Patents

Abstract

The present invention relates to a server, an electronic device, and a method for performing internet of things (IoT) in an electronic device. The method for performing Internet of things in an electronic device comprises: receiving a user input from at least one external electronic device; generating an action plan corresponding to the received user input based on previously stored object information and operation information for each object; and controlling at least one object corresponding to the received user input so that at least one operation according to the generated plan is performed. Other various embodiments are possible.

Description

TECHNICAL FIELD [0001] The present invention relates to a server, an electronic device,

Various embodiments of the present invention are directed to a server, an electronic device, and a method for performing the Internet of things in an electronic device.

The electronic device refers to a device that performs a specific function according to a program loaded from an appliance, such as an electronic notebook, a portable multimedia player, a mobile communication terminal, a tablet PC, a video / audio device, a desktop / laptop computer, . For example, such electronic devices may output stored information as sound or video. As the degree of integration of electronic devices increases, ultra-high-speed and large-capacity wireless communications become popular, and various functions are incorporated in electronic devices.

As the user's peripheral IoT (Internet of Things) devices become diverse, there are many tasks that the user needs to manage for the IoT devices that can be controlled according to the user's input.

According to various embodiments of the present invention, a control plan for IoT devices can be established and executed to receive user input, identify various tasks associated with the user input, and perform the identified tasks.

According to various embodiments of the present invention, a server includes: a communication unit; Memory; And a processor operable to receive user input from at least one external electronic device via the communication unit and to receive, based on the object information stored in the memory and the operation information of each object, Generate a corresponding action plan, and control at least one object corresponding to the received user input such that at least one action according to the generated plan is performed.

A method for performing object Internet in a server and an electronic device according to various embodiments of the present invention includes the steps of generating an action plan corresponding to the received user input upon receiving a user input from at least one external electronic device, By controlling at least one object corresponding to the received user input according to the plan, various objects can be controlled even with a minimum input of the user and operations according to the intention of the user can be performed.

1 is a diagram illustrating an example of a system structure according to various embodiments of the present invention.
2 is a diagram illustrating an operation for receiving a user input between a server and an electronic device and establishing a plan based on the received input, in accordance with various embodiments of the present invention.
3 is a flow diagram illustrating an example of an operation for creating and performing a plan for controlling IoT devices, in accordance with various embodiments of the present invention.
4 is a diagram illustrating an example of a database structure in accordance with various embodiments of the present invention.
5 is a flow diagram illustrating an example of operation for controlling an IoT device in accordance with user input, in accordance with various embodiments of the present invention.
6 is a flow diagram illustrating an example of operation for controlling an IoT device in accordance with user input, in accordance with various embodiments of the present invention.
7 is a flow diagram illustrating an example of an operation for configuring a database for controlling an IoT device in a server, in accordance with various embodiments of the present invention.
8 is a flow diagram illustrating an example of an operation for establishing and performing a plan for controlling an IoT device at a server, in accordance with various embodiments of the present invention.
9 to 14 are diagrams showing various examples of a database for controlling the IoT device.
Figure 15 is a diagram illustrating an example of a system for controlling an IoT device according to user preferences, in accordance with various embodiments of the present invention.
16 is a diagram illustrating an example of an execution screen of an IoT device that recommends related operations according to various embodiments of the present invention.
17 is a diagram showing an example of an execution screen of an IoT device for controlling the display of the operations included in the routine according to the addition or deletion of the IoT device, according to various embodiments of the present invention.
18 is a diagram illustrating various examples of situations in which IoT devices are controlled, according to various embodiments of the present invention.
19 is a diagram showing an example of an execution screen of an application recommending the operation of the IoT device according to user input according to various embodiments of the present invention.
20 is a flow chart illustrating an example of an operation of controlling an IoT device by checking operation time information from a user input according to various embodiments of the present invention.
21 is a diagram showing an example of an execution screen of an application recommending tasks in relation to an IoT device in an electronic device, according to various embodiments of the present invention.
22 is a diagram showing an example of an execution screen of an application that generates a schedule with tasks recommended in association with an IoT device in an electronic device, according to various embodiments of the present invention.
23 is a diagram showing an example of an execution screen of an application that generates a schedule from a routine related to an IoT device in an electronic device, in accordance with various embodiments of the present invention.
24 is a diagram illustrating an example of a network environment, in accordance with various embodiments of the present invention.
25 is a block diagram illustrating an example of an electronic device configuration according to various embodiments of the present invention.
26 is a block diagram illustrating an example of a program module structure according to various embodiments of the present invention.
27 is a flowchart illustrating an example of an operation of registering an IoT device in a server according to various embodiments of the present invention.
28 is a flow diagram illustrating an example of an operation for performing a plan in a server, in accordance with various embodiments of the present invention.
29 is a flow diagram illustrating an example of an operation for performing a plan in a server, in accordance with various embodiments of the present invention.
30 is a flow diagram illustrating an example of an operation to perform a plan in a server, in accordance with various embodiments of the present invention.
31 is a flow diagram illustrating an example of an operation to perform a plan in a server, in accordance with various embodiments of the present invention.
32 is a flow diagram illustrating an example of an operation to perform a plan at a server, in accordance with various embodiments of the present invention.
33 is a flowchart illustrating an example of an operation of registering an IoT device in a server according to various embodiments of the present invention.

Various embodiments of the invention will now be described with reference to the accompanying drawings. It should be understood, however, that the description herein is not intended to limit the invention to the particular embodiments, but includes various modifications, equivalents, and / or alternatives of the embodiments of the invention . In connection with the description of the drawings, like reference numerals may be used for similar components.

In the present invention, the expression "having," " having, "" comprising," Quot ;, and does not exclude the presence of additional features.

In the present invention, the expression "A or B," "at least one of A or / and B," or "one or more of A and / or B," etc. may include all possible combinations of the listed items . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) Or (3) at least one A and at least one B all together.

The terms "first," "second," "first," or "second," etc. used in the present invention may be used to denote various components, regardless of their order and / or importance, But is used to distinguish it from other components and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment, regardless of order or importance. For example, without departing from the scope of the invention described in the present invention, the first component can be named as the second component, and similarly, the second component can also be named as the first component.

(Or functionally or communicatively) coupled with / to "another component (eg, a second component), or a component (eg, a second component) Quot; connected to ", it is to be understood that any such element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when it is mentioned that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component) It can be understood that there is no other component (e.g., a third component) between other components.

The phrase " configured to be used "as used in the present invention means that, depending on the situation, for example," having the capacity to, To be designed to, "" adapted to, "" made to, "or" capable of ". The term " configured to (or set up) "may not necessarily mean" specifically designed to "in hardware. Instead, in some situations, the expression "configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) , And a generic-purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding operations.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the other embodiments. The singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art. The general predefined terms used in the present invention can be interpreted in the same or similar meaning as the contextual meanings of the related art, and, unless expressly defined in the present invention, mean ideal or overly formal meanings . In some cases, the terms defined in the present invention can not be construed as excluding the embodiments of the present invention.

An electronic device in accordance with various embodiments of the present invention may be, for example, a smartphone, a tablet personal computer, a mobile phone, a video phone, an e-book reader, Such as a desktop personal computer, a laptop personal computer, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP) A device, a camera, or a wearable device. According to various embodiments, the wearable device may be of the accessory type (e.g., a watch, a ring, a bracelet, a bracelet, a necklace, a pair of glasses, a contact lens or a head-mounted-device (HMD) (E. G., Electronic apparel), a body attachment type (e. G., A skin pad or tattoo), or a bioimplantable type (e.g., implantable circuit).

In various embodiments of the invention, the electronic device may be a home appliance. Home appliances include, for example, televisions, digital video disc (DVD) players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwaves, washing machines, air cleaners, set- Such as a home automation control panel, a security control panel, a TV box such as Samsung HomeSync ^TM , Apple TV ^TM or Google TV ^TM , a game console such as Xbox ^TM and PlayStation ^TM , , An electronic key, a camcorder, or an electronic frame.

In various embodiments of the present invention, an electronic device may be used in a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), a magnetic resonance angiography (MRA) (navigation system), a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile (Eg, navigation devices, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or home robots, ATMs of financial institutions a teller's machine, a point of sale of a store, or an internet of things such as a light bulb, various sensors, an electric or gas meter, a sprinkler device, a smoke alarm, a thermostat ), A streetlight, a toaster, a fitness device, a hot water tank, a heater, a boiler, etc.).

According to various embodiments of the invention, the electronic device may be a piece of furniture or part of a building / structure, an electronic board, an electronic signature receiving device, a projector, (E.g., water, electricity, gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device may be a combination of one or more of the various devices described above. An electronic device according to some embodiments may be a flexible electronic device. In addition, the electronic device according to the embodiment of the present invention is not limited to the above-described devices, and may include a new electronic device according to technological advancement.

1 is a diagram illustrating an example of a system structure according to various embodiments of the present invention.

Referring to FIG. 1, the IoT system may include an IoT server 101, at least one IoT devices 102, a context data server 103, and a dialogue system 104.

Upon receiving the user input (e.g., speech or biometric information), the IoT server 101 can confirm the information of the IoT device related to the user input. For example, the information of the IoT device may include at least one operation information related to the state of the device or the user input.

According to various embodiments of the present invention, the IoT server may perform goal setting, plan generation, operation execution, personalization information reflection, or recommendation inference operation, and may control at least one IoT device through other user input Various operations can be performed.

The goal setting operation may include setting a goal associated with a particular user input using data contained in a service database 116c and an ontology database 116b. For example, the target may include information about a device to be controlled and a status of the device.

The plan generation operation may be performed based on the data included in the ontology database 116b and the data included in the personal things database 116e so that the operation between the IoT devices or the operations of the respective IoT devices Confirming the order, and creating a plan for controlling the IoT device according to the identified order.

  The operation and personalization information reflecting operation controls the operations of the IoT devices and the respective IoT devices based on the generated plan, and the performed IoT devices and operations are stored in the personal thing database 116e .

The recommendation reasoning operation may generate a recommendation action based on data stored in the service database 116c, the personal service database 116d, or the personal thing database 116e. For example, the recommendation action may recommend an action associated with the received user input, taking into account the stored data as the user input is received.

The IOT server 101 may include an IOT automated planner 111 or a thing controller 113. [

Upon receiving the user input (e.g., utterance), the dialog system 104 can confirm the operation of the device related to the received user input and output information for performing the confirmed operation through the electronic device. For example, the user input may be transmitted to the system (e.g., IOT server 101) via an application (e.g., Messenger App) installed on a sensor or an electronic device included in the electronic device.

The IOT server 101 transmits a user input received through the dialog system 104 to the IOT automated planner 111 (e.g., user utterance, user input, device status trigger) 112, .

The IOT automated planner 111 may be included in the IOT server 101 or may operate as a separate configuration from the IOT server 101.

According to various embodiments of the present invention, the IOT automated planner 111 can perform an action planning for grasping an electronic device or an operation to be controlled from a user input and performing the identified apparatus or operations. For example, the Action planning may include an operation sequence between devices or an operation sequence between operations.

In accordance with various embodiments of the present invention, a task is a unit of performing an action, such as an action corresponding to a user input, a mandatory action required to perform the action, or other optional actions ) Can be specified according to the operation order. For example, the required actions may include actions that need to be performed before or after a particular action in order for the action corresponding to the user input to be performed.

According to various embodiments of the present invention, the IOT automated planner 111 may output an Optional Action related to the performed Action and the user's intention through an electronic device. For example, the Optional Action may include an item for operations that can be performed in association with user input.

Information on items selected by the user may be stored in the system as Preference / Pattern, and in the Optional Action when performing the action planning, the stored information is information on the devices and operations preferred by the user . The electronic device may control to execute the previously stored information (for example, information for performing roughness lowering and noise control related operation) when performing an action planning (e.g., movie viewing), or may output the selected information as a user selectable item.

2 is a diagram illustrating an operation for receiving a user input between a server and an electronic device and establishing a plan based on the received input, in accordance with various embodiments of the present invention.

2, an IoT system may include an IoT control device (e.g., dialog manager 202) or a server (e.g., action planner 201).

The dialog manager 202 may send the user's confirmation of the user's intention or recommendation action corresponding to the user input to the action planner 201. [

The action planner 201 may confirm at least one operation in which a user's preference is confirmed based on a previously input user input or an acknowledgment, and transmit the confirmed operation to the electronic device as a recommendation operation.

The recommending action (for example, the roughness related to the movie viewing and the option for performing the cleaner) may be output to the execution screen of the application included in the electronic device.

3 is a flow diagram illustrating an example of an operation for creating and performing a plan for controlling IoT devices, in accordance with various embodiments of the present invention.

Referring to FIG. 3, in operation 310, the server may receive the user's intent / command data via the electronic device.

In operation 320, the server may automatically perform operations according to the transmitted intent and command. Priority and parameters are determined based on the existing plan and Personal Service DB, and the server can control the IoT device so that at least one operation confirmed by the operation according to the intention and the command is automatically performed.

For example, when there is a plan having a schedule of a certain priority or more, the server can control the corresponding operation to be performed automatically.

As a result of performing the above-described operation 320, if the operation to be performed automatically is not confirmed, the server in the 321 operation can provide a plurality of plans associated with the user's intent / command through the user's electronic device. For example, when the plan to be automatically executed is not confirmed, it may include a case where the priority is less than or equal to a predetermined priority, or a case where priority between plans is matched.

In operation 323, the server may confirm that selection of at least one of the plans provided through the user's electronic device is received and perform the selected plan.

If the operation to be automatically performed is confirmed as a result of the above-described operation 320, the server can determine whether a plan according to the usage history (e.g., pattern / preference) can be generated.

As a result of performing the above-described 330 operation, if the usage history is confirmed, the server in 340 operation can determine whether a user's opinion on the plan selection is needed.

In operation 350, the server may verify user feedback (e.g., a selection action or voice feedback for a user's specific action).

In operation 360, the server can update the personal service DB of the user using the result of user selection.

In operation 370, the server may generate a plan with a preset default value and perform the generated plan.

If the use history is not confirmed as a result of performing the above-described operation 330, or if the user's opinion is not required as a result of performing the operation 240 described above, the operation 370 may also be performed. 4 is a diagram illustrating an example of a database structure in accordance with various embodiments of the present invention.

4, a database may include a task 410, a precondition 420, or a sub-task 430 to recommend actions that can be performed according to user input. The task 410 may include information on operations that can be performed by a specific device related to user input, based on capabilities or attributes of each device.

Each of the task 410, precondition 420 or sub-task 430 may include information related to the intent of the device information or user utterance data (e.g., device context 441), personalized service or object information (e.g., personal KB 442), or a combination of at least one or at least two of the commonly specified motion information (e.g., common sense 443) for an object.

The precondition may include condition information for performing the task.

The sub-task may include detailed operation information configuring the task.

The sub-task may include information of a mandatory operation (e.g., a first operator 451), which is information of a predetermined operation for a specific operation, or an operation (e.g., a second operator 452) As shown in FIG. The mandatory or optional information may include information on execution performance, precondition information (e.g., 451a or 452a) for performing the performance operation, or effects information (e.g., 451b or 452b) .

5 is a flow diagram illustrating an example of operation for controlling an IoT device in accordance with user input, in accordance with various embodiments of the present invention.

Referring to FIG. 5, in operation 510, the user 501 may input the firing data (e. G., "Hot now") via the electronic device. The electronic device may send the input speech data to a system (e.g., IoT server) 502.

In operation 520, the system 502 may generate a plan associated with the utterance data (e. G., Lowering the room temperature) and control operations to perform the generated plan (e. G., Heater off or windows open) . For example, the generated plan may be output through the user's electronic device via voice data (e.g., turn off the heater and open the window) or through the application launch screen.

In operation 530, the user 501 may input a confirmation input (e. G., "Yes") for the plan through the electronic device and the system performs an operation on the generated plan as the user confirmation input is received (E.g., "have performed an operation") to inform the IoT devices that they are in control at 540 operation via the electronic device.

According to various embodiments of the present invention, in the system, the IoT devices may store a controlled history and process user input subsequently entered based on the stored history.

For example, at 550 operation, the user may again input the firing data (e.g., "is it hot?") Through the electronic device. The electronic device may transmit the input speech data to the system.

The system may consider the stored history to generate a plan associated with the input speech data and to determine actions to perform the generated plan. For example, the system can generate a plan including the operation of controlling the air conditioner among the IoT devices in association with the utterance data, and can confirm whether the operation relating to the air conditioner control is performed on the stored history.

For example, if the stored history includes an operation related to air conditioner control, "open window" is included, the system can control the air conditioner and recommend control of the window to the user.

In operation 560, the system may perform an action according to user input (e.g., "I will turn on the air conditioner a moment") through the electronic device and a recommendation action associated with the stored history (e.g., Can you? ").

In operation 570, when the user enters a confirmation input (e.g., "yes") for performing the action and recommending action, the electronic device may send the confirmation input to the system.

In response to the confirmation input, in 580 operation, the system may control the output of data (e.g., "Performed action") indicating that the action is performed through the electronic device. The type of data output to the electronic device may be voice, text, or image data.

6 is a flow diagram illustrating an example of operation for controlling an IoT device in accordance with user input, in accordance with various embodiments of the present invention.

Referring to FIG. 6, in the system, it is recommended to check the status information and perform the operation of the IoT device as a user based on the confirmed status information.

The system checks weather information among the status information, and if it is determined that the amount of fine dust is out of a preset range, the system can confirm the IoT device related to air cleaning and the operations that can be performed by the device and recommend it through the electronic device.

In operation 610, the system may output data informing the electronic device of a recommended action (e.g., "Would you like to turn on the air purifier now?").

In operation 620, the electronic device may receive an acknowledgment for the recommendation operation. For example, the electronic device may analyze the input data (e.g., "yes") entered from the user to determine whether the operation is an affirmative or negative response to the operation.

If an affirmative response (e.g., "yes") is received for the recommendation operation, the system may control the associated IoT device (e.g., air purifier) to perform the recommendation operation.

In operation 630, the system 602 may output data (e.g., "Performed Action") indicating that the referral operation is performed through the electronic device of the user 601. [

Thereafter, in operation 640, user's utterance data (e.g., "I want to listen to music ...") to control the IoT device may be received.

In operation 650, the system performs an action (e.g., TV off or speaker playback) that can be performed in association with the received speech data, and data that recommends actions (e.g., window closing) , "I want to turn off the TV now and play the song with speakers, can I close the window?").

In operation 660, the system may store the received information upon receipt of a user ' s acknowledgment (e.g., "NO") to the recommendation data.

For example, if the user inputs a negative response (e.g., "NO") to the recommendation data, the system 602 stores the negative response in the personalization data as a negative response to the recommendation data, Can be confirmed by information for determining the preference.

7 is a flow diagram illustrating an example of an operation for configuring a database for controlling an IoT device in a server, in accordance with various embodiments of the present invention.

Referring to FIG. 7, in operation 710, the server may register at least one IoT device.

The at least one IoT device may include devices that are located within a certain distance from a hub device registered in the server or that can be determined by the hub device, and the electronic device registered in the server and the electronic device Controllable IoT devices may be included.

The server may include a thing controller, and in the 720 operation, the thing controller can check the data that a particular IoT device can perform from the thing database.

In operation 730, the thing controller may configure a personal thing database based on the identified data. For example, the thing controller may identify a combination of the IoT device and operation information corresponding to the received user input among the operations that the specific IoT device can perform, and may store the combination information as personalization information for the corresponding user.

8 is a flow diagram illustrating an example of an operation for establishing and performing a plan for controlling an IoT device at a server, in accordance with various embodiments of the present invention.

According to various embodiments of the present invention, the server may include at least one of a goal generator, a plan generator, a plan ranker, a decision maker, a plan executor, and a personalizer. When at least one of the configurations is omitted or changed, it is possible to perform the operation of the configurations in one configuration.

The server may deliver command information to the goal generator for the particular IoT device (e.g., air conditioner) corresponding to the user input (e.g., "cool") based on user input (e.g., utterance data) .

The goal generator may set a goal based on the command information. For example, the goal generator may be set to target the operation of adjusting the room temperature in response to the 'cool down' input.

The plan generator can establish a plan for the order between the IoT devices and the operations to be controlled based on the set goals. For example, the plan generator may identify an IoT device (e.g., a window, a fan or a heater) associated with the operation to be controlled (e.g., air conditioner) and the operation of the associated IoT device You can establish a plan to carry out.

The plan ranker can determine the ranking of established plans. For example, the order may be set as an air conditioner, a window, an electric fan, and a heater in order according to a goal of 'room temperature control'.

The decision maker can inquire the user whether to perform the established plans. For example, for an operation identified as a recommendation operation (e.g., heater off) with respect to the control operation (e.g., air conditioner), the decision maker may send data (e. Yes, "I want to turn off the heater, what plan do you like?") In the form of voice or text data.

The plan executor can check the established plan and control the corresponding IoT devices so that the actions included in the plan are performed.

The personalizer may store the results of the plan execution and execution in the personal thing / service database of the server. Also, the personalizer may store the user's acknowledgment of the recommendation operation, and may confirm the IoT device or the operation order to be controlled in the plan establishment operation.

For example, the personal thing database stores information about the IoT devices controlled or related to a specific user or the operation of the corresponding IoT devices, and the personal service database stores planning information related to a specific target (e.g., temperature control) , A window, a fan or a heater) can be stored.

9 to 14 are diagrams showing various examples of a database for controlling the IoT device.

Referring to FIG. 9, the thing database 900 may include information of all IoT devices that can be controlled by the IoT server. The information of the IoT device includes the product name (e.g., Samsng LED F7100 Series TV 910a or Hue bulb 910a), device type (e.g., TV 920a or bulb 920b), IoT device (940a, 940b), which is state information of each capability, or command information 950a, which is command information related to each capability, have.

According to various embodiments of the present invention, the state of 'audio control' of attributes may be changed as 'setVolume' is set among the commands included in the TV. For example, a value set according to the 'setVolume' may be included in the 'number' field of 'volume' included in the 'audio control'.

Referring to FIG. 10, the ontology database 1000 may include information on operations to be performed before or after a specific operation.

The ontology database 1000 includes an operation 1010 for driving a heater, a heater.set.temperature operation 1020, an aircondition.set.temperature operation 1030, 1040, 1040, 1040, 1060, 1060, 1060, or 1060) of the air conditioner 1040, before or after the air conditioner 1040, And may include information of operations.

For example, when 'cold' is input as a user input, the server confirms to control the heater among the IoT devices for temperature control, and the temperature control (e.g., heater.set.temperature) operation 1010 (E.g., "heater.turnned.on (swich: on)" operation) or a post condition that is to be performed subsequently to perform a pre- Information on the operation (e.g., temperature value setting ("parameter: number") operation) can be confirmed through the ontology database.

According to various embodiments of the present invention, the precondition or post condition may include information about specific actions and command information associated with the action.

When a target according to user input is set in the server, a sequence of operations stored in the ontology database and a condition of the stored operation are checked to generate a combination of operations, and an operation sequence can be planned according to the combination.

Referring to FIG. 11, the service database 1100 may store service information related to a specific user input. For example, the service information is information for providing a service corresponding to the user input (for example, setting the room temperature to a temperature that the user feels cool 1110 or air cleaner driving 1120) And the performance information of the IoT device to be controlled (e.g., "performedby"). The performance information may include a condition for performing the corresponding operation, a sequence of operations among the operations, or a relation indicating whether the operation is necessary for the service provision.

The relation may be set to 'required' indicating execution information to be performed for essential service provision or 'suggestion' indicating execution information recommended in connection with a specific service provision.

For example, if a user input (e.g., "cool down") is entered, the server checks the temperature control service (e.g., 1110) associated with the user input and checks the 'temperature' , And the value of the parameter (for example, 25 degrees) set in advance can be confirmed.

In accordance with various embodiments of the present invention, the server may identify specific performance information (e.g., an air circulation operation ("make.aircirculation")) and determine whether the condition of the performance information (eg, external temperature <internal temperature) And determine whether to perform the operation according to a relationship with the service (e.g., suggestion). For example, when the provision of a specific service is requested, the server can control to perform an operation corresponding to performance information set to 'suggestion' or 'required' in relation to the service.

The server can check the order information and control the corresponding operation to be performed in accordance with the set order.

Referring to FIG. 12, the personal service database 1200 may include a service database for a specific user. For example, a service database for a first user may be included and the first user may set a temperature to a particular value (e.g., 23 degrees) for a particular user input (e.g., "cool" When air circulates to the temperature, changes in air status, or fans, the order for each operation and operation may be stored in the personal service database 1200.

The personal service database 1200 may be included in the service database 1100 as service information of the first user.

In the information included in the personal service database, a parameter value for a specific operation is changed based on a user input of a first user (for example, the number of negative response inputs) and a user's preference according to a plan execution , The priority of the operations can be set.

Referring to FIG. 13, the personal thing database 1300 may include IoT device information associated with a particular user. For example, information about each device may be stored in the personal thing database 1300 based on the thing database 900 as information of the first user's electronic device or an accountable IoT device.

The device information stored in the personal thing database 1300 includes information such as a thing ID, a product name 1310, a device type 1320, owner 1330, capabilities 1340, and attributes 1350 information.

When a specific IoT device is operated as a user input is received, an attribute value may be updated in the personal thing database 1300 based on operation information of the IoT device.

Referring to FIG. 14, the plan 1400 may include operation information between devices executed to perform a set target.

The plan may be stored for each execution plan (e.g., execute plan1 1410 or execute plan2 1420), and each execution plan (e.g. execute plan1 1410) includes plan identification information (ID) 1411, And a plan step (1412) indicating the order between the two. For example, in the case of an execution plan for controlling temperature (eg, execute plan1), the plan step may be airconditioning (eg, "aircondition.turn.on"), air conditioning temperature setting (E.g., "temperature (23)") and an operation to turn off the air conditioner (e.g., "aircondition.turn.off") and data representative of each operation.

The plan executor of the server can control to perform the corresponding operations in order according to the plan step based on the information stored in the plan.

Figure 15 is a diagram illustrating an example of a system for controlling an IoT device according to user preferences, in accordance with various embodiments of the present invention.

15, a routine according to a specific situation is provided in an IoT device (e.g., smart TV) 1501 of the IoT system 1500. When each routine is selected, May be performed in at least one IoT device 1502 of the IoT system 1500. For example, when a specific routine is selected, the operations set to mandatory execution are performed, and the operations set to the recommendation execution can be output through the IoT device as items that the user can select.

The routine includes a 'Good morning' routine for specifying operations to be performed in the IOT apparatus 1502 in the morning, a 'Good night' routine for specifying operations to be performed in the pre-IOT apparatus 1502, a pre-outgoing IoT apparatus 1502 I 'back' routine that specifies actions to be performed by the IoT device 1502 when the user comes home after going out or a 'Goodbye' routine that specifies actions to be performed by the user You can create routines in combination.

In accordance with various embodiments of the present invention, when a new IoT device is identified, the server can check the routine information associated with the new IoT device and output it as a user selectable item as a recommendation run when the routine is selected .

According to various embodiments of the present invention, if the previously identified IoT device is not located in a pre-set range or the connection is not verified, the server checks the routine containing the IoT device and the connection is not confirmed in the routine IoT devices can be deleted.

According to various embodiments of the present invention, a user may perform an updated routine to add or remove a corresponding IoT device entry by the server without performing additional operations or additions to the newly connected or disconnected IoT device Can be confirmed.

16 is a diagram illustrating an example of an execution screen of an IoT device that recommends related operations according to various embodiments of the present invention.

16, an execution screen 1600 output from an IoT device (e.g., TV) includes an item (e.g., 1611) for selecting at least one predetermined routine, a recommended action list 1602 ) Or buttons 1625 for selecting and storing recommendation actions.

If the user selects an item (e.g., 1611) for a particular routine (e.g., 'good night') on the execution screen 1600, the operations included in the selected routine may be performed in the IoT device.

Based on the data contained in the service database or the things database of the database, the server can check the information of the operations that can be performed with respect to the specific routine. For example, when the 'play movie' routine is selected, it is possible to identify actions related to lighting settings (eg, 'light off') or volume settings (eg, user Bluetooth speacker) (E.g., 1621, 1622, 1623, or 1624) through an IoT device (e.g., TV).

In the server, the 'play movie' routine 1611 may be selected, and the information of the operation performed by the user more than a predetermined number of times or performed over a predetermined time may be checked and output as a selection item. In the case of an operation (e.g., play video) 1623 performed more than the preset number of times, the video can be output through the IoT device (e.g., TV) in a state where a visual effect (e.g., highlight) is displayed.

17 is a diagram showing an example of an execution screen of an IoT device for controlling the display of the operations included in the routine according to the addition or deletion of the IoT device, according to various embodiments of the present invention.

17, an execution screen 1700 output from the IoT device (e.g., TV) includes an item (e.g., 1711 or 1712) for selecting at least one predetermined routine, (1720), or buttons (1725) for selecting and storing recommendation actions.

When the server confirms that the hue light is found as a new IoT device, it can confirm the routine related to the hue light. For example, if an item of a routine (e.g., 1711) associated with the newly discovered hue light is selected, the server displays an action in the hue light (e.g., dim down ) 1721 may be included. In addition, for the hue light recommendation operation, an indication indicating that it is newly installed ('added hue' phrase addition or highlight processing) may be included and output.

Based on the data contained in the service database or the things database of the database, the server can check the information of the operations that can be performed with respect to the specific routine.

The server may indicate to the item of the associated routine that it is updateable (e.g., 'U') (eg, 1711) or may indicate that the routine is updateable (eg, 'N') .

For example, when a specific routine is selected according to a user's input, in the case of a routine (e.g., 'automatic light') in which the user performs all operations without selecting a specific operation, (E.g., 1712) may include an 'N' indication.

For example, in the case of a routine (e.g., 'play movie') for performing a user-selected operation when a specific routine is selected according to a user's input, a selection item (e.g., 1711) Quot; U &quot; may be included.

18 is a diagram illustrating various examples of situations in which IoT devices are controlled, according to various embodiments of the present invention.

Referring to FIG. 18, when the same user input is inputted to two assumptions (for example, Tom 1801 and Mary 1802), various operations can be performed according to the configuration of the IoT device installed in the home.

Tom's home may include an illumination 1814, a window 1815, two air conditioners 1811a and 1811b, a TV 1813, or a robot cleaner 1812a as IoT devices.

Mary's home may include an illumination 1824, a window 1825, an air conditioner 1821, or a speaker 1827 as an IoT device.

In accordance with various embodiments of the present invention, the user may input the speech data corresponding to 'Good morning' through the electronic device to execute the 'Good morning' routine 1803, A selection item corresponding to a specific routine can be selected through the selection button 1804.

In accordance with various embodiments of the present invention, Tom's home 1801 may control lighting 1814 according to the indoor lighting condition (e.g., darkness) at Tom's wake-up time as the 'Good morning' , The lighting is turned on), the window 1815 is kept closed in accordance with the weather condition (e.g., rain), the first air conditioner 1811a (or the air purifier) is executed in accordance with the window condition , And the second air conditioner 1811b can be executed according to the external temperature (for example, 32 degrees). The server may also be controlled to power on the TV 1813 at a specific time in Tom's home 1801.

On the other hand, in the Mary's home 1802, the lighting 1824 is controlled (e.g., kept turned off or turned off) according to the indoor lighting state (e.g., brightness) ), It is possible to output data recommending the operation related to the window control through the mary's electronic device. In addition, the server may be controlled so that the speaker 1827 is turned on at a specific time (e.g., at the time of the weather) in the Mary's home 1802, and the predetermined music is reproduced.

19 is a diagram showing an example of an execution screen of an application recommending the operation of the IoT device according to user input according to various embodiments of the present invention.

Referring to FIG. 19, the application may analyze the user's utterance data and output the analyzed result as operation information of the IoT device that can perform the analysis. For example, the execution screen 1900 of the application may include an area 1910 in which the user's inputted data and an input of the server are output, a recommended action list 1920 related to the inputted input data, And a button 1930 for activating the microphone of the electronic device to activate the microphone.

For example, a user may input utterance data (e.g., 'play movie') through the application, and the electronic device in which the application is executed may include a list of recommended operations including the IoT device associated with the utterance data, 1920).

According to various embodiments of the present invention, the recommendation action list 1920 may include information 1924, 1924, or 1923, or an item 1924 that indicates whether to perform a particular action . The information on the recommendation operation may be information on the location (e.g., living room) where the IoT device is installed, the type of IoT device (e.g., robot cleaner or bulb) . &Lt; / RTI &gt; Entry 1924 may include selections to select whether to perform (e.g., "go ahead!") Or deny (eg, no thanks) at least one recommended action selected.

When the user selects at least one of the output operation information, the electronic device transmits the selected operation information to the server, and the server controls the IoT device associated with the selected operation information to perform an operation selected by the user can do.

20 is a flowchart illustrating an example of an operation of controlling an IoT device by confirming operation time information from a user input according to various embodiments of the present invention.

The user can input action command data (e.g., data input via speech data or UI) for performing a specific operation together with operation time information through the electronic device. The user can input the utterance data corresponding to the data, or select a selection item related to a specific operation from the execution screen of the electronic apparatus, and input the command data. The input data may be transmitted to the server through an electronic device.

Referring to FIG. 20, in the 2010 operation, the server can confirm that the user inputs action instruction data including operation time and setting information through the electronic device.

In operation 2020, the server receives data input from the electronic device, and confirms the operation time and operation information to be performed by the user in the received data.

Table 1 relates to speech data including the operation time and examples of operations corresponding to corresponding speech data, according to various embodiments of the present invention.

Run time
Setting Type Ignition action Time (minutes / day) Give me music for 30 minutes . 1) Music Play
2) After 30 minutes Stop Temperature I need air conditioning until 24 degrees . 1) Air conditioner on / cooling
2) When the temperature reaches 24 ℃ through real-time sensing, the air conditioner off calendar Take out mode during summer vacation 1) Detect summer vacation in calendar and start date out mode on
2) Switch to coming home mode on the last day shower Give me the music while I shower . 1) When entering the bathroom, play music
2) Music Stop movie Reduce the brightness of the bulb while watching the movie . 1) TV video play detect to reduce bulb brightness
2) Increase bulb brightness by stopping TV video

In operation 2030, the server may determine the end time of the operation from the operation time, perform the inactive action as the end operation, and register the inactive action as the end operation with the end time in the scheduler application . In the scheduler application, when an end time for a specific operation has elapsed, an inactive action of the operation can be controlled to be performed. The inactive action may be deactivated when the corresponding action is performed, and may be performed to terminate the specific action.

In operation 2040, when the server confirms the operation time from the user's utterance data, the server sets an inactive action to be performed after the operation time and can perform an active action.

Table 2 relates to targets and active and inactive actions for the targets, according to various embodiments of the present invention.

Task Name Active action Inactive action Play music Speaker.play Speaker.stop Make air cool Airconditioner.coolmode.activate Airconditioner.power.off Dim down Bulb.dim Bulb.brighten Activate away mode away mode Coming home mode Activate keep mode Refrigerator.keepmode.activate Refrigerator.notify.expiredTime

Referring to Table 2, when performing 'play music' as an example of a task, a speaker, which is an example of the IoT device, can operate. The server can check the operation time included in the user's utterance data and control the operation of the speaker (play) through a speaker as an active action during the confirmed operation time. When the determined operation time has elapsed, the server may control the operation of the music playback stop (speaker.stop) through the speaker as an inactive action.

In 2050 operation, the server can determine whether the termination of an active action is detected.

Table 3 relates to a kind of operation, a start operation, a start operation detection method, a termination operation and a termination operation detection method in association with a specific operation, according to various embodiments of the present invention.

Type start Start detection method End Termination detection method Time (minutes / day) Now Save current time Current time + Set time After calculating the end time, timer management Temperature Current temperature temperature Senser Set temperature Comparison of current temperature and set temperature with temperature sensor calendar schedule
Start date and time Calendar sync search schedule
End date and time Calendar sync shower Shower start Detect motion sensor in bathroom Shower end Detect motion sensor bathroom out movie Start Movie TV video play Movie ends TV video stop

Referring to Table 3, in the case of the operation related to the temperature control, the type of operation is time (minute / day), the server controls to perform the operation of setting the time, and when the time It can be determined that the operation is completed. As a result of the determination, if the active action is performed during the operation time included in the user's utterance data, the server can confirm that the active action is completed.

In the 2060 operation, the server can perform an inactive action if it is determined that the active action is terminated.

According to various embodiments of the present invention, active actions and inactive actions may be mapped for each action and stored in the ontology database of the server.

21 is a diagram showing an example of an execution screen of an application recommending tasks in relation to an IoT device in an electronic device, according to various embodiments of the present invention.

Referring to FIG. 21, the schedule management application may include a menu for generating a schedule for a specific operation. When the menu is selected, an execution window 2101 displays an input window 2111 for inputting information related to schedule, An item 2112 for setting a start and end time of a schedule, and an item 2113 for setting whether a schedule is repeated.

In the electronic device, the user input can be received through the input window 2111 of the schedule management application. For example, when text data is inputted through the input window 2111, the electronic device can confirm the IoT device associated with the input text data. The electronic device may transmit the input text data to the server and request the server to confirm the IoT device corresponding to the data.

The data input through the input window may be various types of data including information on a task or routine to be performed by the user such as voice data and biometric information data.

According to various embodiments of the present invention, the electronic device or server can determine the type of data, such as IoT devices, tasks, routines, etc., by checking the text data being input.

According to various embodiments of the present invention, an electronic device or server can identify the type of data of the input text data and recommend an IoT device, job or routine associated with the input text. The recommended data may be displayed in a partial area of the execution screen in the form of a list, or may be output in a separate screen. Also, based on the information of the IoT device and the work previously stored in the user account or the server, the work related to the inputted text date can be recommended.

For example, when the user inputs 'air' through the input window, the electronic device or the server judges that the 'air' corresponds to the 'air conditioner' of the IoT device and the 'air conditioner' Air conditioner cooling or air conditioner air cleaning) can be controlled to be output through the electronic device.

For example, when the user inputs the 'mode' through the input window, the electronic device or the server determines that the 'mode' corresponds to at least one of the preset routines, and the related routines (eg, Mode, movie mode, cleaning mode, sleep mode, etc.) 2114b is output through the electronic device. For example, if one of the recommended routines is selected, pre-configured IoT devices and tasks for that routine may be performed.

According to various embodiments of the present invention, a user may create a schedule comprising at least one of the recommended IoT devices, tasks or routines.

The schedule may include information about a task or a routine performed at a specific date (e.g., August 3, 2015), and the information about the task and the routine may be classified according to an operation time. For example, if one of the stored tasks and routines (e.g., 2121) is selected, a screen 2122 containing detailed information about the task and the routine may be output through the electronic device.

The screen 2122 including the detailed information may include an operation state for a specific task and a routine, a selection item for asking whether to perform a predetermined operation, and information on an additional configurable operation. If one of the additional setting information is selected, the corresponding IoT device can perform the selected operation after the operation being performed is terminated.

22 is a diagram showing an example of an execution screen of an application that generates a schedule with tasks recommended in association with an IoT device in an electronic device, according to various embodiments of the present invention.

Referring to FIG. 22, an execution screen 2201 may include items for generating a schedule for a specific job. For example, the execution screen includes an input window 2231 for inputting a schedule title (e.g., a puppy lunch), an item 2232 for setting a start or end date of a schedule, and an item 2233 for setting a repetition .

The user can input 'dog' in the input window to enter a schedule title. In the electronic device or server, it is possible to control the output of the list 2231a of the tasks related to the input 'puppy' (for example, 'puppy feed' or 'puppy feeder') through the electronic device. The list of recommended jobs 2231a may be displayed adjacent to the input window or displayed on a separate screen.

According to various embodiments of the present invention, when one of the recommended jobs is selected, detailed information or operation time for performing the job can be set. The task information includes at least one task name (eg, puppy feed), a device type (eg feed automatic feeder), an active action (eg, food distribution), an inactive action, (For example, a first sub-action: provided as an amount of food corresponding to the third step), or a task type indicating whether or not the task is repeated (for example, a one-time operation or repeatedly performed every day at a specific time) And various information related to the operation method.

According to various embodiments of the present invention, when the user creates the 'puppy feed' job as a schedule, the electronic device generates a schedule 2241 of the job at a date corresponding to the operation time of the job, ) Can be displayed corresponding to the corresponding date. For example, when at least one schedule is created, the electronic device or the server can control the IoT devices associated with the job so that the job is performed at a time corresponding to the operation time of the specific job.

According to various embodiments of the present invention, when the user selects one of the generated schedules (e.g., 2241), the electronic device may output the operational status for the selected schedule. For example, if it is confirmed that the schedule 2241 of the 'puppy feed' is selected, the electronic device outputs data (e.g., "Feed automatic feeding machine . The output data may be output as image data on a screen that is additionally displayed on the execution screen, or output as voice data through a speaker.

23 is a diagram showing an example of an execution screen of an application that generates a schedule from a routine related to an IoT device in an electronic device, in accordance with various embodiments of the present invention.

Referring to FIG. 23, an execution screen 2301 may include items for generating a schedule for a specific routine. For example, the schedule management application may include a menu for generating a schedule for a specific operation. When the menu is selected, an execution window 2301 displays an input window 2311 for inputting information related to the schedule, An item 2312 for setting an end time, and an item 2313 for setting a predetermined iteration.

For example, according to a user input (e.g., an operation of tapping the input window 2311) inputted through the execution screen 2301, a list of IoT devices that can be set in association with a corresponding routine (e.g., sleep mode) And a screen 2311a including items for changing the settings of the respective IoT devices.

The user can change the settings of the IoT device and the respective devices for the specific routine through the execution screen 2301, set the operation start or end time, and store the schedule.

According to various embodiments of the present invention, the stored schedule is stored in a date corresponding to the operation time, and in the electronic device, the stored schedule information may be outputted as a date-specific schedule screen 2302 through the schedule management application . For example, the schedule screen 2302 may indicate that the schedule 2321 is stored on a date corresponding to the predetermined operation time (e.g., August 3, 2015).

According to various embodiments of the present invention, the electronic device and the server are configured to check the current time, perform a routine in which an operation start time is set corresponding to the current time, and at least one Lt; RTI ID = 0.0 &gt; IoT &lt; / RTI &gt;

According to various embodiments of the present invention, when the user selects a routine (e. G., Sleep mode) 2321 that is stored as a schedule through the scheduling application, the electronic device may display the state of the selected routine in a portion of the execution screen Or through a screen 2321a displayed separately. The screen 2321a in which the state of the routine is output is displayed on the display unit 2321a when an item for selecting whether to terminate the routine (e.g., "sleep mode is ended " Do you want to run the morning mode? ") Or an item to select whether to perform at least one IoT device or operation pre-designated to perform the routine.

24 is a diagram illustrating an example of a network environment, in accordance with various embodiments of the present invention.

24, the network environment 2400 includes an electronic device 2401, at least one electronic device (e.g., a first electronic device 2402 or a second electronic device 2404) or a server 2406 Each of which may be connected via a network 2462 or connected to the electronic device 2401 via a communication module 2470 of the electronic device 2401. [

The electronic device 2401 may include a bus 2410, a processor 2420, a memory 2430, an input / output interface 2450, a display 2460, and a communication interface 2470. In some embodiments, the electronic device 2401 may omit at least one of the components or additionally include other components.

Bus 2410 may include circuitry, for example, to connect components 2410-2470 to one another and to communicate communications (e.g., control messages and / or data) between the components.

Processor 2420 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). Processor 2420 may perform computations or data processing related to, for example, control and / or communication of at least one other component of electronic device 2401. [

Memory 2430 may include volatile and / or non-volatile memory. The memory 2430 may store instructions or data related to at least one other component of the electronic device 2401, for example. According to one embodiment, the memory 2430 may store software and / or programs 2440. [ The program 2440 may include one or more of the following: a kernel 2441, a middleware 2443, an application programming interface (API) 2445, and / or an application program . &Lt; / RTI &gt; At least a portion of the kernel 2441, middleware 2443, or API 2445 may be referred to as an operating system (OS).

The kernel 2441 may include system resources used to execute an operation or function implemented in other programs (e.g., middleware 2443, API 2445, or application program 2447) : Bus 2410, processor 2420, or memory 2430). The kernel 2441 also provides an interface to control or manage system resources by accessing individual components of the electronic device 2401 in the middleware 2443, API 2445, or application program 2447 .

Middleware 2443 can act as an intermediary for API 2445 or application program 2447 to communicate with kernel 2441 to exchange data, for example.

Middleware 2443 may also process the one or more task requests received from application program 2447 in priority order. For example, middleware 2443 may use system resources (e.g., bus 2410, processor 2420, or memory 2430) of electronic device 2401 in at least one of application programs 2447 Priority can be given. For example, the middleware 2443 can perform the scheduling or load balancing of the one or more task requests by processing the one or more task requests according to the priority assigned to the at least one task.

The API 2445 is an interface for the application 2447 to control the functions provided by the kernel 2441 or the middleware 2443, for example, Control or the like, for example, instructions.

The input / output interface 2450 may serve as an interface through which commands or data input from, for example, a user or other external device can be transferred to another component (s) of the electronic device 2401. [ The input / output interface 2450 can also output commands or data received from other component (s) of the electronic device 2401 to a user or other external device.

Display 2460 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) A microelectromechanical systems (MEMS) display, or an electronic paper display. Display 2460 may display various content (e.g., text, image, video, icon, or symbol, etc.) to a user, for example. Display 2460 may include a touch screen and may receive touch, gesture, proximity, or hovering input, for example, using an electronic pen or a portion of a user's body.

Communication interface 2470 may be used to establish communication between electronic device 2401 and an external device such as first external electronic device 2402, second external electronic device 2404, or server 2406 . For example, communication interface 2470 may be connected to network 2462 via wireless or wired communication to communicate with an external device (e.g., second external electronic device 2404 or server 2406).

Wireless communications may include, for example, cellular communication protocols such as long-term evolution (LTE), LTE Advance (LTE), code division multiple access (CDMA), wideband CDMA (WCDMA) mobile telecommunications system, WiBro (Wireless Broadband), or Global System for Mobile Communications (GSM). The wireless communication may also include, for example, local communication 2464. The short range communication 2464 may include at least one of, for example, wireless fidelity (WiFi), Bluetooth, near field communication (NFC), or global navigation satellite system (GNSS). GNSS can be classified into two types according to the use area or bandwidth, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (Beidou) And may include at least one. Hereinafter, in this document, "GPS" can be interchangeably used with "GNSS ". The wired communication may include at least one of, for example, a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), or plain old telephone service (POTS). Network 2462 may include at least one of a telecommunications network, e.g., a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 2402 and 2404 may be the same or a different kind of device as the electronic device 2401. [ According to one embodiment, the server 2406 may include one or more groups of servers. According to various embodiments, all or a portion of the operations performed in the electronic device 2401 may be performed in one or more other electronic devices (e.g., electronic devices 2402, 2404, or server 2406). According to the example, in the event that the electronic device 2401 has to perform some function or service automatically or on demand, the electronic device 2401 may be capable of executing at least (E. G., Electronic device 2402, 2404, or server 2406) may request some functionality from other devices (e. G., Electronic devices 2402 and 2404 or server 2406) Function or an additional function and transmit the result to the electronic device 2401. The electronic device 2401 can directly or additionally process the received result to provide the requested function or service. Yes For example, cloud computing, distributed computing, or client-server computing technology may be utilized.

25 is a block diagram illustrating an example of an electronic device configuration according to various embodiments of the present invention.

Referring to Fig. 25, the electronic device 2501 may include all or part of the electronic device 2401 shown in Fig. 24, for example. The electronic device 2501 may include one or more processors (e.g., an application processor (AP)) 2510, a communication module 2520, a subscriber identification module 2524, a memory 2530, a sensor module 2540, ), A display 2560, an interface 2570, an audio module 2580, a camera module 2591, a power management module 2595, a battery 2596, an indicator 2597, and a motor 2598 have.

The processor 2510 may, for example, control an operating system or application programs to control a plurality of hardware or software components coupled to the processor 2510, and may perform various data processing and operations. The processor 2510 may be implemented with, for example, a system on chip (SoC). According to one embodiment, the processor 2510 may further include a graphics processing unit (GPU) and / or an image signal processor. Processor 2510 may include at least some of the components shown in FIG. 25 (e.g., cellular module 2521). Processor 2510 may load and process instructions or data received from at least one of the other components (e.g., non-volatile memory) into volatile memory and store the various data in non-volatile memory have.

The communication module 2520 may have the same or similar configuration as the communication interface 2470 of Fig. The communication module 2520 includes a cellular module 2521, a WiFi module 2523, a Bluetooth module 2525, a GNSS module 2527 (e.g., a GPS module, a Glonass module, a Beidou module, or a Galileo module) An NFC module 2528, and a radio frequency (RF) module 2529.

The cellular module 2521 may provide voice, video, text, or Internet services, for example, over a communication network. According to one embodiment, cellular module 2521 may utilize a subscriber identity module (e.g., a SIM card) 2524 to perform the identification and authentication of electronic device 2501 within the communication network. According to one embodiment, the cellular module 2521 may perform at least some of the functions that the processor 2510 may provide. According to one embodiment, the cellular module 2521 may include a communication processor (CP).

Each of the WiFi module 2523, the Bluetooth module 2525, the GNSS module 2527 or the NFC module 2528 may include a processor for processing data transmitted and received through the corresponding module, for example. At least some (e.g., two or more) of the cellular module 2521, the WiFi module 2523, the Bluetooth module 2525, the GNSS module 2527, or the NFC module 2528, according to some embodiments, (IC) or an IC package.

The RF module 2529 can, for example, send and receive communication signals (e.g., RF signals). RF module 2529 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 2521, the WiFi module 2523, the Bluetooth module 2525, the GNSS module 2527, or the NFC module 2528 transmits / receives an RF signal through a separate RF module .

The subscriber identity module 2524 may include, for example, a card containing a subscriber identity module and / or an embedded SIM and may include unique identification information (e.g., an integrated circuit card identifier (ICCID) Subscriber information (e.g., international mobile subscriber identity (IMSI)).

The memory 2530 (e.g., memory 2430) may include, for example, an internal memory 2532 or an external memory 2534. The built-in memory 2532 may be a volatile memory such as a dynamic RAM (DRAM), a static random access memory (SRAM), or a synchronous dynamic RAM (SDRAM), a non-volatile memory Programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash) A hard drive, or a solid state drive (SSD).

The external memory 2534 may be a flash drive such as a compact flash (CF), a secure digital (SD), a micro secure digital (SD), a mini secure digital (SD) digital, a multi-media card (MMC), a memory stick, and the like. The external memory 2534 can be functionally and / or physically connected to the electronic device 2501 via various interfaces.

The sensor module 2540 may, for example, measure a physical quantity or sense the operating state of the electronic device 2501 and convert the measured or sensed information into an electrical signal. The sensor module 2540 includes a gesture sensor 2540A, a gyro sensor 2540B, an air pressure sensor 2540C, a magnetic sensor 2540D, an acceleration sensor 2540E, a grip sensor 2540F, 2540G, a color sensor 2540H (e.g. an RGB (red, green, blue) sensor), a biosensor 2540I, a temperature / humidity sensor 2540J, a luminance sensor 2540K, ) Sensor 2540L or a touch sensor 2540M. Additionally or alternatively, the sensor module 2540 may include, for example, an E-nose sensor, an electromyography sensor, an electroencephalogram sensor, an electrocardiogram sensor, , An infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 2540 may further include a control circuit for controlling at least one or more sensors belonging thereto. In some embodiments, the electronic device 2501 further includes a processor configured to control the sensor module 2540, either as part of the processor 2510 or separately, so that while the processor 2510 is in a sleep state, The sensor module 2540 can be controlled.

The input device 2550 may include, for example, a touch panel 2552, a (digital) pen sensor 2554, a key 2556, or an ultrasonic input device 2558). As the touch panel 2552, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type can be used. Further, the touch panel 2552 may further include a control circuit. The touch panel 2552 may further include a tactile layer to provide a tactile response to the user.

(Digital) pen sensor 2554 may be part of, for example, a touch panel or may include a separate recognition sheet. Key 2556 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 2558 can sense the ultrasonic wave generated by the input tool through the microphone (e.g., the microphone 2588) and confirm the data corresponding to the sensed ultrasonic wave.

Display 2560 (e.g., display 2460) may include a panel 2562, a hologram device 2564, or a projector 2566. Panel 2562 may comprise the same or similar configuration as display 2460 of FIG. The panel 2562 can be embodied, for example, flexible, transparent, or wearable. The panel 2562 may be composed of one module with the touch panel 2552. [ The hologram device 2564 can display stereoscopic images in the air using the interference of light. The projector 2566 can display an image by projecting light onto a screen. The screen may, for example, be located inside or outside the electronic device 2501. According to one embodiment, the display 2560 may further include control circuitry for controlling the panel 2562, the hologram device 2564, or the projector 2566.

The interface 2570 may be a digital interface such as, for example, a high-definition multimedia interface (HDMI) 2572, a universal serial bus (USB) 2574, an optical interface 2576, or a D- ) &Lt; / RTI &gt; The interface 2570 may be included in the communication interface 2470 shown in Fig. 24, for example. Additionally or alternatively, the interface 2570 may be, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card / multi-media card (MMC) data association standard interface.

The audio module 2580 can, for example, convert sound and electrical signals in both directions. At least some of the components of the audio module 2580 may be included in, for example, the input / output interface 2445 shown in Fig. The audio module 2580 may process sound information input or output through, for example, a speaker 2582, a receiver 2584, an earphone 2586, a microphone 2588, or the like.

The camera module 2591 is a device capable of capturing, for example, a still image and a moving image. According to one embodiment, the camera module 2591 includes at least one image sensor (e.g., a front sensor or a rear sensor) , Or a flash (e.g., an LED or xenon lamp, etc.).

The power management module 2595 can manage the power of the electronic device 2501, for example. According to one embodiment, the power management module 2595 may include a power management integrated circuit (PMIC), a charger integrated circuit (PMIC), or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging scheme may include, for example, a magnetic resonance scheme, a magnetic induction scheme, or an electromagnetic wave scheme, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 2596, the voltage during charging, the current, or the temperature. The battery 2596 may include, for example, a rechargeable battery and / or a solar battery.

The indicator 2597 may indicate a particular state of the electronic device 2501 or a portion thereof (e.g., processor 2510), such as a boot state, a message state, or a state of charge. The motor 2598 can convert the electrical signal into mechanical vibration, and can generate vibration, haptic effects, and the like. Although not shown, the electronic device 2501 may include a processing unit (e.g., a GPU) for mobile TV support. The processing unit for supporting the mobile TV can process media data conforming to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or media flow ( ^TM ).

Each of the components described in the present invention may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device.

In various embodiments of the present invention, an electronic device may comprise at least one of the components described in the present invention, some components may be omitted, or may further include additional other components. In addition, some of the components of the electronic device according to various embodiments of the present invention may be combined into one entity, so that the functions of the components before being combined can be performed in the same manner.

26 is a block diagram illustrating an example of a program module structure according to various embodiments of the present invention.

26, program module 2610 (e.g., program 140) includes an operating system (OS) that controls resources associated with an electronic device (e.g., electronic device 2401) and / (E.g., application programs 147) running on the system.

The operating system may be, for example, android, iOS, windows, symbian, tizen, or bada.

The program module 2610 may include a kernel 2620, a middleware 2630, an application programming interface (API) 2660, and / or an application 2670. At least a portion of the program module 2610 may be preloaded on the electronic device or may be downloaded from an external electronic device such as the electronic device 2402 or 2404 or the server 2406 or the like.

The kernel 2620 (e.g., the kernel 141) may include, for example, a system resource manager 2621 and / or a device driver 2623. The system resource manager 2621 can perform control, allocation, or recovery of system resources. According to one embodiment, the system resource manager 2621 may include a process management unit, a memory management unit, or a file system management unit. The device driver 2623 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication .

The middleware 2630 may provide various functions commonly required by the application 2670 or may be provided through the API 2660 in various ways to allow the application 2670 to efficiently use limited system resources within the electronic device. Functions may be provided to the application 2670. According to one embodiment, the middleware 2630 (e.g., middleware 143) includes a runtime library 2635, an application manager 2641, a window manager 2642, a multimedia manager A data manager 2643, a resource manager 2644, a power manager 2645, a database manager 2646, a package manager 2647, a connectivity manager 2643, 2642, a notification manager 2649, a location manager 2650, a graphic manager 2651, or a security manager 2652 can do.

The runtime library 2635 may include, for example, a library module that the compiler uses to add new functionality through a programming language while the application 2670 is running. The runtime library 2635 may perform input / output management, memory management, or functions for arithmetic functions.

The application manager 2641 can manage the life cycle of at least one of the applications 2670, for example. The window manager 2642 can manage GUI resources used on the screen. The multimedia manager 2643 can recognize the format required for playing various media files and can encode or decode the media file using a codec suitable for the format. The resource manager 2644 can manage resources such as source code, memory or storage space of at least one of the applications 2670.

The power manager 2645 operates in conjunction with a basic input / output system (BIOS), for example, to manage a battery or a power source, and to provide power information and the like necessary for the operation of the electronic device. The database manager 2646 may create, retrieve, or modify a database for use in at least one of the applications 2670. The package manager 2647 can manage installation or update of an application distributed in the form of a package file.

The connection manager 2648 can manage wireless connections, such as, for example, WiFi or Bluetooth. The notification manager 2649 may display or notify events such as arrival messages, appointments, proximity notifications, etc. in a manner that does not interfere with the user. The location manager 2650 can manage the location information of the electronic device. The graphic manager 2651 may manage a graphical effect to be presented to the user or a user interface associated therewith. The security manager 2652 can provide all security functions necessary for system security or user authentication. According to one embodiment, when an electronic device (e.g., electronic device 2401) includes a telephone function, middleware 2630 further includes a telephony manager for managing the voice or video calling capabilities of the electronic device can do.

Middleware 2630 may include a middleware module that forms a combination of various functions of the above-described components. The middleware 2630 can provide a module specialized for each type of operating system in order to provide differentiated functions. Middleware 2630 may also dynamically delete some existing components or add new ones.

API 2660 (e.g., API 145) may be provided in a different configuration, for example, as a set of API programming functions, depending on the operating system. For example, for Android or iOS, you can provide one API set per platform, and for tizen, you can provide more than two API sets per platform.

An application 2670 (e.g., an application program 147) may include, for example, a home 2671, a dialer 2672, an SMS / MMS 2673, an instant message 2674, a browser 2675, A camera 2676, an alarm 2677, a contact 2678, a voice dial 2679, an email 2680, a calendar 2681, a media player 2682, an album 2683 or a clock 2684, or one or more applications capable of performing functions such as health care (e.g., measuring exercise or blood glucose), or providing environmental information (e.g., providing atmospheric pressure, humidity, or temperature information, etc.).

According to one embodiment, an application 2670 is an application that supports the exchange of information between an electronic device (e.g., electronic device 2401) and an external electronic device (e.g., electronic devices 2402 and 2404) For convenience, an "information exchange application"). The information exchange application may include, for example, a notification relay application for communicating specific information to an external electronic device, or a device management application for managing an external electronic device.

For example, the notification delivery application may send notification information generated in other applications (e.g., SMS / MMS applications, email applications, healthcare applications, or environmental information applications) of the electronic device to external electronic devices , 2404). Further, the notification delivery application can receive notification information from, for example, an external electronic device and provide it to the user.

The device management application may be capable of performing at least one function (e.g., a turn-on or turn-off) of the external electronic device itself (or some component) of an external electronic device (e.g., electronic device 2402, 2404) (E.g., on / off-off, or adjusting the brightness (or resolution) of the display), managing applications (e.g., , Or updated).

According to one embodiment, the application 2670 may include an application (e.g., a healthcare application of a mobile medical device, etc.) designated according to an attribute of an external electronic device (e.g., electronic device 2402, 2404). According to one embodiment, the application 2670 may include an application received from an external electronic device (e.g., server 2406 or electronic devices 2402 and 2404). According to one embodiment, May include a preloaded application or a third party application downloadable from the server. The names of the components of the program module 2610 according to the illustrated embodiment may include a type of operating system Therefore, it can be changed.

According to various embodiments of the invention, at least some of the program modules 2610 may be implemented in software, firmware, hardware, or a combination of at least two of them. At least some of the program modules 2610 may be implemented (e.g., executed) by, for example, a processor (e.g., processor 2510). At least some of the program modules 2610 may include, for example, modules, programs, routines, sets of instructions or processes, etc., to perform one or more functions.

27 is a flowchart illustrating an example of an operation of registering an IoT device in a server according to various embodiments of the present invention.

Referring to FIG. 27, in operation 2710, the server may register the information of the IOT device to be used by the individual in the system.

In 2720 operation, the server can check related specifications such as operation, plan, or operation that specific IoT device can perform from Thing Database.

In operation 2730, the server may configure the Personal Thing Database using the verified data.

In 2740 operation, the server can monitor the status of registered IoT devices. For example, you can check the updated information as a result of monitoring and update the database associated with the IoT device.

28 is a flow diagram illustrating an example of an operation for performing a plan in a server, in accordance with various embodiments of the present invention.

28, in 2810 operation, the server may receive commands (e.g., 'turn on air conditioning') from user utterances or UI inputs.

In the 2820 operation, the server may set a Goal (eg, an air conditioner).

In 2830 operation, the server can check for possible plans (eg, turn on air conditioning (a plan) or open windows).

In 2840 operation, the server can determine the priority of the plan.

In 2850 operation, the server may determine a plan (e.g., a plan) to perform among the determined plans if a plurality of plans are determined.

In 2860 operation, the server can perform the determined plan (eg, turn on the air conditioner).

29 is a flow diagram illustrating an example of an operation to perform a plan in a server, in accordance with various embodiments of the present invention.

29, in operation 2910, the server may receive command data (e.g., turn on the air conditioner) from user utterance or UI input.

In 2920 operation, the server can set Goal (eg, air conditioner).

In 2930 operation, the server can establish a possible plan. For example, the possible plans may include a plan (close window and turn on air conditioner) or plan b (turn on air conditioner).

In 2940 operation, the server can rank the plan. For example, the priority of each plan is 0.10, and the same situation can be assumed.

In the 2950 operation, the server can check if the priority of the plans to be performed is low, and includes a recommendation action (eg, do you want to close the window)?

In 2960 operation, the server can perform the determined plan.

In 2970 operation, the server can reflect the plan execution and the results in the personalization database. For example, in the personalization database, information of a high frequency of closing a window when a specific user turns on the air conditioner is stored, and the priority of closing the window can be raised and stored.

30 is a flow diagram illustrating an example of an operation to perform a plan in a server, in accordance with various embodiments of the present invention.

Referring to FIG. 30, in operation 3010, the server may receive command data (e.g., turn on the air conditioner) from user utterance or UI input.

In 3020 operation, the server may set Goal (eg, air conditioner).

In 3030 operation, the server can establish an actionable plan. For example, the possible plans may include a plan (close window and turn on air conditioner) or plan b (turn on air conditioner).

In 3040 operation, the server can rank between plans. The priority of the a plan can be set to 0.8, and the priority of the plan b can be set to 0.1.

In the 3050 operation, the server can control to automatically perform a plan having a high priority or a predetermined priority greater than a predetermined value.

In operation 3060, the server may perform a determined plan and control that data (e.g., closing the window and turning on the air conditioner) to be performed is output through the electronic device.

31 is a flow diagram illustrating an example of an operation to perform a plan in a server, in accordance with various embodiments of the present invention.

Referring to FIG. 31, in operation 3110, the server can confirm the user's intention from user utterance (e.g., cool) or from the UI.

In 3120 operation, the server may set a Goal (e.g., a room temperature of 25 degrees).

In 3130 operation, the server can establish a plan and prioritize. For example, set a ranking of 0.3 for a plan (turn on air conditioning -> room temperature to 25 degrees), b to rank b plan (turn heater off -> turn on air conditioning -> room temperature to 25 degrees) Set to 0.3, set the order to 0.1 for c plan (close window -> window when the room temperature is 25 degrees), d plan (turn the fan on -> turn off the fan when the room temperature drops) To 0.1, and the e-plan (turn on the fan and air conditioner -> turn off both when the temperature is low) can be set to 0.1.

In the 3140 operation, the server can ask the user if user feedback is required (low priority difference) among the plans to be performed. For example, the server may output the data of " Would you like to turn on the air conditioner? Turn off the heater? &Quot; through the electronic device and determine whether to perform the affirmative response (e.g., yes) .

In 3150 operation, the server can perform the determined plan (eg, plan b).

In the 3160 operation, the server can reflect the results of the execution in the personalization database. For example, the server may store a high priority on a plan to turn on the air conditioner and turn off the heater, depending on the frequency with which the user turns off the heater when the user turns on the air conditioner in the personalization database.

32 is a flow diagram illustrating an example of an operation to perform a plan at a server, in accordance with various embodiments of the present invention.

Referring to FIG. 32, in operation 3210, the server can confirm the user's intention from user utterance (e.g., cool) or from the UI.

In 3220 operation, the server may set a Goal (e.g., a room temperature of 25 degrees).

In the 3230 operation, the server can establish a plan and determine a priority. For example, set a ranking of 0.3 for a plan (turn on air conditioning -> room temperature to 25 degrees), b to rank b plan (turn heater off -> turn on air conditioning -> room temperature to 25 degrees) Set to 0.8, set the order to 0.1 for c plan (close window -> window when the room temperature is 25 degrees), d plan (turn off the fan -> turn off the fan when the room temperature drops) To 0.1, and the e-plan (turn on the fan and air conditioner -> turn off both when the temperature is low) can be set to 0.1.

In the 3240 operation, the server can select a plan having a high priority among plans to be executed. In addition, the server can automatically perform plans (eg, plan b) with a priority higher than a predetermined value.

In 3250 operation, the server can perform a determined plan (e.g., plan b).

33 is a flowchart illustrating an example of an operation of registering an IoT device in a server according to various embodiments of the present invention.

Referring to FIG. 33, in operation 3310, the server can confirm the user's intention from user utterance (e.g., cool) or from the UI.

In the 3320 operation, the server may set a Goal (e.g., a room temperature of 25 degrees).

In operation 3330, the server can establish a plan and determine a priority. The plan is: a Plan (turn on the air conditioner -> adjust the room temperature to 25 degrees), b plan (turn off the heater -> turn on the air conditioner -> adjust the room temperature to 25 degrees), c plan Closing the window when the temperature is 25 ° C), plan d (turn the fan on -> turn off the fan when room temperature drops), e plan (turn off the fan and air conditioner -> turn off both temperatures) have.

In the 3340 operation, the server may select a plan that does not have preference information from among the plans to be performed.

In 3350 operation, the server may perform a determined plan (e.g., plan b).

The term "module" as used herein may mean a unit comprising, for example, one or a combination of two or more of hardware, software or firmware. A "module" may be interchangeably used with terms such as, for example, unit, logic, logical block, component, or circuit. A "module" may be a minimum unit or a portion of an integrally constructed component. A "module" may be a minimum unit or a portion thereof that performs one or more functions. "Modules" may be implemented either mechanically or electronically. For example, a "module" may be an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable-logic devices And may include at least one.

At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments of the present invention may be, for example, a computer-readable storage media). When the instruction is executed by a processor (e.g., processor 120), the one or more processors may perform a function corresponding to the instruction. The computer readable storage medium may be, for example, memory 130. [

The computer readable recording medium may be a hard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), an optical media (e.g., a compact disc read only memory (CD-ROM) digital versatile discs, magneto-optical media such as floptical disks, hardware devices such as read only memory (ROM), random access memory (RAM) Etc. The program instructions may also include machine language code such as those produced by the compiler, as well as high-level language code that may be executed by a computer using an interpreter, etc. The above- May be configured to operate as one or more software modules to perform the operations of various embodiments of the invention, and vice versa.

Modules or program modules according to various embodiments of the present invention may include at least one or more of the elements described above, some of which may be omitted, or may further include other additional elements.

Operations performed by modules, program modules, or other components in accordance with various embodiments of the invention may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. Accordingly, the scope of the present invention should be construed as including all modifications or various other embodiments based on the technical idea of the present invention.

Claims (1)

In the server,
A communication unit;
Memory; And
And a processor,
Wherein the processor receives user input from at least one external electronic device via the communication unit,
Generating an action plan corresponding to the received user input based on object information stored in the memory and operation information of each object,
And to control at least one object corresponding to the received user input such that at least one action according to the generated plan is performed.

Images