KR102429334B1 - An Implicature-Based Interaction Model to Modify Automation Rules in IoT Environments - Google Patents

Abstract

An implicit-based interaction method and device for modifying automation rules in an IoT environment are presented. The method proposed by the present invention includes the steps of: recognizing a user command by converting a user command in voice form into text using a natural language processing model by a user command recognizing unit, and then extracting identifiers including object, operation, parameter, and location to recognize the user command; if the user command is recognized by a user command recognizing unit, providing the recognized content to a user terminal through a control unit, and controlling to display a message to the user terminal to add or stop the user command; taking the user command without redundant phrases for a previous IoT service called through mashup storage according to a message to add or stop the user command so as to iteratively add user commands, combine IoT services, and create and store mashups; if a recognition error occurs in a generated mashup, performing recovering from an error by canceling the execution of the corresponding user command without restarting the entire conversation through an error correction unit; and generating feedback for implicit confirmation through a feedback generation unit to inspect suggestion confirmation, and progress confirmation of the IoT service related to the recognized user command to check the need for implicit confirmation, suggestion confirmation, and progress confirmation, and confirmation, and delivering the generated feedback to the user terminal if necessary. IoT service mashups can be configured using simple and intuitive user commands.

Description

An Implicature-Based Interaction Model to Modify Automation Rules in IoT Environments

The present invention relates to an implication-based interaction method and apparatus for modifying automation rules in an IoT environment.

In the Internet of Things (IoT) environment, such as smart homes and smart offices, users aim to achieve their goals by accessing and personalizing the functions of multiple IoT devices. This requires the ability to combine the capabilities of IoT devices from their point of view at a high level without considering low level technical details [1]. Some prior technologies have proposed an IoT service mashup method that allows users to combine the functions of IoT devices into advanced services [2, 3, 4, 5]. The most common approach for users to create and manage IoT service mashups is through a visual interface. Here, each IoT service is represented by a visual component such as an icon or block, and the execution order is specified as a connection flow between IoT services or a simplified rule.

Recently, there is an increasing demand from users to manage IoT service mashups through voice-enabled IoT devices such as smart speakers [6, 7, 8]. Users want an easy way of traditional visual approaches when they need to create, modify and remove service mashups in their daily life [6]. In some prior art, the use of interactive agent has been proposed for the needs of interactive IoT service mashup [8, 9]. Conversational agents are natural language-based interaction systems such as Alexa1 or Google Assistant2. Users can interact with interactive agents to control simple functions of IoT services in natural language [8, 10].

On the other hand, according to the recent prior art, about 1/3 of the IoT service mashups created by skilled users in the smart home environment connect three or more IoT devices or services [11]. Existing interactive agents for IoT service mashups rely on a single utterance approach, where the interaction between the user and the agent mainly consists of one request command and one response. However, when constructing such complex mashups using existing agents, a single command is not sufficient to express complex user needs, which must be met by coordinating multiple IoT services with subsequent actions. Therefore, in order for a user to create an IoT service mashup consisting of three or more IoT services, it is necessary to support the user using an interactive agent.

In addition to the problem of the single utterance approach, there is another problem that existing agents do not support modifying IoT service mashups. Modifying the generated mashup is common for users of IoT environments because they are not professional developers [12, 13, 14]. Because users refine the mashup through an iterative process of modifying the mashup, it is important to support the modification of the created IoT service mashup. However, existing agents do not support modifications as they are considered challenging in terms of usability. In order to modify the mashup, it was considered essential for the user to listen to a list of currently distributed mashups and select one of them through an interactive interface [15]. This approach is inefficient for the user and has a usability problem that imposes a large cognitive load on the user [16]. Therefore, it is necessary to support the modification of the IoT service mashup while minimizing the length of unnecessary interactions, such as listening to unnecessary mashup information.

In the Internet of Things (IoT) environment, such as smart homes and smart offices, users aim to achieve their goals by accessing and personalizing the functions of multiple IoT devices. This requires the ability to combine the capabilities of IoT devices from their point of view at a high level without considering low level technical details [1]. Some prior technologies have proposed an IoT service mashup method that allows users to combine the functions of IoT devices into advanced services [2, 3, 4, 5]. The most common approach for users to create and manage IoT service mashups is through a visual interface. Here, each IoT service is represented by a visual component such as an icon or block, and the execution order is specified as a connection flow between IoT services or a simplified rule.

Recently, there is an increasing demand from users to manage IoT service mashups through voice-enabled IoT devices such as smart speakers [6, 7, 8]. Users want an easy way of traditional visual approaches when they need to create, modify and remove service mashups in their daily life [6]. In some prior art, the use of interactive agent has been proposed for the needs of interactive IoT service mashup [8, 9]. Conversational agents are natural language-based interaction systems such as Alexa1 or Google Assistant2. Users can interact with interactive agents to control simple functions of IoT services in natural language [8, 10].

On the other hand, according to the recent prior art, about 1/3 of the IoT service mashups created by skilled users in the smart home environment connect three or more IoT devices or services [11]. Existing interactive agents for IoT service mashups rely on a single utterance approach, where the interaction between the user and the agent mainly consists of one request command and one response. However, when constructing such complex mashups using existing agents, a single command is not sufficient to express complex user needs, which must be met by coordinating multiple IoT services with subsequent actions. Therefore, in order for a user to create an IoT service mashup consisting of three or more IoT services, it is necessary to support the user using an interactive agent.

In addition to the problem of the single utterance approach, there is another problem that traditional agents do not support modifying IoT service mashups. Modifying the generated mashup is common for users in IoT environments because they are not professional developers [12, 13, 14]. Because users refine the mashup through an iterative process of modifying the mashup, it is important to support the modification of the created IoT service mashup. However, existing agents do not support modifications as they are considered challenging in terms of usability. In order to modify a mashup, it was considered essential for the user to listen to a list of currently distributed mashups and select one of them through an interactive interface [15]. This approach is inefficient for the user and has a usability problem that imposes a large cognitive load on the user [16]. Therefore, it is necessary to support the modification of the IoT service mashup while minimizing the length of unnecessary interactions, such as listening to unnecessary mashup information.

In one aspect, the implication-based interaction method for modifying an automation rule in the IoT environment proposed by the present invention is an object, operation after a user command recognition unit converts a user command in voice form into text using a natural language processing model. , extracting each identifier including a parameter and a location to recognize a user command, when the user command is recognized by the user command recognition unit, the recognized content is provided to the user terminal through the control unit, and further user commands are added or Controlling a stop message to be displayed on the user terminal, adding more user commands or adding user commands repeatedly by adding user commands without overlapping phrases for the previous IoT service called through the mashup storage unit according to the message to stop The steps to create and save a mashup by combining IoT services, and to recover from the error by undoing the execution of the user command without restarting the entire conversation through the error correction unit when a recognition error occurs in the created mashup and feedback Through the generator, feedback for implicit confirmation of the IoT service regarding the recognized user command, proposal confirmation, and progress confirmation is generated to check whether there is a need for implicit confirmation, proposal confirmation and progress confirmation, and confirmation is required and transmitting the generated feedback to the user terminal in case of case.

If a recognition error occurs in the generated mashup, the step of recovering from the error by canceling the execution of the user command without restarting the entire conversation through the error correction unit is to classify the mashup that is currently deployed in the IoT environment and needs correction localizing to the mashup and performing an interaction to modify the classified mashup.

The step of localizing to classify a mashup that is deployed in the current IoT environment and needs modification is to search for a mashup by utilizing hints about user utterances, including utterance types, commands, expressions, and questions, and perform interactions. localizes the mashup in a usable manner through do.

The suggestion-based backtracking process additionally provides a device keyword for searching for a mashup using a specific IoT device in user speech, and infers hints about IoT environment properties to be changed from user speech input through a natural language processing model Inferring a list of IoT action services that affect users by backtracking the ontology including the mapping between the IoT environment property to be changed and the action of a specific IoT device, and querying the recently executed IoT service mashup history to select the corresponding mashup Searches, prioritizes using call timestamps, and presents to users according to the priority of the mashup.

The step of localizing to classify the mashup that is deployed in the current IoT environment and needs to be modified is started by a user's command or a user's utterance, and returns a hint about the user's utterance to infer the characteristics of the current IoT environment. Then, according to the trigger information of the search result of the IoT service mashup that affected the current IoT environment characteristics, the modification step or the reason for executing the action is asked, and if the search result is not satisfactory, another mashup is searched, or Alternatively, by going back to previous results, it minimizes unnecessary information through the interaction of implicit-based traceback process and localization.

The step of performing the interaction to modify the classified mashup may include receiving a user command for modifying the classified mashup and replacing the service type, parameter, keyword, location to modify, and mashup for the user command for modifying the mashup. or add a new mashup component or replace a pre-established mashup component or parameter according to the required information including the addition.

Based on a plurality of command patterns including adding a new trigger for the user command, adding a new action, replacing an existing trigger, replacing an existing action, modifying a trigger parameter, modifying an action parameter, and deleting a service mashup, the service type of the user command is a parameter Whether to set, increment or decrement, and if there are components that use the parameter in the mashup to modify it, and if no user command to modify an existing parameter can be found, where to modify and replace the mashup to modify it. Or, by checking whether to add or not, the ambiguity of essential information including service types, parameters, keywords, locations to modify, and substitution or addition of mashups for user commands for modifying the mashup is improved.

Through the feedback generating unit, by generating feedback for implicit confirmation of the IoT service regarding the recognized user command, confirmation of proposals, and confirmation of progress, check whether there is a need for implicit confirmation, confirmation of proposals, and confirmation of progress, and confirm If this is necessary, the step of delivering the generated feedback to the user terminal checks whether implicit confirmation, which is the natural language form of the recognized IoT service, is necessary, and when the interaction counter reaches a predetermined number, the IoT service is added according to the user command If the number of interaction counters is greater than a predetermined number and the mashup creation is completed and the user's utterance is finished, it is checked whether the suggestion confirmation is necessary, and the progress confirmation is checked so that the user can track the mashup progress. , when confirmation is required, the generated feedback is transmitted to the user terminal.

In another aspect, the implication-based interaction device for modifying an automation rule in the IoT environment proposed by the present invention converts a user command in a voice form into text using a natural language processing model, A user command recognition unit for recognizing a user command by extracting each identifier including a parameter and a location, when the user command is recognized by the user command recognition unit, provides the recognized contents to the user terminal, and adds or stops the user command A control unit that controls to display a message to the user terminal, adding more user commands or adding user commands without overlapping phrases to the previous IoT service called according to a message to stop repeatedly adding user commands and running IoT services A mashup storage unit that creates and stores a service mashup by combining, an error correction unit that recovers from the error by undoing the execution of the user command without restarting the entire conversation when a recognition error occurs in the generated mashup, and the recognized user command By generating feedback for implicit confirmation, proposal confirmation, and progress confirmation of IoT service regarding It includes a feedback generator to transmit.

When a plurality of IoT services are connected through an implicit-based interaction model for modifying automation rules in an IoT environment according to embodiments of the present invention, a user can configure an IoT service mashup using a simple and intuitive user command, , it is possible to avoid redundant syntax in user commands, and to achieve simplification of the information passed in the response. It can also assist in maintaining mashup creation progress without incurring significant cognitive load on the user in the interaction of the mashup creation.

1 is a flowchart illustrating an implicit-based interaction method for modifying an automation rule in an IoT environment according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of an implicit-based interaction device for modifying an automation rule in an IoT environment according to an embodiment of the present invention.
3 is a schematic diagram for explaining the interaction process of CoMMA according to an embodiment of the present invention.
4 is a diagram for explaining the concept of a suggestion-based traceback process according to an embodiment of the present invention.
5 is a diagram for explaining an example of a suggestion-based traceback process according to an embodiment of the present invention.
6 is a diagram for explaining the interaction of localization according to an embodiment of the present invention.
7 is a diagram for explaining an example of an interaction for searching for another mashup or returning to a previous mashup in the localization process according to an embodiment of the present invention.
8 is a diagram illustrating a command pattern for modifying a mashup according to an embodiment of the present invention.
9 is a diagram for explaining a problem of ambiguity of a user command according to an embodiment of the present invention.
10 is a view for explaining a process of clarifying a user command according to an embodiment of the present invention.
11 is a view for explaining a process of performing an interaction to modify a mashup according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating an implicit-based interaction method for modifying an automation rule in an IoT environment according to an embodiment of the present invention.

The implication-based interaction method for modifying an automation rule in an IoT environment according to an embodiment of the present invention is performed through a Conversational, Multi-dialog Mashup Agent (ComMA), which is a multi-conversational method.

In the proposed IoT environment, the implication-based interaction method for modifying the automation rules is that the user command recognition unit converts the user command in the form of speech into text using a natural language processing model, and each Step 110 of recognizing the user command by extracting the identifier of the user command, when the user command is recognized by the user command recognition unit, the recognized contents are provided to the user terminal through the control unit, and a message to add or stop the user command is sent to the user Step 120 of controlling to be displayed on the terminal, adding user commands repeatedly by adding user commands without overlapping phrases with respect to the previous IoT service called through the mashup storage unit according to a message to add or stop more user commands, Creating and saving a mashup by combining IoT services (130), when a recognition error occurs in the created mashup, recovering from the error by canceling the execution of the user command without restarting the entire conversation through the error correction unit ( 140) and through the feedback generator, generate feedback for implicit confirmation of the IoT service regarding the recognized user command, confirmation of proposals, and confirmation of progress, and check whether there is a need for implicit confirmation, confirmation of proposals, and confirmation of progress, and , when confirmation is required, transmitting the generated feedback to the user terminal (150).

In step 110, the user command recognition unit using a natural language processing (NLP) model 112 to convert the user commands (user commands) 111 in the form of speech into text objects (Object), Recognize the user command by extracting each identifier (Identifiers) 113 including the operation (Operation), parameter (Parameter) and position (Position).

When the user command is recognized by the user command recognition unit, the content recognized in step 120 is provided to the user terminal through the control unit, and a message to add or stop the user command is controlled to be displayed on the user terminal.

In step 130, according to the message to add or stop more user commands, by adding user commands without overlapping phrases to the previous IoT service called through the mashup storage, the user command is repeatedly added, and the IoT service is combined. Create and save the mashup.

When a recognition error occurs in the generated mashup, the error correction unit does not restart the entire conversation through the error correction unit, but cancels the execution of the user command to recover from the error.

Step 140 includes localizing to classify a mashup that is currently deployed in the IoT environment and needs modification, and performing an interaction to modify the classified mashup.

The step of localizing to classify a mashup that is deployed in the current IoT environment and needs modification is to search for a mashup by utilizing hints about user utterances, including utterance types, commands, expressions, and questions, and perform interactions. localizes the mashup in a usable manner through do.

The suggestion-based backtracking process additionally provides a device keyword for searching for a mashup using a specific IoT device in user speech, and infers hints about IoT environment properties to be changed from user speech input through a natural language processing model do. After that, backtrack the ontology including the mapping between the IoT environment property to be changed and the action of a specific IoT device to infer a list of IoT action services that affect users, and retrieve the recently executed IoT service mashup history to determine the corresponding mashup. Searches, prioritizes using call timestamps, and presents to users according to the priority of the mashup.

The step of localizing to classify the mashup that is deployed in the current IoT environment and needs to be modified is started by a user's command or a user's utterance, and returns a hint about the user's utterance to infer the characteristics of the current IoT environment. and the reason for performing the modification step or executing the action according to the trigger information of the search result of the IoT service mashup that has affected the current IoT environment characteristic is asked. Minimize unnecessary information through the interaction of a hint-based traceback process and localization, either by retrieving another mashup if the search results are not satisfactory, or by returning to the previous result.

The step of performing the interaction to modify the classified mashup may include receiving a user command for modifying the classified mashup, and replacing the service type, parameter, keyword, location to modify, and mashup for the user command for modifying the mashup. Or add a new mashup component or replace a pre-established mashup component or parameter according to the required information including the addition.

Based on a plurality of command patterns including adding a new trigger to the user command, adding a new action, replacing an existing trigger, replacing an existing action, modifying a trigger parameter, and modifying an action parameter, the service type of the user command sets and increases the parameter Or whether there is a component that uses the parameter in the mashup to reduce and modify it. If a user command to modify an existing parameter cannot be found, the service type, parameter, keyword, location to modify and the mashup for the user command for modifying the mashup by checking the location to modify and whether to replace or add the mashup to modify. Remedies ambiguity of essential information, including substitutions or additions.

In step 150, feedback for implicit confirmation of the IoT service regarding the recognized user command, proposal confirmation, and progress confirmation is generated through the feedback generator. The feedback generated includes feedback for tacit confirmation, feedback for suggestion confirmation and progress confirmation, and feedback for simple errors. Thereafter, it is checked whether there is a need for implicit confirmation, proposal confirmation, and progress confirmation (151), and when confirmation is necessary, generated feedback including the implicit confirmation, proposal confirmation and progress confirmation to the user terminal (Generated Feedback with) Check Suggestion). If confirmation is not required, only feedback on simple errors is provided.

More specifically, it is checked whether an implicit confirmation, which is a natural language form of a recognized IoT service, is required, and when the interaction counter reaches a predetermined number and an IoT service is added according to a user command, and when the interaction counter reaches a predetermined number After completion of mashup creation and before the end of user utterance, it is checked whether or not it is necessary to confirm the suggestion, and it is checked whether it is necessary to confirm the progress so that the user can track the progress of the mashup. provide feedback.

2 is a diagram illustrating a configuration of an implicit-based interaction device for modifying an automation rule in an IoT environment according to an embodiment of the present invention.

The configuration of an implicit interaction device for modifying an automation rule in an IoT environment according to an embodiment of the present invention may be included in a multi-conversational, multi-dialog mashup agent (ComMA).

Implication-based interaction devices for modifying automation rules in the proposed IoT environment are a User Command Indicator 210, a Controller 220, a Mashup Store 230, and an error It includes a correction unit (Error Corretor) 240 and a feedback generator (Feedback Generator) 250 .

The user command recognition unit 210 converts user commands in voice form into text using a Natural Language Pprocessing (NLP) model, and then converts the user commands into text objects, operations, and parameters. ) and position (Position) by extracting each identifier (Identifiers) to recognize the user command.

When the user command is recognized by the user command recognition unit 210, the control unit 220 provides the recognized content to the user terminal, and controls to display a message to add or stop more user commands to the user terminal.

The mashup storage unit 230 repeatedly adds user commands by adding user commands without overlapping phrases to the previous IoT service called according to a message to add or stop more user commands, and combines IoT services to create a mashup to save

When a recognition error occurs in the generated mashup, the error correction unit 240 recovers from the error by canceling the execution of the corresponding user command without restarting the entire conversation.

The error correction unit 240 performs localization to classify a mashup that is currently deployed in the IoT environment and requires correction, and performs interaction to correct the classified mashup.

The error correction unit 240 is currently deployed in the IoT environment and mashups by utilizing hints about user utterances including utterance types, commands, expressions, and questions to perform localization to classify mashups that require correction , localize the mashup in a usable way through interaction, infer the current IoT environment characteristic from the hint about the user utterance, and retrieve the IoT service mashup that affected the current IoT environment characteristic Perform an implicit-based traceback process.

The error correction unit 240 additionally provides a device keyword for searching for a mashup using a specific IoT device in a user's utterance through a hint-based backtracking process, and an IoT that wants to change from a user's utterance input through a natural language processing model Infer hints about environment properties. After that, backtrack the ontology including the mapping between the IoT environment property to be changed and the action of a specific IoT device to infer a list of IoT action services that affect users, and retrieve the recently executed IoT service mashup history to determine the corresponding mashup. Searches, prioritizes using call timestamps, and presents to users according to the priority of the mashup.

The error correction unit 240 is started by a user's command or a user's utterance, returns a hint about the user's utterance, infers the current IoT environment characteristic, and a search result of the IoT service mashup that affects the current IoT environment characteristic Depending on the trigger information of Minimize unnecessary information through the interaction of a hint-based traceback process and localization, either by retrieving another mashup if the search results are not satisfactory, or by returning to the previous result.

The error correction unit 240 receives a user command for correcting the classified mashup, and the service type for the user command for correcting the mashup, parameter, keyword, location to be modified, and replacement or addition of the mashup according to essential information including Add new mashup components or replace predefined mashup components or parameters.

The error correction unit 240 is based on a plurality of command patterns including a new trigger addition, a new action addition, an existing trigger replacement, an existing action replacement, a trigger parameter modification, and an action parameter modification for the user command, the service of the user command Check whether a type sets, increments or decrements a parameter and if there is a component that uses that parameter in the mashup to modify it. If a user command to modify an existing parameter cannot be found, the service type, parameter, keyword, location to modify and the mashup for the user command for modifying the mashup by checking the location to modify and whether to replace or add the mashup to modify. Remedies ambiguity of essential information, including substitutions or additions.

The feedback generator 250 generates feedback for implicit confirmation of the IoT service related to the recognized user command, confirmation of proposals, and confirmation of progress. The feedback generated includes feedback for tacit confirmation, feedback for suggestion confirmation and progress confirmation, and feedback for simple errors. Thereafter, it is checked whether there is a need for implicit confirmation, proposal confirmation, and progress confirmation, and when confirmation is required, generated feedback with Check Suggestion including the implicit confirmation, proposal confirmation and progress confirmation to the user terminal to convey If confirmation is not required, only feedback on simple errors is provided.

The feedback generator 250 checks whether implicit confirmation, which is a natural language form of the recognized IoT service, is required, and when the interaction counter reaches a predetermined number and an IoT service is added according to a user command, and the interaction counter is set in advance After the mashup creation is completed and the user's utterance is over, it is checked whether or not there is a need for confirmation of the proposal, and whether the confirmation is necessary so that the user can track the progress of the mashup. The generated feedback is transmitted.

3 is a schematic diagram for explaining the interaction process of CoMMA according to an embodiment of the present invention.

The traditional way of talking about IoT service mashups has taken the form of interacting with a single utterance. The problem with this limited form was that the user had to connect multiple IoT services and repeat the mashup creation process to remember the connections between the created mashups. In order to solve the problems of the existing approach, the following requirements through a Conversational, Multi-dialog Mashup Agent (ComMA) including an implicit-based interaction device for modifying automation rules in an IoT environment according to an embodiment of the present invention We want to satisfy:

[R1] When 3 or more IoT services are connected, users should be able to configure IoT service mashups using simple and intuitive commands.

[R2] Duplicate syntax in user commands must be avoided and the response must not contain too much information.

[R3] Interactions in mashup creation should support maintaining mashup creation progress without incurring significant cognitive load on the user.

Referring to FIG. 3 , the left side 310 is a series of speech baloons and represents the interaction between the user (or user terminal) 311 and the CoMMA 312 . The right 320 shows how user commands are recognized at each step, processed as identifiers, and converted into IoT services.

The iterative and incremental mashup model is a key feature of CoMMA, enabling users to construct multi-action IoT service mashups through the interaction of multiple dialog boxes. We first describe how user commands are converted from CoMMA to IoT services, and then describe the details of the features in two steps: starting to create a mashup and iterating to add a mashup.

An example of how a user command is converted into an IoT service is shown on the right side 320 of FIG. 3 . First, in the step of recognizing a user command (330), ComMA converts a voice command into text (331) using a natural language processing model, and then extracts displayed identifiers (Identifiers) (332). The identifier 332 may be classified into four types including an object, an operation, a parameter, and a position.

More specifically, object identifiers are used to figure out which IoT devices are needed or which attributes are to be controlled. Then, the operation identifier is used to identify the action required for the object. The parameter identifier is additional information required to provide an IoT service, and is a time (eg, 8:00 pm) for activating a time-related service. The location identifier is used to determine the execution order of the services in the mashup and whether the IoT service is a trigger or action type.

With a few exceptions, such as time-related services, most IoT services can be identified by a combination of object and operational identifier. Location identifiers are used to determine the order in which IoT services will be run, and these services are organized in order to create a mashup. For example, identifier 332 of FIG. 3 shows that the combination of "windows" and "close" is identified as "CloseWindowFrame Action", while "if" and "rain" are identified as "WeatherConditionsTrigger". The generated mashup is stored internally in the form of a directed graph where the node is an IoT service, and it increases incrementally as the user adds commands.

In the embodiment of the present invention, based on EUPont ontology [18] that triggers and categorizes IoT service types based on actual user data, IoT service types to be identified in ComMA are configured.

When the user command is recognized by the user command recognition unit, the recognized contents are then provided to the user terminal through the control unit, and a message to add or stop the user command is further controlled to be displayed on the user terminal.

Next, in the mashup creation step 350 according to the embodiment of the present invention, the first step of starting the mashup creation of CoMMA is to designate a trigger service, and the first action service is "trigger service→action service". Thus, a user using a single-utterance approach can easily move to a multi-talk approach.

Step 330 shows the user's interaction to create the mashup. If the input to the user command is correctly processed by ComMA, the recognized content is provided as feedback to the user terminal and a message to add or stop more commands is displayed. Then, in step 340, the user may repeatedly add commands to create a service mashup with multiple tasks.

In step 340 , using the repeat add feature, it is possible to combine additional IoT services after the previous IoT service called without duplicate phrases. In the present invention, ComMA is designed so that the user can designate one IoT service in the dialog box. Therefore, each user command in the multi-dialog box interaction can be concise with high cohesion, avoiding problems caused by long user commands.

In step 340, the user repeatedly adds IoT services to show the interaction to create a mashup. Users can create IoT service mashups by adding the required IoT services one by one within a multi-dialog interaction. Therefore, the repeating additional feature satisfies the user requirements [R1] and [R2] described above.

According to an embodiment of the present invention, recovery from errors is an essential function in multi-dialog box interaction [17]. It is also important to make the recovery process easy to use. If the correction command is not easy, users are more likely to make more errors while correcting errors [16].

Therefore, in the error correction step 360 according to the embodiment of the present invention, an Undo function for removing the last added IoT service from the current mashup is added to ComMA so that the user can easily perform error recovery. Step 360 shows an example of an undo interaction. The user can recover from the error by undoing 361 the erroneous command when a recognition error occurs without restarting the entire conversation.

Next, the feedback mechanism according to the embodiment of the present invention is another key feature of CoMMA. Components of the feedback mechanism according to an embodiment of the present invention include feedback for implicit confirmation, proposal confirmation, and progress confirmation.

In order to support creation of IoT service mashups, it is important to properly recognize services through user verification. Misunderstood IoT services, such as user commands from "then deactivating the security system" to "then THE activating the security system" can affect There are two ways to check that the recognized service is correct. That is, there is an implicit method and an explicit method [17, 19]. In the present invention, an implicit confirmation method is used in order not to bother the user too much.

In feedback for implicit confirmation according to an embodiment of the present invention, as a user command is recognized as an IoT service, an implicit confirmation is transmitted as a response of CoMMA, and the content of the implicit confirmation is a natural language form of the identified IoT service and a recognized parameter to be. For example, 332 of FIG. 3 shows a user input of “close the windows if it rains” to be identified as “WeatherConditionsTrigger” and “CloseWindowFrameAction”. An implicit confirmation of this recognized service is "closing the windows if the weather condition is raining" as in (333). After acknowledgment feedback is provided, ComMA prompts the user to speak a follow-up command.

The bold dialog in Figure 3 illustrates how implicit confirmation is provided within a multi-dialog interaction. The user can check the recognition error in the feedback and proceed to the next command. Thus, implicit validation makes multi-dialog interactions reliable and understandable, meeting the user requirements [R2] and [R3] described above.

One of the problems identified in the single utterance approach according to the prior art is that it is difficult to track the progress of the mashup creation. In order to meet the user requirement [R3], CoMMA according to an embodiment of the present invention has a suggestion check that recommends proactively checking in multiple dialog box interactions.

According to an embodiment of the present invention, whenever a user adds or undos an IoT service, the interaction counter is incremented by 1, except for the initial interaction where two IoT services, a trigger and an action must be specified. There are two moments at which the proposal confirmation is presented. For example, if the interaction counter reaches 5 first and the user adds IoT services, the proposed interval 5 is based on the heuristic where the information capacity of human short-term memory is between 5 and 9 items [20]. Second, if you have two or more interaction counters and the user has finished creating the mashup, check for suggestions before the conversation ends.

In step 350 of FIG. 3 , an interaction for confirming a suggestion is shown to the user. The IoT service added by the user increments the interaction counter by 1 ( 351 ), and the user receives a suggestion confirmation from the CoMMA ( 352 ). The user can either choose to accept or reject the proposal, and the current status of the mashup, as shown in 353, is displayed if accepted. This allows the user to periodically identify how the mashup occurs, thus satisfying the user requirement [R3].

A feedback mechanism according to an embodiment of the present invention includes a progress check that supports a user to track the progress of a mashup. However, a progress check may be necessary on its own and is helpful in general cases. For example, if a user accidentally forgets the progress of creating a mashup, it can be helpful to call the progress. Users can ask CoMMA to check the progress of the mashup whenever they want by saying "tell me about the mashup state" or "current mashup".

According to an embodiment of the present invention, when a user utters a user command (User Commands) for generating a mashup, the necessary identifiers are a natural language processing model (NLP Model) such as the example of 331 of FIG. 3 . recognized using Through the recognized identifier, CoMMA provides the type of IoT service that the user wants to use, as shown in the example 332 of FIG. 3 . If a problem occurs while inferring a service type, such as a user mentioning a service that cannot be used in the current IoT environment, ComMA feeds back an error to the user through the feedback generator. The estimated service type is stored in the mashup storage and fed to the feedback generator. The mashup storage unit is used to store the mashup created by the user in the form of a graph indicated, such as the mashup data of 353 of FIG. 3 . The proposed feedback generator generates feedback based on the inferred information as in the example of 333 of FIG. 3 , and checks whether confirmation is required for the current interaction counter. When confirmation is required, both the generated feedback and confirmation suggestion are transmitted to the user as illustrated in 352 of FIG. 3 .

According to an embodiment of the present invention, natural language processing of CoMMA such as identifier matching was performed using the Google Dialogflow NLP model. The commands identified were sent to a ComMA backend server running on a computer with an Intel Xeon E3-1230v6 CPU and 16 GB of RAM. The backend server considers generating ComMA responses and managing mashups in graph form. The backend server was programmed with Flask 1.1.22 and NetworkX 2.23 in Python 3.5.

The interaction between a user (or a user terminal) and CoMMA for modifying a mashup according to an embodiment of the present invention is divided into two main phases, namely, a localization phase and a modification phase. The localization stage is a stage to localize the mashup that is currently deployed in the smart home (that is, the IoT environment) and needs to be modified, and the user can classify the problematic mashup at this stage. An interaction is then made to modify the localized mashup in the edit phase.

It is neither efficient nor effective in terms of usability for the user to check the list of currently deployed mashups through a conversational interface. Instead of telling CoMMA and the user a list of all available mashups and selecting one, the present invention uses CoMMA to retrieve a mashup based on the user's suggestion of the user. The implied meaning is the meaning implied in the user's utterance [21]. You can leverage these user hints to localize the mashup in a usable way through an interactive interface. In addition, we propose an "implicit-based traceback process" that infers target environment characteristics from user hints and retrieves IoT service mashups that have influenced environment characteristics.

According to the taxonomy of pragmatics [22], which is influenced by the type of speech, five speech actions are possible during the interaction between the user and ComMA. Two of these voice actions (eg, command and expression) can be utilized for implicit reasoning. This is because both voice actions can include an intention to change the environment in a direct or indirect way [23]. In addition, both were used by the prior art [8, 24, 25] in the smart home environment, and the NLP model for implicit inference was trained to support the relevant sentence types. The following utterance types are supported:

For example, a command might contain a sentence such as "make this room cool". This type of sentence is most prevalent in the interaction between the user and CoMMA [23].

For example, the expression may include sentences such as "the room is really hot" or simply "it's hot". Based on Gricean's maximum relevance [21], it reflects not only the user's feelings about the current environment, but also the user's desire to improve the current environment.

For example, a question might contain sentences such as "why is the room so hot". These questions take the form of essentials in that the user asks the conversationalist for reasons about the current environmental characteristics. At the same time, they also have the property of expressions in that they act implicitly as negative feedback on current environmental characteristics.

For the type of environment attribute that can affect users in the smart home environment according to the embodiment of the present invention, reference was made to EUPont ontology [18]. For example, in the ontology, six characteristics such as air quality, brightness, humidity, noise, temperature, and security are defined, and each IoT action service increases or decreases one or more environmental characteristics. For example, turning on air clear defined as "EnableAirPurifierSystemAction" in the ontology increases the air quality of the environment, so its main effect is defined as HighAirQuality.

However, not only the main effects of IoT services, but also user hints may occur in side effects. For example, turning on Air Clear not only improves the quality of the air in the environment, but also has a side effect of increasing noise levels in the environment. Since Ontology defines only the main effects of each IoT service, the classification has been expanded in consideration of potential side effects.

4 is a diagram for explaining the concept of a suggestion-based traceback process according to an embodiment of the present invention.

In order to start a mashup search for modifications according to an embodiment of the present invention, the user can say any representation that the NLP model 410 can support. For example, the user may additionally provide a Device Keyword to search for a mashup using a particular device, such as "the air conditioner is not cooling enough". Thereafter, the NLP model 410 infers user implicature from the input user utterances. Here, the user hints at information about the environment attribute that he wants to change. By tracing back onto the ontology 420 containing the mapping between environment properties and IoT actions, it is possible to infer a list of IoT action services that may have affected the user (named “Attributable Actions” in the figure). . Finally, by querying the recently executed IoT service mashup history 430 , the attributable mashup is searched and priority is given using the call timestamp. The most recently executed mashup has the highest priority because it is most likely to affect users and historical records that exceed the time threshold are excluded from the search. Mashups retrieved through this process are presented to users in priority order.

5 is a diagram for explaining an example of a suggestion-based traceback process according to an embodiment of the present invention.

For example, in 510 of FIG. 5 , the user said "the room is hot now", and the implication of "HighTemperature" is inferred from here. A list of contributing behaviors can be obtained through an ontology traceback with the implication of "HighTemperature". In 520 of FIG. 5 , the list of IoT actions that may affect the temperature characteristic include turning off a heater (Disable Heating System), turning on a dehumidifier (Enable Dehumidifier System), and opening a window (Open Window Frame). This list is queried as a mashup call record as shown at 530 in FIG. 5 to prioritize the attribute mashup.

6 is a diagram for explaining the interaction of localization according to an embodiment of the present invention.

Referring to FIG. 6 , how the interaction between the user (or user terminal) and CoMMA works at the front end will be described. The localization process according to an embodiment of the present invention is initiated by the user's command or expression utterance. Thereafter, CoMMA according to an embodiment of the present invention returns a user suggestion inferred from the utterance as shown in 610 of FIG. 6 . This tacit consent not only ensures that the inferred implication is correct, but also serves to grant access to the IoT service mashup record. This is because accessing records is a privacy-sensitive behavior [26, 6]. Next, a brief answer is given to the search result as shown in 620 of FIG. 6 . If the user remembers the trigger information of the search result, the user can directly proceed with the correction step. Otherwise, the user may ask the reason for executing the action as shown in 630 of FIG. 6 , and CoMMA according to an embodiment of the present invention responds using all information about the result.

7 is a diagram for explaining an example of interaction for searching for another mashup or returning to a previous mashup in the localization process according to an embodiment of the present invention.

If the current search result is not satisfactory, as shown in FIG. 7( a ), a rejection command to make CoMMA search for the next mashup may be given. Also, it is possible to return to the previous result as shown in FIG. 7(b). Through this localization interaction with the user implicit reasoning process, IoT service mashups can be localized with simple and intuitive sentences, and unnecessary information can be minimized in the ComMA response.

In the error correction step according to an embodiment of the present invention, the user refers to a command to correct the localized mashup. The command may be to add a new mashup component (eg "add setting the temperature to 24") or replace an existing mashup component or parameter (eg "trigger condition"). change the triggering condition to whenever I leave the office"). To clarify what types of modifications CoMMA should support, we have grouped 7 types of modifications as shown in Figure 8.

8 is a diagram illustrating a command pattern for modifying a mashup according to an embodiment of the present invention.

Adding a new trigger to a user command according to an embodiment of the present invention (Appending a new Ttrigger), adding a new action (Appending a new action), replacing the existing trigger (Replacing the existing trigger), replacing the existing action (Replacing the existing action) , based on a plurality of command patterns including Modifying the trigger parameter, Modifying the action parameter, and Removing the service mashup, the service type of the user command sets the parameter , whether to increment or decrement, and if there is a component that uses the parameter in the mashup to modify. If a user command to modify an existing parameter cannot be found, the type of service, parameters, keywords, location to modify, and substitution or Ambiguity of essential information including additions can be improved.

As described above, it was confirmed that four pieces of information were needed to modify the IoT service mashup based on the classified type of user input. A description of each piece of information follows:

Among the information for modifying the IoT service mashup, the "service type" is needed to identify the service used to perform user commands. For example, if the user says "change it to when I leave home," then you want to use a location-based exit detection service. When a user says "increase temperature by 3" or "when the room temperature increased by 3", this indicates that a temperature sensing or activation service is used. The type of service required by the user is resolved through intent detection in the NLP model.

In addition, among the information for modifying the IoT service mashup, “parameters” and “keywords” are important information in user input. In the example above, "home" and "3" are recognized as parameters as they are necessary to operate the IoT service correctly. This information is addressed through the slot filling function of the NLP model.

Also, we need a "modify location" to check whether the user wants to modify the mashup's trigger component or action component. This information may be explicitly specified in a user command (e.g., "change the trigger to ..." If it cannot be checked with , it can be solved in a trivial way, otherwise an ambiguity problem arises.

In addition, among the information for modifying the IoT service mashup, “replace” or “add” indicates whether the user will replace the existing mashup component or add a new component. Similar to the location of the fix, there is also an ambiguity problem with this information.

According to the Gricean maximum [21], people can omit information that can be easily inferred from the current conversation. Thus, the user can simplify the command by omitting some essential information, which creates an ambiguity problem.

9 is a diagram for explaining a problem of ambiguity of a user command according to an embodiment of the present invention.

In the mashup as in the example of FIG. 9 , the user may “change the action to turning the dehumidifier on (A1)” to replace the existing action. Here, the user can omit the keyword "action" by simply saying "change it to turning on the humidifier (A2)". Users can sometimes further simplify their speech by using phrases such as (A3) or irregular grammars. As another example, the user may modify the humidity parameter shown in FIG. 9 by issuing a command "change humidity to 60% (P1)" or saying "change to 60 (P2)".

A pure way to solve the ambiguity problem is to ask the user a follow-up question whenever a problem arises. In other words, when the user says "change it to dehumidifier turn on," ComMA asks "which one do you want to change, trigger or action" can be Follow-up questions are not clear [27], but frequent follow-up questions increase conversation turns and make interactions unnatural. Therefore, in terms of ease of use, it is important to solve the ambiguity problem while minimizing the number of follow-up questions in the course of the revision interaction. In a smart home environment, there are different triggers and actions, and there can be multiple IoT service mashups that are combinations of triggers and actions. In the present invention, the ambiguity problem is solved by considering the ambiguity strategy based on the user's seven command patterns as a generalized solution that does not depend on a specific type of natural language command, trigger, or action.

10 is a view for explaining a process of clarifying a user command according to an embodiment of the present invention.

Except for the command pattern “delete service mashup” in FIG. 8 , the process of classifying the types of user commands is divided into three steps as shown in FIG. 10 . Depending on the process, the details of each step are as follows:

First, it is checked whether a parameter is changed by the command (1010). At this point, you need to check two conditions: whether the service type of the user command sets, increments or decrements the parameter, and whether there is a component using the parameter in the currently selected mashup. CoMMA according to an embodiment of the present invention checks whether the target parameter type is specified in the user command whenever both conditions are satisfied. If the user does not specify a type (eg, (P2) in FIG. 9 ), the component using the parameter in the selected mashup is modified. Otherwise, a component that uses a parameter of the same type as the user-specified type is modified (eg, (P1) in FIG. 9). If ambiguity arises during this process (for example, both trigger and parameter use the same type of parameter), the user is presented with a follow-up question. If there is no matching component in the above process, the user command is regarded as one of the command patterns (a-d) in Fig. 8 and the process proceeds to the next step.

Next, it is necessary to clarify the location of the correction (1020). If a user command to modify an existing parameter cannot be found (that is, the type of modification is one of the command patterns (a-d) in FIG. 8), it is necessary to check the location of the modification and whether to replace or add it. To solve this, the present invention uses contextual information within the conversation recommended by the Human-AI guidelines [27]: (1) currently selected IoT service mashup information, (2) Conversation record between the user and CoMMA, (3) We use 3 contextual information such as keywords spoken by the user.

The highest priority is whether the user has ever explicitly said a location keyword. If the user said a location keyword in the current command or in a previous sentence, use that information to clarify it. If not, it is checked whether the user has ever heard a detailed response ( 630 in FIG. 6 ) in the localization step. In this case, the user's conversation record can be identified and checked. CoMMA considers that users are trying to modify the working component if they have never heard of the triggering details. If the ambiguity has not been resolved in the above procedure, CoMMA should ask a follow-up question to clarify the location of the correction.

Also, the substitution or addition should be made explicit (1030). As with the previous step, using contextual information, it is not clear whether the user will replace an existing component or add a new component. Keywords explicitly spoken by the user have the highest priority. For example, if the user said a command such as "change..." or "replace...", they replaced an existing component. Commands like "add..." and "include..." are considered to add a new component. In the absence of an explicit keyword, the service type is compared to the target component type. If the input command and the current target component refer to the same type of service, the command is considered to replace the existing component. Otherwise, follow-up questions for disambiguation are generated.

11 is a view for explaining a process of performing an interaction to modify a mashup according to an embodiment of the present invention.

11 shows an example of a modification interaction between a user (or user terminal) and CoMMA. The edit interaction is initiated by the user issuing a command to modify the currently selected mashup. The recognized user utterances are processed by the NLP model, and available information is extracted from the utterances. In the example 1110 of FIG. 11 , the service type in which the user said "change it to turning on the dehumidifier" is device activation, and the device parameter is recognized as dehumidifier activation, replacement, or additional information. Clarification proceeds based on this accepted information. First, in this example, the example 1120 of FIG. 11 is skipped because the user command does not modify the parameters in the existing mashup. Second, since the user has heard the triggering information and the user has not specified which mashup component to modify, a follow-up question is generated as to where to modify, as in example 1130 of FIG. 11 . Substitution or addition of information is automatically disambiguated in the user's voice, as in example 1140 of FIG. 11 . Finally, an explicit confirmation request is made before applying the user's edit command to the mashup. Once the edit is complete, the user can either search for another mashup to edit or complete the edit process.

Natural language processing such as implicit type detection or identifier matching according to an embodiment of the present invention was performed using the Google Dialog flow1 NLP engine. The verified user commands were sent to a ComMA backend server running on a computer with an Intel Xeon E3-1230v6 CPU and 16 GB of RAM. The backend server was programmed in Python 3.5 using Flask 1.1.22.

As described above, when a plurality of IoT services are connected through an implicit-based interaction model for modifying automation rules in the IoT environment according to embodiments of the present invention, the user can configure the IoT service mashup using simple and intuitive user commands. It is possible to avoid duplicate syntax in user commands, and to achieve simplification of information conveyed in responses. It can also assist in maintaining mashup creation progress without incurring significant cognitive load on the user in the interaction of the mashup creation.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

<References>

[1] Giuseppe Ghiani, Marco Manca, Fabio Patern'o, and Carmen Santoro. Personalization of context-dependent applications through trigger-action rules. ACM Transactions on Computer-Human Interaction (TOCHI), 24(2):1-33, 2017.

[2] Dominique Guinard, Vlad Trifa, Stamatis Karnouskos, Patrik Spiess, and Domnic Savio. Interacting with the soa-based internet of things: Discovery, query, selection, and on-demand provisioning of web services. IEEE Transactions on Services Computing, 3(3):223-235, 2010.

[3] Janggwan Im, Seonghoon Kim, and Daeyoung Kim. Iot mashup as a service: Cloud-based mashup service for the internet of things. In Proceedings of the 2013 IEEE International Conference on Services Computing, pages 462-469, 2013.

[4] In-Young Ko, Han-Gyu Ko, Angel Jimenez Molina, and Jung-Hyun Kwon. Soiot: Toward a user-centric iot-based service framework. ACM Transactions on Internet Technology (TOIT), 16(2):8, 2016.

[5] Marwa Boulakbech, Nizar Messai, Yacine Sam, Mohammad Hammoudeh, et al. Iot mashups: From iot big data to iot big service. In Proceedings of the International Conference on Future Networks and Distributed Systems, page 20. ACM, 2017.

[6] Tawfiq Ammari, Jofish Kaye, Janice Y Tsai, and Frank Bentley. Music, search, and iot: How people (really) use voice assistants. ACM Transactions on Computer-Human Interaction (TOCHI), 26(3):1-28, 2019.

[7] Dave Braines, Nick O'leary, Anna Thomas, Daniel Harborne, Alun Preece, and Will Webberley. Conversational Homes: A Uniform Natural Language Approach for Collaboration Among Humans and Devices. International Journal on Advances in Intelligent Systems, 10(3):223-237, 2017.

[8] Andr'e Sousa Lago, Joao Pedro Dias, and Hugo Sereno Ferreira. Conversational interface for managing non-trivial internet-of-things systems. In International Conference on Computational Science, pages 384-397. Springer, 2020.

[9] Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S Lam. Almond: The architecture of an open, crowdsourced, privacy-preserving, programmable virtual assistant. In Proceedings of the 26th International Conference on World Wide Web, pages 341-350, 2017.

[10] Frank Bentley, Chris Luvogt, Max Silverman, Rushani Wirasinghe, Brooke White, and Danielle Lottridge. Understanding the long-term use of smart speaker assistants. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(3):91, 2018.

[11] Yoonjeong Cha, Jincheul Jang, Younghyun Hong, and Mun Yong Yi. "jack-of-all-trades": A the matic analysis of conversational agents in multi-device collaboration contexts. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-6, 2019.

[12] A.J. Bernheim Brush, Bongshin Lee, Ratul Mahajan, Sharad Agarwal, Stefan Saroiu, and Colin Dixon. Home automation in the wild: Challenges and opportunities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, page 2115-2124, New York, NY, USA, 2011. Association for Computing Machinery.

[13] Sarah Mennicken and Elaine M Huang. Hacking the natural habitat: An in-the-wild study of smart homes, their development, and the people who live in them. In Pervasive Computing: 10th International Conference, Pervasive 2012, Newcastle, UK, June 18-22, 2012. Proceedings, volume 7319, page 143. Springer, 2012.

[14] Jong-bum Woo and Youn-kyung Lim. User experience in do-it-yourself-style smart homes. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp '15, page 779-790, New York, NY, USA, 2015. Association for Computing Machinery.

[15] Luigi De Russis and Alberto Monge Roffarello. Personalizing iot ecosystems via voice. In Proceedings of the 1st International Workshop on Empowering People in Dealing with Internet of Things Ecosystems co-located with International Conference on Advanced Visual Interfaces (EMPATHY@AVI 2020), volume 2702 of CEUR Workshop Proceedings, pages 37-40, 2020.

[16] Ben Shneiderman. The limits of speech recognition. Communications of the ACM, 43(9):63-65, 2000.

[17] Christine Murad, Cosmin Munteanu, Leigh Clark, and Benjamin R Cowan. Design guidelines for hands-free speech interaction. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pages 269-276. ACM, 2018.

[18] Fulvio Corno, Luigi De Russis, and Alberto Monge Roffarello. A High-Level Approach Towards End User Development in the IoT. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '17, pages 1546-1552, New York, New York, USA, 2017. ACM Press.

[19] Nicole Yankelovich, Gina-Anne Levow, and Matt Marx. Designing speechacts: Issues in speech user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 369-376. ACM Press/Addison-Wesley Publishing Co., 1995.

[20] George Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63:81-97, 1956.

[21] Herbert P Grice. Logic and conversation. Syntax and Semantics, 3:41-58, 1975.

[22] Andrea Scarantino. How to do things with emotional expressions: The theory of affective pragmatics. Psychological Inquiry, 28(2-3):165-185, 2017.

[23] Minha Lee. Speech acts redux: Beyond request-response interactions. In Proceedings of the 2nd Conference on Conversational User Interfaces, CUI '20, New York, NY, USA, 2020. Association for Computing Machinery.

[24] Dave Braines, Nick O'Leary, Anna Thomas, Daniel Harborne, Alun David Preece, and William M Webberley. Conversational homes: A uniform natural language approach for collaboration among humans and devices. International Journal on Advances in Intelligent Systems, 10(3/4):223-237, 2017.

[25] Mahda Noura, Sebastian Heil, and Martin Gaedke. Growth: Goal-oriented end user development for web of things devices. In International Conference on Web Engineering, pages 358-365. Springer, 2018.

[26] Josephine Lau, Benjamin Zimmerman, and Florian Schaub. Alexa, are you listening? privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proc. ACM Hum.-Comput. Interact., 2 (CSCW), November 2018.

[27] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. Guidelines for human-AI interaction. In Conference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, May 2019.

Claims (16)

delete localizing to classify mashups that are currently deployed in the IoT environment and require modification; and
performing an interaction between the suggestion-based traceback process and localization through an error correction unit to correct the classified mashup;
An implicit-based interaction method for modifying automation rules in an IoT environment that includes 3. The method of claim 2,
The step of localizing to classify a mashup that is deployed in the current IoT environment and requires modification is,
retrieving a mashup using hints about user utterances, including utterance types, commands, expressions, and questions, localizing the mashup in a usable manner through the interaction, and from hints about the user utterance to the current Performing a hint-based traceback process of inferring IoT environment characteristics and searching for IoT service mashups that have affected the current IoT environment characteristics
An implicit-based interaction method for modifying automation rules in an IoT environment. 4. The method of claim 3,
The suggestion-based traceback process is
provide additional device keywords to search for mashups that use specific IoT devices in user utterances;
Inferring hints about the IoT environment properties you want to change from user utterances input through the natural language processing model,
Inferring a list of IoT action services that affect users by tracing back the ontology including the mapping between the IoT environment property to be changed and the action of a specific IoT device,
Searches the recently executed IoT service mashup history, searches for the corresponding mashup, gives priority using the call timestamp, and provides users according to the priority of the corresponding mashup.
An implicit-based interaction method for modifying automation rules in an IoT environment. 5. The method of claim 4,
The step of localizing to classify a mashup that is deployed in the current IoT environment and requires modification is,
A modification step according to trigger information of a search result of an IoT service mashup that is initiated by a user's command or user's utterance, returns a hint about the user's utterance, infers the current IoT environment characteristic, and has affected the current IoT environment characteristic , or by asking why an action was executed, asking for another mashup if the search results are unsatisfactory, or returning to the previous result, providing unnecessary information through the interaction between the suggestion-based traceback process and localization. to minimize
An implicit-based interaction method for modifying automation rules in an IoT environment. 3. The method of claim 2,
The step of performing the interaction between the suggestion-based traceback process and localization through the error correction unit to correct the classified mashup,
Receive a user command for modifying the classified mashup, and add a new mashup component according to essential information including service type, parameter, keyword, location to modify, and replacement or addition of mashup for the user command for modifying the mashup or to replace pre-formulated mashup components or parameters.
An implicit-based interaction method for modifying automation rules in an IoT environment. 7. The method of claim 6,
Based on a plurality of command patterns including adding a new trigger for the user command, adding a new action, replacing an existing trigger, replacing an existing action, modifying a trigger parameter, modifying an action parameter, and deleting a service mashup, the service type of the user command is a parameter Whether to set, increment or decrement, and if there are components that use the parameter in the mashup to modify it, and if no user command to modify an existing parameter can be found, where to modify and replace the mashup to modify it. or to improve the ambiguity of essential information, including service types, parameters, keywords, locations to modify, and substitution or addition of mashups for user commands for modifying the mashup by checking whether to add
An implicit-based interaction method for modifying automation rules in an IoT environment. Recognizing the user command by the user command recognition unit using a natural language processing model to convert the user command in voice form into text and then extract each identifier including an object, an operation, a parameter, and a location;
When the user command is recognized by the user command recognition unit, providing the recognized content to the user terminal through the control unit, and controlling the user terminal to display a message to further add or stop the user command;
By adding user commands without overlapping phrases to the previous IoT service called through the mashup storage in response to a message to add or stop more user commands, iteratively adds user commands, and combines IoT services to create and store mashups. step;
recovering from the error by undoing the execution of the user's command without restarting the entire conversation through the error correction unit when a recognition error occurs in the generated mashup; and
Through the feedback generator, feedback for implicit confirmation of the IoT service regarding the recognized user command, confirmation of proposals, and confirmation of progress is generated to check whether there is a need for implicit confirmation, confirmation of proposals and confirmation of progress, and confirmation Transmitting the generated feedback to a user terminal if necessary
including,
Through the feedback generating unit, by generating feedback for implicit confirmation of the IoT service regarding the recognized user command, confirmation of proposals, and confirmation of progress, check whether there is a need for implicit confirmation, confirmation of proposals, and confirmation of progress, and confirm If necessary, the step of delivering the generated feedback to the user terminal,
Check whether there is a need for implicit confirmation, which is a natural language form of the recognized IoT service,
When the interaction counter reaches a predetermined number and IoT services are added according to user commands, and when the interaction counter is greater than or equal to a predetermined number, after completing the creation of the mashup and before the end of the user's utterance, check whether it is necessary to confirm the suggestion;
It checks whether progress confirmation is necessary so that the user can track the progress of the mashup, and delivers the generated feedback to the user terminal when confirmation is required.
An implicit-based interaction method for modifying automation rules in an IoT environment. delete An error correction unit that performs localization to classify a mashup that is currently deployed in the IoT environment and requires modification, and performs an interaction between the implicit-based traceback process and localization to correct the classified mashup
An implicit-based interaction device for modifying automation rules in an IoT environment comprising: 11. The method of claim 10,
The error correction unit,
Retrieve mashups using hints about user utterances, including utterance types, commands, expressions, and questions, to perform localization to classify mashups that are currently deployed in IoT environments and require modification, and the interactions localizes the mashup in a usable way through performing
Implication-based interaction devices for modifying automation rules in IoT environments. 12. The method of claim 11,
The error correction unit,
Further providing a device keyword for searching for a mashup using a specific IoT device in user utterances through the suggestion-based backtracking process,
Inferring hints about the IoT environment properties you want to change from user utterances input through the natural language processing model,
Inferring a list of IoT action services that affect users by tracing back the ontology including the mapping between the IoT environment property to be changed and the action of a specific IoT device,
Searches the recently executed IoT service mashup history, searches for the corresponding mashup, gives priority using the call timestamp, and provides users according to the priority of the corresponding mashup.
Implication-based interaction devices for modifying automation rules in IoT environments. 13. The method of claim 12,
The error correction unit,
IoT that starts localization for classifying the mashup by the user's command or the user's utterance, returns a hint about the user's utterance, infers the current IoT environment characteristic, and affects the current IoT environment characteristic Implicit-based traceback by taking corrective steps or asking why an action was executed according to the trigger information in the search result of the service mashup, and then searching for another mashup if the search result is not satisfactory or returning to the previous result Minimize unnecessary information through interactions between processes and localization
Implication-based interaction devices for modifying automation rules in IoT environments. 11. The method of claim 10,
The error correction unit,
Receive a user command for modifying the classified mashup, and add a new mashup component according to essential information including service type, parameter, keyword, location to modify, and replacement or addition of mashup for the user command for modifying the mashup or to replace pre-formulated mashup components or parameters.
Implication-based interaction devices for modifying automation rules in IoT environments. 15. The method of claim 14,
The error correction unit,
Based on a plurality of command patterns including adding a new trigger for the user command, adding a new action, replacing an existing trigger, replacing an existing action, modifying a trigger parameter, modifying an action parameter, and deleting a service mashup, the service type of the user command is a parameter Whether to set, increment or decrement, and if there is a component that uses the parameter in the mashup to modify it, and if a user command to modify an existing parameter cannot be found, where to modify it and the substitution of the mashup to modify it. or to improve the ambiguity of essential information, including service types, parameters, keywords, locations to modify, and substitution or addition of mashups for user commands for modifying the mashup by checking whether to add
Implication-based interaction devices for modifying automation rules in IoT environments. a user command recognition unit that converts a user command in voice form into text using a natural language processing model and then extracts each identifier including an object object, operation, parameter, and location to recognize the user command;
When a user command is recognized by the user command recognition unit, the control unit provides the recognized content to the user terminal, and controls to display a message to add or stop the user command further on the user terminal;
A mashup storage unit that repeatedly adds user commands by adding user commands without overlapping phrases to the previous IoT service that is called according to a message to add or stop more user commands, and combines IoT services to create and store a service mashup ;
an error correction unit that recovers from the error by undoing the execution of the user's command without restarting the entire conversation when a recognition error occurs in the generated mashup; and
By generating feedback for implicit confirmation of the IoT service regarding the recognized user command, confirmation of proposals, and confirmation of progress, it checks whether implicit confirmation, confirmation of proposals and confirmation of progress are necessary, and when confirmation is necessary, to the user terminal A feedback generator that transmits the generated feedback
including,
The feedback generator,
Checking whether there is a need for implicit confirmation, which is a natural language form of the recognized IoT service,
When the interaction counter reaches a predetermined number and IoT services are added according to user commands, and when the interaction counter is greater than or equal to a predetermined number, after completing the creation of the mashup and before the end of the user's utterance, check whether it is necessary to confirm the suggestion;
It checks whether progress confirmation is necessary so that the user can track the progress of the mashup, and delivers the generated feedback to the user terminal when confirmation is required.
Implication-based interaction devices for modifying automation rules in IoT environments.

Images