CN113156827A

CN113156827A - Intelligent equipment control method and system based on semantics

Info

Publication number: CN113156827A
Application number: CN202110359565.9A
Authority: CN
Inventors: 张腾翔; 曾馨; 陈益强
Original assignee: Institute of Computing Technology of CAS
Current assignee: Institute of Computing Technology of CAS
Priority date: 2021-04-02
Filing date: 2021-04-02
Publication date: 2021-07-23
Anticipated expiration: 2041-04-02
Also published as: CN113156827B

Abstract

The invention provides a semantic-based intelligent device control method and a semantic-based intelligent device control system, which comprise the following steps: constructing an internet of things comprising intelligent equipment and a control end, wherein the control end extracts the functions of the intelligent equipment; the control end extracts input semantics of a user through an input sensor, screens functions of the intelligent equipment based on the input semantics to obtain a function alternative set only matched with the input semantics, and controls the intelligent equipment to execute corresponding functions according to the functions selected by the user in the alternative set. Compared with the prior art, the input association mechanism based on the semantics can reduce the complexity of the interaction task, and can be applied to various ubiquitous input technologies.

Description

Intelligent equipment control method and system based on semantics

Technical Field

The invention relates to the technical field of human-computer interaction and intelligent device input, in particular to a semantic-based intelligent device control method and system.

Background

With the development of the internet of things in recent years, more and more intelligent devices appear in the lives of people and play an important role. The input interaction mode with the intelligent equipment mainly comprises the following steps: voice input, gesture input, etc. A control command for an intelligent device can be divided into three parts: (1) equipment: refers to a device, such as a light, a television, a computer, etc., with which a user is to interact. (2) The device atomic function: what the user wants to operate with and cannot be further subdivided is called a device atomic function, hereinafter simply referred to as "function". Such as a light switch, a "last channel" button on a television, a volume slider on a computer. (3) And (3) input function control: it is referred to how the user wants to operate a function, which is hereinafter referred to as "input" for short, such as turning on/off a lamp, tuning to the last channel of a television, sliding a volume bar so that the volume of the computer is brought to a certain value.

Most of the existing various control commands are composed of the three parts, and the control flow of the traditional intelligent device can be summarized as follows: after the user generates the interactive intention, the user selects the equipment to be controlled, selects the function to be controlled and inputs a specific control value. As shown in figure 1, a user feels hot air temperature, then wants to lower the air temperature, selects air conditioning equipment, selects a temperature control function, and finally operates input temperature reduction to be used as function control, and the total number of the steps is 5. Based on this way of interaction, the most critical issue is how to pair with the device to be controlled. There are various forms, for example, the most common devices are each equipped with a remote controller or an operation panel for control; in an intelligent home environment, all devices are generally controlled by one APP, and the device to be controlled is selected and then specifically controlled; photographing by using a smart phone and identifying an object by using a computer vision algorithm to bind equipment; and capturing the direction of the user's gaze by using a front camera of the mobile phone as a basis for binding the equipment. In addition, device identification is also performed based on user location or contextual context.

"naturalness" and "efficiency" are the goals of interactive process optimization. The mode of firstly carrying out equipment and then carrying out functions is really the most suitable for the input habit of the user, however, as the technology of the internet of things is gradually integrated into the life of people, more and more intelligent equipment is provided. This approach has the following problems:

the first disadvantage is that: a redundant control device or a complicated device pairing method is required, and development and learning costs are high. Conventional control flow requires first associating context with a particular device or application and then associating inputs and functions. How to solve the problems of device, function and input association is still a problem. As described in the related background, the present methods predetermine the relationships between input technologies, interfaces, and functions, which do not meet the requirements of ubiquitous input systems in the context of the present internet of things. Or a novel pairing technology needs to be developed, although the interaction efficiency or experience can be improved, the requirement on environment or operation is high mostly, or a user needs to learn a new interaction mode additionally, so that the requirement on a ubiquitous system is difficult to be met.

The second disadvantage is that: the equipment association method greatly influences the interactive experience, and the interaction lacks spontaneity. Currently, most steps of device, function and input association are still used as an additional interaction step, and such an interaction paradigm requires a user to explicitly select a device, so that the interaction process is lack of spontaneity. In addition, different device association methods and effects will greatly affect the interactive experience.

Disclosure of Invention

The invention aims to solve the problems that the traditional equipment input association mode is complex in association method, low in interaction efficiency and not spontaneous in interaction, and provides an intelligent equipment control method based on semantics.

Aiming at the defects of the prior art, the invention provides a semantic-based intelligent device control method, which comprises the following steps:

step 1, constructing an internet of things comprising intelligent equipment and a control end, wherein the control end extracts the functions of the intelligent equipment;

and 2, the control end extracts input semantics of the user through the input sensor, screens the functions of the intelligent equipment based on the input semantics to obtain a function alternative set only matched with the input semantics, and controls the intelligent equipment to execute corresponding functions according to the functions selected by the user in the alternative set.

The intelligent equipment control method based on the semantics is characterized in that a display device is arranged in or externally connected with the control end to display the functions of the intelligent equipment; and the display device displays the functions of the intelligent device.

The intelligent equipment control method based on the semantics is characterized in that a display device is arranged in or externally connected with the control end to display the functions of the intelligent equipment; and the step 2 comprises: and continuously triggering the input sensor to switch the current function displayed by the display device.

The intelligent device control method based on the semantics, wherein the input sensor comprises: a forward button, a back button, a ok button, and a touch slide, the forward button and the back button representing a "previous" and "next" semantic, respectively, the ok button representing an "ok" semantic, and the relative sliding of the touch slide representing an "increase" or "decrease" semantic.

The intelligent equipment control method based on the semantics is characterized in that the intelligent equipment is a refrigerator, a sound box, a television and a desk lamp with the function of the Internet of things.

The intelligent device control method based on the semantics is characterized in that the control end comprises a Bluetooth transceiver module which is used for connecting the intelligent device.

The intelligent equipment control method based on the semantics is characterized in that the display device is an intelligent watch.

The intelligent device control method based on the semantics is characterized in that the content displayed by the display device comprises the name of the current intelligent device, the function of the current intelligent device and the state of the function of the current intelligent device.

The invention also provides a semantic-based intelligent device control system, which comprises:

the intelligent equipment has the function of the Internet of things and a control end connected with the intelligent equipment;

the control end is used for extracting the functions of the intelligent equipment, extracting the input semantics of the user through the input sensor, screening the functions of the intelligent equipment based on the input semantics to obtain a function alternative set only matched with the input semantics, and controlling the intelligent equipment to execute the corresponding functions according to the functions selected by the user in the alternative set.

According to the scheme, the invention has the advantages that:

compared with the prior art, the semantic-based input association mechanism can reduce the complexity of interaction tasks, and can be applied to various ubiquitous input technologies, particularly resource-limited ubiquitous input interfaces. According to the mechanism, an input system is realized, the system utilizes a smart watch and a semantic-based input association mechanism, and presents advantages compared with a traditional device-based input mode, and the application potential of the technology is shown.

Drawings

FIG. 1 is a diagram illustrating an interaction method of inputting after a device functions in the prior art;

FIG. 2 is a diagram illustrating a function-before-input interaction of the present invention;

fig. 3 is a schematic structural diagram of the controller according to the present invention.

Fig. 4a to c are schematic diagrams of an embodiment of the present invention.

Detailed Description

Based on the problems, the invention utilizes the semantic information hidden in the function to invent a semantic-based input association mechanism. The semantics are easy to understand, express and extract, and can be used for controlling the equipment of the Internet of things. For example, buttons imply discrete binary inputs, such as on/off of a light, on/off of a microphone; the state switching input implies discrete state switching selection, such as last song/next song selection of a sound box; while a slider bar implies a continuous change of value, for example to adjust the volume of a television. Different equipment functions contain different semantics, and by utilizing the characteristics, the invention provides an interaction mode of firstly inputting the function and then inputting the function, associates the equipment selection with the function through the function semantics, reduces the interaction complexity and the equipment specificity, reduces the learning cost of a user, and improves the interaction efficiency of multiple intelligent equipment in an intelligent environment. As shown in fig. 2, the user feels hot air temperature, then wants to lower the air temperature, and then operates the input cooling as function control, and inputs the function control to the air conditioning equipment for 4 steps in total.

By using the invention, the user can select from the semantically matched function subsets, thereby improving the interaction efficiency and reducing the interaction burden of the user. In order to achieve the technical effects, the application comprises the following key technical points:

the key point 1 is an input association mechanism based on semantics; the technical effects are as follows: performing device association through semantic implicit functions;

the key point 2 is based on a semantic input system of the intelligent watch; the technical effect is that an input system is realized by utilizing an intelligent table and a semantic-based input association mechanism, and the input system can be used as an input mode of intelligent equipment.

Specifically, the invention provides a semantic-based intelligent device control method, which comprises the following steps:

step 1s, constructing an internet of things comprising intelligent equipment and a control end, wherein a display device is arranged in or externally connected to the controller to display the functions of the intelligent equipment;

and 2s, the controller comprises at least one input sensor, the input sensor is used for representing specific semantics, and the user completes the control of the current display function of the display device by triggering the input sensor.

The intelligent equipment control method based on the semantics is characterized in that the display device displays the functions of the intelligent equipment.

The intelligent device control method based on the semantics, wherein the step 2 comprises the following steps: and continuously triggering the input sensor to switch the current function displayed by the display device.

In order to make the aforementioned features and effects of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.

The following embodiments are mainly based on the semantic input system of the smart watch, but it should be noted that the present application is not limited to the semantic input mode of the smart watch, and for example, the AR-based spaced gesture operation is also possible.

(1) Designing touch interactive interface

The touch interactive interface determines how the user will express the semantics that they want to input. In the present system, we design users to touch different widgets to express their input semantics.

As shown in fig. 3, it is designed for the touch interface of the present system. The touch interface includes one bluetooth transceiver module (byte E73-2G4M04S1B embedded in NRF52832 chip, powered using CR 2032) and four copper touch widgets. The touch widgets are made of thin and soft copper strips, can be easily deployed on the surfaces of different objects, and are convenient to fuse with other ubiquitous input interfaces. The touch widget is connected with an analog input pin of the Bluetooth module and used for self-capacitance sensing.

In the system, three types of touch widgets are designed, and three types of semantics are included. (1) The user can touch the "previous" button to indicate that "up/forward/previous" etc. contain the semantics of "previous"; touching the "next" button to indicate that "down/back/next" etc. contain the semantics of "next"; (2) touching the "confirm" button to indicate that "confirm/allow" and the like contain the semantics of "confirm"; (3) touching the "slider bar" and sliding to the right means that "increase" or the like contains the semantics of "increase"; touching the "slider bar" and sliding to the left means "decrease" or the like contains the semantic of "decrease".

Three types of information are encoded as bluetooth broadcast information, broadcast every 20 ms. The smart watch may continuously scan the bluetooth broadcast and decode the touch state of each touch widget. And the instruction for specifically controlling the intelligent device can be generated by the intelligent watch and directly or indirectly sent to the intelligent device, or generated by the control end and directly or indirectly sent to the intelligent device.

2) Design selection display interface and selection mechanism

In the present system, a smart watch is used as a display device of the system. The reason for selecting the watch is that the watch can be worn on the wrist of the user, normal function selection of the user is not affected, and when the user touches a small component, attention does not need to be transferred between the touch interface and the display interface, so that one-hand touch input can be supported.

On the screen of the intelligent watch, one function is displayed at a time, each function is sequentially displayed within fixed time, and the circular progress bar at the edge of the screen is used for displaying time progress.

As shown in fig. 4a, the display layout consists of three parts. (1) The top is the name of the device currently to be manipulated. (2) The middle is an icon of a function, occupying a larger position in the layout. (3) The bottom most is the state of the function. The above display design makes it sufficient for the user to recognize the currently displayed device, function, and state at a glance, reducing the mental burden on the user.

In the present system, the selection of devices has been integrated into the selection of functions using a semantic-based input mechanism, avoiding additional effort by the user.

In the primitive system, the selection of a function is triggered by the pressing of a widget, a release or drag event, in which events semantics are included, which in turn correspond to the function. For example, as shown in fig. 4a to 4c, when the user presses the "confirm" widget, the system detects the behavior, the display interface of the watch uses a recommendation algorithm (making a recommendation according to context, user location, history information, etc.), and the recommendation includes a "confirm" semantic correspondence function (e.g., a function that does not include a binary semantic "confirm" and does not appear in the recommendation list). At the same time, the circle at the edge of the watch screen will indicate a countdown, if not the function the user wants to select is currently displayed, the user will keep holding the widget's action. The countdown ending system may recommend the next function until the user releases the touch widget, indicating that the user confirms the selection.

Specifically, in fig. 4a, when a user wants to control the switching of the desk lamp, he needs to press the confirmation button first, and at this time, the wristwatch may display a control interface of the television switching function, and continue to press the confirmation button, and the control interface is switched from the television switching function to the desk lamp switching function, and at this time, the desk lamp can be turned on or off by releasing the button;

fig. 4b shows that when the user wants to watch the tv program of the next channel, he needs to press the "next" button first, and at this time, a control interface for playing the next song by the speaker may be displayed on the wristwatch, and then the "next" button is pressed continuously, and the control interface switches from the function of playing the next song by the speaker to the function of the next channel of the tv, and at this time, the user can switch the tv program to the next program by releasing the button;

fig. 4c shows that, when a user wants to increase the sound volume of the sound box, the user needs to first press the sliding bar and slide the sliding bar along the specific direction of the sliding bar, at this time, a control interface which may display the brightness of the desk lamp is displayed on the wristwatch, the sliding bar is continuously pressed, the control interface is switched from the brightness of the desk lamp to the sound box volume, at this time, the sliding bar is released, the sound box volume can be increased, if the user forgets to slide the sliding bar along the specific direction of the sliding bar before, the user waits to switch to the sound box volume, and then the sliding bar is released after sliding along the specific direction of the sliding bar, so that the sound box volume can be increased.

The following are system examples corresponding to the above method examples, and this embodiment can be implemented in cooperation with the above embodiments. The related technical details mentioned in the above embodiments are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the above-described embodiments.

The invention also provides a semantic-based intelligent device control system which is characterized by comprising an intelligent device and a control end connected with the intelligent device, wherein a display device is arranged in or externally connected with the controller to display the function of matching the input semantics of the user, the controller comprises at least one input sensor, the input sensor is used for representing specific semantics, and the user can control the display function of the display device by triggering the input sensor.

And the display device periodically displays the function of matching the semantics input by the user. And continuously triggering the input sensor to switch the current function displayed by the display device. The input sensor includes: a forward button, a back button, a ok button, and a touch slide, the forward button and the back button representing a "previous" and "next" semantic, respectively, the ok button representing an "ok" semantic, and the relative sliding of the touch slide representing an "increase" or "decrease" semantic. The intelligent device can be a device with the function of the Internet of things, such as a refrigerator, a sound box, a television, a desk lamp and the like. The control end comprises a Bluetooth transceiver module used for connecting intelligent equipment. The display device is a smart watch. The content displayed by the display device comprises the name of the intelligent device, the function of the intelligent device and the state of the function of the intelligent device, wherein the name of the intelligent device matches the input semantics of the user.

Claims

1. a semantic-based intelligent device control method is characterized in that, comprising:

Step 1. Build an Internet of Things including smart devices and a control terminal, and the control terminal extracts the functions of each smart device;

Step 2, the control terminal extracts the user's input semantics through the input sensor, and filters the functions of each smart device based on the input semantics to obtain a function candidate set that only matches the input semantics. The function selected in the selection set controls the smart device to execute the corresponding function.

2 . The semantic-based smart device control method of claim 1 , wherein a display device is built-in or externally connected to the control terminal to display the function of the smart device; and the display device displays the function of the smart device. 3 .

3. The semantic-based intelligent device control method according to claim 1, wherein a display device is built-in or externally connected to the control terminal to display the function of the intelligent device; and the step 2 comprises: continuously triggering the input sensor to switch the function currently displayed on the display device.

4. The semantic-based intelligent device control method according to claim 1, wherein the input sensor comprises: a forward button, a back button, a confirmation button and a touch slider, the forward button and the back button respectively indicate "up". The semantics of "one" and "next", the OK button represents the semantics of "confirm", and the relative sliding of the touch slider represents the semantics of "increase" or "decrease".

5 . The semantic-based smart device control method according to claim 1 , wherein the smart device is a refrigerator, a speaker, a TV and a desk lamp with Internet of Things function. 6 .

6 . The semantic-based smart device control method according to claim 1 , wherein the control terminal comprises a Bluetooth transceiver module for connecting to the smart device. 7 .

7 . The semantic-based smart device control method of claim 1 , wherein the display device is a smart watch. 8 .

8 . The semantic-based smart device control method according to claim 1 , wherein the content displayed by the display device includes the name of the current smart device, the function of the current smart device and the function status of the previous smart device. 9 .

9. A semantic-based intelligent device control system, comprising:

Smart devices with Internet of Things functions, and control terminals connected to them;

Among them, the control terminal is used to extract the functions of each smart device, and extract the user's input semantics through the input sensor, and filter the functions of each smart device based on the input semantics to obtain a function candidate set that only matches the input semantics, According to the function selected by the user in the candidate set, the smart device is controlled to execute the corresponding function.