KR20180105301A - Image keyboard system for IoT device control - Google Patents

Abstract

Disclosed is a method for controlling an internet of things (IoT) device using an image keyboard by a mobile device. The method for controlling an IoT device includes the steps of: displaying a conversation window for a conversation between a user of the mobile device and a counterpart user; analyzing conversation contents inputted to the conversation window and selecting one or more IoT devices that need to be controlled according to an analysis result; displaying a virtual keyboard including an icon list corresponding to the selected one or more IoT devices; and displaying a remote controller image for controlling the corresponding IoT device on the image keyboard if the user selects a desired icon on the virtual keyboard. Accordingly, the present invention can easily control the IoT device through a graphic interface.

Description

Image keyboard system for IoT device control Image keyboard system for IoT device control

The present invention relates to control of an IoT device, and more particularly, to an image keyboard system for controlling an IoT device with a virtual keyboard using an SNS and an image, and a method for controlling the IoT device using the system.

IoT (Internet of Things) is also called Internet of Objects or Internet, and refers to the concept that all objects that can communicate can be connected to the network to perform mutual communication. Every systemically recognizable entity is classified as Things or Objects, which can include objects or persons that are equipped with local and remote communication capabilities and can produce and use data, such as sensors.

In order to operate or monitor each IoT device in the IoT environment, various methods such as a dedicated application, a web access, and a SNS (Social Network Service) control are used.

In particular, a method for controlling an IoT device through an instant messenger service, which is one of the SNS services, has been actively researched. According to a method of giving an SNS account to an IoT device and exchanging status information in the form of an SNS message, There is a problem that it is cumbersome and difficult to use because a message is required to be input.

In addition, there are few IoT devices that users need to control in the present situation where IoT products are not normalized or diversified. Therefore, there is no inconvenience to select necessary devices, operation method and operation for each device. However, with the rapid development of IoT technology and the prospect that nearby home appliances, convenience products and electronic devices will become IoT, especially in the near future, up to 20 billion IoT products will be produced and consumed. As the types and numbers of devices that users need to control increase explosively, it is difficult to select them one by one and to control them appropriately. When the life convenience products to which IoT technology is applied become common, IoT devices can be shared among several persons or organizations Will be.

In this case, it is difficult to find the necessary IoT device right now, to easily find the devices to be sequentially used from now to a specific time by operating time, and to easily manage it with family, friends, There is a problem that is difficult to share.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems of the prior art, and it is an object of the present invention to provide an instant messenger service such as a karaoke or a line, IOT devices that need to be controlled according to the context, meaning, or specific words are automatically recommended instead of inputting, and the virtual keyboard is changed to the IoT device controller image according to the situation. .

The present invention also provides an IoT device control method for allowing a specific IoT device to be shared with a specific user for a predetermined period by using an instant messenger service, or allowing a plurality of users such as a family member and an employee to share and use a specific IoT device.

According to an embodiment of the present invention, a method for controlling an IoT device using an image keyboard system according to an embodiment of the present invention includes displaying a dialog for a user of the mobile device and a user of the other party to communicate with each other ; Analyzing conversation contents inputted in the dialog window and selecting one or more IoT devices that need to be controlled according to the analysis result; Displaying a virtual keyboard including a list of icons corresponding to the selected one or more IoT devices; When the user selects a desired icon from the virtual keyboard, displaying a remocon (Remote controller) image for controlling the corresponding IoT device.

The IoT device control method includes: performing control of a corresponding IoT device according to a user's input through the remote control image; And inputting information on the control of the performed IoT device into the dialog of the dialog window.

The control method of the IoT device may further include the step of, when a control phrase including " grant control authority to the counterpart user " of the IoT device corresponding to the selected icon is input into the dialog of the dialog, Waiting to be received from the counterpart user via the dialog window; And controlling the IoT device according to a control message received from the counterpart user.

The control phrase granting the control authority may include information on at least one of a controllable range and a controllable time for the corresponding IoT device.

The step of displaying the remote controller image may include receiving information on a current state from the corresponding IoT device; Generating a remote control image including the received status information and a graphical interface for controlling the corresponding IoT device; And displaying the remote control image on the virtual keyboard.

The other user may be a registered virtual chat buddy for controlling the IoT device.

The IoT device control method includes: searching IoT devices controllable through the mobile device when opening a dialog for chat with a virtual chat buddy; And displaying a virtual keyboard including a list of icons corresponding to the retrieved IoT devices.

According to another embodiment of the present invention, a method for a mobile device to control an IoT device using an image keyboard system comprises the steps of: displaying a dialog for a user of the mobile device to talk with a remote user; Receiving a control statement including a meaning of " granting control authority " of a predetermined IoT device from the counterpart user via the dialog window; When the control phrase is received, displaying an image virtual keyboard including a remote control image for controlling the IoT device; Generating a control message for the IoT device according to the input of the user via the remote control image; And transmitting the generated IoT device control message to the counterpart user via the dialog window or performing control of the IoT device.

The control phrase giving the control authority includes information on the controllable range for the IoT device, and the remote control image can be generated reflecting the controllable range.

The control phrase granting the control authority includes information on the controllable time for the IoT device, and the remote control image may be deactivated when the controllable time elapses.

According to the present invention, an IoT device requiring control according to a dialogue context, a meaning, and a specific word in a dialog window is automatically recommended, so that control of the IoT device is performed while conversing, or a specific word or command entered in the dialog window is analyzed IoT devices that require control depending on the situation can be selected and grouped for convenient use.

In addition, according to the present invention, since the IoT device can be selected through the graphic interface of the image keyboard connected to the dialogue window and the control command can be issued, the IoT device can be easily controlled in an intuitive manner, Efficient control is possible even in situations where input is impossible, volume size, brightness level, location on a map, etc., even if precise control is impossible with text input.

In addition, according to the present invention, the operation status of current IoT devices can be noticed at a glance through the graphic interface of the image keyboard connected to the dialogue window, the control commands by touch, drag and swiping on the graphic are visually displayed, IOT device's usage period and use amount are changed because it recalculates control command by characterizing and storing in dialog window to prevent operation error and to store or analyze past history or required requirements to recommend or control optimal requirements. As the number of users increases, more convenient and convenient recommendation selection control becomes possible, and device development and marketing utilizing accumulated data become possible.

In addition, according to the present invention, a specific IoT device can be shared with a family member, shared with a certain number of friends or employees, and used together, and the sharing selection, sharing range, and sharing time of the IoT device can be easily adjusted.

1 is a diagram for explaining a network configuration of an image keyboard system for controlling an IoT device according to an embodiment of the present invention.
2 is a flowchart illustrating a method of controlling an IoT device using an image keyboard system according to an embodiment of the present invention.
FIG. 3 illustrates a screen on which an image keyboard for controlling an IOT device is generated according to an embodiment of the present invention.
4 is a flowchart illustrating a process of controlling an IoT device using an image keyboard system according to an embodiment of the present invention.
5 illustrates a remote control image of an image keyboard according to an embodiment of the present invention.
6 is a diagram illustrating a screen in which a dialog window provides IoT control information using an image keyboard according to an embodiment of the present invention.
FIGS. 7 to 9 are exemplary screens for explaining a process of granting a control right of an IoT device to a counterpart user according to an embodiment of the present invention.
FIG. 10 illustrates a screen for selecting and controlling an intra-home IoT device using an image keyboard according to an embodiment of the present invention.
11 to 13 illustrate screens for selecting and controlling IoT devices in a vehicle using an image keyboard system according to an embodiment of the present invention.
14 is a diagram for explaining an example of recommending IoT devices requiring control by analyzing conversation contents according to an embodiment of the present invention.
15 to 17 illustrate a screen for selecting and controlling an intra-home IoT device using an image keyboard according to an embodiment of the present invention.

The terms used in this specification will be briefly described and the present invention will be described in detail.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. Also, in certain cases, there may be a term selected arbitrarily by the applicant, in which case the meaning thereof will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term, not on the name of a simple term, but on the entire contents of the present invention.

When an element is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements as well, without departing from the spirit or scope of the present invention. The term " means ", "part "," module ", etc. in the specification means units for processing at least one function or operation, Lt; / RTI >

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

1 is a diagram for explaining a network configuration of an image keyboard system for controlling an IoT device according to an embodiment of the present invention.

1, an image keyboard system includes at least one IoT device 12, a user terminal 10, and a user of a user terminal 10 that can transmit and receive data using a wired / wireless network such as the Internet, And a terminal 11 of the other user.

The IoT device 12 is a smart grid system that manages energy such as electricity, gas, and water supply, a home network system that integrally manages home air conditioning, security, and home appliances, door lock and remote control of automobiles, Connected devices, connected car systems, smart watches, smart glasses, smart shoes, and other smart devices and sensors that are subject to IoT technology.

The user terminal 10 and the counterpart user terminal 11 may be mobile devices capable of controlling the IoT device 12 by a wireless communication method such as a smart phone, a tablet PC, a notebook computer, a wearable computer, The IoT device control method of the present invention may be applied to a personal computer (PC).

The user terminal 10 and the counterpart user terminal 11 are terminals capable of using an instant messenger service capable of exchanging conversations with other users and installing and executing a dedicated application for controlling the IoT device .

The user terminal 10 and the counterpart user terminal 11 can perform a service-wide operation such as configuration of a service screen, data input, data transmission / reception, and data storage under the control of a dedicated application.

The user terminal 10 is a user terminal having control authority of the IoT device 12, and the user registers IoT devices to be controlled in a dialog window or a setting window of the instant messenger service.

The IoT device 12 is controlled by a wireless communication method and performs a corresponding function when receiving a control signal from the user terminal 10. [

1, an IoT control platform may be included in the image keyboard system, in which case the user terminal 10 sends a control signal to control the particular IoT device to the IoT control platform, The IoT control platform will send a control signal to the IoT device 12 to perform control. Data transmission / reception between the user terminal 10 and the IoT device 12 in addition to the control signal will also be performed through the IoT control platform.

The user executes a dedicated application in the user terminal 10, opens a dialog window to communicate with the other user, and the dedicated application analyzes the conversation contents of the two users and selects which IoT device to control according to the analysis result. The selected IoT device includes at least a part of all IoT devices that the user terminal 10 can control, and a list of icons representing the selected one or more IoT devices is displayed on the screen. Here, for the sake of convenience of explanation, 'two users' is exemplified, but there may be a plurality of conversation partners.

A virtual keyboard is a virtual input device for inputting characters in a smart device without a keyboard or keyboard, and a virtual keyboard connected to input characters in a dialogue when using an instant message service. On the other hand, unlike the virtual keyboard, which is mainly used for typing, the image keyboard displays a remote controller (controller), a drawing, a structure diagram, and a map in a graphical image so as to efficiently control the IoT device in addition to a keyboard State, and inputting control instruction contents and IoT device control by touch and drag operation without character typing are possible.

The icon list is displayed through a virtual keyboard for inputting a dialog in a dialogue box, and the icon list can be displayed in the form of being added to the virtual keyboard or overlaid on the virtual keyboard while the virtual keyboard is displayed.

On the other hand, the image keyboard may include a user interface such as a button for listing controllable IoT devices. When the user selects the corresponding button, a list of icons of all controllable IoT devices will be displayed on the image keyboard. This is to allow the user to manually invoke the desired IoT device control.

When the user selects a specific icon from the icon list of the image keyboard, a remote controller image for controlling the corresponding IoT device, i.e., a virtual remote controller is displayed as an image keyboard. When the user manipulates the remote control image, a corresponding control signal is generated and transmitted to the IoT device. Information on which control has been performed is automatically input into the dialog in the dialog window so that the user and the opponent user can check the control contents.

The control of the IoT device 12 using the image keyboard will be described in more detail below with reference to FIG. 2 to FIG.

2 is a flowchart illustrating a method of controlling an IoT device using an image keyboard according to an embodiment of the present invention.

Referring to FIG. 2, when a user executes a dedicated application and opens a dialog window, a dialog window for communicating with the opposite user is displayed on the screen (S42). The opening of the dialog for the start of the conversation can also be done by the other user.

Here, 'user' is a person having control authority of the IoT device, and the corresponding IoT device will operate according to a control signal outputted from the user terminal 10.

The 'other user' does not have the control authority of the IoT device as the user of the user, but it can induce the control of the IoT device or delegate the control of the IoT device by talking with the user.

The control unit 10 analyzes the conversation contents of the user and the user of the other party, that is, the sentence input by the user and / or the sentence entered by the user of the other party, to determine whether there is content related to the control of the IoT device. Lt; RTI ID = 0.0 > IoT < / RTI > The analysis of the conversation contents is for automatically determining which of the IoT devices the user or the user wants to control.

Next, a virtual keyboard including the icon list corresponding to the one or more IoT devices selected in the above is displayed (S22). It is to recommend the appropriate IoT device (s) according to the context of the conversation.

FIG. 3 illustrates a screen on which an image keyboard for controlling an IOT device is generated according to an embodiment of the present invention.

Referring to FIG. 3, a dialog window for conversation with the conversation partner 'Kim OO' is opened, and the contents of the corresponding sentence are analyzed when 'Kim OO' inputs a sentence '(35). Various methods for analyzing the context can be used for analysis of the conversation contents.

A list of icons representing the selected IoT devices is displayed at the bottom of the virtual keyboard 36 according to the results of the analysis.

3, a home icon 31 indicating IoT devices on the home network associated with 'home', 'boiler', 'off', an icon 32 indicating a light control device, as well as an icon 30 corresponding to a heating device ), Icons 33 indicating security related devices such as door locks, and so on. If you are talking about whether you turned off the boiler, you will need to check the status of the lamps, door locks, appliances that worked together, and show them as an association icon.

In addition, if past and previous status information is compared and analyzed, IOT devices that are operating in a state different from the previous one may be recommended as a 'state of observation'. That is, if there is an air-conditioning system in which the electric oven or electric lamp which is not operating in the current time zone is operated or the temperature which is different from the present date (season) is detected as a result of comparing and analyzing the past usage data, .

Another example is that when a user enters "break me now" into the chat room, the dedicated application analyzes the conversation, recognizes the context, and synthesizes the current issue (time, location, weather, traffic, etc.) The necessary IoT devices are selected and a corresponding icon list is applied, and they are sequentially arranged and displayed by time zone.

At this time, the icon list is displayed by sorting the related IoT devices according to the time period or sequential operation and order of use from "preparation for leaving work-> leaving work-> home arrival", and the user can select which icon , You can see at a glance which icon to keep as it is.

IoT devices that need to be controlled at work may be set in advance, but the same devices may be selected after storing information on IoT devices that the user has previously controlled at the time of work.

Referring again to FIG. 2, when the user selects a desired icon on the virtual keyboard, a remocon (Remote controller) image for controlling the corresponding IoT device is displayed (S23). In order to display the remote control image, first, the information about the current state or the last used information is received from the corresponding IoT device, and a remote control image including the received state information and a graphic interface for controlling the corresponding IoT device is generated , And display the generated remote control image on the image keyboard.

In the example of Fig. 3, when the user touches and selects the heating device icon 30, the remote control image for the control operation of the heating device will be displayed. When the user selects the home icon 31, the IoT device (s) belonging to the home network will be listed, and if one is selected from this list, the remote control image for the control operation of the corresponding IoT device will be displayed. When the user selects the light control device icon 32, a remote control image for light control of each room will be displayed. Also, when the user selects the door lock icon 33, a remote control image capable of controlling opening and closing of the door lock will be displayed.

At this time, detailed information of the selected icon (current state information, past last information or previously set information) is displayed on the dialog window displayed together with the image keyboard, and detailed information of the dialog window (dialog information with the IoT device ), Or delete the detailed information by using the remote control image of the image keyboard or modify the control command.

The remote control image for controlling the IoT device is a virtual remote control that simulates a real control device, and is generated and displayed differently according to the type of the device. However, the controller, the operation button, the control panel, , It is possible to control the IoT device intuitively by minimizing the confusion and malfunction of the usage method.

The remote control image may be displayed in place of the keyboard for inputting characters of the virtual keyboard 36, or may be generated and displayed as an image keyboard separate from the existing virtual keyboard 36.

4 is a flowchart illustrating a process of controlling an IoT device using an image keyboard according to an embodiment of the present invention.

Referring to FIG. 4, a user can input an input for controlling a desired device through a remote controller image of the image keyboard (S40). The user terminal controls the IoT device according to the user's input received through the remote controller image (S41). To this end, the user's input is converted into a control signal for the corresponding IoT device and transmitted to the IoT device, and the IoT device will operate according to the received control signal.

5 illustrates a remote control image of an image keyboard according to an embodiment of the present invention. In the example of FIG. 3, when the user having the control right selects the heating device icon 30 from his / her user terminal, the remote control image 50 as shown in FIG. 5 may be displayed.

The remote control image may include one or more user interfaces for control and status information of the corresponding IoT device. A user interface button for control, a switch, a jog shuttle, and the like.

The user can generate a control command by touching, dragging, or swiping a predetermined portion of the remote control image.

At this time, when the user touched or dragged, the user can confirm the operation state of the button on the remote control image by the visual, auditory, or tactile by giving the vibration and the sound effect. .

5, the remote controller image 50 corresponding to the heating device includes status information indicating the current state of the heating operation, current temperature, desired temperature, etc., temperature control buttons 51 and 52, and power button 53 As shown in FIG.

When the user touches and selects the power button 53, a control signal to turn off the heating device is generated and transmitted to the corresponding IoT device. The IoT device can perform a control operation according to the control signal and return the result of the operation.

Referring again to FIG. 4, information on control of the IoT device performed through the remote controller image is input into the dialog window dialog (S42).

Instead of typing control commands or dialogs through the keyboard, users can easily turn on the lights in the home, adjust the temperature, add destinations to the vehicle's navigation, etc. via the remote control image, The control contents are displayed in the dialogue contents. Since the content of the control is converted into text and entered as a dialog, the other user can also see which device is controlled.

However, when the communication partner is not designated, only the user can confirm the device control state and the display contents.

In addition, since the control and execution histories are recorded as texts, it is possible to browse later or utilize the accumulation of the device control information, and the content controlled by the remote control image is re-confirmed as the content of the dialog in the dialog window, Input errors can be prevented and prevented.

6 is a diagram illustrating a screen in which a dialog window provides IoT control information using an image keyboard according to an embodiment of the present invention.

In the remote controller image shown in FIG. 5, when the power button 53 is touched to turn off the heating device, the corresponding information 60 will be input into the dialog window in dialog with the other user as shown in FIG. The user who has cut off the power of the heating device automatically inputs the information into the conversation even if the user does not directly input the information into the chat room.

As described above, in the IoT device control method according to the present invention, a user interface (IoT device selection button, IoT device control remote controller, etc.) for controlling the IoT device is coupled to the virtual keyboard displayed for dialog input in the dialog window. You can view and control the current status, control method, and control history of all IoT devices from the image keyboard.

In addition, by analyzing the dialog between the users, it is possible to estimate the IoT device requiring control and automatically display the icon of the corresponding IoT device to the user having the control authority, thereby inducing device control and controlling the touch, drag and / It is easy to control the IoT device by pinging.

When the use of the hand is not free, it is possible to input a control command by voice. In this case, the input speech is recognized and replaced with text, and then a control signal can be generated using it and a conversation to be input to the dialog window can be generated.

FIGS. 7 to 9 are exemplary screens for explaining a process of granting a control right of an IoT device to a counterpart user according to an embodiment of the present invention.

7 is a mobile device of a child who is a user of the other party, and the other user "son" in the dialog box with the user "Dad" &Quot; I will watch TV ".

The IoT device control application executing on the terminal of the user 'father' analyzes the context of the input dialog, determines that it is necessary to control the IoT device called TV, and transmits the TV control icon 74 to the image It will display on the keyboard.

When the user selects the TV control icon 74, the TV remote control image will be displayed on the image keyboard, and a control signal will be generated and transmitted to the TV according to the user operation through the TV remote control image. However, in the example of FIG. 8, the user 'Dad' gives the control authority to the opponent user 'son' instead of directly controlling the TV.

As shown in Fig. 8, when the user clicks " OK. Quot; one hour only " 72, the dedicated application can grasp the context, grant control authority to the other user, and further determine that the other user is allowed to control the TV for one hour.

At this time, it is recognized that the dialog input by the user includes a control phrase for giving the control authority of the corresponding IoT device to the counterpart, and the application of the user terminal waits for the control message from the counterpart user via the dialog window.

The control phrase granting the control authority may include information on the controllable time for the corresponding IoT device, and may also include information on the controllable range.

The user terminal of the other party sends a message "OK. Quot; only one hour " is received, it can be judged as reception of the control phrase for authorization by context analysis. When the remote control user touches and selects the received control phrase, the remote controller image for controlling the corresponding IoT device is displayed on the image keyboard connected to the dialog window of the opposite user terminal.

If the control phrase includes information on the controllable range, a remote control image capable of operating only the controllable functions will be generated by reflecting the information. For example, it is possible to prevent an adult channel from being used or to effectively restrict a pay channel, and only the remaining channels are displayed in the generated remote control image after the forbidden channel is deleted.

If the control phrase includes information on the controllable time, the device controllable time through the remote control image is limited.

As shown in FIG. 9, the dedicated application of the counterpart user terminal analyzes the conversation of the user entered in the chat room, recognizes that the TV control authority is temporarily granted to the counterpart user for one hour, and displays the remote control image 90 ) On the image keyboard.

The opponent user 'Son' can perform control operations such as turning on the power of the TV and selecting a desired channel through the image keyboard, and a control message corresponding to the control operation performed at this time is generated and automatically inputted into the dialog window.

The user terminal 'Dad' performs control according to the received control message. That is, the user terminal converts the control message received through the dialog window into a control signal, and transmits the control message to the corresponding IoT device, thereby performing the control requested by the counterpart user. Also, the user 'dad' can know the control contents by the user of the other party through the dialog window.

In this embodiment, since there is a time limit for authorization, when the time limit of 1 hour has elapsed, the power of the corresponding IoT device is automatically shut off, the TV control remote controller 90 is erased from the terminal of the opposite user 'son' The virtual keyboard will reappear.

8, instead of inputting a dialog for authorization, an authorization icon may be given to the user terminal so that a remote control image capable of controlling the corresponding IoT device is displayed on the terminal of the other user .

For example, it is possible to provide a TV-shaped TV control authorization icon on the virtual keyboard displayed on the user terminal of FIG. 8, to control the remote control image for TV control to be displayed on the terminal of the other user instead of the user terminal, It is possible to control the TV remote controller to be displayed on both the user terminal and the terminal of the other user. In this way, the user can take the TV remote control out of the dialog window and give it to the other party so that the control right can be handed down in a more intuitive way.

In this case, a control phrase for granting a control right is generated and input to the dialogue window, or an emoticon is transmitted from the instant messenger service, a TV icon is input to the dialogue window, and the application of the user terminal receives a control message .

The user of the other party can control the TV using the TV control remote controller while the corresponding window is open. If there is a time limit, the TV can be controlled within a limited time even if the dialog window is opened. The TV control remote controller image 90 assigned to the terminal will be deactivated or erased.

At this time, the remote controller image displayed on the image keyboard can be enlarged to the entire screen of the terminal device by pinching out (opening two fingers at the same time) for convenience of control operation of the buttons and the like.

Also, in the above example, a method of sharing the control authority between the father and the son has been described, but the same method can be used to grant authority to a plurality of users. In other words, in addition to my dad and son, IoT device control privileges can be shared by several family members. In addition, it is possible to designate the usage limit of control authority such as sharing time and shared channel at times, so that it is convenient for family members, friends and employees It is possible.

According to another embodiment, a user interface can be provided so that an IoT control command can be issued through an artificial intelligence (AI) user who is not an actual user, that is, a chat window with a virtual chat buddy.

In this embodiment, when an AI user for controlling an IoT device is registered as a buddy (chat buddy) and starts a conversation with an AI user, the conversation contents are analyzed, an IoT device to be controlled according to a context is recommended, It is possible to support control of the IoT device by displaying the corresponding remote control image on the image keyboard.

When opening a dialog for chatting with a virtual chat buddy registered for control of the IoT device, it is possible to search IoT devices controllable through the user terminal and display a virtual keyboard including an icon list corresponding to the retrieved IoT devices have. When the user selects the desired IoT device from this list, the corresponding remote control image is displayed on the image keyboard, thereby allowing the device to be controlled.

At this time, it is possible to maximize the efficiency of the device control by receiving the accumulated information of the AI user who is the communication partner, or the currently selected issue (weather information, location information, traffic information, other user information, etc.) .

FIG. 10 illustrates a screen for selecting and controlling an intra-home IoT device using an image keyboard system according to an embodiment of the present invention.

Referring to FIG. 10, IoT devices that are recommended according to the analysis of the conversation contents in the virtual keyboard 101 associated with the instant messenger service dialog can be checked (100). If the user selects one of them, And converted into the remote control image 106 of the device.

A plurality of registered IoT devices can be classified according to the category 104. 10 is a screen when the user selects the home icon 105. Detailed information about the IoT devices connected to the home network is displayed in the dialog window 103. When the audio 107 is selected, Remote control image 106 is displayed.

The user can input a control command by touching or dragging from the remote control image 106 as if the user operated the real controller, the remote controller, or the switch in off-line. In general, it is possible to control the control such as temperature, time, speed, on / off by typing or dictation which is easy to judge by understanding accurate value or reference value. However, The remote control image using the graphic image is very useful because it is difficult to precisely control the lighting brightness of the room and the position of the specific portion on the map by typing or dictation.

 The input control command is converted into a control signal by the application and transmitted to the corresponding IoT device.

When the user succeeds in powering off the audio device through the remote control image 106, the control history 108 is automatically input in the dialog window, so that the user and the user of the other party can confirm this.

11 to 13 illustrate screens for selecting and controlling IoT devices in a vehicle using an image keyboard system according to an embodiment of the present invention.

Referring to FIG. 11, when the vehicle icon 111 is selected in the virtual keyboard 110 in which icons of the registered IoT devices are displayed, a remote control image 112 for controlling the vehicle navigation apparatus is displayed. When the user touches the waypoint in the remote control image 112, the corresponding control signal is generated and transmitted to the vehicle navigation apparatus, and information on the control signal is displayed in the dialog 113 of the dialog window.

According to another embodiment, when the user shares the remote control image (FIG. 11 (a)) on which the navigation is displayed with the user of the other party as the conversation partner, The navigation image already set by the user is displayed. Referring to FIGS. 11A and 11B, it can be seen that the current position, the current traveling speed, the traveling route, and the destination of the other party as well as the current position of each other are displayed and displayed in real time between the navigation image of the user terminal and the navigation image of the counterpart terminal have.

The present navigation system only exchanges the location of the final destination with the map image or text and gives only the map, address and name of the final destination. However, according to the present invention, And the current operation information of the user and the other party is shared in real time. Therefore, even if the vehicle departs at different times or different positions, the traveling information such as route change, traveling delay, destination change, There is a convenience in that it can check in real time and cope with and deal with when a stopover place such as a rest stop is newly created in the middle of a destination, a route is changed, or an accident or an emergency situation is delayed.

In addition, it is possible to efficiently modify and adjust driving speed, driving direction, route, etc. based on the traveling information of the other party.

Fig. 12 shows an example of selecting the waypoint by the text input 121 in the virtual keyboard, and Fig. 13 shows an example of an image keyboard for inputting waypoint information via the remote controller image 112 including the map image will be.

14 is a diagram for explaining an example of recommending IoT devices requiring control by analyzing conversation contents according to an embodiment of the present invention.

Referring to FIG. 14, the virtual keyboard connected to the dialog window 144 includes an IoT icon 140, and if it is selected, the list of registered IoT devices and the current status of each IoT device can be confirmed.

Also, when a dialog 141 of " leaving work " is input to the virtual keyboard, information on at least one of the content of the input character, the current time information, the current position information and the previous IoT device control history is referred to, Selects IoT devices suitable for the current situation among the IoT devices and displays corresponding icons on the virtual keyboard 142. [

According to an embodiment of the present invention, the grouping of IoT devices is not performed manually but the input sentences are analyzed to select and sort the related IoT devices, and then the current state, the previous history, the issue , Traffic, etc.). In the example of FIG. 14, information about the selected IoT devices is displayed in the dialog window area 144.

In addition, the details controlled by the user through the image virtual keyboard are converted into conversations and automatically input into the dialog window 144. [ The details of the control may include what the IoT device selected by the user is, and what control is indicated for this IoT device.

Specifically, when the user selects the vehicle icon 143 from the virtual keyboard 142, the selection information is automatically input into the dialog 146 and the corresponding remote-controller image 148 is displayed as the image keyboard. When a specific position 147 is selected as the stop position on the remote control image 148, a corresponding control signal is generated and transmitted to the vehicle navigation apparatus, and the control details are input to the dialog 149.

Referring to FIG. 14, two routes are displayed in the original travel route. When the stop route 147 is selected by the user's touch, the past use history, the current issue (weather information, Traffic, accidents, other user information, etc.).

Thereafter, the IoT devices associated with " off work " can be preset to operate sequentially in time zones. For example, it is possible to set the temperature controller to be operated so that the room temperature becomes the predetermined value at the arrival time, prepare the unlocking of the door lock and other security devices, and then adjust the ON / OFF time of the TV or the lighting device. That is, it can be implemented not only by analyzing the conversation contents to recommend the associated IoT devices, but also by the application itself to judge the progress schedule of the IoT devices in conversation form and notify the user.

The user can add and / or delete the IoT device to be controlled in the interface as shown in FIG. 14 or modify the reserved control operation. Specifically, control such as changing the desired indoor temperature or changing the reserved TV channel by touching, dragging and / or swapping can be performed on the remote control image, similar to changing the traveling route by adding the waypoint in the previous example have.

15 to 17 illustrate a screen for selecting and controlling an intra-home IoT device using an image keyboard according to an embodiment of the present invention.

15, when the home icon 151 of the virtual keyboard is selected, a list of the IoT devices connected to the home network and the current status thereof are displayed in the dialog window area 150, and the IoT device icon list and the control of the selected IoT device 153 An image keyboard 152 containing a remote control image for the remote control is displayed.

By touching each room on the image keyboard 152 and selecting it, the A / V device existing at the corresponding location can be controlled, and the control details will be displayed in the dialog 154.

Fig. 16 shows an example of controlling the light devices in each room in the home. You can touch each room on the image keyboard to reserve the time and illuminance to turn on / off the light at that location. At this time, since the control contents predicted based on the last used or pre-set information and the current situation (departure time and arrival time) and issues are presented, it is possible to operate the separate remote control image The modified IoT device will be controlled with the changed contents.

17, when the home icon 151 is selected in the virtual keyboard, the selection history is automatically input into the conversation 171, and the image keyboard 170 including the icon list of the IoT devices belonging to the home category and the remote control area, Was displayed. The remote controller image corresponding to the IoT device selected on the image keyboard 170 will be displayed in the remote control area and the user can control the desired IoT device through remote control operation. The details of the control will also be automatically entered into the conversation dialog.

The method according to an embodiment of the present invention can be implemented in the form of a program command which can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

10: User terminal
11: terminal of the other user
20: IoT (Internet of Things) device

Claims (12)

A method for a mobile device to control an Internet of Things (IoT) device using an image keyboard system,
Displaying a dialog for the user of the mobile device and the user of the other party to communicate;
Analyzing conversation contents inputted in the dialog window and selecting one or more IoT devices that need to be controlled according to the analysis result;
Displaying a virtual keyboard including a list of icons corresponding to the selected one or more IoT devices;
And displaying the remote controller image for controlling the corresponding IoT device on the image keyboard when the user selects a desired icon on the virtual keyboard, and an IoT device control method using the same . The method according to claim 1,
Performing control of the corresponding IoT device according to a user input through the remote control image;
Further comprising the step of inputting information on control of the performed IoT device into a dialog of the dialog window, and an IoT device control method using the same. The method according to claim 1,
A control message for giving a control right of the IoT device corresponding to the selected icon to the counterpart user or a control message for the IoT device being received from the counterpart user via the dialog window when the icon is input to the dialog window ;
And controlling the IoT device in accordance with a control message received from the counterpart user. The image keyboard system for controlling an IoT device and the IoT device control method using the same. The method of claim 3,
Wherein the control phrase giving the control right includes information on at least one of a controllable range and a controllable time for the corresponding IoT device, and an IoT device control method using the same. The method according to claim 1,
Wherein the step of displaying the remote-
Receiving information on a current state and predetermined control information from the corresponding IoT device;
Generating a remote control image including the received status information and control information and a graphical interface for controlling the corresponding IoT device;
Displaying an image of the remote controller on the image keyboard; and controlling the IoT device using the image keyboard system. The method according to claim 1,
Wherein the counterpart user is a virtual chat buddy registered for controlling the IoT device, and an IoT device control method using the same. The method according to claim 6,
Retrieving controllable IoT devices through the mobile device when opening a dialog for chat with the virtual chat buddy;
Further comprising the step of displaying a virtual keyboard including a list of icons corresponding to the searched IoT devices, and an IoT device control method using the same. An image keyboard system for controlling an IoT device according to any one of claims 1 to 7 and a computer program recorded on a recording medium for performing an IoT device control method using the same. A method for a mobile device to control an IoT device using an image keyboard system,
Displaying a dialog for the user of the mobile device to communicate with the other user;
Receiving a control statement or icon from the counterpart user through the dialog window to give control authority of a predetermined IoT device;
When the control phrase or icon is received, displaying an image keyboard including a remote control image for controlling the IoT device;
Generating a control message for the IoT device according to the input of the user via the remote control image;
Transmitting the generated IoT device control message to the counterpart user via the dialog window or performing control of the IoT device, and an IoT device control method using the same. 10. The method of claim 9,
Wherein the control phrase for granting the control right includes information on a controllable range for the IoT device, and the remote controller image is generated reflecting the controllable range. IoT device control method. 10. The method of claim 9,
Wherein the control statement for giving the control authority includes information on the controllable time for the IoT device and the remote control image is inactivated when the controllable time elapses. IoT device control method. An image keyboard system for controlling an IoT device according to any one of claims 9 to 11 and a computer program recorded on a recording medium for performing an IoT device control method using the same.

Images