KR102073839B1 - An Automatic Control System for Smart devices and Method thereof - Google Patents

Abstract

According to the present invention, a smart device interlocking control system comprises a smart device that can be controlled through the Internet and a device control server that can control and monitor the smart device through the wired/wireless Internet, and interlocks and automatically controls all smart devices, regardless of whether a smart device manufacturer provides an API, through an application control means which includes an automatic execution part automatically executing an application (hereinafter referred to as ″app″) for the smart device and a recognition processing part capturing and recognizing a specific part of a screen on which a result value of the executed app is displayed to determine whether a condition is satisfied.

Description

{An Automatic Control System for Smart devices and Method}

The present invention relates to a smart device interworking control system and control method, and automatically control all smart devices interworking regardless of whether a smart device manufacturer provides an application programming interface (API) for smart device interworking control. It relates to a system and a method thereof.

Smart devices can be controlled via the Internet, and the smartphone app provided by the manufacturer refers to a device that can be remotely controlled using a smartphone.

Examples of smart devices include smart home appliances such as fans, refrigerators, televisions, air conditioners, and air purifiers that can be controlled through the Internet, and smart sensor products such as temperature sensors, humidity sensors, gas sensors, fine dust sensors, carbon dioxide sensors, and motion sensors. In addition, there are smart actuators such as window switch, blinder regulator, hot water valve switch, and gas valve switch. The makers of these smart devices provide apps for smartphones to remotely control, monitor and control the devices, and have simple automation functions such as adjusting according to time zones or responding to problem messages.

In order for smart devices to help us in our lives, users need to be able to program in order to perform more various functions depending on the situation. It is an application programming interface (API) required when receiving. For example, in the case of a smart gas valve controlled via the Internet, the program can check the status by knowing how to use the API to request the current status and the API used to close the valve. You can write a program to close the.

However, most smart devices currently on the market can be controlled by apps provided by the manufacturer, but since the API is not provided, it is impossible for the user to control other smart devices.

For example, if you want to write an internet application control program that uses a sensor to open a smart window when the carbon dioxide concentration is high, close the window when the temperature is low, and close the gas valve when a propane gas is detected, Without an API, automation is not possible.

1 shows a general smart device 30 control and is controlled and monitored using a smart phone app 21 in a smart phone 20 through a smart device control server 40 via wired / wireless internet. Smart devices do not provide an API, but rarely do not provide an app 21 for a smartphone. The smartphone app is generally controlled by the user touching the smartphone screen, but can be automatically performed by the macro program as if the user touched it.

In addition, using the smartphone emulator can run the app of the smartphone on the user or administrator computer screen and control the smart device by the effect of touching the screen of the app displayed on the computer screen using a macro program.

1.Korean Registered Patent No. 10-1605507 (2016.03.16.)

The present invention has been made in view of the above necessity, to provide a system and method for automatically controlling all smart devices interworking regardless of whether a smart device manufacturer provides an application program interface (API) for smart device interworking control. do.

Smart device interworking control system according to the present invention, smart device interlocking control system includes a smart device that can be controlled via the Internet and a device control server that can control and monitor the smart device via the wired / wireless Internet, for smart devices App control means including an automatic execution unit for automatically executing an application (hereinafter referred to as an 'app') and a recognition processing unit for capturing and recognizing a specific part of the screen on which the result value of the executed app is displayed and determining whether the condition is satisfied. Through, it is characterized in that the automatic control by interlocking all smart devices regardless of whether the smart device manufacturer provides an API.

The app control means is characterized in that executed on the smartphone.

The smart device interworking control system is characterized in that it further comprises a service server that can control the smartphone over the Internet.

Characterized in that the app control means using a smartphone emulator on a user or administrator computer.

It is characterized in that by using the smartphone emulator on the service server to integrate and execute the app control means.

As another embodiment of the present invention, a smart device interlocking control method may include: receiving an API command from an interface unit, and extracting information of the smart device by interpreting the received API command; Selecting and driving a smartphone emulator provided according to a model of a smart device in a user computer or a service server, and driving a required smart device app to replace the smart phone; Interpreting an API command from the interface unit to extract user information and input a user ID and password; Interpreting an API command in the interface unit to extract control information to prepare location information necessary for app control; And executing the API command by controlling the app in order using the app information, the user information, and the control information of the device extracted from the API command in the app execution unit.

Even smart devices that do not provide essential APIs for Internet programming can be controlled by interlocking all smart devices by providing API control means by utilizing the smartphone app provided by the smart device.

The present invention can automate the smart device by time zone or condition for various external conditions by interlocking control of all smart devices.

1 is a block diagram of a general smart device control system.
2 is a block diagram of a smart device interworking control system according to an embodiment of the present invention.
3 is a block diagram of the app control means according to the present invention.
4 is a block diagram of a smart device interworking control system according to another embodiment of the present invention.
5 is a block diagram of a smart device interworking control system according to another embodiment of the present invention.
6 is a block diagram of a smart device interworking control system according to another embodiment of the present invention.
7 is a flowchart illustrating a linkage control using a smart device linkage control system according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and the contents described in the accompanying drawings, but the present invention is not limited or limited to the embodiments.

The Internet application control program checks the status of smart devices to be used through the Internet and repeats the process of giving control commands to specific devices through the Internet as necessary.

To program through the Internet, use the API (Application Programming Interface) provided by the server. An API is a collection of functions as a language or message format used for communication between an operating system and an application control program to make it easier for application control program developers. The most popular API protocol is the Hyper Text Transfer Protocol (http), which allows commands and data to be exchanged between the server and client programs.

The smart device manufacturer operates the device control server 40 for the smart device (30). If a smart device manufacturer provides an API for controlling each smart device 30, a user or a service provider may develop an application service in conjunction with these smart devices. However, at present, most smart device manufacturers provide an app 21 for controlling a device, but there are not many manufacturers that provide APIs for internet programming.

In order to solve the above problems, even if a smart device does not provide an essential API for Internet programming, all smart devices are interlocked by providing API control means by utilizing a smart phone app provided by a smart device. To provide a control system.

2 is a configuration diagram of a smart device interworking control system according to the present invention, the smart device 30 that can be controlled through the Internet, by executing the application provided by the smart device manufacturer (hereinafter referred to as "app") The smart phone 20 for controlling the smart device and the device control server 40 that can control and monitor the smart device 30 through the wired / wireless Internet.

The smart device 30 is a device that can be controlled via the Internet and equipped with a smartphone app provided by the manufacturer in the smart phone 20 can be remotely controlled.

Examples of smart devices 30 include smart home appliances such as fans, refrigerators, televisions, air conditioners, and air purifiers that can be controlled through the Internet, and include temperature sensors, humidity sensors, gas sensors, fine dust sensors, carbon dioxide sensors, and motion sensors. There are smart sensor products, and there are smart driving devices such as window switches, blinder regulators, hot water valve switches, and gas valve switches. The manufacturers of these smart devices provide apps for smartphones to remotely control, monitor and control the devices, and have simple automation functions such as control over time, control over conditions, or control over problem messages.

The device control server 40 controls the smart device 30 through a wired / wireless internet network.

The smart phone 20 controls the smart device 30 remotely through the device control server 40 connected to the mobile communication network and the Internet network by installing the app 21 provided by each smart device manufacturer as a mobile communication terminal do.

The present invention provides an app control means (100) to perform a driving API command according to whether or not the smart device manufacturer in accordance with whether or not the smart device manufacturers provide the API of the smart device in accordance with the condition. The app control means 100 is downloaded to a smartphone to operate as a single app and by running the app provided by each smart device manufacturer in the smartphone by a command from the outside of the smartphone and the result value is displayed on the specific screen Capturing the part and converting it into letters or numbers by image recognition process to determine whether the condition is satisfied and complete the command. The command consists of one or more combinations and becomes the API of the app control means 100. By using the app control means 100 API of the present invention, the smart device manufacturer can control in conjunction with other smart devices regardless of whether the smart device API is provided.

The app control means 100 is the input unit 110, the service server 10, the user or administrator computer 60 or the device control server 40 for setting the user login and conditions for smart period interlocked control as shown in FIG. Interface unit 120 for receiving an API command from the application control program of the app, the app execution unit 130 for executing the app for the smart device and the database unit 140 for updating and storing the information of the smart device.

 The app executor 130 recognizes whether the condition is satisfied by recognizing a specific part of the screen on which the auto-execution unit bundles and executes a plurality of commands as a single key and executes the result value of the executed app. It includes a processing unit.

The interface unit 120 includes a command interpreter that interprets an API command from an external application control program and who controls how and which smart device, and a server / client interface that exchanges API commands with a server or a client.

The input unit 110 logs-in with user information and authenticates for security, and sets a target condition and an operation.

4 may utilize the service server 10 as another embodiment of the present invention. It is a way for general users to request a professional service provider to provide services because it is very difficult because they do not have professional knowledge. The professional operator can produce and supply the app control means 100 according to the user's order.

The app control means may be produced by operating in the Android or iPhone system, or may be produced by an application control program operating in the operating system, such as a server or a window of the PC. In addition, by using the application control program 11 that can be used independently of the app of the smart device can be linked to control the smart devices (30).

FIG. 5 illustrates a method of interlocking and controlling a computer 60 of a user or an administrator using the smartphone emulator 50 without using the smartphone 20.

The smartphone emulator 50 executes an app for a smartphone on the emulator by operating the smartphone 20 on the display of the service server 10 or the user / manager computer 60 as if the desired app is selected and executed on the smartphone. do. That is, the app control means 100 is executed in the service server 10 or the user / manager computer 60 without directly using the smartphone 20.

There are many types of smartphone emulators, so the app for your smartphone may or may not run well, so choose the best emulator for each app for your smartphone.

The app control means 100 may be provided in plurality for each device or model. App control means 100 having a similar function and operation can be grouped, one app control means 100 can drive a plurality of smart device models. In this embodiment, the app control means 100 is operating inside the virtual smartphone of the Android operating system, but in another embodiment may be performed as a computer application running on the Windows operating system.

App execution unit 130 of the present invention drives the image recognition program to find and recognize the app for the smart device in the displayed screen, and executed by capturing a portion of the screen to interpret and convert to a letter or number and meets the condition Determine and execute the API command. Image recognition can be determined by coordinates or by shapes or patterns. These functions are the values that a macro finds and executes an app on a specific smart device, and is displayed on another smart device, just as a person finds, executes, controls, and reads the value of an app on a smart device on a smartphone. You can read the value stored in the database or read the value stored in the database and execute the following command set by time zone or when the condition is satisfied.

Image recognition is performed by scanning the shape of the displayed app, if necessary, and extracting the shape. Shape extraction extracts important features by simplifying an image, such as edge detection, and omits unnecessary parts. Image recognition may apply deep learning technology.

The state of the smart device 30 is also periodically checked through the smartphone app and if there is a change, update the state of the smart device 30 in the database unit 140. Through this process, the current state of the smart device 30 is always recorded in the database unit 140. Since the smart device 30 may be directly controlled through an app or may be controlled through another app, the smart device 30 should always be updated periodically.

The API command interpreter analyzes which smartphone emulator to use, which app to use, what is the user ID and password, and whether to add it to the information update list according to the model of the smart device. The emulator driver drives the smartphone emulator that is optimal for the app you want to use and runs the appropriate version of the smartphone app on it.

The auto-execution unit may perform a set of commands for hotkeys, in which order, at which position on the smartphone app screen, according to a control command.

6 provides an integrated service in a method of controlling interworking with the service server 10 using the smartphone emulator 50 without using the smartphone 20 as another embodiment of the present invention.

Interoperability between devices is essential for the spread of the Internet of Things (IoT), but there is no global standardization yet, so they are not compatible or interoperable with each device. According to the present invention, all smart devices are automatically linked to each other regardless of API. Can be monitored.

On the other hand, Figure 7 is a flow diagram of the interlocking control using a smart device interlocking control system using a user or administrator computer.

An interworking control method using a smart device interworking control system according to the present invention comprises the steps of: receiving an API command from an interface unit, interpreting the received API command and extracting information of the smart device;

Selecting and driving a smartphone emulator provided according to a model of a smart device in a computer, and driving a required smart device app to replace the smart phone;

Interpreting an API command from the interface unit to extract user information and input a user ID and password;

Interpreting an API command in the interface unit to extract control information to prepare location information necessary for app control;

And performing an API command by controlling the app in order using the app information, the user information, and the control information of the device extracted from the API command in the app execution unit.

An embodiment of a control and monitoring method of a smart device according to the present invention is as follows. The smart device of the embodiment is an air conditioner and a temperature sensor. The temperature sensor may be a temperature sensor built in the air conditioner, and may be controlled in a specific condition by using an air conditioner and an external sensor as an external sensor installed independently in a specific place separately from the air conditioner.

Example 1 Control Function

A. Receive API command to control the temperature of the air conditioner to a specific temperature to the interface of the app control means. API commands are entered according to the rules set by the service server. For example, if you say "GET-http: //TechAndSlow.com", you are asking for data about your homepage. You can also get the file by displaying the data according to the rules.

The smartphone emulator periodically updates the status information of the smart device in the information update list and stores it in the database so that the user can get the temperature data of the internal or external sensor by API command.

B. The service server runs the Android smartphone emulator so that the computer acts as a smartphone instead of a smartphone.

C. The service server checks the air conditioner model and runs the app for the smartphone that matches the model.

D. If you log in using user information and set the target temperature, you can start the air conditioner if the conditions are met according to the specific time zone or the status information of the temperature sensor.

E. Quit the smartphone app and the smartphone emulator.

Example 2 Air Conditioning Monitoring Function

A. The interface unit of the app control unit receives an API command to update the state of the temperature sensor every 10 minutes.

B. Register information update list of database and send update command to command interpreter when time is up.

C. The service server runs an Android smartphone emulator.

D. The service server checks the air conditioner model and runs the app for the smartphone that matches the model.

E. Login by using user information and read data and store it in database.

F. Quit the smartphone app and the smartphone emulator.

In the above embodiment, an Android smartphone emulator is exemplified. However, when using a service server, a user, or an administrator computer, an application control program or an emulator operating in a Windows operating system may be used.

The smart device interworking control system according to the present invention may be utilized as a platform of a smart device manufacturer, a communication service provider or a third service provider.

10: Service Server 11: Application Control Program
20: Smartphone 21: App for Smartphone
30: smart device 40: device control server
50: smartphone emulator 60: user / administrator computer
100: app control means

Claims (6)

In the smart device interworking control system comprising a smart device that can be controlled through the Internet and a device control server that can control and monitor the smart device through the wired / wireless Internet,
Captures the result of the smart device by capturing a specific part of the screen displayed on the executed auto-execution unit and the executed app to automatically execute the smart device application (hereinafter referred to as "app"), and the result value of the smart device App control means including a recognition processing unit for determining whether or not to meet the operating conditions of the smart device,
The app control means utilizes a smart device app, and executes a set command when an operating condition is satisfied, regardless of whether the smart device manufacturer provides an API or not, and automatically controls all smart devices by interlocking the smart device. Interlock control system. The method of claim 1,
The app control means is a smart device interlocking control system, characterized in that executed on a smart phone app installed for the smart device. The method of claim 1,
The smart device interworking control system further comprises a service server that can control a smartphone or a user / administrator computer over the Internet smart device interlocking control system. The method of claim 3,
Smart device interlocking control system, characterized in that for executing the app control means using a smartphone emulator on a user or administrator computer. The method of claim 3,
Smart device interlocking control system, characterized in that by using the smartphone emulator on the service server to integrate and execute the app control means. In the linkage control method using a smart device linkage control system,
Receiving an API command from a smartphone, a user / administrator computer or a service server at an interface unit, and interpreting the received API command to extract information of the smart device, user information and control information;
Checking the information of the smart device, selecting a smartphone emulator provided according to the model of the smart device, driving it on a display of the user / administrator computer or a service server, and driving and executing the required smart device app on the emulator;
Logging in using the user information in the interface unit and preparing location information necessary for controlling the app using control information; and
Including the step of performing the API command by controlling the app in order using the app information, the user information and the control information of the device extracted from the API command in the app execution unit,
App control means including the interface unit and the app execution unit is smart device interlocking control method characterized in that executed in the user / manager computer or a service server.

Images