KR102086512B1 - A remote control system, a mobile device and method of controlling a digital device thereof - Google Patents

Abstract

In the present specification, disclosed is a remote control system, a mobile terminal, and a device control method for performing data communication. Here, an example of a mobile device for controlling a digital device in a home network system may include a communication module supporting connection and data communication with the home network system; A receiver configured to receive status information on a digital device in the connected home network system through the communication module; A display configured to output the received digital device state information, an interactive user interface for controlling the digital device, and a first user input for controlling the digital device through the interactive user interface; And a controller configured to execute a remote control application for the remote control, generate a display control and a first control signal according to the first user, and transmit the generated control signal to the digital device.

Description

A REMOTE CONTROL SYSTEM, A MOBILE DEVICE AND METHOD OF CONTROLLING A DIGITAL DEVICE THEREOF

The present invention relates to a remote control system, a mobile device and a device control method for performing data communication.

With the recent rapid growth of the mobile market, people own at least one mobile terminal.

Conventional mobile terminals have been used exclusively for traditional functions such as message transmission and voice call. However, with the recent development of information technology (IT) technology and convergence technology, the conventional mobile terminal is transforming into a complex digital device, for example, the development into a smart phone. have.

As mobile terminals become smart phones, the Internet can be accessed through the network, and applications have been downloaded and personal usage, such as performing various functions or work-related tasks, has increased.

Mobile terminals are beginning to be used not only for personal utility but also for controller functions, such as for device control over a home network. However, there are still problems in controlling a home device through a home network through a mobile terminal, such as limitations of function control and inconvenience in use.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to provide digital device (s) belonging to one or more home network systems through a mobile device. To provide a remote control system (remote control system) to remotely control the digital device (s).

Another object of the present invention is to provide an application of a mobile device for controlling the digital device (s) and a UI / UX (User Interface / User eXperience) for controlling the digital device used for the application.

Still another object of the present invention is to provide a remote control system that makes full use of an existing system to secure expandability, backward compatibility, and the like at low cost.

Another object of the present invention is to provide a convenient and accurate remote control system in consideration of the state of a mobile device, the state of a home network, or the state of a digital device in a home network.

In this specification, a new standard model of a remote control system and a mobile device is proposed through the above-mentioned problems or any combination thereof, thereby improving product satisfaction and increasing purchase desire by maximizing user convenience.

On the other hand, the technical problem described herein is not limited to only the above-mentioned technical problem (s), other technical problems that are not mentioned from the following description to those skilled in the art to which the present invention belongs. It will be clearly understood.

An example of a method of controlling a digital device in a home network system in a mobile device according to the present invention for solving the above problems includes: executing a remote control application; Connecting with the home network system; Receiving and outputting status information on a digital device in the connected home network system; Outputting an interactive user interface for controlling the digital device; Outputting a first user input for controlling the digital device from the output interactive user interface; And transmitting a first control signal according to the first user input to the digital device.

According to the present invention, an example of a mobile device for controlling a digital device in a home network system includes: a communication module supporting connection and data communication with the home network system; A receiver configured to receive status information on a digital device in the connected home network system through the communication module; A display configured to output the received digital device state information, an interactive user interface for controlling the digital device, and a first user input for controlling the digital device through the interactive user interface; And a controller configured to execute a remote control application for the remote control, generate a display control and a first control signal according to the first user, and transmit the generated control signal to the digital device.

The technical solutions obtained in the present invention are not limited to the above-mentioned solutions, and other solutions not mentioned above will be apparent to those skilled in the art from the following description. It can be understood.

According to the invention,

First, there is an effect of providing a remote control system for remotely controlling digital device (s) belonging to one or more home network systems through a mobile device.

Secondly, there is an effect of providing an application of the mobile device for controlling the digital device (s) and a UI / UX for controlling the digital device used for the application.

Third, there is an effect of providing a remote control system that utilizes the existing system to maximize the expansion, compatibility and the like at low cost.

Fourth, there is an effect of providing a convenient and accurate remote control system in consideration of the state of the mobile device, the state of the home network or the state of the digital device in the home network.

Fifth, by presenting a new standard model of a remote control system and a mobile device, the user's convenience is maximized, thereby improving product satisfaction and increasing purchase desire.

Effects obtained in the present invention are not limited to the above-mentioned effects, and other effects not mentioned above may be clearly understood by those skilled in the art from the following description. will be.

1 is a block diagram schematically illustrating an example of a remote control system;
FIG. 2 is a block diagram illustrating an example of a detailed configuration of a management unit in the home network system of FIG. 1 described above; FIG.
3 is a block diagram of a mobile device in accordance with one embodiment of the present invention;
4A and 4B are front and perspective views of an example of a mobile device or a mobile terminal according to the present invention, respectively;
5 is a diagram illustrating a connection relationship in a home network system;
6 is a diagram for explaining a connection relationship in a remote control system;
7 and 8 illustrate a method of controlling a digital device via a remote control application in a mobile device;
9 to 14 illustrate a remote control scenario in the above-described mobile device; And
15 to 18 are flowcharts illustrating the remote control method.

Hereinafter, a remote control system, a mobile device, and a digital device control method will be described in detail with reference to the accompanying drawings.

The suffixes "module" and "unit" for components used in the following description are given or mixed in consideration of ease of specification, and do not have distinct meanings or roles. On the other hand, ordinal numbers such as "first-", "second-", "third-" and the like are not only sequential meanings, but also used for the purpose of distinguishing the equivalent components.

As used herein, the term “remote control system” refers to data communication between at least one home network system and at least one mobile device, thereby supporting the home communication through the mobile device. It includes any system supporting remote control of one or more digital devices in a network system. In the above-mentioned system, various devices such as a base station, a relay, a switching circuit, a gateway for connecting or pairing and data communication between the home network system and the mobile device or System may be included. The control may include power on / off, various function control, and the like for the digital device in the home network system.

Meanwhile, in order to support data communication in the remote control system, wireless Internet technologies for wireless Internet access include WLAN (Wi-Fi), Wibro (Wireless broadband), and Wimax (World Interoperability for Microwave Access). , High Speed Downlink Packet Access (HSDPA), Global System for Mobile communications (GSM), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Long Term Evolution (LTE), LTE-Advanced (LTE-A) And a short range communication technology for short-range communication, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), UWB ( Ultra Wideband, ZigBee, RS, etc. can be supported.

As used herein, the term "mobile device" includes a mobile phone, a smart phone, a tablet PC, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a PMP ( It may include any type of digital device used as a control means in a remote control system, including a portable multimedia player, navigation and the like. On the other hand, it is easy for those skilled in the art that the configuration according to the embodiments described herein may also be applied to a stand terminal such as a digital TV, a desk-top computer, etc., except that the configuration according to the present invention is applicable only to a mobile device. You will know. On the other hand, the mobile device may perform a function as the control means through itself or another digital device.

FIG. 1 is a block diagram schematically illustrating an example of a remote control system 100.

A mobile device for remotely controlling a digital device in a home network system may include a communication module supporting connection and data communication with the home network system, and receiving status information about the digital device in the connected home network system through the communication module. Outputting a first user input for controlling the digital device through a receiver, the received digital device state information, an interactive or conversational user interface (UI) for controlling the digital device, and the interactive user interface And a controller configured to execute a remote control application for the remote control, to generate a first control signal according to the display control and the first user, and to transmit and control the display to the digital device. .

The controller may detect a key word from the output first user input, compare the detected keyword with function data for controlling the digital device, and compare the result. Determine the function data corresponding to the detected keyword, and generate the first control signal based on the determined function data, and include text input, voice or speech recognition, and emoticon. The first user input may be received through at least one of an emoticon and a gesture. The controller may be further configured to control the digital device according to an attribute and a status of the digital device based on emoticon data for a messenger application stored in the mobile device. An emoticon may be provided, and at least one of an item indicating whether the digital device is currently on / off, a detail information display item according to a property or a location of the digital device, and an image or video data display item The state information of the digital device may be configured to include one or more items. The controller may receive a signal including an alarm message according to an event occurrence from the digital device, parse the alarm message in the received signal, and parse the parsed alarm message into the interactive user. Control to output on the interface, and generate and transmit a second control signal to the digital device according to a second user input for controlling the digital device according to the occurrence of the event, wherein the image or video data display item Upon receiving a third user input for selection, requesting an image obtained from at least one of a closed caption television server in the home network system and a sensor of a digital device other than the digital device, Recall the image received on request in the form of a pop-up. Control to output on an interactive user interface, the first control signal and the second control including at least one of a power on / off of the digital device, a function control currently performed and a function control reservation message of the digital device; At least one of the signals may be generated.

The mobile device may be any one of a mobile terminal, a tablet PC, and a wearable device.

Referring to FIG. 1, the remote control system 100 includes a client including at least one home network system 120 based on a network 110, and a cloud processor 132. ), A wearable device 134, mobile terminals 136, etc., to control the client.

In FIG. 1, each component in the remote control system 100 may perform data communication according to a predefined communication protocol with each other. In addition, although not shown, the remote control system 100 may further include at least one of a processor, a repeater, a switching circuit, a gateway, and the like necessary for supporting the data communication.

First, the client, that is, the home network system 120 will be described in detail as follows.

Home network system 120 supports building a home network including one or more digital devices. The home network system 120 may include a management module 122 that manages digital devices belonging to the home network. The management unit 122 will be described with reference to FIG. 2, and a detailed description thereof will be omitted. Each digital device in the home network system 120 may be controlled through the manager 122.

As illustrated in FIG. 1, the home network system 120 performs data communication with an external server through the network 110, and the digital device (s) in the home network according to a control signal of an external server through the data communication. ) Can change the status or control the function.

Meanwhile, the home network may be implemented in various forms. For example, the home network may be implemented in the form of a gateway or a repeater such as a Zigbee gateway or the like.

In addition, each digital device belonging to the home network system 120 preferably includes a communication module for data communication. In addition, each digital device in the home network system 120 may perform a control operation by directly performing data communication with the server through the external network 110 separately from the above-described management unit 122 through the communication module.

Next, a detailed description of the server side composed of means for controlling the aforementioned home network system 120 is as follows.

The wearable device 134 generates a signal for controlling at least one digital device of the home network system 120 connected through the network 110 and transmits the signal to the home network system 120.

The mobile terminal 136 generates a signal for controlling at least one digital device of the home network system 120 connected through the network 110 and transmits the signal to the home network system 120.

Meanwhile, any one of the wearable device 134 and the mobile terminal 136 may transmit a control signal to the home network system 120 through the other control means. For example, a user wearing the wearable device 134 does not transmit a control signal for a home network system 120 selected on the wearable device and a digital device selected for control in the selected home network system through the communication network of the wearable device. The control device may control the digital device by transmitting the control signal through the mobile communication network of the connected mobile terminal 136. This is, for example, because a user wearing and using the wearable device 134 may be a device that is not registered in the remote control system 100 or / and the home network system 120, that is, a device without control authority. In addition, UI / UX data may be received through other control means connected to the wearable device and registered in the systems 100 and 120 to output, select, and transmit the control signal on the wearable device. In addition to the above registration, transmission and reception of control signals may be performed in various paths according to various factors such as temporary network failure, network speed to efficiency, and data amount.

Meanwhile, the cloud processor 132 supports or controls the home device of the wearable device 134 or / or the mobile terminal 136 when the management unit 122 in the home network system controls the digital device in the home network system. It can support network system control.

FIG. 2 is a block diagram illustrating an example of a detailed configuration of the manager 122 in the home network system of FIG. 1.

The management unit 122 performs data communication with each component in the home network system or participates in data communication between each component, and manages all events occurring in the home network system such as schedules and functions of the respective components. / Indirect control.

Referring to FIG. 2, the management unit 122 of the home network system includes a voice service processing unit 212, a message service processing unit 214, and a context awareness processing unit. 216, a device management unit 218, a command parsing unit 220, a Session Initiation Protocol (SIP) processor 222, a Universal Plug and Play (UPnP) processor 224, And an output unit 226 and the like.

The voice service processing unit 212 is configured to perform not only voice service through the PSTN network but also voice over an IP network or a communication network other than the PSTN (eg, long-term evolution (LTE) or LTE-A (dvanced)). Refers to a configuration that handles services. Voice services over the latter IP network include services such as Voice over Internet Protocol (VoIP), mobile-VoIP (m-VoIP), and Voice over LTE (VoLTE).

The message service processor 214 may receive and process a control message generated by the manager 122 or for a device of a home network system through an external network. When the message service processing unit 214 receives the control message, the message service processing unit 214 may help to perform the requested control function in comparison with previously stored message service data.

The situation recognition processor 216 processes the situation awareness event occurring in the home network system. Here, context awareness refers to devices in or around a home network system based on communication and computing capabilities. The situation recognition processor 216 recognizes and determines an event generated through situation recognition in the home network system, and outputs the output unit 226, an output unit provided in the corresponding device, or a movement held by a user regardless of time or place. Useful information may be provided to a user through a display of a terminal, a wearable device, or the like.

The device manager 218 manages the status of at least one device related to the home network system, such as all schedules such as various events, situations, reservations, or processes of the devices in the home network system and whether or not the device malfunctions. The data related to the management may be temporarily stored.

The command parser 220 may be provided through the device manager 218 or the voice service processor 212, the message service processor 214, the situation recognition processor 216, the SIP 222, the UPnP 224, and the output unit. Parse the command received via 226. For example, the parsing may be performed by comparing a command table stored in a memory or a database in the management unit 122 although not shown.

The SIP 222 processes protocols for managing sessions or calls in connection with multimedia communications made in home network systems. In this case, the multimedia communication is handled by a real-time protocol (RTP) based on real-time transmission, and SIP may process an application level protocol. The protocol stack of the SIP may be as defined in the standard.

The UPnP processing unit 224 supports or processes UPnP, which is an interface standard for accessing or connecting to home network systems of various devices.

The output unit 226 may include various audio, video, image, etc. related to the home network system related service or service processing through an external network, such as management and processing of the management unit 122 such as an event in the home network system and processing of the event. Output the data.

3 is a block diagram of a mobile device in accordance with an embodiment of the present invention.

Hereinafter, a mobile terminal will be described in more detail with reference to the accompanying drawings.

The mobile terminal 300 may include a wireless communication unit 310, an A / V input unit 320, a user input unit 330, a sensing unit 340, an output unit 350, and a memory 360. ), An interface unit 370, a controller 380, a power supply unit 390, and the like. The mobile terminal 300 may be implemented by omitting or adding some of the components shown in FIG. 3 as necessary. On the other hand, the mobile terminal 300 may be implemented as a single module of some components of the components shown in FIG. 3, and some of the components may be implemented as individual components.

Referring to FIG. 3, each component of the mobile device 300 will be described in detail as follows.

The wireless communication unit 310 may include one or more modules that enable wireless communication between the mobile device 300 and the mobile communication system or between the mobile device and the network in which the mobile device is located. For example, the wireless communication unit 310 may include a broadcast receiving module 311, a mobile communication module 311, a wireless internet module 313, a short range communication module 214, and a location information module 315. .

The broadcast receiving module 311 receives a broadcast signal and / or broadcast related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel, a terrestrial channel, a cable channel, and the like. Two or more broadcast receiving modules 311 may be provided to the mobile terminal 300 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The broadcast management server may mean a server that generates and transmits a broadcast signal and / or broadcast related information or a server that receives a pre-generated broadcast signal and / or broadcast related information and transmits the same to a terminal. The broadcast signal may include not only a TV broadcast signal, a radio broadcast signal, and a data broadcast signal but also a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

The broadcast related information refers to information related to a broadcast channel, a broadcast program or a broadcast service provider. The broadcast related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 312.

The broadcast related information may exist in various forms. For example, the broadcast related information may exist in the form of Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB), Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld (DVB-H), and the like.

The broadcast receiving module 311 may include, for example, Digital Multimedia Broadcasting-Terrestrial (DMB-T), Digital Multimedia Broadcasting-Satellite (DMB-S), Media Forward Link Only (MediaFLO), and Digital Video Broadcast-Handheld (DVB-H). ), Convergence of Broadcasting and Mobile Service (DVB-CBMS), Open Mobile Alliance-BroadCAST (OMA-BCAST), China Multimedia Mobile Broadcasting (CMMB), Mobile Broadcasting Business Management System (MBBMS), Integrated Services Digital Broadcast Digital broadcasting signals may be received using digital broadcasting systems such as Terrestrial, Advanced Television System Committee-Mobile Handheld (ATSC-M / H), and File Delivery over Unidirectional Transport (FLUTE). Of course, the broadcast receiving module 311 may be configured to be suitable for not only the above-described digital broadcast system but also other broadcast systems.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 311 may be stored in the memory 260.

The mobile communication module 312 may include at least one of a base station, an external terminal, and a server on a mobile communication network such as, but not limited to, Global System for Mobile communications (GSM), Code Division Multiple Access (CDMA), and Wideband CDMA (WCDMA). Send and receive wireless signals with one. The wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text / multimedia message.

The wireless internet module 313 refers to a module for wireless internet access, and may be embedded or external to the mobile device 300. Wireless Internet technologies include Wireless LAN (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), GSM, CDMA, WCDMA, and LTE (Wireless Internet). Long Term Evolution), LTE-Advanced (LTE-A), and the like may be used. Herein, in view of the fact that the wireless Internet access by Wibro, HSDPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is made through a mobile communication network, the wireless Internet module for performing wireless Internet access through the mobile communication network ( 313 may be understood as a kind of mobile communication module 312.

The short range communication module 314 refers to a module for short range communication. As short range communication technology, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, RS, etc. may be used. Can be.

The location information module 315 is a module for obtaining a location of a mobile device, and a representative example thereof is a global positioning system (GPS) module. According to the current technology, the GPS module 315 calculates distance information and accurate time information away from three or more satellites, and then applies a triangulation method to the calculated information so that the three-dimensional current position according to latitude, longitude, and altitude is obtained. Information can be calculated accurately. Currently, a method of calculating position and time information using three satellites and correcting the error of the calculated position and time information using another satellite is widely used. In addition, the GPS module 315 may calculate speed information by continuously calculating the current position in real time.

The A / V input unit 320 is for inputting an audio signal or a video signal, and may include a camera 321 and a microphone 322.

The camera 321 processes image frames such as still images or moving images obtained by an image sensor in a video call mode or a photographing mode. The processed image frame may be displayed on the display unit 351. The image frame processed by the camera 321 may be stored in the memory 360 or transmitted to the outside through the wireless communication unit 310. Two or more cameras 321 may be provided according to a use environment.

The microphone 322 receives an external sound signal by a microphone in a call mode, a recording mode, a voice recognition mode, etc., and processes the external sound signal into electrical voice data. The processed voice data may be converted into a form transmittable to the mobile communication base station through the mobile communication module 312 and output in the call mode. The microphone 322 may be implemented with various noise removing algorithms for removing noise generated while receiving an external sound signal.

The user input unit 330 generates input data for the user to control the operation of the terminal. The user input unit 330 may be composed of a button 336 and a touch sensor (static pressure / capacitance) 337 positioned at the front / rear or side of the mobile terminal 300, and although not shown, a keypad ), A dome switch, a jog wheel, and a jog switch may be further included.

The sensing unit 340 detects a current state of the mobile terminal 300 such as an open / closed state of the mobile terminal 300, a location of the mobile terminal 300, presence or absence of a user contact, orientation of the mobile terminal, acceleration / deceleration of the mobile terminal, and the like. To generate a sensing signal for controlling the operation of the mobile terminal 300. Examples of the sensing unit 340 may include a gyroscope sensor, an acceleration sensor, a geomagnetic sensor, and the like. For example, when the mobile terminal 300 is a slide phone type, it may sense whether the slide phone is opened or closed. Also, whether the power supply unit 390 is supplied with power or whether the interface unit 370 is coupled to an external device may be sensed. The sensing unit 340 may include a proximity sensor 341.

The output unit 350 is used to generate an output related to sight, hearing, or tactile sense, and includes a display unit 351, a sound output module 352, an alarm unit 353, and a haptic module 354. Can be.

The display unit 351 displays (outputs) information processed by the mobile device 300. For example, when the mobile terminal is in a call mode, the mobile terminal displays a user interface (UI) or a graphic user interface (GUI) related to the call. When the mobile device 300 is in a video call mode or a photographing mode, the mobile device 300 displays a photographed and / or received image, a UI, or a GUI.

The display unit 351 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible). and at least one of a 3D display.

Some of these displays may be configured to be transparent or light transmissive so that they can be seen through them. This may be referred to as a transparent display. A representative example of the transparent display is TOLED (Transparent OLED). The rear structure of the display unit 351 may also be configured as a light transmissive structure. With this structure, the user can see the object located behind the terminal body through the area occupied by the display unit 351 of the terminal body.

According to an implementation form of the mobile terminal 300, two or more display units 351 may exist. For example, a plurality of display units may be spaced apart or integrally disposed on one surface of the mobile terminal 300, or may be disposed on different surfaces.

When the display unit 351 and a sensor for detecting a touch operation (hereinafter, referred to as a touch sensor) form a mutual layer structure (hereinafter referred to as a touch screen), the display unit 351 may be used in addition to an output device. Can also be used as an input device. The touch sensor may have, for example, a form of a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 351 or capacitance generated at a specific portion of the display unit 351 into an electrical input signal. The touch sensor may be configured to detect not only the position and area of the touch but also the pressure at the touch. If there is a touch input to the touch sensor, the corresponding signal (s) is sent to a touch controller (not shown). The touch controller processes the signal (s) and then transmits the corresponding data to the controller 380. As a result, the controller 380 may determine which area of the display unit 351 is touched.

The proximity sensor 341 may be disposed in an inner region of the mobile terminal covered by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object present in the vicinity without using a mechanical contact by using electromagnetic force or infrared rays. Proximity sensors have a longer life and higher utilization than contact sensors. Examples of the proximity sensor include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the touch screen is configured to detect the proximity of the pointer by a change in an electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, an act of allowing the pointer to be recognized without being in contact with the touch screen so that the pointer is located on the touch screen is referred to as a "proximity touch", and the touch The act of actually touching the pointer on the screen may be referred to as a "contact touch." The position where the proximity touch is performed by the pointer on the touch screen may mean a position where the pointer is perpendicular to the touch screen when the pointer is in proximity touch. The proximity sensor senses a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, and a proximity touch movement state). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

The sound output module 352 may output audio data received from the wireless communication unit 310 or stored in the memory 360 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output module 352 may also output a sound signal related to a function (for example, a call signal reception sound or a message reception sound) performed in the mobile terminal 300. The sound output module 352 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 353 outputs a signal for notifying occurrence of an event of the mobile device 300. Examples of events occurring in the mobile device include call signal reception, message reception, key signal input, and touch input. The alarm unit 353 may output a signal for notifying an occurrence of an event by vibration, in addition to a video signal or an audio signal. The video signal or the audio signal may also be output through the display unit 351 or the audio output module 352. In this case, the display unit 351 and the audio output module 352 may be a kind of alarm unit 353. May be classified.

The haptic module 354 generates various tactile effects that a user can feel. Vibration is a representative example of the haptic effect generated by the haptic module 354. The intensity and pattern of vibration generated by the haptic module 354 can be controlled. For example, different vibrations may be synthesized and output or may be sequentially output. In addition to the vibration, the haptic module 354 may be used to stimulate pins that vertically move with respect to the contact skin surface, jetting force or suction force of air through the jetting or suction port, grazing to the skin surface, electrode contact, electrostatic force, and the like. Various tactile effects can be generated, such as the effects of the heat-absorption and the reproduction of the sense of cold using the element capable of generating heat. The haptic module 354 may not only deliver the haptic effect through direct contact, but may also be implemented to allow the user to feel the haptic effect through a muscle sense such as a finger or an arm. Two or more haptic modules 354 may be provided according to a configuration aspect of the mobile device 300.

The memory unit 360 may store a program for processing and controlling the control unit 380, and temporarily stores input / output data (for example, a phone book, a message, an audio, a still image, a video, etc.). It can also perform a function for. The memory unit 360 may store a frequency of use of each of the data (for example, each telephone number, each message, and frequency of use of each multimedia). In addition, the memory unit 360 may store data regarding vibration and sound of various patterns output when a touch input on the touch screen is performed. The memory 360 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM Storage medium of at least one of random access memory (SRAM), static RAM (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EPEROM), programmable ROM (PROM), magnetic memory, magnetic disk, and optical disk. It may include. The mobile terminal 200 may operate in association with a web storage that performs a storage function of the memory 360 on the Internet.

The interface unit 370 serves as a path with all external devices connected to the mobile device 300. The interface unit 370 receives data from an external device or receives power and transfers the data to each component in the mobile terminal 300, or transmits data in the mobile terminal 300 to the external device. For example, wired / wireless headset ports, external charger ports, wired / wireless data ports, memory card ports, ports for connecting devices with identification modules, audio input / output (I / O) ports, The video input / output (I / O) port, the earphone port, and the like may be included in the interface unit 370.

The identification module is a chip that stores various types of information for authenticating the use authority of the mobile device 300, and includes a user identification module (UIM), a subscriber identify module (SIM), and a universal user authentication module (SIM). USIM: Universal Subscriber Identity Module, and the like. A device equipped with an identification module (hereinafter referred to as an 'identification device') may be manufactured in the form of a smart card. Therefore, the identification device may be connected to the terminal 300 through a port.

The interface unit 370 may be a passage through which power from the cradle is supplied to the mobile device 300 when the mobile device 300 is connected to an external cradle, or various command signals input from the cradle by a user. May be a passage that is delivered to the mobile device. Various command signals or power input from the cradle may be operated as signals for recognizing that the mobile device is correctly mounted on the cradle.

The controller 380 typically controls the overall operation of the mobile device. For example, perform related control and processing for voice calls, data communications, video calls, and the like. The controller 380 may include a multimedia module 381 for multimedia playback. The multimedia module 381 may be implemented in the controller 380 or may be implemented separately from the controller 380. The controller 380 may perform a pattern recognition process for recognizing a writing input or a drawing input performed on the touch screen as text and an image, respectively.

The power supply unit 390 receives an external power source and an internal power source under the control of the controller 380 to supply power for operation of each component. The battery may be a built-in battery made to be chargeable, and may be detachably coupled to the terminal body for charging. The connection port may be configured as an example of the interface 370 to which an external charger for supplying power for charging the battery is electrically connected.

The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware or a combination thereof.

According to a hardware implementation, embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and the like. It may be implemented using at least one of a processor, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented by the controller 380 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented in software applications written in a suitable programming language. The software code may be stored in the memory 360 and executed by the controller 380.

4A is a front perspective view of an example of a mobile device.

The disclosed mobile device 300 has a terminal body in the form of a bar. However, the present invention is not limited thereto and may be applied to various structures such as a slide type, a folder type, a swing type, a swivel type, and two or more bodies are coupled to be relatively movable.

The body of the mobile device 300 includes casings 301, 302, and 303 which form an appearance. In this embodiment, the case may be divided into a front case 301 and a rear case 202. Various electronic components are built in a space formed between the front case 301 and the rear case 302.

In some cases, an electronic component may be mounted on the surface of the rear case 302. Electronic components mounted on the surface of the rear case 302 include a battery, a USIM card, a memory card, and the like, which the user needs to remove. In this case, the rear case 302 may further include a rear cover 303 for covering the surface thereof. The rear cover 303 is formed of a structure that is easily removable by the user. When the rear cover 303 is removed, the surface of the rear case 302 is exposed.

As shown in FIG. 4A, when the rear cover 303 is coupled, only a part of the side surface of the rear case 302 may be exposed. When the size of the rear cover 303 is reduced, the rear surface of the rear case 302 may be exposed. Some may be exposed. When the rear cover 303 covers the entire rear surface of the rear case 302, the rear cover 303 may include an opening 303 ′ for exposing the camera 321 ′ or the sound output unit 352 ′ to the outside.

The cases may be formed by injecting synthetic resin, or may be formed to have a metal material such as stainless steel (STS), titanium (Ti), or the like.

The display unit 351, the audio output unit 352, the camera 321, the user input units 330/331, 332, the microphone 322, and the interface 370 may be disposed in the terminal body, mainly the front case 301. have.

The display unit 351 occupies most of the main surface of the front case 301. The sound output unit 352 and the camera 321 are disposed in regions adjacent to one end of both ends of the display unit 351, and the user input unit 331 and the microphone 322 are disposed in regions adjacent to the other end. The user input unit 332, the interface 370, and the like may be disposed on side surfaces of the front case 301 and the rear case 302.

The user input unit 330 is manipulated to receive a command for controlling the operation of the mobile device 300, and may include a plurality of manipulation units 331 and 332. The manipulation units 331 and 332 may also be collectively referred to as a manipulating portion.

Content input by the first or second manipulation units 331 and 332 may be variously set. For example, the first operation unit 331 receives a command such as start, end, scroll, and the like, and the second operation unit 332 adjusts the volume of the sound output from the sound output unit 352 or the display unit 351. Command), such as activating / deactivating the touch recognition mode.

4B is a rear perspective view of the mobile device shown in FIG. 4A.

Referring to FIG. 4B, a camera 321 ′ may be additionally mounted on the rear of the terminal body, that is, the rear case 302. The camera 321 ′ has a photographing direction substantially opposite to that of the camera 321 (see FIG. 4A), and may be a camera having the same or different pixels as the camera 321.

For example, the camera 321 has a low pixel so that the user's face is photographed and transmitted to the counterpart in case of a video call, and the camera 321 'does not immediately transmit a general subject. It is preferable to have a high pixel in many. The camera 321 'may be installed in the terminal body to be rotatable or pop-up.

The flash 323 and the mirror 324 may be further disposed adjacent to the camera 321 ′. The flash 323 emits light toward the subject when the subject is photographed by the camera 321 ′. The mirror 324 allows the user to see his / her own face or the like when photographing (self-taking) the user using the camera 321 '.

An acoustic output unit 352 'may be further disposed on the rear surface of the mobile device body. The sound output unit 352 ′ may implement a stereo function together with the sound output unit 352 (see FIG. 4A), and may be used to implement a speaker phone mode during a call.

In addition to the antenna for a call or the like, an antenna 316 for receiving a broadcast signal may be further disposed on the side of the mobile device body. An antenna 316 constituting a part of the broadcast receiving module 311 (see FIG. 3) may be installed to be pulled out of the mobile device body.

The mobile device body is equipped with a power supply unit 390 for supplying power to the mobile device 300. The power supply 390 may be configured to be embedded in the mobile device body or directly detachable from the outside of the mobile device body.

The rear case 302 may be additionally equipped with a touch pad 335 for sensing a touch. The touch pad 335 may be configured as a light transmissive type for the display unit 351. In this case, if the display unit 351 is configured to output visual information on both sides (i.e., in the direction of both the front and rear surfaces of the mobile terminal), the visual information may be recognized through the touch pad 335. do. The information output on both surfaces may be controlled by the touch pad 335.

Meanwhile, since a display dedicated to the touch pad 335 is separately mounted, the touch screen may also be disposed on the rear case 302.

The touch pad 335 operates in association with the display unit 351 of the front case 301. The touch pad 335 may be disposed parallel to the rear of the display unit 351. The touch pad 335 may have a size equal to or smaller than that of the display unit 351.

Hereinafter, embodiments related to a control method that can be implemented in the mobile device 300 configured as described above will be described with reference to the accompanying drawings.

For convenience of description, it is assumed that the mobile device 300 mentioned below includes at least one of the components shown in FIG. 3. In detail, the mobile device 300 according to the present invention may include a user input unit 330, a display unit 351, a memory 360, and a controller 380. In some cases, the mobile device 300 according to the present invention may further include a proximity sensor 341. When the display unit 351 is implemented as the touch screen 351, the present invention can be implemented more easily. Hereinafter, the present invention will be described on the assumption that the display unit 351 is the touch screen 351. do. In addition, when the user input unit 330 includes the touch sensor 337, the touch screen 351 may also be used as the user input unit 330.

In the present specification, the mobile device 300 may be a calling terminal or a receiving terminal according to an embodiment.

In order to control digital devices in the home network, each device in the remote control system must be connected to each other. However, as described above, the control means for controlling the digital device does not need to be directly connected to the digital device.

Hereinafter, the connection relationship in the home network system will be described first. Here, for convenience, the home network systems will be described with respect to each other based on the management unit 122 as an embodiment.

At least one digital device, a repeater, and a manager included in the home network may be connected to each other.

The digital device may include a set-top box, a digital TV set (SET), a microwave oven, a refrigerator, a washing machine, a cleaner, an air conditioner, an lighting device, an electric iron, and the like. On the other hand, such a digital device may include a communication module for communication and data communication with a repeater or / and a management unit, and a processing unit for processing data transmitted and received through the communication module.

The repeater supports data communication between the digital device and the management unit. The repeater may function as a kind of gateway. When the communication protocol for data communication is different between each digital device and the management unit, the repeater may appropriately convert data to be recognized by the corresponding communication protocol. For example, assume that the digital device supports only the first communication protocol and performs data communication accordingly, and the management unit supports only a second communication protocol different from the first communication protocol and thus performs data communication. At this time, even if the management unit transmits control data according to the second communication protocol to the digital device supporting only the first communication protocol, the digital device can properly process the data transmitted from the management unit even if it is impossible or unreceivable. There will be no. In such a case, the repeater converts and processes the data appropriately to prevent malfunctions or errors that may occur due to differences in communication protocols between the digital device and the management unit so that the operation can be performed according to the intention of each device. It is. However, as described above, the repeater may not be an essential component when there is no problem in data communication between the digital device and the management unit.

The management unit controls the connection and status with the digital device in the home network system via a repeater or directly.

First, a description will be given of the connection process between the digital device and the management unit.

The management unit transmits a packet A-1 including a message requesting to connect to the home network system, that is, the management unit, to the digital device when the power is turned on or at the request of the user.

Although not shown, the digital device may first transmit a packet for connection permission before receiving the A-1 packet from the management unit. This may be for reporting when there is a state change of the digital device or for additional control. In the above, the change of the state of the digital device is different from the current state of the digital device, such as power on or off, start or end of execution of a reserved or requested function, battery state, or change of the device state level to a predetermined level or less. All events that change to state can be included.

When the A-1 packet is received from the management unit, the digital device parses the corresponding A-1 packet to identify a message included in the parsed A-1 packet. The digital device transmits a response packet B-1 to the connection request according to the device status when the A-1 packet is the content of the connection request to the management unit. According to the device state, the digital device may transmit a B-1 response packet including a connectable message and a B-1 response packet including a disconnectable message. The latter non-connection message may be included in the B-1 response packet and transmitted to the management unit when the digital device performs a different function, is below a predetermined level of a battery, or a bad network state.

When the management unit receives the B-1 response packet from the digital device, the management unit parses the received B-1 response packet to identify the content of the message. If the B-1 response packet includes a connectable message, the manager transmits an A-2 packet including a connection permission message to the digital device. The connection permission message may include data for allowing the connection of the digital device, data on a channel or communication protocol to be used for connection of the digital device, connection time data of the digital device, and the like.

When the digital device receives the A-2 packet including the connection permission message of the management unit, the digital device parses the message of the packet. The digital device attempts to connect with the management unit by using time data, channel or communication protocol data included in the message of the parsed packet.

When the connection with the manager is completed, the digital device transmits a B-2 packet including a message indicating that the connection with the manager is successful to the manager.

When the B-2 packet is received from the digital device, the management unit stores the completion of the connection of the digital device according to the received B-2 packet, and terminates the connection process with the corresponding digital device. Thereafter, the management unit may continuously transmit the A-3 packet including the message for confirming the connection status to the corresponding digital device periodically / aperiod.

The connection relationship of the remote control system is made between the first device, network management and the second device.

Here, the first device corresponds to the server side of FIG. 1 described above and will be described as a mobile device for convenience. In addition, the second device corresponds to the client side of FIG. 1 described above and will be described below as a management unit for convenience. Meanwhile, network management may refer to the network of FIG. 1 or the control processor in the network described above.

When the mobile device is powered on, the mobile device executes an application for connecting to a remote control system (hereinafter referred to as a remote control application).

When the remote control application is executed, the mobile device transmits a control signal according to a home network connection or connection selection for remote control through the UI according to the execution.

The network management receives a mobile device or a control signal of the mobile device through a mobile communication server and transmits the control signal to the management unit.

The management unit parses the transmitted control signal of the mobile device and checks a connection state with the digital device in the home network signal according to the parsed control signal.

If it is determined that the connection state is not connected to the corresponding digital device, the management unit performs the connection process as described above. Otherwise, the management unit transmits a packet including the digital device control message based on the parsed control signal.

The manager receives a packet including a feedback message of the digital device according to the transmitted packet and returns a response signal to the control signal to the network management.

The network management transmits the response signal of the returned management unit to the corresponding mobile device directly or through a mobile communication server.

When the response signal of the management unit is received, the mobile device parses the response signal and provides the UI / UX to be provided to the user based on the message in the parsed response signal. Thereafter, the mobile device may perform data communication by transmitting and receiving a control signal with the management unit through the above-described process in addition to the user's input or selection through the provided UI / UX.

Meanwhile, in the above, the process of controlling one digital device in the home network system through the mobile device has been described, but the user simultaneously or sequentially controls a plurality of digital devices in the home network system through the UI / UX of the mobile device. You may. In the case of controlling the plurality of digital devices sequentially, the overlapping process of the previous digital device control process may be omitted. Meanwhile, in the case of controlling a plurality of digital devices at the same time, each digital device control message may be included in one control signal and transmitted. In this case, each of the digital device control messages may be identified by using a header portion of the control signal. You can make it possible. For example, in order to efficiently operate a remote control system, the management unit may allocate or manage identifier information of a digital device that can be registered or registered in a home network system, and transmit the same to a mobile device to share the information. The identifier information may be, for example, unique device number data of the device itself, ID (entifier) data, IP / UDP address data, MAC address data, home network system ID data, etc., which are arbitrarily assigned by the management unit or the mobile device. The identification information may be determined by including at least one data and combining the data.

On the other hand, unlike the foregoing, the mobile device may connect to the cloud server through a remote control application or through an internet application after power on. The cloud server may correspond to a network, for example. In this case, when the mobile device uploads a remote control message to the cloud server, the cloud server notifies the management unit that the remote control message has been uploaded.

5 to 7 illustrate a method of controlling a digital device through a remote control application in a mobile device.

Referring to FIG. 5, when the user executes a remote control application, the mobile device provides a UI as shown in FIG. 5A. Here, as described above, a UI and a process for registering or setting a home network system that is a remote control target (hereinafter, 'registered') or a connection process with the corresponding home network system may be performed before the screen of FIG. 5 (a). have.

Referring to FIG. 5A, the mobile device selects items 511, 512, and 513 that represent a home network system registered as a remote control target. In FIG. 5 (a), although the registered home network system shows three My Home, Parent Home, and Friend Home, this is only one embodiment and may be at least one. . Meanwhile, each item 511, 512, 513 may be displayed together in the form of a number of control targets in the corresponding home network system.

When the user selects the my home 511 in FIG. 5 (a), as shown in FIG. 5 (b), the room 1 (521), the room 2 (522), and the remote controllable room in the my home, as shown in FIG. Four second-order items are provided for selection in the kitchen 523 and the bathroom 524. In FIG. 5B, when the user selects the kitchen item 523 from the four next-order items 521, 522, 523, and 524, three next-order items 531, 533, and 535 are provided as shown in FIG. 5C.

Referring to FIG. 5 (c), as described above, items of a light 531, a refrigerator 533, and a dishwasher 535 that can be connected / controlled in the kitchen (top) of the home (top) are output. Here, each item of FIG. 5C may be provided in various forms so as to intuitively recognize whether the item is on or off. In FIG. 5C, whether a title item is turned on or off next to the item title name (lighting) is provided in text form. However, the present invention is not limited thereto. Meanwhile, in FIG. 5C, thumbnail images 532, 534, and 536 representing the corresponding item may be provided beside each item. The thumbnail images 532, 534, and 536 may serve to recognize a user's current state as well as intuitive recognition and identification of the item itself. Meanwhile, the thumbnail images 532, 534, and 536 may be provided in various forms such as audio, video, and text as well as an image form, and may be selected by the user, and a pop-up window is provided upon selection. It may be provided as an enlarged screen or may be converted to a full screen and provided. On the other hand, such thumbnail images (532, 534, 536), for example, in conjunction with the CCTV, various sensors, etc. provided in the home network system can immediately identify and control the current status in real time. The sensor may include a sensing sensor, a temperature sensor, a distance sensor, a camera sensor, and the like, but the camera sensor may use not only a camera sensor of a corresponding digital device but also other camera sensors around the digital device.

When the user selects either the lighting item 531 or the representative thumbnail image 532 in FIG. 5C, the mobile device provides the UI illustrated in FIG. 5D. The UI of the mobile device of FIG. 5 (d) includes a fluorescent lamp 541 and a dining table lamp 543. In addition, representative UI images 542 and 544 may be provided together.

As illustrated in FIGS. 5A through 5D, the user may provide a UI according to the execution of the remote control application in the mobile device to enable intuitive recognition and control of the home network system (s) that is the user's remote control target.

When any one item is selected in FIGS. 5 (a) to 5 (d), the mobile device provides a controllable sub-item at a corresponding item level through a pop-up window as shown in FIG. Here, when the user selects a predetermined sub item, the mobile device identifies the selected item, generates a control signal corresponding to the identified item, and transmits the control signal to the corresponding home network system.

In FIG. 6, the pop-up window 610 including sub-items for a specific item (table, etc.) is illustrated using FIG. 5 (d) as an example. Sub-items included in the pop-up window 610 of FIG. 6 may have different contents provided by the type, state, etc. of a corresponding upper item. For example, the table lamp 543 illustrated in FIG. 8 is currently off. Accordingly, the sub-item of FIG. 6 may be provided with a power-on, a level control item, and the like. Here, the power-on may be regarded as the default value when the level is 100%. On the other hand, the level control item is to turn on the power of the table lamp, but to control to a level other than the 100%. On the other hand, Figure 6 is defined as an embodiment of providing a level in the form of a number or a mode other than the number and the selection control is also possible. The mode type may be defined in various modes such as an outing mode and a meal mode.

On the other hand, as shown in Figure 5 (b) to (d) sequentially according to the level, the screen itself may be switched, it may be provided to be selected in a pop-up form within the screen without switching the screen. .

Although each item is provided in the form of an icon in FIGS. 5 (a) to 5 (d), this is only one embodiment for the user's identification and / or selection convenience, and may be implemented in other forms.

In addition, as illustrated in FIGS. 5A to 5D, the mobile device may provide a user to intuitively identify whether each item can be connected or controlled. In this case, unlike in FIG. 5, the mobile device may provide only items that can be connected or controlled in initially providing each item. The mobile device may change texts, emoticons, color changes, density / saturation changes, etc. to connect / disconnect each item to help the user intuitively identify. For example, in FIGS. 5 (a) to 5 (d), the items that can be connected / controlled in one embodiment help the user to intuitively recognize using solid lines and dark colors, while the items that cannot be connected / controlled have dotted lines and light colors. It is shown to be differentiated from the connection / controllable items. Referring to FIG. 5A, the user may intuitively know that the current friend home item 513 is not connected / controlled among the my home 511, the parent home 512, and the friend home 513.

Meanwhile, in FIGS. 5A to 5D, the control target or the control level of the home network system provided on the main screen may be displayed at the top of the screen so as to be easily recognized. Referring to FIG. 5 (d), the items “542-543 and 544-545” currently displayed on the main screen are displayed as “Remote Control App (My Home / Kitchen / Lighting)” at the top of the screen. It is easy to recognize that the lighting items of the kitchen items are provided among the four items of.

7 is similar to the above-described FIGS. 5 to 6. Therefore, hereinafter, description of the contents overlapping with FIGS. 5 to 6 will use the above-described contents, will be omitted here, and the description will be mainly focused on different points.

In FIG. 5 to FIG. 6, the mobile device provides a user interface according to a location such as a room, a kitchen, a bathroom, etc. when My Home is selected, whereas when My Home is selected in FIG. A user interface as shown in FIG. 7B is provided based on the type of the digital device.

Referring to FIG. 7 (b), the next higher item is provided for lighting, a refrigerator, a TV, a washing machine, a robot cleaner, and the like based on the type of the digital device. In this case, a representative thumbnail image may be provided together with the next higher item.

The mobile device provides the next higher order items of FIG. 7C when the user selects an illumination item among the next higher items of FIG. 7B.

Here, FIG. 7 (c) is provided in FIG. 7 (c) as long as it belongs to the home network system as described above, which is based only on the type of digital device, and is output regardless of the position of the lighting. . However, each light should be provided in an identifiable form. Thus, as shown in Fig. 7 (c), the mobile device may display the eight respective lights belonging to the lighting item selected in Fig. 7 (b), “room 1 / fluorescent lamp (on)”, “room 2 / fluorescent lamp (on)”. , “Living Room / Fluorescent (on)”, “Bathroom / Fluorescent (off),“ Kitchen / Dinner (off) ”etc. Therefore, the user can intuitively identify and control which lighting is at which position in the lighting only by using the UI of FIG. 7C.

8 illustrates another embodiment of a UI / UX of a remote control application provided by a mobile device.

In FIGS. 5 to 7, if the UI is provided so that the remote control items can be selected from the upper level to the lower level step by step, FIG. 8 illustrates another embodiment provided by the mobile device according to a user's request to execute a remote control application. Alternatively, FIG. 8 may be a UI provided when a user wants to simultaneously control or simultaneously monitor a plurality of home network systems.

Hereinafter, the basic concept will be omitted from FIG. 5 to FIG. 7 and the detailed description of the overlapping content will be omitted, and the description of FIG. 5 to 7 will be used, and different parts will be mainly described.

When the mobile device receives a request for executing a remote control application, the mobile device configures and provides information about the connected home network system as shown in FIG. 8. 8 illustrates a case where two connected home network systems are my home and a parent home. However, the connected home network system may be configured in one or three or more similar home systems. On the other hand, the mobile device has a large number of control targets for one home network system and cannot be output on the screen all at the same time or on different pages for control targets for each home network system. It can be provided, or it can be provided as a vertical scroll function within a single page.

Hereinafter, the UI / UX of the remote control application provided by the mobile device of FIG. 8 will be described in detail. As described above, the UI / UX of FIG. 8 is divided into UI / UX (A) 810 for my home and UI / UX (B) 850 for the parent home.

The UI / UX for the My Home output to the A area 810 will be described as an example.

In the A region 810, three items 820, 830, and 840 are output for each control target.

The first item 820 identifies whether the control target is on or off. The identification may be made by implementing a form of color, size, 2D / 3D implementation, blinking, or the like for the corresponding item, or may be made through text or by writing the text together.

The second item 830 identifies the control object itself. Identification of the control object itself may be made through text data. Meanwhile, the second item 830 further includes count data of the identified control object. In this case, the count data may be provided in an on count / total count form. For example, if the total number of lights in my home is eight and five of them are turned on, it is provided as 5/8. The count data may be linked with the first item 810. For example, if more than half of the total number of lights is on, the first item 820 is determined to be on. In the case of the region 850 of FIG. 10, when the illumination is 3/8, the first item may be displayed as off, unlike the region A 810.

The second item 830 may additionally provide detailed data about the item. For example, when the control object is a light, a total of eight lights are provided separately. Referring to FIG. 10, for each lighting, such as living room (2/3), room 1 (2/3), and room 2 (1/2), the number of turned on / total number of each light can be easily Can be identified. In addition, even if each of the divided lights does not exceed half of the total number of the sub-items 832 for the corresponding lights may be provided to be distinguished from other sub-items. For example, referring to TV as another control object in FIG. 8, it is possible to indicate that one of two TVs is currently turned on, and that any TV program can be identified that is watching any program on a certain channel. (“18:30, watching MBC Infinite Challenge”). For example, the refrigerator may suggest a recommended dish of the day based on the contents contained in the refrigerator, and may provide the user with data on contents of the contents that have passed an expiration date or have a shelf life. You can also In addition, the washing machine is currently in a state of being rinsed, and may display that 15 minutes remain until the washing is completed. The robot cleaner may indicate that there is a reservation information to start cleaning the living room at 20 o'clock or off. As such, the second item 830 may be provided so that the user can identify not only the information on the control target but also detailed information without a separate screen or page switching on the screen itself. On the other hand, even if the detailed additional information is not included in the second item 830 can be provided in a pop-up form according to the selection of the item.

The third item 840 is provided with a representative thumbnail image / video of the control target of the second item 830. In this case, the image / video provided to the third item 840 may be received from the corresponding control target. If the control target is a TV, a back channel is automatically captured by automatically capturing the screen currently being watched on the TV. It can be displayed by transmitting to the mobile device through, and in the case of the control target other than the TV, it is possible to provide an image / video captured by a camera sensor of another digital device or the surrounding CCTV provided in the home network system. Can be. When the corresponding item is selected, the data of the third item 840 requests the mobile device to update the image / video to the corresponding control object, and updates the latest image / video data to the user by responding to the control object. can do. In addition, such a thumbnail image / video may be deactivated when photographing or the like is impossible as described above, and a user may download and display a related substitute image from a server such as the Internet for convenience of item identification.

On the other hand, when there are a plurality of thumbnail images / videos of the third item 840, the mobile device displays the number in the corresponding item to provide the user with identification of the plurality of data. The mobile device may simultaneously provide each data in the form of a pop-up or the like when the user accesses the third item 1040. In this case, each data provided at the same time may be mixed with the image data and the video data, and in the case of the video data may be reproduced and provided at the same time even if there is no separate selection.

Meanwhile, the item data and the detailed additional data of FIG. 8 may be updated in real time by continuously requesting status information to the home network system or each digital device of the home network system in a polling manner in the mobile device. Even if there is no polling request from the mobile device, if there is a change in the current state in each digital device, the user may be provided by providing information on the state change and requesting or suggesting further control.

It will be apparent that each item of the my home of FIG. 8 may be implemented in the same form in FIGS. 5 to 7.

9 to 14 are diagrams for explaining a remote control scenario in the above-described mobile device. Here, FIGS. 9 to 14 are exemplary embodiments for interactively performing remote control interactively with a specific device or a home network system manager such as a conversation with another person through a messenger or the like in a conventional mobile device.

9 illustrates an embodiment in which a user executes a remote control application on a mobile device to control a cleaner of my home.

Referring to FIG. 9A, when the cleaner of my home is selected as described above, the mobile device outputs a text of "hello" to the chat window 910 informing the start of control of the selected cleaner (S910). In this case, the mobile device may provide a text to the conversation window 910 and transmit a control signal including a message requesting to join the cleaner of the home and / or to participate in the remote control process requested by the user. Here, the step S910 may be automatically provided by default in the mobile device even if the user does not directly input.

The cleaner of my home sends a response signal according to the control signal reception, and the mobile device responds to the response signal. Should I start cleaning? ” The phrase is provided to the conversation window 910 (S920).

Here, "Would you like to start cleaning?" In the phrase "convenience to perform the most basic function or the frequency of use by the user in consideration of the function, property, type, etc. of the corresponding digital device or based on the user's previous remote control history. Is chosen to provide. Thus, it may be replaced with a phrase for other functions depending on the digital device in question.

Thereafter, when the user inputs the phrase “yes” to agree to start cleaning through the input window 930 and presses the transmit button 940, the mobile device outputs the phrase to the conversation window 910 (S930). . At this time, the mobile device transmits a control signal including a control message according to the phrase, that is, a message meaning to start cleaning, through the cleaner or the management unit at the same time as the output.

Thereafter, when the cleaning completion request is received from the polling or the cleaner whether the cleaning is completed according to the cleaning start request at a periodic / non-period, the mobile device outputs it to the chat window 910 to inform the user that the requested event is completed (S940).

When the user selects the input window 930, the control process of step S930 may be performed through a text input by outputting a virtual keyboard. Alternatively, when the user presses the emoticon button 950, an emoticon related to the corresponding digital device is provided in a pop-up form as shown in “Yes”, and when the user presses a predetermined emoticon and presses the send button 940, the emitter In addition to being output to the dialog window 910, a control signal including a message requesting to perform a function mapped with a corresponding corresponding emoji may be transmitted to the cleaner through the cleaner or the manager to perform the same function as described above. .

In the above, when the user inputs text through the input window 930, the mobile device compares the function recognition terms of the table stored in advance from the input text with the control function through the corresponding text data to meet the user's intention. Do it. Meanwhile, the mobile device extracts a keyword from the full text sentence for the comparison and compares the keyword with the text data in the table to identify a user's intention or desired control function. The identification process is also applied to the text input step. For example, when a user inputs text through a virtual keyboard or the like, based on the consonant and vowel data of the text input in units of words, the relevant text word is provided in the table in a predetermined area of the input window 930. Can help with input.

Meanwhile, FIG. 9 (b) may be for the interactive control embodiment through voice while FIG. 9 (a) described above is for text data input.

Referring to FIG. 9 (b), when the user selects the voice icon 960 at the bottom, an item 970 for voice recognition is popped up on the chat window 910. When a user outputs a voice through the voice recognition item 970, the user may recognize the voice and output the text data based on the recognized voice data on the lower input window 930, and may be provided to the chat window when the user presses the transmit button. Alternatively, when the voice recognition is performed, the text data converted from the voice may be output in the voice recognition item and may be directly output on the chat window 910 instead of the input window 930 according to the user's selection. Meanwhile, the voice data may be provided to be mapped to the corresponding emoticon of FIG. 10 to be described later rather than text data.

Meanwhile, referring to FIG. 9A, a user may add or change another device through the device add button 960 in an interactive remote control process. In this case, in the case of adding another device, the dialog window keeps the conversation contents being output as is and prepares to output the interactive control contents for the added devices. Meanwhile, in the case of a change to another device, the chat window may maintain or delete the existing chat contents and configure and provide a new chat window. In this case, when the mobile device maintains the existing conversation contents, when the user reselects the output control-related text data, the mobile device may provide the interactive control function for the existing device to the current device as it is. That is, when a user requests the same control function newly based on a previously performed function after adding or changing a device, control input is easily performed based on history data even if voice, text, etc. are not input again for convenience of input. It can also be implemented.

FIG. 10 illustrates an embodiment of an emoticon mapped to a message for interactive remote control in a mobile device.

Here, FIG. 10 (a) illustrates an embodiment of mapping emoticons previously stored for a general messenger on a mobile device to a message, and FIG. 10 (b) illustrates an embodiment of a newly defined emoticon for interactive remote control in a mobile device. to be.

First, referring to FIG. 10 (a), a mobile device has a previously stored emoticon while executing a messenger application such as a text message or KakaoTalk. In this case, when the user launches a remote control application through a mobile device and initiates interactive remote control with a specific digital device, input takes time and text to directly power on / off and control the function. You may wonder whether the content will perform the function you want. Accordingly, the mobile device identifies the selected digital device, pre-maps the previously stored emoticons with the control functions of the identified digital devices, and then provides the emoticon button when providing the interactive window.

In the above, the mobile device may adopt and provide only related emoticons so as to identify functions mapped only with emoticons for the convenience of the user when mapping the previously stored emoticons and the control functions of the identified digital devices.

Meanwhile, when the emoticon selected through FIG. 10 (a) is provided to the chat window 910 of FIG. 9 (a), text data may be written together to identify a corresponding control function together with the selected emoticon. For example, it can be output in a format such as “Emoticon (Power On)”.

Next, referring to FIG. 10 (b), unlike FIG. 10 (a), the mobile device generates a new emoticon for the convenience of the interactive remote control, and can easily identify a control function of the generated emoticon. You can write text data below for.

Meanwhile, referring to FIG. 10 (b), an emoticon is provided at the bottom to enable a phone call to a parent, an A / S article, or a 119, which is provided for immediate response in case of a malfunction or emergency of the corresponding digital device.

Meanwhile, even if the emoticon is not newly generated in FIG. 10 (b), the emoticon stored in the mobile device may be used as it is, and a control function may be placed under the emoticon to assist identification convenience.

Referring to FIG. 11, a description will be given of how the user processes an event in a control process for a digital device of My Home.

Referring to FIG. 11A, a digital device of a refrigerator, a TV, and an iron is output to a control target in My Home.

The mobile device not only monitors the control object but also notifies the user when a monitoring result or a predetermined event occurs. As described above, this may be done based on polling of the mobile device or signal transmission of the corresponding digital device.

Referring to FIG. 11 (b), the user may transmit a monitoring signal to notify the user when an iron is used for a predetermined time or more among control targets of my home through a chat window of the mobile device. In this case, as shown in FIG. 11 (b), when the mobile device outputs a message “Please inform when used over 8 minutes” and receives a response signal in response to the control signal including the monitoring request message transmitted to the iron at the same time as the output, Print a response message (“Yes, sir.”).

The message allows the iron to monitor its status on a predetermined basis. In this case, when an event does not correspond to the control content of the user as a result of the monitoring of the iron that is the control target of FIG. 11 (a), the message “it has been turned on for too long” may be notified to the user.

On the other hand, when a predetermined time elapses for the remote control target such as an iron, a notification message as shown in FIG. 11 (a) is provided, and when the user checks the message of FIG. 11 (a), the user receives a message as shown in FIG. It can also be set and used for monitoring.

Referring to FIG. 12 (a), there are two homes and a parent home in a home network system to be controlled remotely. If FIG. 11 is for My Home, FIG. 12 is a control procedure for the digital device in the parent home.

As illustrated in FIG. 12, the control of the digital device in the parent home other than the my home may require an operation such as an authority setting, an authority request, and an authentication process prior to proceeding with the control process. This must be preceded in the process of controlling the parent home or the friend home, not the my home, because of unintended control due to overlapping control commands with the main control mobile device or malfunctions caused by arbitrary control of others, This is to prevent errors.

Basically, when the mobile device connects to the home network system, the manager determines whether the mobile device has access / control authority to the home network system. This can be done through various methods, such as whether the user is a mobile device registered in the system, a password, a fingerprint scan, or a voice recognition. If there is access / control right through the above process, the mobile device outputs information on whether to access the home network system. Meanwhile, the access right and the control right may be separately determined and processed.

On the other hand, if the management unit determines that the mobile device does not have access / control authority, as determined by the access / control access of the mobile device, the administrator indicates that there is an unauthorized person's access to the mobile device of a user who has pre-registered main access / control authority. Alarm messages can be output.

In the above process, if the determination results that the mobile device has the right to access / control the parent home, the mobile device may control one or more control targets for the parent home network system, control contents for the control targets, and control targets. Outputs information such as the current state.

Here, as shown in FIG. 12A, when the user selects two washing machines and two cleaners at the same time, a dialogue window as shown in FIG. 12B is provided. In addition, as described above, in the washing machine, which is a digital device of the parent home, the cleaner may control the start of cleaning by the user through text, voice, and emoticon.

13 is a diagram of controlling a home network system through an interactive remote control.

While the above description mainly describes the contents of selecting a specific digital device through the mobile device and controlling the selected device, FIG. 13 shows a message outputted through a chat window after executing a remote control application of the mobile device. Accordingly, the control of one or more related digital devices is performed. In this case, the user may select a specific digital device.

Even if the user does not select a specific device and / or a specific home network system in FIG. 13 (a), if the message A of “Arrive at home at 7:00” is input through the dialog of FIG. 13 (b), The mobile device may automatically select the corresponding home network system and the corresponding digital device through the phrases “7:00”, “home” and “arrival” from the message, and transmit the related control contents.

Referring to FIG. 13B, when a control signal according to the user's A message input is transmitted to the corresponding home network system, the management unit transmits the control signal to the entire digital device or the related digital device. In this case, the control signal may be transmitted as it is, or the control content of the mobile device is converted into control content suitable for each digital device. On the other hand, this may be controlled by transmitting a control message converted in advance for the corresponding digital device from a mobile device other than the management unit.

Referring to FIG. 13 (b), the mobile device notifies the reception of the message and the acceptance of the control according to the message A (“Yes, I see.”), And the control contents of the first device (air conditioner) and the second device (vacuum cleaner). (“I will turn on and clean my air conditioner on time.”).

On the other hand, the home network system may propose a message according to the current status, status, etc. of each digital device through the mobile device. Referring to Figure 13 (b), if the milk has dropped into the refrigerator, since the milk is dropped, it is suggested to purchase milk. In addition, in FIG. 13 (b), information on a mart available to a user may be provided by searching for surrounding information by accessing a network. In FIG. 13 (b), for example, “xx mart is closed, so use yy mart.”

Referring to FIG. 14A, when a user selects a parent home and has access / control authority, the user may control or request to perform a reservation function for a predetermined digital device in the parent home network system.

As shown in FIG. 14 (b), when the user selects the parent home, the parent home management unit transmits a message about a parent's recent situation, and the mobile device transmits a message after the user receives the message. Show it to me. ”

The mobile device or the management unit generates a control signal from the user transmission message so that the concert program can be reserved and watched / reserved from the TV from "Today", "6", "Nahuna concert", and "Show me". Can be controlled.

In the meantime, the above description has mainly described the process after the remote control application of the mobile device is executed, but even when the remote control application of the mobile device is not running, the mission previously controlled by the mobile device is completed or predetermined. When an event occurs, you can report it. For example, if an electric iron overheats or poses a fire hazard, you can report it to the user, report it via CCTV or camera sensors, or send a message to a sprinkler or 119 to make a call.

In addition, although not shown, the UI / UX of FIGS. 5 to 14 may not be provided in a text form, but may be provided in a map form to allow a user to more intuitively control the functions of each digital device in the map. have. In addition, when a plurality of devices are located in a predetermined area, and thus selection or identification is not easy, the user may enlarge / reduce the mobile device through pinch-in / pinch-out. You can also implement UI / UX for easy control.

In addition, when control commands through a plurality of mobile devices are received by the same device or the home network system at the same time, the home network system considers the reliability, priority (whether or not the main device) of each mobile device, and the like. The control command may be ignored. Alternatively, in the above case, it is possible to ignore all of the plurality of control commands and to notify that each mobile device controls the same device or the home network system. Meanwhile, when the same or the opposite control command is received from another mobile device within a predetermined time range, the mobile device or the manager may ignore the control command and inform the mobile device of the reason why the current function is being performed or ignored.

Meanwhile, the above-described embodiments are mainly focused on direct input of a user, but the present invention is not limited thereto, and remote control may be performed based on a specific gesture of the user together with or separately from the user's touch input. .

15 to 18 are flowcharts illustrating a remote control method of a digital device in a home network system through a mobile device.

Referring to FIG. 15, a method of remotely controlling a digital device in a home network system through a mobile device is as follows.

The mobile device executes a remote control application to remotely control a predetermined digital device of a home network system in the remote control system according to a user's request (S1510) and connects to the home network system (S1520). In this case, when the remote control application is not provided, the mobile device may be downloaded from a web server and executed after installation.

The mobile device receives and outputs state information on the digital device in the connected home network system (S1530).

Thereafter, the mobile device outputs an interactive user interface for digital device control (S1540), and receives and outputs a first user input for controlling the digital device from the outputted interactive user interface (S1550).

The mobile device controls the digital device by generating a first control signal according to the first user input and transmitting the generated first control signal to the digital device (S1560).

Referring to FIG. 16, in operation S1560, the first control signal may be generated and transmitted to the digital device according to the following method.

The mobile device detects a keyword from the first user input output on the interactive user interface (S1610), and compares the detected keyword with pre-stored function data for controlling the digital device (S1620).

The mobile device determines function data corresponding to the detected keyword according to the comparison result (S1630), and generates the first control signal based on the determined function data (S1640).

FIG. 17 may be illustrated to describe a scenario after transmitting an initial control signal from a mobile device to a digital device as shown in FIG. 15 or 16.

After the control signal is transmitted from the digital device, the mobile device receives a signal including an alarm message according to an event occurrence (S1710), and parses an alarm message in the received signal (S1720).

The mobile device outputs the parsed alarm message on the interactive user interface (S1730), receives a second user input for controlling the digital device according to the occurrence of the event, and outputs the second user input on the interactive user interface (S1740). .

The mobile device generates and transmits a second control signal according to the second user input to the corresponding digital device after step S1740 or at the same time (S1750).

Referring to FIG. 18, the mobile device receives a third user input for selecting an image or video data display item (S1810), and includes at least one of a CCTV server in the home network system and a sensor of a digital device other than the digital device. In operation S1820, an image acquired from one may be received and the received image may be output on the interactive user interface in a pop-up form.

In the above, the first user input is made through at least one of a text input, a voice recognition, an emoticon, and a gesture, wherein the emoticon is an attribute of the digital device based on emoticon data for a messenger application previously stored in the mobile device. It may be provided for the digital device control depending on the state and.

Meanwhile, in the above description, the state information of the digital device may include at least one item indicating whether the digital device is currently on / off, a detailed information display item according to the property or location of the digital device, and an image or video data display item. It may include.

The at least one of the first control signal and the second control signal may include at least one of power on / off of the digital device, a function control currently performed by the digital device, and a function control reservation message.

In addition, the above-described mobile device may be any one of a mobile terminal, a tablet PC, and a wearable device.

According to the present invention described above, a remote control system for remotely controlling the digital device (s) belonging to one or more home network systems through the mobile terminal, the application of the mobile terminal for controlling the digital device (s) and the UI / UX for controlling digital devices used in applications, remote control systems that make full use of existing systems to achieve expandability, backward compatibility at low cost, mobile terminal status and home network status, or It is possible to provide a convenient and accurate remote control system in consideration of the status of digital devices in the home network. In addition, according to the present invention, by presenting a new standard model of the remote control system, the mobile terminal, it is possible to improve the product satisfaction and increase the purchase desire by maximizing the user's ease of use.

According to one embodiment of the present invention, the above-described method (operation flowchart) can be embodied as processor readable code on a medium in which a program is recorded. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may be implemented in the form of a carrier wave (for example, transmission over the Internet). Include.

The mobile terminal 200 described above is not limited to the configuration and method of the above-described embodiments, the embodiments may be a combination of all or part of each embodiment selectively so that various modifications can be made It may be configured.

On the other hand, it has been described with respect to the preferred embodiments of the present invention through the specification and drawings, although specific terms are used, it is only used in a general sense to easily explain the technical details of the present invention and to help the understanding of the invention. It is not intended to limit the scope of the present invention. It will be apparent to those of ordinary skill in the art that other modifications based on the technical spirit of the present invention may be implemented even in the embodiments disclosed herein.

100: remote control system 110: network
120: home network system 132: cloud processor
134: wearable device 136: mobile terminal

Claims (18)

delete delete delete delete delete delete delete delete delete A mobile device for remotely controlling a digital device in a home network system, the mobile device comprising:
A communication module supporting connection and data communication with the home network system;
A receiver configured to receive status information on a digital device in the connected home network system through the communication module;
A display for displaying the received digital device status information, an interactive user interface for remote control of the digital device, and a first user input for remote control of the digital device via the interactive user interface; And
And a controller configured to execute an application for remote control of the digital device, generate a first control signal according to the display control, and the first user input, and transmit the generated control signal to the digital device.
The control unit,
Displaying on the interactive user interface information suggesting performance of a function most frequently used by the first user among functions available in the digital device, based on a previous remote control history of the first user for the digital device; and,
And generating and transmitting the first control signal according to the first user input to command the performance of the function to the digital device. The method of claim 10,
The control unit,
Detecting a keyword from the displayed first user input, comparing the detected keyword with pre-stored function data for controlling the digital device, and determining function data corresponding to the detected keyword according to the comparison result, And generate the first control signal based on the determined function data. The method of claim 11,
The control unit,
And receive the first user input through at least one of text input, speech recognition, emoticon, and gesture. The method of claim 12,
The control unit,
And providing the emoticon for controlling the digital device according to the attribute and state of the digital device based on emoticon data for a messenger application previously stored in the mobile device. The method of claim 10,
The control unit,
Configuring the state information of the digital device, including at least one of an item indicating whether the digital device is currently on / off, a detailed information display item according to a property or a location of the digital device, and an image or video data display item. Mobile device. The method of claim 10,
The control unit,
Receiving a signal including an alarm message according to an event occurrence from the digital device, parsing an alarm message in the received signal, controlling to output the parsed alarm message on the interactive user interface, and And generate and transmit a second control signal to the digital device according to a second user input for controlling the digital device. The method of claim 14,
The control unit,
Upon receiving a third user input for selecting the image or video data display item, requesting an image obtained from at least one of a CCTV server in the home network system and a sensor of a digital device other than the digital device, and requesting the request. And output the received image on the interactive user interface in a pop-up form. The method of claim 15,
The control unit,
And at least one of the first control signal and the second control signal, including at least one of the digital device power on / off, the currently performing function control of the digital device, and a function control reservation message. . The method of claim 10,
The mobile device,
A mobile device, characterized in that any one of a mobile terminal, a tablet PC and a wearable device.

Images