KR20050119368A - Method and system for controlling computer using mobile communication terminal - Google Patents

Abstract

The present invention relates to a computer control system and method using a mobile communication terminal, the system according to the present invention is connected to the computer to be controlled via a network, IP database server for managing IP address information of the connected computer; A mobile communication terminal that aggregates control command information of a user on a computer to be controlled and transmits the information to a computer, and reconfigures information on screen contents transmitted from the computer to be controlled according to a display environment; Replaces the screen driver of the computer to be controlled with itself to intercept information on the screen change, and replaces the event monitor of the computer to be controlled with itself to intercept the user interface information of the application and the state change information of the user interface; And a computer application program configured to reconfigure the virtual screen using the chan information and to sequentially execute a task instructed by a user in response to information about a task execution provided from the terminal.

Description

Method and System for Controlling Computer Using Mobile Communication Terminal

The present invention relates to a method and system for controlling a computer using a mobile communication terminal, and more particularly, to a method and system for performing an operation required by controlling an application program installed in a computer at a remote location using a mobile communication terminal. .

Due to the rapid spread of computers, the use of computers has become commonplace in daily life and work, and this has led to the use of many computers such as companies and homes. As a result, the documents or data required by the users are often distributed and stored on various computers, and the necessity of viewing or storing necessary information stored in the computers at remote locations is raised.

In order to overcome this problem, an idea of accessing a computer remotely through a wired telephone network, a wireless telephone network or an internet network and transmitting necessary control information has been raised through a patent document or a magazine. However, there are many problems for this idea to be realized in reality, so that no actual computer remote control service is provided.

Recently, since the computer operating system is not a text mode but a GUI mode, it is difficult to realistically reproduce the computer screen of the GUI environment in a terminal that does not use an operating system such as a computer.

Although a method of imaging a screen of a computer and transmitting it to a terminal has been proposed, such a method is difficult to apply to a current terminal due to limitations in data processing speed and transmission speed, and due to a small display screen of the terminal. It is difficult to identify the screen of the computer through the image in the terminal.

In particular, when importing an image of a computer as it is, it is necessary to control the computer through a mouse, but there is a problem that a mouse function is not supported in the current terminal.

In the present invention, in order to solve the problems of the prior art as described above, by transmitting a computer output screen meaningful to the user to the terminal and reconfiguring the user interface in the terminal, the computer can effectively control the computer even in a terminal with a low processing speed and a small screen. We propose a remote control system.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of a computer control method and system using a mobile communication terminal according to the present invention.

1 is a diagram showing the configuration of a computer control system using a mobile communication terminal according to an embodiment of the present invention.

As shown in FIG. 1, a computer control system using a mobile communication terminal according to an embodiment of the present invention may include a mobile communication terminal 100, an IP database server 102, and a computer 104.

The mobile communication terminal 100 may include a device having a communication function as a portable device such as a mobile phone or a PDA.

The mobile terminal 100 communicates with the IP database server 102 via a wireless network to receive information on a computer to be controlled from the server, and provides control information input by the user to the computer 104. Do it. It will be apparent to those skilled in the art that the mobile communication terminal 100 may communicate with the server 102 or the computer 104 using a wireless internet network, and may use a wireless network other than the wireless internet network.

The mobile communication terminal 100 is provided with firmware for receiving information about a computer from a server and providing computer control information to the computer. Detailed module configuration of the firmware installed in the mobile communication terminal will be described later through separate drawings.

The server 102 is connected to a computer to be controlled through a network, and when the terminal requests information about a computer to be controlled, the server 102 provides the mobile communication terminal 100 with IP address information of the corresponding computer. The mobile communication terminal can access the computer to be controlled by using the address information received from the server 102.

The computer 104 is provided with a program for receiving control information from the terminal and providing the current status information to the terminal. The installed program determines the current screen display information of the computer through communication with the computer 104 operating system, stores the screen display information in a queue, and transmits the screen display information to the terminal upon request. The method of determining the screen display information of the connected computer from the server will be described in detail later. In recent years, since the operating system of the computer is not a text environment but a GUI (Graphic User Interface) environment, it is not easy to display the screen display information of the computer on the terminal. Therefore, according to a preferred embodiment of the present invention, the firmware installed in the terminal includes a module for reconfiguring the screen display information stored in the computer in the terminal.

With this system, you can use your mobile phone to email or directly send files stored on your computer to others at a remote location, and you can also transfer files from your computer to your mobile phone or transfer files stored on your mobile phone to your computer. It is possible to send to. In addition, it is possible to check and reply to e-mail or edit documents remotely.

2 is a diagram showing a module configuration of a program installed in a computer according to an embodiment of the present invention.

As shown in FIG. 2, the control program according to an embodiment of the present invention may include a screen change tracking unit 200, a user interface event tracking unit 202, a virtual screen storage unit 204, and a network communication unit 206. , A task executor 208, a system input queue 210, and an application program interface 212.

In FIG. 2, the screen change tracking unit 200 tracks the content displayed on the screen of the computer and information related to the content, and provides the tracked information to the virtual screen storage unit 204. Here, the displayed content may include information about the position and size of the output screen. The screen change tracking unit 200 registers itself as a display driver of a computer operating system in order to track the contents displayed on the computer screen and the change of the contents. When the program is installed, the screen change tracking unit is registered as a screen driver in the registry instead, thereby replacing the existing screen driver with the screen change tracking unit. At this time, the existing screen driver is stored in the form of a separate file.

When the screen change tracking unit 200 is registered as a screen driver of a computer, when a screen output event occurs, the screen change tracking unit 200 is called instead of the original screen driver and the changed screen output information is provided to the screen change tracking unit. Change screen information can be tracked.

The user interface event tracking unit 202 functions to track the state information and the change of state of the user interface element displayed on the computer screen, and provide the tracked information to the virtual screen storage unit 204. For example, when Outlook, which is an email application, is executed on a user's computer, the user interface event tracker 202 extracts information about various menus and folders of Outlook and provides them to the virtual screen storage. will be. In addition, when the user inputs a control command regarding the user interface through the terminal, the user interface tracking unit 202 receives it from the job execution unit 208 to generate an event input by the user in the corresponding user interface element. do.

In order to track state information and state changes of the user interface elements, the user interface event tracker 202 registers itself as an event monitor of the operating system. As in the case of the screen change tracking unit, when the program is installed, the user interface event tracking unit is registered as an event watcher in the registry instead, so that the existing event monitor can be replaced with the user interface event tracking unit. At this time, the existing screen driver is stored in the form of a separate file. When the event monitor is registered with the user interface event tracker 202, the user interface event tracker 202 may track information that a user interface related event has occurred.

The virtual screen storage unit 204 extracts meaningful contents from contents currently displayed on the computer by using screen output information and user interface status information provided from the screen change tracking unit 200 and the user interface event tracker 202. It functions to reconstruct the virtual screen, and provides the reconstructed virtual screen to the network communication unit 206 when the terminal requests a screen view. The terminal changes and displays the screen display information provided from the virtual screen storage unit 204 in a format suitable for the terminal.

When the computer control information input by the user through the terminal is provided through the network communication unit, the task execution unit 208 functions to sequentially control the application program of the computer according to the received control information. For example, when a user invokes an email creation application to create an email, the task executor 208 receives the information and provides information to the computer's operating system to execute the email creation application.

The network communication unit 206 is responsible for communication between the terminal and the computer. In detail, the network communication unit 206 receives an authentication processing request from the terminal, control information input by the user through the terminal, a screen viewing request, and transmits the same to the corresponding module, and stores the virtual screen storage unit 204 or the job of the server. It delivers the information output from the execution unit 208 to the terminal. As described above, the network communication unit may perform communication with the terminal using a general wireless internet protocol.

3 is a block diagram illustrating an operation structure of a screen change tracker according to an exemplary embodiment of the present invention, and FIG. 4 is a flowchart illustrating an operation of the screen change tracker according to an exemplary embodiment of the present invention.

The detailed operation of the screen change tracking unit will be described with reference to FIGS. 3 and 4 as follows.

First, the screen change tracking unit accesses an operating system of a computer to be controlled to store existing screen driver information and separately store the information (S400).

After storing the information of the existing screen driver, the screen change tracking unit registers itself as a screen driver (S402). As described above, the screen change tracking unit registers itself as the screen driver of the operating system in order to intercept the screen content output by the application program of the computer. When the screen change tracker is registered as the screen driver, the operating system of the computer to be controlled recognizes the screen change tracker as the screen driver and calls the screen change tracker when an event related to the screen output occurs.

When the screen output is made to the application (S404), the operating system detects the occurrence of the screen output event (S406), calls the screen change tracker registered as the screen driver, and the screen change tracker intercepts information about the screen output. (S408). The screen output information intercepted by the screen change tracking unit is information about whether it is graphic or text, information about the screen position where the output is displayed and the width and height occupied by the screen, and what is displayed on the screen is covered by other content. It includes information about the lost area, information about the area where the screen is erased, and information about style, style, and font size when the output content is text.

The screen change tracking unit intercepting the information on the screen output determines whether the information on the screen output is meaningful to the user (S410).

If the information transmitted to the screen change output unit is meaningful to the user, the screen change output unit stores the transferred information in a queue (S412). In addition, the screen change output unit transmits information that a screen output event has occurred to the virtual screen storage unit (S414).

After the screen output information is not meaningful to the user or the screen output information is transferred to the virtual screen storage, the screen change tracking unit calls the existing screen driver stored separately (S416).

The existing screen driver executes the screen output so that the actual screen is displayed on the monitor (S418).

5 is a diagram illustrating an operation structure of a user interface tracking unit of the present invention.

Referring to FIG. 5, the user interface tracking unit registers itself as the user interface element event monitor 502 in the event monitoring unit 500 of the operating system. By registering itself with the operating system, the operating system forwards various events occurring in the operating system to a user interface tracker, which is a registered event monitor. Thus, the UI tracer can detect a change in state of the UI element. Here, the user interface elements may include buttons, check boxes, edit boxes, various menus of application programs, and the like.

6 is a flowchart illustrating an operation flow of a user interface tracker when there is a state change of a user interface element in a computer.

As shown in Figure 6, the user interface tracking unit first registers itself as an event monitor of the monitoring unit of the operating system (S600).

The user interface tracking unit registered as an event monitor detects whether a user generates an event for an application program (S602).

When an event occurs in the application program, the user interface tracking unit registered as an event monitor of the operating system receives the generated event information (S604).

The user interface tracking unit determines whether the event information is an event related to a user interface element (S606).

If the event is related to the user interface element, the user interface tracking unit transmits the generated event information to the virtual screen storage unit (S608).

The user interface element-related event occurrence information is transmitted to the virtual screen storage, or if the generated event is not a user interface-related event, the user interface tracking unit transmits it to the operating system monitoring unit (S610).

7 is a flowchart illustrating an operation of a user interface tracking unit when a user who remotely controls a computer executes a user interface related task through a terminal.

As shown in FIG. 7, when a work command for a user interface element occurs, it is transmitted to the user interface tracking unit (S700).

The user interface tracking unit determines whether the delivered work command is a valid user interface element ID (S702).

8 is a diagram illustrating an operation structure of a virtual screen storage unit according to an exemplary embodiment of the present invention, and FIG. 9 is a flowchart illustrating an operation flow of the virtual screen storage unit when the screen output content of the computer is changed.

As shown in FIG. 9, when the output screen is changed, the virtual screen storage unit receives information on the changed screen output content from the screen change tracking unit (S900).

When receiving the information on the screen change from the screen change tracking unit or the user interface tracking unit, the virtual screen storage unit sequentially retrieves the information on the screen output from the queue (S902).

After the information on the screen output is obtained from the queue, the virtual screen storage unit determines whether the obtained information is information on screen deletion (S904). If the obtained information is information about screen deletion, the virtual screen storage unit determines information on the position and size of the area to be deleted and reconfigures the corresponding area to be deleted from the virtual screen (S906).

The corresponding area is deleted from the virtual screen, and the virtual screen storage unit transmits information about the deletion of the corresponding area to the terminal through the server (S914).

If the information obtained from the queue is not information about screen deletion, the virtual screen storage unit determines whether the current virtual screen contents and the screen output contents obtained from the queue overlap (S908).

When the current virtual screen overlaps the screen output obtained from the queue, the virtual screen is reconfigured so that the contents of the virtual screen of the overlapping region are deleted (S910).

If the current virtual screen contents and the screen output contents obtained from the queue do not overlap or the overlapping area is deleted, the virtual screen storage unit inserts the screen output contents obtained from the queue into the virtual screen (S912).

The virtual screen storage unit provides the terminal with the information on the reconfigured virtual screen by reflecting the screen output content obtained from the queue (S914).

FIG. 10 is a flowchart illustrating an operation of a virtual screen storage unit when a state change occurs in a user interface element.

As shown in FIG. 10, when there is a state change in the user interface element, the user interface tracking unit notifies the virtual screen storage unit of the change in the state (S1000).

After receiving the event occurrence information from the user interface tracking unit, the virtual screen storage unit determines whether it is a valid user interface element ID (S1002).

If it is not a valid user interface element ID, the virtual screen storage performs error handling

If it is a valid user interface element ID, the virtual screen storage unit accesses the user interface element of the application program and obtains state information of the user interface element through communication with the application program (S1006). For example, if the application program is a word processor program, the user interface element may be a menu for saving or loading, and the state information of the interface element may be information on whether each menu is activated. Since the method of obtaining the interface element and the state information of each element from the application program is already known, a detailed description thereof will be omitted.

The virtual screen storage unit reconfigures the virtual screen by using a user interface element obtained from an application program and its state information, and provides the terminal with information about the virtual screen (S1008).

11 is a flowchart illustrating a process of delivering a virtual screen through the network communication unit by the virtual screen storage unit according to an exemplary embodiment of the present invention.

Referring to FIG. 11, a user makes a request for viewing a current screen to a computer through a terminal (S1100).

When the screen view request information is received from the computer, the virtual screen storage unit determines whether it is a full screen view request or a recent change screen view request (S1102).

In the case of a full screen view request, the virtual screen storage unit fetches information on the stored screen output contents (S1104).

If the screen view request is a request for a changed recent screen, the virtual screen storage unit brings only information on the recently changed screen (S1106).

After obtaining information about the screen content, it is determined whether a user interface element is output on the current screen (S1108). That is, it is determined whether there is a user interface element and its state information transmitted from the user interface event tracker.

If the user interface elements are output on the screen, the transferred user interface elements and their status information are taken (S1112).

Information on the imported screen content and information on the user interface element are transmitted to the terminal through the network communication unit (S1114).

12 is a flowchart illustrating an operation flow of a task execution unit according to an exemplary embodiment of the present invention.

Referring to FIG. 12, first, a user inputs computer control information through a terminal, and the control information input by the user is transferred to a job execution unit through a network communication unit (S1200).

The job executor sequentially executes jobs corresponding to the control information transmitted from the terminal (S1202), and the job executor executes job contents sequentially by calling an application program corresponding to the input control information.

The job execution unit determines whether a job command end signal is received from the terminal, and determines whether the requested job command is a text input, a job associated with a user interface element, or an application program call job (S1204).

If the input job command is a job for a user interface element, the job executor calls the user interface event tracker (S1206), and receives an event (for example, a mouse click) corresponding to the control information entered in the user interface element. To generate a state change of the corresponding user interface (S1208).

If the input job command relates to text input, the job executor inserts the text information transmitted from the terminal into the system input queue (S1212).

If the input work command relates to an application program call, the task execution unit calls the corresponding application program (S1210).

After executing the work command input by the user, the work execution unit determines whether there is a screen request for the execution result from the terminal (S1214).

If the screen request information on the execution result is not additionally transmitted from the terminal, the task execution unit transmits the task execution end information to the terminal through the network communication unit (S1216).

If there is a screen request for the execution result from the terminal, the task execution unit requests the terminal to provide information on the changed screen to the virtual screen storage unit (S1218). The virtual screen storage unit received the request from the job execution unit transmits the information on the changed screen output to the terminal through the network communication unit (S1220).

FIG. 13 is a block diagram illustrating an operation structure of a network communication unit according to an exemplary embodiment of the present invention, and FIG. 14 is a flowchart illustrating an operation flow of the network communication unit according to an exemplary embodiment of the present invention.

As shown in Fig. 13, the network communication unit provides information received from the terminal to the task execution unit, and transmits the information provided from the virtual screen storage unit or the task execution unit to the terminal. In addition, the network communication unit functions to process the authentication process of the user when the user connects to the computer using the terminal.

Referring to FIG. 14, first, the network communication unit receives access request information from a terminal (S1400). The access request information includes user authentication information such as ID and password.

The network communication unit performs user authentication processing using the user authentication information included in the connection request information (S1402). If the user authentication result is not a valid user, the network communication unit rejects the access of the terminal (S1404).

If the user authentication result is a valid user, the terminal is allowed to access the terminal and receives control information from the terminal, and it is determined whether the received control information relates to a job execution request or a screen view request (S1406).

If the screen view request is related, the network communication unit requests the information on the current screen output content to the virtual screen storage unit to obtain it (S1408). As described above, the virtual screen storage unit determines whether the request is for a full screen or for a changed screen, and provides the corresponding screen output to the network communication unit.

The network communication unit transmits information on the requested screen output content to the terminal (S1410).

When the control information input by the user through the terminal relates to the job execution request, the network communication unit provides the corresponding job execution request to the job execution unit in operation S1412.

The job execution unit sequentially executes the transferred job execution requests, and when the job execution is completed, delivers the job completion information to the network communication unit, and the network communication unit transmits the job completion information to the terminal (S1414).

As described above, according to the method and system for remote computer control according to the present invention, there is an advantage that even an application program in a GUI environment can be controlled remotely without being greatly influenced by the environment of the terminal.

1 is a diagram showing the configuration of a computer control system using a mobile communication terminal according to an embodiment of the present invention.

2 is a diagram showing the module configuration of a program installed in a computer according to an embodiment of the present invention.

3 is a block diagram illustrating an operation structure of a screen change tracking unit according to an exemplary embodiment of the present invention.

 4 is a flowchart illustrating an operation of a screen change tracking unit according to an exemplary embodiment of the present invention.

5 is a view showing the operation structure of a user interface tracking unit of the present invention.

6 is a flowchart illustrating an operation flow of a user interface tracker when there is a state change of a user interface element in a computer.

7 is a flowchart illustrating an operation of a user interface tracking unit when a user who remotely controls a computer executes a user interface related task through a terminal.

8 is a diagram illustrating an operation structure of a virtual screen storage unit according to an exemplary embodiment of the present invention.

9 is a flowchart showing an operation flow of the virtual screen storage unit when the screen output contents of the computer are changed.

10 is a flowchart showing the operation of the virtual screen storage unit when a state change occurs in a user interface element.

11 is a flowchart illustrating a process of delivering a virtual screen through the network communication unit by the virtual screen storage unit according to an exemplary embodiment of the present invention.

12 is a flowchart illustrating an operation flow of a task execution unit according to an exemplary embodiment of the present invention.

13 is a block diagram showing an operation structure of a network communication unit according to an embodiment of the present invention.

14 is a flowchart illustrating the operation flow of the network communication unit according to the preferred embodiment of the present invention.

Claims (1)

In a computer remote control system, An IP database server connected to the computers to be controlled through a network and managing IP address information of the connected computers; A computer connection module for accessing the IP database server to find out address information of the computer to be controlled and to access the computer to be controlled using the found address information; A work command synthesis module for synthesizing control command information of a user for a computer connected through the computer connection module; A screen reconstruction module for reconstructing information on screen contents transmitted from the computer according to a display environment; and A mobile communication terminal including a network communication unit which transmits job execution information output from the job command synthesis unit to a computer to be controlled and receives information on the screen contents transmitted from the computer and provides the information to the screen reconstruction unit; A screen change tracker that intercepts information about the screen change by replacing the screen driver of the computer to be controlled with itself, A user interface event tracker that intercepts the event monitor of the computer to be controlled by itself and intercepts the user interface information of the application and state change information of the user interface, A virtual screen storage unit configured to reconstruct a virtual screen using information provided by the screen change tracker and the user interface event tracker; A task execution unit that sequentially executes a task instructed by a user through interworking with an operating system or the user interface event tracker in response to the information on the execution of the task provided from the terminal, and And a computer application program including a network communication unit responsible for transmitting and receiving data between the terminal and the computer to be controlled.