JP2002111893A - Remote operation system, method therefor and recording medium with remote operation program recorded therein - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote operation system where an outer computer can remotely be operated with GUI even by a small portable communication terminal such as a portable telephone set and PHS. SOLUTION: The system is provided with a picture analysis part 103 analyzing a content displayed on a picture, a picture information transmission part 104 transmitting data on picture information that the picture analysis part outputs to terminal equipment 101, an operation performing part 106 receiving an operation from a user and performing a prescribed processing corresponding to the operation, and a picture change detection part 105 detecting the change of the display of the picture. The picture analysis part 103 detects GUI parts, obtains the types and the names of the GUI parts and data on a position in the picture and has the setting of hyper links for the respective GUI parts. When the hyper links corresponding to the respective GUI parts are selected, a web page for transmitting data identifying the GUI parts and data for identifying an operation corresponding to the GUI parts is generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a remote control system for remotely controlling an information processing device from another information processing device via a communication network, a remote control method thereof, and a recording medium storing a remote control program. About.

[0002]

2. Description of the Related Art Conventionally, in an electronic computer system, a remote operation system for operating a server from an external terminal via a communication network has been widely used.

[0003] By this remote operation, a user can freely operate a computer or equipment at a remote place.
Also, many users can share a high-performance server and operate the server from a plurality of (and unspecified terminals connected to a communication network).

In a conventional remote operation system, a server which receives a remote operation is provided with a function of receiving a user's login and operation via a network.
Remote control is realized by equipping the remote control terminal with virtual terminal software and a terminal emulator for remote control.

For example, virtual terminal software for remotely controlling a server from such an external terminal includes telnet.
Etc. telnet is a CUI (character based user interface) which is a method of inputting command commands by characters.
Remote control, and a computer corresponding to such a CUI operation such as UNIX (registered trademark) is set as a target of the remote operation, and an operation screen of the CUI environment of the computer to be operated is displayed on the user's original terminal. Then, the user performs a remote operation by inputting a command to the operation screen.

As described above, in the CUI environment remote operation system, the operation screen of the CUI environment of the computer to be operated is displayed as the entire display of the terminal or as one of the frames, and the command input by the user is operated. And displays a response corresponding to the execution on an operation screen.

[0007] Recently, a GUI (graphical operation) for operating a graphical element such as an icon or a window displayed on a screen with a pointing device such as a mouse.
user interface: Graphical user interface)
Computers with an operating environment of
Computers in such a GUI environment often do not support operations by inputting command commands in the CUI.

[0008] However, even for a computer in such a GUI environment, by providing virtual terminal software corresponding to the GUI environment in a terminal on the side of remote control, C
Remote control can be performed as in the case of the UI environment.

The virtual terminal software in the case of the GUI environment connects to the computer to be operated, acquires the data of the display screen, and displays the data on the display of the terminal. Then, when the user performs a GUI operation such as a mouse click on the display screen, a remote operation is performed by instructing a computer to be operated to execute a process corresponding to the operation.

Thus, the display screen of the computer to be operated can be displayed on the user's terminal, and the computer to be operated can be remotely operated by operating the GUI.

The virtual terminal software corresponding to such a GUI environment includes, for example, “WinShare” (NE
C), "Desktop On-Call" (Japan I)
BM) and "pcANYWHERE32" (manufactured by Symantec). In addition, the prior art of a remote control system for remotely controlling a computer in a GUI environment includes:
For example, there is a technology of a remote computer operating device disclosed in Japanese Patent Application Laid-Open No. 05-035622.

[0012]

As described above, the conventional remote control system for a computer in a GUI environment has the following problems.

First, in the conventional remote control system, it is necessary that the terminal on the remote control side has special functions such as software for remote processing or a semiconductor circuit for realizing such functions. Met.

The GUI environment of the computer to be operated depends on the model of the computer and the OS (operat
The virtual terminal software and the like need to use software corresponding to the computer to be operated in advance.

Further, such software or the like for performing remote control using a GUI requires detailed and complicated settings at the time of installation or the like, occupies a large amount of space in a hard disk, and may be expensive. For example, there are many troublesome points in providing this in a terminal.

For this reason, the user cannot freely select a terminal for remote operation, and must use a special terminal having a remote operation function corresponding to the computer to be operated. . For example, a remote operation cannot be performed by freely selecting an information processing terminal or a communication terminal at hand.

As a conventional technique for solving this problem, there is a remote control method disclosed in Japanese Patent Application Laid-Open No. 10-301874.

In the prior art disclosed in Japanese Patent Application Laid-Open No. Hei 10-301874, a web page for displaying image data of a display screen of a server which is a computer to be operated is automatically generated, and an image of the display screen is displayed by a user. Display it on the terminal's web browser.

The operation by the user from the GUI is as follows:
By using the JAVA (registered trademark) applet to display the image data on the display screen, when the user performs an operation such as clicking with the mouse on the image data on the display screen, this is automatically transmitted to the server side. And the server executes the corresponding process. Further, the image on the display screen of the server, which is displayed on the web browser of the terminal of the user, is updated when an operation from the user is received.

Therefore, the user terminal only needs to have a communication function such as the Internet and a web browser, and does not need to have a special function for remote control.

However, the image data on the display screen of the computer requires a considerably large amount of data, and when transmitting the data to the user's terminal in order to receive a remote operation, a high-speed communication line and a long communication time are required. In need.
Although there is no need to provide the user's terminal with a special function, the communication frequently communicates a large amount of image data, and is not suitable for the current communication infrastructure such as the Internet.

Second, in the conventional remote control system, it is necessary that the terminal on the remote control side has a display capable of displaying the display screen of the computer to be operated as it is.

For example, it is necessary to use an information processing terminal having a wide display screen such as a personal computer or a notebook personal computer, and it is necessary to use a mobile phone or a PHS (personal handy-pho
With a small communication terminal such as a ne system), it was not possible to remotely control an external computer.

In these communication terminals such as mobile phones and PHSs, the display is considerably narrow, the number of colors that can be displayed is small, and some of them can only display characters but cannot display images. Can not do.

However, these portable communication terminals are widely used and used by many users, are convenient to carry, and are therefore extremely useful if they can be used for remote control of an external computer. It is.

A first object of the present invention is to solve the above-mentioned drawbacks of the prior art, and to enable an external computer to be connected to a GU even with a small portable communication terminal such as a portable telephone or a PHS currently on the market.
It is an object of the present invention to provide a remote operation system, a remote operation method, and a recording medium on which a remote operation program is recorded, which can be remotely operated by I.

A second object of the present invention is to solve the above-mentioned drawbacks of the prior art, and it is not necessary for a terminal which performs remote control to have a special function for remote control, so that a user can freely use the terminal. It is an object of the present invention to provide a remote control system capable of remotely controlling an external computer by a GUI using a portable communication terminal or the like at hand, a remote control method thereof, and a recording medium storing a remote control program.

[0028]

To achieve the above object, a remote operation system according to the present invention is a remote operation system for remotely operating an operation target device from an operation terminal via a communication network. Screen analysis means for analyzing information displayed on a screen, and screen information transmission means for transmitting data of the analysis result of the screen analysis means to the operation-side terminal for display on the operation-side terminal, And

According to the second aspect of the present invention, there is provided a remote control system comprising:
In a remote operation system for remotely operating an operation target device from an operation side terminal via a communication network, the operation target device includes a screen analysis unit configured to analyze a content displayed on a screen of the operation target device; The information processing apparatus further includes a screen information transmission unit that transmits data of an analysis result of the unit to the operation-side terminal to display the data on the operation-side terminal.

According to a third aspect of the present invention, there is provided a remote control system comprising:
An operation execution unit is provided that causes the operation target device to execute a predetermined process in response to a user operation performed using the operation-side terminal.

According to a fourth aspect of the present invention, there is provided a remote control system comprising:
Screen information conversion means for converting the data of the analysis result of the screen analysis means to data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal, the screen information transmission means, The data of the conversion result of the screen information conversion means is transmitted to the operation side terminal for display on the operation side terminal.

According to a fifth aspect of the present invention, there is provided a remote control system according to the first aspect.
A conversion rule registration unit for registering and recording a rule for converting the data of the analysis result of the screen analysis unit into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal; The information conversion unit converts data of the analysis result of the screen analysis unit based on the conversion rule recorded in the conversion rule registration unit.

According to a sixth aspect of the present invention, there is provided a remote control system comprising:
Device characteristic detecting means for detecting a device characteristic of the operation side terminal, wherein the screen information converting means is based on the device characteristic detected by the device characteristic detecting means and a conversion rule recorded in the conversion rule registering means. The data of the analysis result of the screen analysis means is converted.

According to a seventh aspect of the present invention, there is provided a remote control system according to the first aspect.
Screen change detecting means for detecting a change in the display of the screen of the operation target device, and when the generated change of the display of the screen ends, the screen analysis means analyzes the current screen of the operation target device. It is characterized by the following.

The remote control system according to the present invention of claim 8 is
The operation target device is an information processing device including a GUI,
The screen analysis unit detects a GUI component displayed on a screen of the operation target device, and acquires attribute data of the GUI component.

According to a ninth aspect of the present invention, there is provided a remote control system comprising:
The attribute data of the GUI component includes at least one of a type of the GUI component, a name of the GUI component, and a position of the GUI component on a screen.

In another preferred construction, the conversion rule registration means includes a registration of a rule for converting the data of the analysis result of the screen analysis means into a web page that can be displayed on a web browser. The screen information conversion means converts the data of the analysis result of the screen analysis means into a web page based on the rule, and the web page is operated by a user in a GUI component detected by the screen analysis means. GU is applicable to
A hyperlink corresponding to the I component is included, and when the hyperlink is selected, data for identifying a GUI component corresponding to the hyperlink and data for identifying an operation on the GUI component are transmitted to the operation execution unit. It is characterized by the following.

An eleventh aspect of the present invention is the remote operation system according to the eleventh aspect, wherein the conversion rule registration means includes a rule registration for converting the data of the analysis result of the screen analysis means into text of an electronic mail. .

According to a twelfth aspect of the present invention, in the remote operation system, the operation executing means receives an e-mail including information on an operation performed by a user using the operation-side terminal, and transmits the received e-mail. Analysis is performed to identify an operation from a user, and the operation is executed.

According to a thirteenth aspect of the present invention, in the remote operation system according to the present invention, the data of the analysis result of the screen analysis means is XML format data.

According to a fourteenth aspect of the present invention, the remote operation system is characterized in that the operation side terminal is a portable communication terminal having a function of transmitting and receiving character data.

According to a fifteenth aspect of the present invention, the communication network is the Internet.

In the remote operation system according to the present invention, when the moving image or the still image is displayed on the screen of the operation target device, the screen analyzing means detects the display and converts the video data of the display. The screen information transmitting means extracts and transmits the video data extracted by the screen analyzing means to the operation side terminal for display on the operation side terminal.

In the remote operation system according to the present invention, the screen analysis means detects a display when a moving image or a still image is displayed on the screen of the operation target device, and converts the image data of the display. Extracting, the screen information conversion means converts the video data extracted by the screen analysis means into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. And

In the remote operation system according to the present invention, the device characteristics of the operating terminal detected by the device characteristic detecting means include a unique ID, a model, a processing speed, a communication speed, and a unit of the operating terminal. Communication cost per data amount, communication cost per unit time, data format that can be displayed on the screen,
Playable sound data format, executable program and script formats, amount of data that can be received at one time, displayable colors, displayable character fonts, screen resolution, physical length and width of the screen, It is characterized by including at least one of the number of vertical and horizontal characters that can be displayed in one screen.

According to a nineteenth aspect of the present invention, in the remote operation method for remotely operating an operation target device from an operation side terminal via a communication network, the information displayed on the screen of the operation target device is analyzed. And a screen information transmission step of transmitting data of an analysis result of the screen analysis step to the operation side terminal to display the data on the operation side terminal.

A twentieth aspect of the present invention provides the remote operation method according to the present invention, further comprising an operation execution step of causing the operation target device to execute a predetermined process in response to an operation performed by the user using the operation side terminal. It is characterized by the following.

According to a twenty-first aspect of the present invention, in the remote operation method according to the present invention, the data of the analysis result of the screen analysis step is converted into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. It has a screen information conversion step,
In the screen information transmitting step, data of a conversion result of the screen information converting step is transmitted to the operation side terminal for displaying on the operation side terminal.

According to a twenty-second aspect of the present invention, in the remote operation method of the present invention, the data of the analysis result in the screen analysis step is converted into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. A conversion rule registration step of registering a rule, wherein in the screen information conversion step, data of an analysis result of the screen analysis step is converted based on the conversion rule registered in the conversion rule registration step. .

A remote operation method according to a twenty-third aspect of the present invention includes a device characteristic detecting step of detecting a device characteristic of the operation-side terminal, and in the screen information conversion step, the device characteristic detected by the device characteristic detecting step. And converting the data of the analysis result of the screen analysis step based on the conversion rule registered in the conversion rule registration step.

A remote control method according to a twenty-fourth aspect of the present invention includes a screen change detecting step of detecting a change in the display of the screen of the operation target device. In the analyzing step, the contents displayed on the screen of the operation target device at the present time are analyzed.

According to a twenty-fifth aspect of the present invention, in the remote operation method, the operation target device is an information processing device having a GUI, and in the screen analysis step, a GUI component displayed on a screen of the operation target device is detected. It is characterized in that attribute data of the GUI component is acquired.

According to a twenty-sixth aspect of the present invention, the attribute data of the GUI component includes at least one of a type of the GUI component, a name of the GUI component, and a position of the GUI component on the screen. It is characterized by including one.

In a preferred embodiment of the remote control method according to the present invention, in the conversion rule registration step, a rule for converting the data of the analysis result of the screen analysis step into a web page displayable by a web browser is registered. In the screen information conversion step, the data of the analysis result of the screen analysis step is converted into a web page based on the rule, and the web page is operated by a user among the GUI components detected in the screen analysis step. If possible, include a hyperlink corresponding to the GUI part,
In the operation executing step, a predetermined process is performed using data for identifying a GUI component corresponding to the hyperlink and data for identifying an operation on the GUI component, which are transmitted when the hyperlink is selected. It is characterized in that it is executed by the side terminal.

The remote operation method according to the present invention is characterized in that, in the conversion rule registration step, a rule for converting data of an analysis result of the screen analysis step into text of an electronic mail is registered.

According to a twenty-ninth aspect of the present invention, in the screen analysis step, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected and the video data of the display is detected. In the extracting and transmitting the screen information, the video data extracted in the screen analyzing step is transmitted to the operating terminal in order to be displayed on the operating terminal.

According to a thirty-fifth aspect of the present invention, in the screen analyzing step, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected and the video data of the display is detected. Extracting and converting the video data extracted in the screen analysis step into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal in the screen information conversion step. And

According to a thirty-first aspect of the present invention, the device characteristics of the operating terminal detected by the device characteristics detecting step include a unique ID, a model, a processing speed, a communication speed, and a unit of the operating terminal. Communication cost per data amount, communication cost per unit time, data format that can be displayed on the screen, sound data format that can be played, format of executable programs and scripts, amount of data that can be received at one time, colors that can be displayed And at least one of a displayable character font, a screen resolution, a vertical and horizontal physical length of the screen, and the number of vertical and horizontal characters that can be displayed in one screen.

A remote control program according to a thirty-second aspect of the present invention is a remote control program for remotely controlling an operation target device from an operation side terminal via a communication network by controlling an information processing system. Screen analysis processing for analyzing the information displayed on the screen of
A screen information transmission process for transmitting data of an analysis result of the screen analysis process to the operation-side terminal for displaying the data on the operation-side terminal is performed.

According to a thirty-third aspect of the present invention, the remote operation program executes an operation execution process for causing the operation target device to execute a predetermined process in response to an operation performed by a user using the operation-side terminal. It is characterized by making it.

A remote control program according to a thirty-fourth aspect of the present invention converts the data of the analysis result of the screen analysis processing into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. Executing a screen information conversion process, and in the screen information transmission process, executing a process of transmitting data of a conversion result of the screen information conversion process to the operation side terminal to display the data on the operation side terminal. Features.

A remote operation program according to a thirty-fifth aspect of the present invention converts the data of the analysis result of the screen analysis processing into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. A conversion rule registration process for registering a rule is executed, and in the screen information conversion process, a process for converting data of an analysis result of the screen analysis process based on the conversion rule registered by the conversion rule registration process is executed. It is characterized by making it.

The remote control program according to the present invention of claim 36 causes a device characteristic detecting process for detecting a device characteristic of the operation side terminal to be executed, and in the screen information conversion process, the device characteristic detecting process detects the device characteristic. A process for converting the data of the analysis result of the screen analysis process is executed based on the device characteristics and the conversion rule registered by the conversion rule registration process.

A remote control program according to a thirty-seventh aspect of the present invention executes a screen change detecting process for detecting a change in the display of the screen of the operation target device. A process for analyzing the content currently displayed on the screen of the operation target device by the screen analysis process is executed.

A remote control program according to a thirty-eighth aspect of the present invention provides the remote operation program, wherein the operation target device is an information processing device having a GUI, and the screen analysis processing detects a GUI component displayed on the screen of the operation target device. Then, a process of acquiring the attribute data of the GUI component is executed.

In a remote control program according to a thirty-ninth aspect of the present invention, the attribute data of the GUI component includes at least one of a type of the GUI component, a name of the GUI component, and a position of the GUI component on the screen. It is characterized by including one.

In the remote control program according to the present invention, in the conversion rule registration process, a rule for converting data of an analysis result of the screen analysis process into a web page displayable by a web browser is registered. In the screen information conversion processing, the data of the analysis result of the screen analysis processing is converted into a web page based on the rule, and the web page is a G page detected by the screen analysis processing.
For a UI component that can be operated by the user, a hyperlink corresponding to the GUI component is included,
In the operation execution processing, in order to cause the operation-side terminal to execute the predetermined processing, data for identifying a GUI component corresponding to the hyperlink transmitted when the hyperlink is selected, and the GUI component Characterized by using data for identifying an operation for.

The remote operation program according to the present invention, in the conversion rule registration process, executes a process of registering a rule for converting data of an analysis result of the screen analysis process into an e-mail text. It is characterized by.

The remote operation program according to the present invention, in the screen analysis processing, detects a display when a moving image or a still image is displayed on the screen of the operation target device, and detects the video data of the display. In the screen information transmission process, a process of transmitting the video data extracted by the screen analysis process to the operation side terminal for display on the operation side terminal is performed. .

The remote control program according to the present invention is characterized in that in the screen analysis processing, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected and the video data of the display is detected. In the screen information conversion processing, the video data extracted by the screen analysis processing is converted into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. It is characterized by.

A remote operation program according to the present invention is characterized in that the device characteristics of the operating terminal detected in the device characteristic detecting process include a unique ID of the operating terminal, a model, a processing speed, a communication speed, and a unit. Communication cost per data amount, communication cost per unit time, data format that can be displayed on the screen, playable sound data format, executable program and script format, data volume that can be received at one time,
Displayable colors, displayable fonts, screen resolutions,
It includes at least one of the vertical and horizontal physical lengths of the screen and the number of vertical and horizontal characters that can be displayed in one screen.

[0072]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of the remote control system according to the first embodiment of the present invention.

Referring to FIG. 1, in the remote operation system according to the first embodiment, the computer 100 is a device to be remotely operated (operation target device), and the terminal device 101 is an operation side used by the user for remote operation. As a terminal, the computer 100 and the terminal device 101 are connected via a communication network 102 and perform communication for remote operation.

The computer 100 is, for example, an information processing apparatus having an operation environment using a GUI.

The terminal device 101 is, for example, a mobile phone or a P
HS (personal handy-phone system) and other terminals capable of communicating via the communication network 30;
A terminal used by a user to remotely operate the computer 100.

The terminal device 101 is provided with a data communication function and a display screen on which data can be displayed, as in a commercially available mobile phone, PHS, or the like that is currently in widespread use.

The communication network 102 is an information communication line through which the terminal device 101 of the user and the computer 100 to be remotely controlled perform remote control. For example, the Internet, personal computer communication, i-mode Internet, and others. Dedicated lines, and telephone lines and wireless telephone lines for connecting to these lines.

Referring to FIG. 1, a computer 100 to be remotely operated includes a screen analyzing unit 103 and a screen information transmitting unit 1.
04, a screen change detection unit 105, an operation execution unit 106, and a communication unit 107.

The screen analysis unit 103 is a computer
Analyzes the contents of the display screen and outputs the analysis result.

The screen information transmitting section 104 transmits the screen information obtained as a result of the analysis by the screen analyzing section to the terminal device 101.

The communication unit 107 is connected to the communication network 102
, And communicates with the terminal device 101.

The operation execution unit 106 executes a process specified by a remote operation or issues an instruction to execute the process.

The screen change detection unit 105 is a computer
The change of the display screen at 00 is detected.

FIG. 2 is a flowchart for explaining the remote control processing of the remote control system according to the first embodiment.

Referring to FIG. 2, the remote control system according to the first embodiment first receives a connection from a user terminal device 101 in a computer 100 to be remotely controlled (step 201).

The user is authenticated by a method such as receiving an input of an ID number or a password for identifying the user (step 202), and the user is registered as a user who performs a remote operation and is authenticated. As long as the remote control communication is performed.

Then, the screen analyzing unit 103 analyzes the display screen of the computer 100 to be operated (step 2).
03). For example, when the computer 100 is a personal computer (PC) or a workstation (WS) having a GUI, it analyzes what kind of object is displayed on the current GUI screen, and analyzes the type and name of each object. Get data such as position. However, the objects displayed on the GUI screen are GUI parts such as windows, icons, menus, and buttons. In this analysis, for example, a method of acquiring from the OS is used.

The data obtained by the screen analyzing unit 103 is transmitted to the user's terminal device 101 by the screen information transmitting unit 104, and is transmitted to the user's terminal device 10.
1 is displayed on the screen 1 (step 204).

When the user performs an operation on the computer 100 using the terminal device 101,
This operation information is received by the operation execution unit 106 and executed on the computer 100 (steps 205 and 206).

When the display on the screen of the computer 100 changes, the screen change detecting unit 105 detects the change, and when it is known that the change has been completed, the processing from step 203 onward is executed again. Then, the data displayed on the screen of the user terminal device 101 is updated to the latest data.

Next, the analysis of the contents of the display screen of the computer 100 by the screen analysis unit 103 will be described in detail.

FIG. 3 shows a screen analysis unit 103 according to this embodiment.
3 is a flowchart for explaining the processing of FIG.

The screen analysis unit 103 acquires information such as the name of the icon to be displayed, the display position, and the corresponding application from the OS of the computer 100, for example. As a result, data of various objects shown in icons and windows on the screen can be clearly obtained as character data.

This information acquisition method is based on the OCR (op) that simply converts a character image displayed on a screen into character data.
(tical character recognition) method can also be used. However, in the OCR method of converting an image into character data, there is a danger that the data may be converted into incorrect character data, and a part of a window may protrude to the right or left of the screen and cannot be read. is there. The screen analysis unit 103 of the first embodiment adopts a method of acquiring detailed information of each object from the OS of the computer 100, so that it is not affected by the image resolution, the window display position, and the like. Accurate character data can be obtained instantaneously. This method is also used when implementing an accessibility application system for making a data processing apparatus having a GUI available to a user having a visual or hearing impairment.

The screen analysis unit 103 according to the first embodiment
First, of the windows displayed on the current GUI screen of the computer 100, an active window (active window) is detected, and the title of the detected active window is acquired (steps 301 and 3).
02).

However, an active window is a window that is ready to receive input from the keyboard, and a window title is a character string assigned to the window (sometimes called a caption).

[0098] In the GUI screen, the title of the window is generally displayed in an area above the window (called a title bar). For example, the title of the window 40 in FIG. 4 is “Open File”.

In the first embodiment, when there is one or more windows on the screen as in a general OS, it is assumed that one active window exists.

For example, the current GU of the computer 100
It is assumed that three windows 40 are displayed on the I screen as shown in FIG. Then, one of the three windows 40 is an active window, and the information of each window 40 is managed by the OS. Therefore, the screen analysis unit 103 acquires the information from the OS to obtain the current information. The active window 40 can be detected, and similarly information such as the title can be obtained.

A window 40 is displayed on the current GUI screen 50.
Is not displayed, the desktop which is the entire background area (the entire screen) such as a window is set as the currently active window. Here, when the desktop is detected as an active window, the title of the active window is “desktop”.

The screen analysis unit 103 is a computer
When an active window displayed on the current GUI screen is detected, objects such as menus, icons, and buttons displayed on the active window (hereinafter, these objects are collectively referred to as GUI parts) are further detected. The property information is obtained (Step 303).

However, in the first embodiment, the GU
The property information of the I component is a character string assigned to the GUI component, such as a type (menu, icon, button, etc.) and a label (“OK” or “cancel” of the button), which is called a caption. ) And coordinate values on the screen.

For example, as in the example of FIG. 6, the active window displayed on the current GUI screen is the “memo pad” window 40, and the window 40 includes “file”, “edit”, and “format”. , "Tools" menu item 60 is displayed, the screen analysis unit 1
03 acquires the property information of these four menu items 60 as shown in the example of FIG. FIG.
Each of x1 to x4 and y1 to y4 represents a positive integer.

Here, the coordinate value of the GUI component on the screen is, for example, the coordinate of one point indicated by using the horizontal and vertical distances from the upper left point of the screen to the center (or inside) of the GUI component. Can be indicated by

As shown in FIG. 8, when the window 40 is on the desktop 50 and the GUI component 80 is on the window, the coordinate value of the GUI component is represented by the horizontal coordinate x. It is shown as (x, y) using the vertical coordinate y. Here, x and y are positive integers.

When the active window 40 displayed on the current GUI screen of the computer 100 is the “Notepad” window in FIG. 6, the screen analyzing unit 103 determines that the active window 40 is the “Notepad” window. It detects that there is, acquires the title of this window 40 (the character string “Notepad”), and further detects four menu items 60 of this window 40, and as shown in FIG. Get property information. As a result, data as shown in FIG. 9 is obtained.

As described above, the analysis of the GUI screen by the screen analysis unit 103, such as detection of active windows and acquisition of detailed and accurate information of various GUI parts, is performed by, for example, Hidehiko Okada, Asahi Toshiyuki, "Tool for Evaluating Consistency of GUI Design" GUI Tester 2 "" (Report of IPSJ Human Interface Workshop, Vo
l. 97, No. 63, pp. 7-12, 1997).

Then, the screen analyzer 103 outputs the data as shown in FIG. 9 to the screen information transmitter 104 in a predetermined format (step 304). The predetermined format is a format that can be displayed on the device terminal 11, and in the first embodiment, the format is known before the remote operation by the terminal device 101 is started. Screen information transmission unit 1
04 transmits the data received from the screen analysis unit 103 to the terminal device 101. As a result, the terminal device 101
The data output from the screen analysis unit 103 (ie, the GU displayed on the current screen of the computer 100)
(Data including information on the I-parts) is displayed.

For example, the terminal device 101 is an HTML (Hype
It is assumed that a web browser that displays a web page described in r-Text Markup Language) is provided, and a web page published on a website on a communication network such as the Internet can be displayed. In this case, the screen analysis unit 103 outputs the data of the web page in the HTML format, and this data is displayed on the screen of the terminal device 101, for example, as shown in FIG. In the example of FIG. 10, first, "Notepad", which is the name of the active window, is displayed, and subsequently, in each line, the item name of each menu item 60 is displayed as an HTML hyperlink.
This can be selected on the screen of the terminal device 101.

An HTML data file created by the screen analysis unit 103 and transmitted by the screen information transmission unit 104 is displayed on the screen of the terminal device 101, and when an operation is performed on this screen (for example, a hyperlink is selected). ), The operation information is transmitted to the communication network 102.
And transmitted to the operation execution unit 106 via the communication unit 107.

The operation execution unit 106 determines the operation corresponding to the operation performed on the terminal device 101 from the operation information transmitted from the terminal device 101 and the data of the screen analysis result output from the screen analysis unit 103. , Executed on the computer 100.

For example, when the “file” hyperlink, which is one of the menu item names 1000 displayed on the screen of the terminal device 101 as shown in FIG. Referring to the coordinates ((x1, y1) in FIG. 9) corresponding to the menu from the data of the screen analysis result output from 103, the cursor of a pointing device such as a mouse is moved to the coordinates on the screen of the computer 100. Then, the execution of the click at the position is notified to the OS or the like.

For example, in the example described with reference to FIGS. 6 to 10 as described above, FIG.
It is assumed that the "file" hyperlink is selected.
From the data received from the screen analysis unit 103, the operation execution unit 106 corresponds to the “file” menu 60 in FIG. 6 corresponding to the “file” in FIG. 10 with a hyperlink, and its coordinate value is (x1, y1). You can see that there is. When the operation execution unit 106 receives, from the terminal device 101, information indicating that the “file” hyperlink in FIG. 10 is selected on the screen of the terminal device 101, the operation execution unit 106 moves the cursor of a pointing device such as a mouse to coordinate values (x1, y1). And notifies the OS or the like of the execution of the click at the position. As a result, the “File” menu 60 in FIG. 6 is selected on the screen of the computer 100.

The operation corresponding to the operation performed on the screen of the terminal device 101 is executed by the operation execution unit 1 on the computer 100.
06, the screen change detection unit 105
Detects a change on the GUI screen on the computer 100.

The changes on the GUI screen of the computer 100 detected by the screen change detection unit 105 include, for example, opening a new window, closing a displayed window, and displaying a pull-down menu.

Further, the screen change detecting unit 105 determines that the response of the computer 100 to the operation performed by the operation executing unit 106 has been completed when the screen change is not detected for a predetermined time interval or more.

When the screen change detection unit 105 determines that the response has been completed, the screen analysis unit 103 again executes the
The current GUI screen is analyzed to output screen information data, and the screen information transmitting unit 104 transmits the screen information data to the terminal device 101.

That is, when the user selects one of the menu item names 1000 displayed on the terminal
When the operation of selecting is executed, this
0, the process of clicking “File” of the menu item 60 is executed, and the screen information with the “File” pull-down menu displayed after execution is transmitted as data in a format that can be displayed on the terminal device 101. Then, the screen of the user's terminal device 101 is updated to a screen displaying the new data.

Then, similarly, the remote control can be performed by repeatedly executing the processing described above as one cycle.

This cycle is continued until the communication from the terminal device 101 to the computer 100 ends.

Next, an example of remote control by the remote control system according to the first embodiment will be described.

First, the user operates the terminal device 1 such as a mobile phone.
Using 01, a connection is made via the communication network 102 to the computer 100 to be operated. Here, the user is authenticated by a method such as inputting an ID number or password for identifying the user, and the terminal device 101 is authenticated.
Starts communication for remote control to the computer 100.

FIG. 11 is a diagram showing an example of the display screen 50 at the time of connection to the computer 100 to be operated in this embodiment. In the example of FIG. 11, three icons 1100 of “Notepad”, “Schedule”, and “Fax” are displayed on the desktop, and the window has not been displayed yet.

The state of the display screen shown in FIG. 11 is analyzed by the screen analysis unit 103 of the computer 100. The screen analysis unit 103 acquires information such as the name of the icon to be displayed, the display position, the corresponding application, and the like by a method of acquiring from the OS of the computer 100 or the like.

The screen analysis unit 103 converts the screen information data obtained by the analysis into the screen information in a format that can be displayed on the terminal device 101 of the user (for example, as a web page described by an HTML sentence). Output to transmitting section 104. The screen information transmitting unit 104 transmits the received data to the terminal device 101.

FIG. 12 shows an embodiment of the present invention.
5 is a diagram showing an example in which a display screen of the terminal device 101 is analyzed and the information is transmitted to the terminal device 101, and is displayed on the screen of the terminal device 101. FIG.

In the example shown in FIG. 12, since the application window is not yet displayed on the display screen 50 of the computer 100, a “desktop” indicating that fact is displayed and the icons 120 displayed on the display screen 50 are displayed.
The name of 0 is displayed by a hyperlink.

Next, the user selects the icon name 1200 displayed on the terminal device 101. Here, "Fa
An example in which “x” is selected will be described.

Then, the terminal device 101 notifies the computer 100 of a request for data at the link destination of the “Fax” that is a hyperlink. When the computer 100 receives the request, the operation execution unit 106 sets the “Fax”
An operation of double-clicking the coordinates on the GUI screen of the computer 100 (that is, the coordinates on the “Fax” icon 1100) output from the screen analysis unit 103 for the icon 1100 is executed.

In a normal GUI environment, when the icon on the desktop is double-clicked, the process corresponding to the icon is executed. Therefore, here, execution of double-click is set as the operation corresponding to the icon. It is.

FIG. 13 is a diagram showing a state in which the Fax icon is double-clicked on the screen of FIG. 11 to display the Fax window.

In the screen of FIG. 13, a “Fax” window 40 is displayed, which is the currently active window. On the “Fax” window 40, “File” and “Setting” menu items 60 are displayed.

Here, the screen change detection unit 105 controls the screen 5
When a change in which the window 40 is newly displayed within 0 is detected, and the completion of the response is determined because there is no change in the screen for a certain period of time, the screen analysis unit 103 is instructed to analyze the screen.

The screen analysis unit 103 analyzes the screen again, outputs the analysis result as a web page in HTML, and the web page is displayed on the screen of the terminal device 101.

FIG. 14 is a diagram showing an example in which information on the display screen of FIG.

In the display of the terminal device 101 in FIG. 14, first, Fax which is the name of the active window is displayed.
Subsequently, “File” and “Setting”, which are the names of the menu items in the Fax window, are displayed by hyperlinks.

Here, when the user selects the menu item name 1000 of this "file" which is a hyperlink, the operation is similarly notified to the computer 100 side, and the operation execution unit 106 of FIG. GUI screen 50
Then, an operation of clicking the menu item 60 of “File” is executed.

FIG. 15 is a diagram showing a state in which a menu item of a file is clicked on the screen of FIG. 13 and a pull-down menu is displayed.
"Open", "Send", "End" three sub-menu items 1
500 is displayed.

Similarly, the screen change detection unit 105 detects a change in the GUI screen 50 due to the display of the pull-down menu, and if it determines that the screen has not changed for a certain period of time and the response has been completed, the screen analysis unit 103 Instructs screen analysis.

Then, the screen analysis unit 103 analyzes the screen again, outputs the analysis result as a web page in HTML, and the web page is displayed on the screen of the terminal device 101.

FIG. 16 is a diagram showing an example in which information on the display screen of FIG.

In the screen shown in FIG. 16, the names “open”, “send”, and “end” of three submenu items 1500 displayed in the pull-down menu of FIG. 15 are displayed by hyperlinks.

Here, when the user selects the submenu item name 1600 of “send” which is a hyperlink, the operation is similarly notified to the computer 100 side, and the operation execution unit 106 GUI screen 50
An operation of clicking the submenu item “Send” 1500 in the pull-down menu is executed.

FIG. 17 is a view showing a state in which a transmission item is clicked on the screen of FIG. 15 to display a fax transmission window 40, and this window 40 is the currently active window.

The “Send Fax” window 40 includes a list box 1700 for selecting a file to be used for transmission, and buttons 1701 for instructing transmission and cancellation. A list box 1700 displays a list of files 1702 to be transmitted, which are recorded in a folder designated by the user.

Similarly, the screen change detection unit 105
When a screen change due to the display of the “Send Fax” window 40 is detected, and thereafter it is determined that the screen has not changed for a certain period of time and the response has been completed, the screen analysis unit 103 is instructed to analyze the screen.

Then, the screen analysis unit 103 analyzes the screen again and outputs the analysis result as a web page in HTML, and this web page is displayed on the screen of the terminal device 101.

FIG. 18 is a diagram showing an example in which the information on the display screen of FIG.

On the screen of FIG. 18, the file 1702 displayed in each item in the list box 1700 of FIG.
Etc. and “OK” and “Cancel” buttons 1701
Is displayed by a hyperlink.

Here, when the user selects “file2.dat”, which is a hyperlink, the operation is similarly notified to the computer 100 side, and the operation execution unit 106 displays the GUI screen 50 in FIG. An operation of clicking a file 1702 of “file2.dat” displayed in the list box 1700 is executed.

In FIG. 19, by clicking "file2.dat" in the list box 1700 on the screen of FIG. 17, the colors of the characters and the background are highlighted, and the "file2.dat" file 1702 is in a selected state. FIG.

Similarly, screen change detecting section 105 sets item “file2.dat” in list box 1700.
When a change in the screen due to the reverse display is detected and it is determined that the screen has not changed for a certain period of time and the response is completed, the screen analysis unit 103 is instructed to analyze the screen.

Then, the screen analyzing unit 103 analyzes the screen again, outputs the result of the analysis as a web page in HTML, and the web page is displayed on the screen of the terminal device 101.

FIG. 20 is a diagram showing an example in which information on the display screen of FIG.

In the screen shown in FIG. 20, a symbol "●" is added to the head of the file name to indicate that the file is the currently selected item. That is, in the present embodiment, the screen analysis unit 103 outputs an HTML-format web page with a symbol “●” added to the beginning of the hyperlink of the currently selected item in the list box.

Here, when the user selects the hyperlink 1000 indicating the “OK” button, the operation is similarly notified to the computer 100 side, and the operation execution unit 106 displays on the GUI screen 50 in FIG. Then, an operation of clicking an “OK” button 1701 in the “Send Fax” window 40 is executed.

FIG. 21 shows the screen of FIG.
FIG. 14 is a diagram showing a state in which a window for inputting a fax number, which is a fax transmission destination number, is displayed by clicking a button 1701, and this window 40 is a currently active window.

The window 40 for inputting the fax number
Is an input box 2100 for inputting a fax number.
And “OK” and “Cancel” buttons 1701.

Similarly, the screen change detecting unit 105 sets the Fax
When a screen change due to the display of the window 40 for inputting a number is detected, and when it is determined that the response has been completed because there is no change in the screen for a predetermined time or longer, the screen analysis unit 103 is instructed to analyze the screen.

Then, the screen analyzing unit 103 analyzes the screen again and outputs the analysis result as a web page in HTML, and this web page is displayed on the screen of the terminal device 101.

FIG. 22 is a diagram showing an example in which information on the display screen of FIG.

In FIG. 22, character string input box 2
200 is displayed corresponding to the character string input box 2100 in FIG. That is, in the present embodiment, the screen analysis unit 103 displays the character string input box 2100 on the active window of the GUI screen 50 of the computer 100.
If there is, H is displayed so that the character string input box 2200 is displayed on the screen of the terminal device 101 in response to this.
Output a web page in TML format. In recent web browser screens of mobile phones and PHS terminals,
Usually, a character string input box 22 for inputting characters
00 is provided, and in the present embodiment, a character string input box 220 is provided by using this function.
0 is displayed.

Here, the user operates the character string input box 2
When the user inputs the fax number to 200 and selects the hyperlink 1000 indicating the "OK" button, the operation is similarly notified to the computer 100 side, and the operation execution unit 106 transmits the fax input by the user here. Figure 21
Is input to the character string input box 2100 on the GUI screen 50, and the “OK” button 17
Click 01.

As a result, the computer 100 receives a remote operation for instructing transmission of a fax from the user,
file2. dat file contents to the specified F
The ax number is transmitted to the transmission destination.

FIG. 23 is a diagram showing a state where a window 40 indicating the completion of the transmission is displayed in response to the input of the fax number and the transmission instruction in the screen of FIG. The currently active window.

Similarly, the screen change detection unit 105 sets the Fax
When a screen change due to the display of the window 40 for inputting a number is detected, and when it is determined that the response has been completed because there is no change in the screen for a predetermined time or longer, the screen analysis unit 103 is instructed to analyze the screen.

Then, the screen analyzing unit 103 analyzes the screen again, outputs the analysis result as a web page in HTML, and the web page is displayed on the screen of the terminal device 101.

FIG. 24 is a diagram showing an example in which information on the display screen of FIG.

Then, when the user selects the hyperlink 1000 representing the “OK” button,
When an operation for closing window 40 (FIG. 23) for notifying completion of fax transmission is performed, operation execution unit 106
Is the window 4 on the GUI screen of the computer 100.
Click on an “OK” button 1701 on “0”. Then, the fax transmission completion notification window 40 is closed, and the GUI screen 50 of the computer 100 returns to the state shown in FIG.

Thus, a series of operations for transmitting the fax by remotely operating the computer 100 is completed.

Next, a remote control system according to a second embodiment of the present invention will be described.

In the first embodiment, the terminal device 1
01 is a terminal device that can display an HTML-format web page in advance.
The screen information data that is output to the screen information transmitting unit 104 is a web page in HTML format. On the other hand, in the second embodiment, it is known in advance that the terminal device 101 is a terminal device capable of transmitting and receiving e-mail, and the screen analysis unit 103 transmits the screen information to the screen information transmission unit 104. The data is the text of the email.

FIG. 25 is a block diagram showing a configuration of a remote control system according to the second embodiment. The basic operation of the remote control system according to the second embodiment is exactly the same as that of the remote control system according to the first embodiment, and data for remote control is transmitted not by a web page in HTML text but by electronic mail. The difference is in the interaction.

For example, in the case where the display screen of the computer 100 is the display screen 50 shown in FIG. 11, in the remote operation system of the second embodiment, the screen analyzing unit 2500 converts the display screen 50 shown in FIG. Parse the content and output the result as e-mail text. The screen information transmitting unit 2501 transmits the text of the e-mail to the terminal device 10.
1 to the e-mail address. Terminal device 10
1 receives this electronic mail and is displayed on the screen as shown in FIG. 26, for example.

When the user looks at the e-mail screen and wants to operate, for example, the “Fax” icon, the e-mail shown in FIG. 27 is returned to the received e-mail on the screen shown in FIG. In other words, the number of the item to be operated is input in the column of the text and a reply is made.

When the e-mail is returned from the terminal device in this way, operation execution section 2502 receives the returned e-mail and analyzes the text field of the returned e-mail to determine the operation to be performed. The determination is made and the operation is executed in the same manner as in the first embodiment (in the case of the example of the e-mail in FIG. 27, the coordinates on the “Fax” icon 1100 in FIG. 11 are double-clicked). As a result, the display screen of the computer 100 becomes a display screen 50 of FIG.

For example, when the display screen of the computer 100 is the display screen 50 shown in FIG. 17, in the remote operation system of the second embodiment, the screen analyzing unit 2500
Analyzes the contents of the display screen 50 shown in FIG. 17 and outputs the result as text of an e-mail. The screen information transmitting unit 2501 transmits the text of the e-mail to the e-mail address of the terminal device 101. The terminal device 101 receives the e-mail and displays, for example, FIG.
Is displayed as follows. When the user looks at the e-mail screen and wants to select, for example, the file “file2.dat”, the same e-mail as in FIG. 27 is returned to the received e-mail on the screen in FIG.

When the operation execution unit 2502 receives the returned e-mail, “file2.da” displayed in the list box 1700 on the GUI screen 50 of FIG.
The operation of clicking the file 1702 of “t” is executed. As a result, the display screen of the computer 100 is shown in FIG.
The display screen 50 of FIG.

For example, when the display screen of the computer 100 is the display screen 50 shown in FIG. 21, in the remote operation system according to the second embodiment, the screen analysis unit 2500 is used.
Analyzes the contents of the display screen 50 shown in FIG. 21 and outputs the result as text of an e-mail. The screen information transmitting unit 2501 transmits the text of the e-mail to the e-mail address of the terminal device 101. The terminal device 101 receives the e-mail and displays, for example, FIG.
Is displayed as follows. When the user looks at the e-mail screen and inputs a FAX number and wants to select the "OK" button, the e-mail shown in FIG. 30 is returned to the received e-mail on the screen shown in FIG.

When the operation execution unit 2502 receives the returned e-mail, the facsimile number entered in 1 of the text of the received e-mail in FIG. 30 is entered in the character string input box 2100 on the GUI screen 50 in FIG. , And click an “OK” button 1701 on the window 40. As a result, the display screen of the computer 100 is shown in FIG.
The display screen is as shown in FIG. With the remote control system according to the second embodiment described above, a mobile phone or a PH
Even with a small portable communication terminal such as S, remote control can be performed while referring to the contents of the display screen of an external computer. Also, it is not necessary for the terminal that performs the remote operation to have a special function for the remote operation, and the external computer can be remotely operated freely using a portable communication terminal or the like at hand of the user. be able to.

Next, a remote control system according to a third embodiment of the present invention will be described.

FIG. 31 is a block diagram showing the configuration of the remote control system according to the present embodiment.

In the remote control systems according to the first and second embodiments described above, a terminal such as a mobile phone or a PHS is described as a terminal on which an operation is performed. The terminal is not limited to this as long as it has a transmission / reception function. In the present embodiment shown in FIG. 31, a computer 3100 which is a personal computer, a notebook personal computer, or another information processing device is used as a terminal on which an operation is performed.

This computer 3100 is equipped with a web browser, can display HTML format data on a screen, and has the same G function as the computer 100 to be operated.
A UI screen can be displayed.

However, in the remote operation system according to the present embodiment, the remote operation can be performed using only a web browser or the like. It is not necessary to provide any special software or semiconductor circuit for this purpose.

[0187] In recent years, most of commercially available personal computers and notebook personal computers and the like have been installed with a web browser in advance, and thus these terminals can be used as they are in this embodiment without adding special software. It can be used as the computer 3100.

The operation of the remote operation system according to the present embodiment is the same as the operation according to the first embodiment, and the screen information output by the screen analysis unit 103 in HTML format data is
The operation execution unit 10 is displayed on the screen of the computer 3100 and operates in accordance with the operation performed on the computer 3100.
6 executes the operation of the computer 100.

According to the remote control system of the present embodiment described above, the same effects as those of the first embodiment can be obtained even when a computer is used as the terminal on which the operation is performed.

Unlike a conventional remote operation system using a GUI using a computer, a remote operation system transmits only necessary minimum data indicating information on a screen of a computer to be operated to a remote operation side. Therefore, the communication cost can be significantly reduced.

In a mode in which the terminal on which an operation is performed is provided with a function of displaying image data as in the present embodiment, the entire GUI screen of the computer 100 to be operated or a specified part (application application) is displayed. A function of transmitting image data (in a window or the like) to the user's computer 3100 based on an instruction from the user or a predetermined setting may be provided. As a result, the user can display the detailed display contents of the GUI screen which is unknown only with the character data,
It can be displayed and confirmed in a web browser or the like of the computer 3100 of the user.

Next, a remote control system according to a fourth embodiment of the present invention will be described.

FIG. 32 is a block diagram showing a configuration of a remote control system according to the fourth embodiment. The basic operation of the remote operation system according to the fourth embodiment is exactly the same as that of the remote operation system according to the first embodiment, but the screen analysis unit 3200 and the screen information transmission unit 3
There is a difference in the operation of 201. In the remote control system according to the fourth embodiment, the screen analysis unit 3200 determines that a still image or a moving image is displayed in the active window displayed on the current GUI screen of the computer 100. To detect. Further, when the display of the still image or the moving image is detected, the screen analyzing unit 3200 extracts the display data of the still image or the moving image and outputs the extracted data to the screen information transmitting unit 3201 together with the data of the web page in the HTML format. Screen information transmission unit 3201
Transmits the data received from the screen analysis unit 3200 to the terminal device 101.

For example, assume that the current screen of the computer 100 is as shown in a screen 50 of FIG. However, it is assumed that the image display window 40 is the currently active window, and a still image such as a photograph or a moving image such as a TV is displayed in the image display box 3300 on the image display window 40. In this case, the screen analysis unit 3200 detects that the video display window 40 is the currently active window, analyzes the information of the GUI parts on the window, outputs the data of the HTML-format web page, Window 4
It detects that there is a video display box 3300 on which a still image or a moving image is displayed, and extracts and outputs the video data displayed on the video display box 3300.

The data output by the screen analysis unit 3200 is transmitted to the terminal device 101 by the screen information transmission unit 3201.
Are transmitted to the terminal device 10.
1 is displayed as shown in FIG. The video data displayed in the video display box 3300 of FIG.
Is displayed in the video display box 3400 of the. When the video displayed in the video display box 3300 is a moving image, the video in the video display box 3300 changes with time, and the video in the video display box 3400 changes following the change. That is, when the screen analysis unit 3200 detects the display of a moving image, the screen analysis unit 3200 continuously extracts the data of the image in the image display box 3300 where the moving image is displayed, and continues to the screen information transmission unit 3201. Output. The screen information transmitting unit 3201 continuously transmits the video data of the moving image to the video display box 3400. In the present embodiment, the terminal device 101
Is a terminal device capable of displaying the video display box 3400 and a terminal capable of displaying a still image or a moving image in the video display box 3400.

Next, a remote control system according to a fifth embodiment of the present invention will be described.

FIG. 35 is a block diagram showing a configuration of a remote control system according to the fifth embodiment. Referring to FIG. 35, the remote control system according to the fifth embodiment includes:
In addition to the configuration in FIG. 1 of the remote operation system according to the first embodiment, a screen information conversion unit 3502, a conversion rule registration unit 3
505, and a terminal characteristic detecting unit 3506.

The terminal characteristic detecting section 3506 is connected to the terminal device 10
1 is detected. However, the terminal characteristics are the unique ID of the terminal device, model, processing speed, communication speed, communication cost per unit data amount, communication cost per unit time, data format that can be displayed on the screen, sound data format that can be reproduced, Executable program and script formats, amount of data that can be received at one time, displayable colors, displayable character fonts, screen resolution, physical length and width of the screen,
This refers to the number of vertical and horizontal characters that can be displayed on one screen.

Screen information conversion section 3502 transmits data of a screen analysis result by screen analysis section 3501 to terminal apparatus 101 or terminal apparatus 101 in consideration of the terminal characteristics of terminal apparatus 101 detected by terminal characteristic detection section 3506. The data is converted into data suitable for display at 101. The rules for this conversion are:
It is registered in the conversion rule registration unit 3505.

FIG. 36 is a flowchart for explaining the remote control processing of the remote control system according to the first embodiment.

Referring to FIG. 36, the remote operation system according to the fifth embodiment authenticates a user (step 202).
The steps up to this are the same as in the remote system of the first embodiment.

When the authentication of the user is completed, the terminal characteristic detecting section 3506 detects the characteristic of the terminal device 101 (step 3601). Next, the screen analysis unit 3501 analyzes the current screen of the computer 3500 (step 20).
3, as in the first embodiment). Next, the screen information conversion unit 3502 converts the data resulting from the screen analysis by the screen analysis unit 3501 (step 3602). The converted data is transmitted to the terminal device 101 by the screen information transmitting unit 3503 and displayed on the screen of the user terminal device 101 (step 204).

When the user performs an operation on the computer 3500 using the terminal device 101, the operation execution unit 3504 receives the information on the operation and executes the information on the computer 3500 (steps 205 and 2).
06).

When the display on the computer 3500 changes as a result, the screen change detection unit 105
Detects the change and knows that the change is complete,
The processing of step 203 and subsequent steps is executed again to update the data displayed on the screen of the user terminal device 101 to the latest data.

Next, an example of remote control by the remote control system according to the fifth embodiment will be described.

[0206] The screen analyzing unit 3501
00 is analyzed, and the data of the analysis result is output in a general-purpose format that does not depend on the device characteristics of the terminal device 101, for example, an XML (extensible markup language) format. For example, it is assumed that an example of the display screen 50 of the computer 3500 is as shown in FIG. 11, as in the case of the example of the remote operation in the first embodiment. FIG.
7 is an example of XML format screen information data that the screen analysis unit 3501 analyzes the display screen 50 of FIG. 11 and outputs to the screen information conversion unit 3502. In FIG.
x1 to x3 and y1 to y3 are the values of the x coordinate and the y coordinate of the three icons 1100 in FIG. FIG.
In 7, the event tag represents an operation event, and LWCLICK is a double click with the left mouse button.

In this example, the device characteristic detecting unit 3506 detects the unique ID of the terminal device 101,
In 505, for each unique ID of the terminal device 101, a rule for converting data in a general format output by the screen analysis unit 3501 into data in a format suitable for the terminal device having the unique ID is registered. . For example, the unique I of the terminal device 101
If it is detected that D is Z, and if the unique ID is Z, the screen information data is converted from the XML format to HT.
The rule for converting to the ML format is a conversion rule registration unit 35
05 (that is, in this case, the unique I
The terminal device 101 whose D is Z is a terminal device that can display HTML format data.) In this case, the screen information conversion unit 3502 converts the data of the screen information in the XML format output by the screen analysis unit 3501 into HTML according to the conversion rule for the terminal device having the unique ID Z registered in the conversion rule registration unit 3505. Convert to format data. The data in the HTML format is transmitted to the terminal device 101 by the screen information transmitting unit 3503, and is displayed on the screen of the terminal device 101 as in the example of FIG.

The operation execution unit 3504 receives information on an operation performed on the terminal device 101 and executes an operation corresponding to the operation on the computer 3500. Terminal device 10
Assuming that the operation performed in step 1 is V, the screen information data output by the screen analysis unit 3501 and the screen information conversion unit 3502
From the information of the correspondence between the data when the data format of the screen information is converted (information on which part of the data before conversion and the data after conversion correspond), The operation on the 3500 can be specified.

As another example, the device characteristic detecting section 350
6 detects the model of the terminal device 101, and stores the conversion rule registration unit 3
In 505, for each model of the terminal device 101, a rule for converting data in a general-purpose format output by the screen analysis unit 3501 into data in a format suitable for the model is registered. For example, if the model of the terminal device 101 is detected to be Z, and if the model of the terminal device is Z, the rule for converting the screen information data from the XML format to the mail format is a conversion rule registration unit. 3505 (that is, in this case, the terminal device 101 whose model is Z is a terminal device capable of transmitting and receiving mail). In this case, the screen information conversion unit 3502 operates in accordance with the conversion rule for the terminal device of model Z registered in the conversion rule registration unit 3505.
The data of the screen information in XML format output by the screen analysis unit 3501 is converted into data in mail format. The data in the mail format is transmitted as mail to the terminal device 101 by the screen information transmitting unit 3503, and is displayed on the screen of the terminal device 101 in the same manner as in the example of FIG.

As another example, device characteristic detecting section 350
6 detects the processing speed of the terminal device 101, and the conversion rule registration unit 3505 registers a rule for converting data in a general format output by the screen analysis unit 3501 according to the processing speed of the terminal device 101. ing. For example, the terminal device 10
1 is detected to be Z1, and the conversion rule registration unit 3505 stores (1) the screen information data in XML.
Rules for converting from HTML format to HTML format, and
(2) If the processing speed is lower than Z2, it is assumed that a rule for deleting a part of the screen information is registered. In this case, screen information conversion section 3502 includes conversion rule registration section 350
According to the conversion rules (1) and (2) registered in No. 5, if Z1 <Z2, a part of the XML format screen information data output by the screen analysis unit 3501 is deleted, and then HTML Convert to Z1, Z1 = Z2 or Z
If 1> Z2, the XM output by the screen analysis unit 3501
All data of the screen information in the L format is converted to the HTML format. As can be seen from the above, in this example, the terminal device 101
When the processing speed is low, part of the screen information is deleted, thereby reducing the load on the terminal device 101 for displaying the screen information.

As another example, the device characteristic detecting section 350
6 detects the communication speed of the terminal device 101, and a conversion rule registration unit 3505 registers a rule for converting data in a general format output by the screen analysis unit 3501 according to the communication speed of the terminal device 101. ing. For example, the terminal device 10
It is assumed that the communication speed of Z1 is detected as Z1. The conversion rule registration unit 3505 stores (1) the data of the screen information in XML.
Rules for converting from HTML format to HTML format, and
(2) When the communication speed is lower than Z2, it is assumed that a rule for deleting a part of the screen information is registered. In this case, as in the case of the processing speed detection described above, the screen information conversion unit 3502 determines Z1 <according to the conversion rules (1) and (2) registered in the conversion rule registration unit 3505.
If it is Z2, after deleting a part of the data of the screen information in the XML format output by the screen analysis unit 3501,
The data is converted into the ML format, and if Z1 = Z2 or Z1> Z2, all the screen information data in the XML format output by the screen analysis unit 3501 is converted into the HTML format. As can be seen from the above, in this example, when the communication speed of the terminal device 101 is low, a part of the screen information is deleted to reduce the data transmission amount from the computer 3500 to the terminal device 101.

As another example, device characteristic detecting section 350
6 detects the communication cost per unit data amount of the terminal device 101, and stores the screen analysis unit 35 in the conversion rule registration unit 3505.
01 is output to the terminal device 10
Rules for conversion in accordance with the communication cost per unit data amount are registered. The communication cost per unit data amount of the terminal device 101 may be directly detectable or may be indirectly detectable from the unique ID or model of the terminal device 101. For example, the terminal device 10
Assuming that the communication cost per unit data amount of 1 is detected to be Z1, the conversion rule registration unit 3505 stores (1) a rule for converting the screen information data from the XML format to the HTML format, and ( 2) When the communication cost per unit data amount is Z2 or more, it is assumed that a rule for deleting a part of the screen information is registered. In this case, the screen information conversion unit 3502 uses the Z1 according to the conversion rules (1) and (2) registered in the conversion rule registration unit 3505.
= Z2 or Z1> Z2, the screen analysis unit 350
1 deletes part of the XML format screen information data, converts the data to HTML format, and if Z1 <Z2, deletes all the XML format screen information data output by the screen analysis unit 3501. Convert to HTML format. That is, in this example, when the communication cost per unit data amount of the terminal device 101 is large, a part of the screen information is deleted to reduce the data transmission cost from the computer 3500 to the terminal device 101.

As another example, the device characteristic detecting section 350
6 detects the communication cost per unit time of the terminal device 101, and stores the screen analysis unit 3501 in the conversion rule registration unit 3505.
Is registered in accordance with the communication cost per unit time of the terminal device 101. The communication cost per unit time of the terminal device 101 is the same as the communication cost per unit data amount even if the terminal device 101 can be directly detected.
May be indirectly detectable from the unique ID or the model of the device. For example, if it is detected that the communication cost per unit time of the terminal device 101 is Z1, the conversion rule registration unit 3505 stores (1) the data of the screen information as X
It is assumed that a rule for converting the ML format to the HTML format and (2) a rule for deleting a part of the screen information when the communication cost per unit time is equal to or more than Z2. In this case, the screen information conversion unit 3502
According to the conversion rules (1) and (2) registered in the conversion rule registration unit 3505, Z1 = Z2 or Z1>
If it is Z2, after deleting a part of the data of the screen information in the XML format output by the screen analysis unit 3501,
Convert to ML format, and if Z1 <Z2, screen analysis unit 3
All the data of the screen information in the XML format output by 501 is converted into the HTML format. That is, in this example, when the communication cost per unit time of the terminal device 101 is large, a part of the screen information is deleted, and thus the computer 3500 is deleted.
The cost of transmitting data to the terminal device 101 is reduced (because a reduction in the amount of data to be communicated leads to a reduction in communication time).

As another example, device characteristic detecting section 350
6 detects a data format that can be displayed on the screen of the terminal device 101, and stores the screen analysis unit 350 in the conversion rule registration unit 3505.
1 is output to the terminal device 101.
A rule for converting data into a format that can be displayed on the screen is registered. The data format that can be displayed on the screen of the terminal device 101 is, as in the above example, a case where the data can be directly detected, or a case where the data format can be indirectly detected from the unique ID or the model of the terminal device 101. Is also good. For example, if it is detected that the data format that can be displayed on the screen of the terminal device 101 is the HTML format, the conversion rule registration unit 3505 stores rules for converting the screen information data from the XML format to the HTML format, and It is assumed that a plurality of rules for converting into a format other than HTML (such as a mail format) are registered. In this case, the screen information conversion unit 3502
Converts the data of the screen information in the XML format output by the screen analysis unit 3501 into the data in the HTML format according to the conversion rule registered in the conversion rule registration unit 3505.

As another example, the device characteristic detecting section 350
6 detects the formats of programs and scripts that can be executed by the terminal device 101, and stores, in the conversion rule registration unit 3505, data in a general format output by the screen analysis unit 3501 in the terminal device 101. Rules for converting to a format program or script are registered.
The format of the program and the script that can be executed by the terminal device 101 is the same as in the above example, even if the format can be directly detected, or if the format can be indirectly detected from the unique ID or model of the terminal device 101. There may be. For example, it is assumed that it is detected that the terminal device 101 can execute a program in the Java format and a script in the Java script format.
It is assumed that a rule for converting data in the XML format of the screen information into a script in the Java script format is registered. In this case, the screen information conversion unit 3502 converts the data of the screen information in the XML format output by the screen analysis unit 3501 into a script in the Java script format according to the conversion rule registered in the conversion rule registration unit 3505. XML of screen information in conversion rule registration unit 3505
The same applies to a case where a rule for converting data in the Java format into a program in the Java format is registered.

As another example, the device characteristic detecting section 350
6 detects the amount of data that can be received by the terminal device 101 at one time, and the conversion rule registration unit 3505 indicates that the amount of data to be transmitted to the terminal device 101 is equal to or less than the amount of data that can be received by the terminal device 101 at one time. So that the rules to convert are registered. As in the above example, the amount of data that can be received at one time by the terminal device 101 is a case where the data can be directly detected but can be indirectly detected from the unique ID or model of the terminal device 101. You may. For example, if the data format that can be displayed on the screen of the terminal device 101 is detected to be the HTML format, and the terminal device 101 detects that the amount of data that can be received at one time is Z, the conversion rule registration unit In 3505, (1) data of screen information is converted to XM
It is assumed that a rule for converting from the L format to the HTML format and a rule for deleting (2) a part of the data in the HTML format so that the data amount is equal to or less than Z are registered. In this case, screen information conversion section 3502 performs conversion rule registration section 3
According to the conversion rules (1) and (2) registered in the file 505, the data of the screen information in the XML format output by the screen analysis unit 3501 is converted into the data in the HTML format, and the data amount becomes Z or less. Thus, a part of the data in the HTML format is deleted. That is, in this example, the terminal device 1
01, only a certain amount of data can be received at a time. Therefore, by reducing the amount of data of the screen information to be transmitted to the limited amount or less, even if the terminal device 101 has a limited amount of receivable data, the screen information of the computer 3500 can be obtained. Can be displayed.

As another example, the device characteristic detecting section 350
6 detects a color that can be displayed on the screen of the terminal device 101, and stores, in the conversion rule registration unit 3505, the data in the general format output by the screen analysis unit 3501 in a color that can be displayed on the screen of the terminal device 101. A rule for conversion according to the information is registered.
The colors that can be displayed on the screen of the terminal device 101 are the same as those in the above example, even if the color can be directly detected.
May be indirectly detectable from the unique ID or the model of the device. For example, it is detected that the data format that can be displayed on the screen of the terminal device 101 is the HTML format,
The colors that can be displayed on the screen of the terminal device 101 are black, red,
It is assumed that a total of 256 colors such as blue are detected, and the conversion rule registration unit 3505 stores (1) a rule for converting screen information data from the XML format to the HTML format, and (2) a GUI component of the same type. (Buttons, menus, etc.)
It is assumed that an HTML color designation tag is used so that the information is expressed in the same color, and a rule for selecting the color from colors that can be displayed on the screen of the mobile terminal 101 is registered. In this case, screen information conversion section 3502 includes conversion rule registration section 350
In accordance with the conversion rules (1) and (2) registered in (5), the data of the screen information in the XML format output by the screen analysis unit 3501 is converted into the data in the HTML format, and the same is applied to the conversion into the HTML. An HTML color designation tag is used so that the information of the type of GUI component is expressed in the same color, and the color to be used is selected from colors that can be displayed on the screen of the terminal device 101. Note that the case where the device characteristic detection unit 3506 detects a character font that can be displayed on the screen of the terminal device 101 is exactly the same as the case of the color described above.
The screen information data is converted so as to be displayed on the screen of the mobile terminal 101 using the character font.

As another example, device characteristic detecting section 350
6 detects the screen resolution of the terminal device 101, and a conversion rule registration unit 3505 registers a rule for converting general-purpose data output by the screen analysis unit 3501 in accordance with the screen resolution of the terminal device 101. ing. For example, it is assumed that the data format that can be displayed on the screen of the terminal device 101 is detected to be the HTML format, and that the screen resolution of the terminal device 101 is detected to be horizontal W1 pixels and vertical H1 pixels. 3505 includes (1) rules for converting screen information data from the XML format to the HTML format, and (2) a part of the screen information when the screen resolution is less than W2 pixels horizontally and less than H2 pixels vertically. It is assumed that a rule for deletion is registered. In this case, the screen information conversion unit 3502 uses the W in accordance with the conversion rules (1) and (2) registered in the conversion rule registration unit 3505.
If 1 <W2 and H1 <H2, the screen analysis unit 3501
Deletes part of the data of the screen information in the XML format output by HTML, and converts the data to the HTML format. Otherwise, the data of the screen information in the XML format output by the screen analysis unit 3501 are all converted to the HTML format. Convert to In other words, in this example, when the screen resolution of the terminal device 101 is smaller than a certain size, a part of the screen information is deleted so that the amount of information displayed in the case of the terminal device having a small screen size does not become too large. I have to. Note that the device resolution detection unit 3506 also detects the vertical and horizontal physical lengths of the screen of the terminal device 101 or the number of characters in the vertical and horizontal directions that can be displayed in one screen. If the physical length of the screen vertically and horizontally is smaller than a certain value, or if the number of characters in the vertical and horizontal directions that can be displayed on one screen is smaller than the matching value, the screen information Is deleted and the screen information data is converted.

Next, a remote control system according to a sixth embodiment of the present invention will be described.

FIG. 38 is a block diagram showing a configuration of a remote control system according to the sixth embodiment. The basic operation of the remote control system according to the sixth embodiment is exactly the same as that of the remote control system according to the fifth embodiment, except that a screen analyzing unit 3801 and a screen information converting unit 380 are provided.
2. Screen information transmission unit 3803, terminal characteristic detection unit 380
4 and the operation of the conversion rule registration unit 3805.

In the remote control system according to the sixth embodiment, screen analyzing section 3801 determines whether a still image or a moving image is displayed in the active window displayed on the current GUI screen of computer 100. Detect that. Further, when the display of a still image or a moving image is detected, the screen analysis unit 3801 extracts the display data of the still image or the moving image, and extracts the display data of the still image or the moving image.
802 is output together with the screen information data in the XML format.

[0222] The terminal characteristic detecting unit 3804 can detect the data format of a still image and a moving image that can be displayed on the terminal device 101. The conversion rule registration unit 3805 stores the still image or the moving image extracted by the screen analyzing unit. Are registered in order to convert the display data of the above into still image and moving image data in a format that can be displayed on the terminal device 101.

The screen information conversion section 3802 converts the screen information data in the XML format into data in a format that can be displayed on the terminal device 101 in the same manner as in the fifth embodiment. Converts the extracted still image or moving image display data into still image and moving image data in a format that can be displayed on the terminal device 101, according to the rules registered in the conversion rule registration unit 3805. These data are transmitted to the terminal device 101 by the screen information transmission unit 3803 and displayed on the screen of the terminal device 101.

For example, assume that the current screen of the computer 100 is as shown in a screen 50 of FIG. However, it is assumed that the image display window 40 is the currently active window, and a still image such as a photograph or a moving image such as a TV is displayed in the image display box 3300 on the image display window 40. In this case, the screen analysis unit 3801 detects that the video display window 40 is the current active window, analyzes the information of the GUI parts on the window, outputs the screen information data in the XML format, Detecting that there is a video display box 3300 displaying a still image or a moving image on the window 40,
The video data displayed at 00 is extracted and output.

The screen information data and video data in the XML format output by the screen analysis unit 3801 are converted into data that can be displayed on the terminal device 101 by the screen information conversion unit 3802, and the terminal information is output by the screen information transmission unit 3803. It is transmitted to the device 101. As a result, these data are displayed on the screen of the terminal device 101 as shown in FIG. However, the video data displayed in the video display box 3400 in FIG.
300 video data are converted. When the video displayed in the video display box 3300 is a moving image, the video in the video display box 3300 changes with time, and the video in the video display box 3400 changes following the change. That is, when the screen analysis unit 3801 detects the display of a moving image, the screen analysis unit 3801 continuously extracts the video data in the video display box 3300 where the moving image is displayed, and continues to the screen information conversion unit 3802. Output. The screen information conversion unit 3802 continuously converts the video data of this moving image,
03 continuously transmits the converted video data to the video display box 3400. In the present embodiment, terminal device 101 is a terminal device capable of displaying video display box 3400, and
It is a terminal capable of displaying a still image or a moving image within 0. The data format of the still image or moving image that can be displayed in the video display box 3400 may not be the same as the data format of the still image or moving image displayed in the video display box 3300.

The remote control system of each of the above embodiments is
In addition to realizing the functions of the screen analysis unit, the screen information transmission unit, the operation execution unit, the screen change detection unit, and the like in the computer 100 to be operated, and other functions in hardware, a computer program having each function is implemented. ,
It can be realized by being loaded into the memory of the computer processing device. This computer program
It is stored on a magnetic disk, semiconductor memory, or other recording medium 108. Then, the functions described above are realized by being loaded from the recording medium into the computer processing device and controlling the operation of the computer processing device.

FIG. 39 is a block diagram showing an embodiment of the device configuration of the computer 100 to be operated according to the present invention.

The computer 100 in the example of FIG. 39 is connected to the Internet communication network 102 via the Internet connection device 3912 (modem, terminal adapter, dial-up router, etc.) as the communication unit 107. Thereby, the computer 100 can be remotely operated from a PHS terminal or the like via the Internet. In addition, a display 3910 which is a display device,
A keyboard 3909 and a mouse 3911 are provided as input devices.

In the example of FIG. 39, a computer program having the above-described functions of the present invention is stored in a ROM 3902.
(Read Only Memory), Hard Disk 3906, CD
3905 (Compact Disc), FD3908 (Floppy Dis)
c) and the like, and this is stored in the RAM 3903.
(Random Access Memory), and the CPU 390
1 (Central Processing Unit). In addition, when recorded on CD3905, C
When the data is recorded on the FD 3908 via the D drive 3904, the data can be read via the FD drive 3907.

Although the present invention has been described with reference to the preferred embodiments and examples, the present invention is not necessarily limited to the above-described embodiments and examples, and various modifications may be made within the scope of the technical concept. Modifications can be made.

[0231]

According to the remote control system of the present invention as described above, the following effects can be achieved.

First, an arbitrary GUI application which can remotely control a computer and can be operated on a computer to be operated even by a portable terminal having a small display size and information that can be displayed is only characters. The software can be remotely controlled.

Secondly, in order to appropriately convert the display screen of the computer to be operated into hypertext or a web page displayed by character data, transmit this to the user terminal, and accept the operation, The user's terminal does not need to be provided with a special function for remote operation, and can freely select a communicable terminal having a web browser or the like to remotely control the computer in the GUI environment.

Third, the information on the screen of the computer to be operated is obtained by acquiring data such as the title name of the active window and the name of each item of the menu, and converting the data into text data such as an HTML sentence. Therefore, it is not necessary to notify detailed data such as the display position and shape of each window or object unlike the conventional system, and the communication cost can be significantly reduced.

Fourth, since the data of the screen information of the computer to be operated is converted in accordance with the characteristics of the terminal device used for remote operation, terminal devices having various characteristics can be used for remote operation of the computer. Become. That is, the terminal device that can be used for remote control is not limited to a device having a specific characteristic.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a remote operation system according to a first embodiment of the present invention.

FIG. 2 is a flowchart illustrating a remote control process according to the first embodiment of this invention.

FIG. 3 is a flowchart illustrating a screen analysis process performed by a screen analysis unit according to the first embodiment of this invention.

FIG. 4 is a diagram illustrating an example of a window displayed on a GUI screen of a computer.

FIG. 5 is a diagram illustrating an example of a GUI screen of a computer.

FIG. 6 is a diagram showing an example of a menu in a window.

FIG. 7 is a diagram illustrating an example of property information of a GUI component according to the first embodiment of this invention.

FIG. 8 is a diagram illustrating an example of coordinate values of a GUI component.

FIG. 9 is a diagram illustrating an example of data created by a screen analysis unit according to the first embodiment of this invention.

FIG. 10 is a diagram illustrating an example in which the display screen of FIG. 6 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

FIG. 11 is a diagram illustrating an example of a display screen at the time of connection to a computer to be operated according to the first embodiment of this invention.

FIG. 12 is a diagram illustrating an example in which the display screen of FIG. 11 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

13 is a diagram showing a state in which a Fax window is displayed by double-clicking a Fax icon on the screen of FIG. 11;

FIG. 14 is a diagram illustrating an example in which the display screen of FIG. 13 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

15 is a diagram showing a state in which a menu of a file is clicked on the screen of FIG. 13 to display a pull-down menu.

FIG. 16 is a diagram illustrating an example in which the display screen of FIG. 15 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

17 is a diagram showing a state in which a transmission item is clicked on the screen of FIG. 15 to display a fax transmission window.

FIG. 18 is a diagram illustrating an example in which the display screen of FIG. 17 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

19 is a diagram showing a state where a file in a list box is clicked and selected on the screen of FIG. 17;

FIG. 20 is a diagram illustrating an example in which the display screen of FIG. 19 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

FIG. 21 is a diagram illustrating a state in which a window for inputting a fax number, which is a fax transmission destination number, is displayed.

FIG. 22 is a diagram illustrating an example in which the display screen of FIG. 21 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

FIG. 23 is a diagram showing a state in which a window indicating completion of fax transmission is displayed.

FIG. 24 is a diagram illustrating an example in which the display screen of FIG. 23 analyzed by the screen analysis unit according to the first embodiment of the present invention is displayed on a terminal device.

FIG. 25 is a block diagram illustrating a configuration of a remote operation system according to a second embodiment of the present invention.

FIG. 26 is a diagram illustrating an example in which the display screen of FIG. 11 analyzed by the screen analysis unit according to the second embodiment of the present invention is displayed on a terminal device.

FIG. 27 is a diagram illustrating an example of an electronic mail transmitted from a terminal device to a computer according to the second embodiment of this invention.

FIG. 28 is a diagram illustrating an example in which the display screen of FIG. 17 analyzed by the screen analysis unit according to the second embodiment of the present invention is displayed on a terminal device.

FIG. 29 is a diagram illustrating an example in which the display screen of FIG. 21 analyzed by the screen analysis unit according to the second embodiment of the present invention is displayed on a terminal device.

FIG. 30 is a diagram illustrating an example of an e-mail transmitted from the terminal device to the computer according to the second embodiment of this invention.

FIG. 31 is a block diagram illustrating a configuration of a remote operation system according to a third embodiment of the present invention.

FIG. 32 is a block diagram illustrating a configuration of a remote operation system according to a fourth embodiment of the present invention.

FIG. 33 is a diagram illustrating an example of a GUI screen of a computer to be operated according to the fourth and sixth embodiments of the present invention.

FIG. 34 is an example in which the display screen of FIG. 33 analyzed by the screen analysis unit according to the fourth embodiment of the present invention is displayed on a terminal device;
FIG. 34 is a diagram illustrating an example in which the display screen of FIG. 33 converted by the screen information conversion unit according to the sixth embodiment of the present invention is displayed on a terminal device.

FIG. 35 is a block diagram illustrating a configuration of a remote control system according to a fifth embodiment of the present invention.

FIG. 36 is a flowchart for describing remote control processing according to the fifth embodiment of this invention.

FIG. 37 is a diagram illustrating an example of data created by a screen analysis unit according to the fifth embodiment of the present invention.

FIG. 38 is a block diagram illustrating a configuration of a remote operation system according to a sixth embodiment of the present invention.

FIG. 39 is a block diagram illustrating an embodiment of a device configuration of a computer to be operated according to the present invention.

[Explanation of symbols]

100, 3900 Computer to be operated 101 Terminal device 102 Communication network 103, 2500, 3200, 3501, 3801 Screen analysis unit 104, 2501, 3201, 3503, 3803 Screen information transmission unit 105 Screen change detection unit 106, 2502, 3504 Operation execution Unit 107 communication unit 108 recording medium 201 to 207, 301 to 304, 3601, 3602
Processing step 40 Window 50 Computer display screen 60 Menu item 80 GUI component 1000 Hyper-link 1100 Icon 1500 Sub-menu item 1700 List box 1701 Button 1702 List box item 2100, 2200 Input box 3100 Computer on operation side 3300, 3400 Video display box 3502, 3802 Screen information conversion unit 3505, 3805 Conversion rule registration unit 3506, 3804 Terminal characteristic detection unit 3901 CPU 3902 ROM 3903 RAM 3904 CD drive 3905 CD 3906 Hard disk 3907 FD drive 3908 FD 3909 Keyboard 3910 Display 3911 Mouse 3912 Mouse Connection device

Claims (44)

[Claims] 1. A remote operation system for remotely operating an operation target device from an operation side terminal via a communication network, wherein: a screen analysis means for analyzing information displayed on a screen of the operation target device; A remote control system comprising: a screen information transmitting unit that transmits data of a result of the analysis to the operating terminal in order to display the data on the operating terminal. 2. A remote operation system for remotely operating an operation target device from an operation side terminal via a communication network, wherein the operation target device analyzes a content displayed on a screen of the operation target device. And a screen information transmitting means for transmitting data of an analysis result of the screen analyzing means to the operation side terminal for displaying the data on the operation side terminal. 3. The apparatus according to claim 1, further comprising an operation executing unit that causes the operation target device to execute a predetermined process in response to an operation performed by a user using the operation-side terminal. Item 3. A remote control system according to Item 2. 4. A screen information conversion unit for converting data of an analysis result of the screen analysis unit into data suitable for transmission to the operation side terminal or display on a screen of the operation side terminal, 4. The method according to claim 1, wherein the transmitting unit transmits the data of the conversion result of the screen information converting unit to the operation side terminal for displaying the data on the operation side terminal. 5. The remote control system as described. 5. A conversion rule registration means for registering and recording rules for converting the data of the analysis result of the screen analysis means into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. 5. The remote control according to claim 4, wherein the screen information conversion unit converts data of an analysis result of the screen analysis unit based on a conversion rule recorded in the conversion rule registration unit. 6. system. 6. A device characteristic detecting unit for detecting a device characteristic of the operation-side terminal, wherein the screen information conversion unit records the device characteristic detected by the device characteristic detecting unit and the conversion rule registration unit. 5. The remote operation system according to claim 4, wherein the data of the analysis result of the screen analysis unit is converted based on the conversion rule. 7. A screen change detecting means for detecting a change in the display of the screen of the operation target device, and when the change of the display of the generated screen ends, the screen analysis means sets the current state of the operation target device. The remote control system according to claim 1, wherein the remote control system analyzes a screen. 8. The information processing device having a GUI as the operation target device, wherein the screen analysis unit detects a GUI component displayed on a screen of the operation target device and acquires attribute data of the GUI component. The remote control system according to any one of claims 1 to 7, wherein: 9. The attribute data of the GUI component includes a type of the GUI component, a name of the GUI component, and a name of the GUI component.
The remote control system according to claim 8, wherein the remote control system includes at least one of a position of the component in the screen. 10. The conversion rule registration means includes registration of a rule for converting data of an analysis result of the screen analysis means into a web page displayable by a web browser, wherein the screen information conversion means includes The data of the analysis result of the screen analysis means is converted into a web page based on the web page, and the web page is used for GUI parts detected by the screen analysis means and which can be operated by a user. A hyperlink corresponding to the component, wherein when the hyperlink is selected, data identifying the GUI component corresponding to the hyperlink and the G
The data for identifying an operation on a UI component is transmitted to the operation executing means.
The remote control system according to any one of the above. 11. The method according to claim 1, wherein the conversion rule registration unit includes a registration of a rule for converting data of an analysis result of the screen analysis unit into text of an electronic mail.
The remote control system according to any one of claims 1 to 9. 12. The operation execution means receives an e-mail including information on an operation performed by the user using the operation-side terminal, analyzes the received e-mail, and identifies the operation performed by the user. The remote operation system according to claim 11, wherein the operation is performed. 13. The remote operation system according to claim 1, wherein the data of the analysis result of the screen analysis unit is XML format data. 14. The remote operation system according to claim 1, wherein the operation side terminal is a portable communication terminal having a function of transmitting and receiving character data. 15. The remote control system according to claim 1, wherein the communication network is the Internet. 16. The screen analysis means, when a moving image or a still image is displayed on the screen of the operation target device, detects the display and extracts video data of the display, and the screen information transmitting means includes: The remote controller according to any one of claims 1 to 15, wherein the video data extracted by the screen analysis unit is transmitted to the operation-side terminal for display on the operation-side terminal. Operation system. 17. When the moving image or the still image is displayed on the screen of the operation target device, the screen analysis unit detects the display and extracts video data of the display, and the screen information conversion unit includes: The video data extracted by the screen analysis unit is converted into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. The remote control system according to any one of the above. 18. A device characteristic of the operating terminal detected by the device characteristic detecting means includes a unique I of the operating terminal.
D, model, processing speed, communication speed, communication cost per unit data amount, communication cost per unit time, data format that can be displayed on the screen, sound data format that can be played back, executable program and script format, once At least one of the following: the amount of data that can be received, the colors that can be displayed, the character fonts that can be displayed, the screen resolution, the physical length of the screen vertically and horizontally, and the number of characters that can be displayed vertically and horizontally within one screen. The remote control system according to any one of claims 6 to 17, comprising one. 19. A remote control method for remotely controlling an operation target device from an operation side terminal via a communication network, wherein: a screen analysis step of analyzing information displayed on a screen of the operation target device; A screen information transmission step of transmitting the data of the analysis result of the above to the operation side terminal for displaying the data on the operation side terminal. 20. The apparatus according to claim 19, further comprising an operation execution step of causing the operation target device to execute a predetermined process in response to an operation performed by a user using the operation-side terminal. Remote control method. 21. A screen information conversion step of converting data of an analysis result of the screen analysis step into data suitable for transmission to the operation side terminal or display on a screen of the operation side terminal, 21. The transmission method according to claim 19, wherein the conversion result data of the screen information conversion step is transmitted to the operation-side terminal for display on the operation-side terminal. The remote control method described. 22. A conversion rule registration step of registering a rule for converting data of an analysis result of the screen analysis step into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. 22. The remote operation method according to claim 21, wherein in the screen information conversion step, data of an analysis result of the screen analysis step is converted based on the conversion rule registered in the conversion rule registration step. 23. A device characteristic detecting step for detecting a device characteristic of the operating terminal, wherein the screen information converting step includes registering the device characteristic detected by the device characteristic detecting step and the device characteristic detected by the conversion rule registering step. 22. The remote operation method according to claim 21, wherein data of an analysis result of the screen analysis step is converted based on the conversion rule. 24. A screen change detecting step for detecting a change in the display of the screen of the operation target device, and when the change of the display of the generated screen ends, the screen analysis step detects the change of the display of the operation target device at the present time. 2. The content displayed on the screen is analyzed.
The remote control method according to any one of claims 9 to 23. 25. The information processing apparatus having a GUI as the operation target device, wherein in the screen analysis step, a GUI component displayed on a screen of the operation target device is detected, and attribute data of the GUI component is acquired. The remote control method according to any one of claims 19 to 24, characterized in that: 26. The attribute data of the GUI component includes a type of the GUI component, a name of the GUI component, and a name of the GUI component.
26. The remote control method according to claim 25, wherein the remote control method includes at least one of the positions of the I parts in the screen. 27. In the conversion rule registration step, a rule for converting the data of the analysis result of the screen analysis step into a web page that can be displayed by a web browser is registered, and in the screen information conversion step, And converting the data of the analysis result of the screen analysis step into a web page. The web page corresponds to the GUI parts detected by the screen analysis step and which can be operated by the user. In the operation execution step, the data identifying the GUI component corresponding to the hyperlink and the data identifying the operation on the GUI component transmitted when the hyperlink is selected are included in the operation execution step. 26. The method according to claim 25, wherein the predetermined process is executed by the operation side terminal. 27. The remote control method according to claim 26. 28. The conversion rule registration step according to claim 19, wherein a rule for converting the data of the analysis result of the screen analysis step into an e-mail text is registered. The remote control method described in (1). 29. In the screen analysis step, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected to extract video data of the display, and in the screen information transmission step, The remote control according to any one of claims 19 to 28, wherein the video data extracted in the screen analysis step is transmitted to the operation-side terminal for display on the operation-side terminal. Method of operation. 30. In the screen analysis step, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected to extract video data of the display, and in the screen information conversion step, The video data extracted by the screen analysis step is converted into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. The remote control method according to any one of the above. 31. The device characteristics of the operating terminal detected by the device characteristics detecting step include a unique ID of the operating terminal, a model, a processing speed, a communication speed, a communication cost per unit data amount, and a communication cost per unit time. Communication cost, data format that can be displayed on the screen, sound data format that can be played, format of executable programs and scripts, amount of data that can be received at a time, displayable colors, displayable fonts, screen resolution, 31. Any one of claims 23 to 30, wherein the information includes at least one of the vertical and horizontal physical lengths of the screen and the number of vertical and horizontal characters that can be displayed in one screen. 3. The remote control method according to 1. 32. A remote operation program for remotely operating an operation target device from an operation side terminal via a communication network by controlling an information processing system, wherein information displayed on a screen of the operation target device is analyzed. A remote control program for executing a screen analysis process and a screen information transmission process of transmitting data of an analysis result of the screen analysis process to the operation side terminal for displaying the data on the operation side terminal. 33. An apparatus according to claim 32, further comprising: executing an operation execution process for causing the operation target device to execute a predetermined process in response to an operation performed by the user using the operation-side terminal. Remote control program as described. 34. A screen information conversion process for converting data of an analysis result of the screen analysis process to data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal, 34. The information transmitting process, wherein a process of transmitting data of a conversion result of the screen information converting process to the operation-side terminal to be displayed on the operation-side terminal is executed. The remote control program according to any one of the above. 35. A conversion rule registration process for registering a rule for converting the data of the analysis result of the screen analysis process to data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. 35. The screen information conversion process according to claim 34, further comprising: executing a process of converting data of an analysis result of the screen analysis process based on a conversion rule registered by the conversion rule registration process. Remote control program as described. 36. A device characteristic detecting process for detecting a device characteristic of the operation-side terminal, wherein in the screen information converting process, the device characteristic detected by the device characteristic detecting process and the conversion rule registering process are used. 35. The remote control program according to claim 34, wherein a process for converting data of an analysis result of the screen analysis process is executed based on a registered conversion rule. 37. A screen change detection process for detecting a change in the display of the screen of the operation target device, and when the change in the display of the generated screen ends, the operation target device at the present time is determined by the screen analysis process. The remote control program according to any one of claims 32 to 36, wherein the remote control program executes a process of analyzing the content displayed on the screen of (1). 38. The operation target device is an information processing device having a GUI, and in the screen analysis processing, a GUI component displayed on a screen of the operation target device is detected, and attribute data of the GUI component is obtained. The remote control program according to any one of claims 32 to 37, wherein the remote control program causes a process to be performed. 39. The attribute data of the GUI component includes a type of the GUI component, a name of the GUI component, and a name of the GUI component.
39. The remote control program according to claim 38, comprising at least one of the positions of the I parts in the screen. 40. In the conversion rule registration process, a rule for converting the data of the analysis result of the screen analysis process into a web page that can be displayed by a web browser is registered, and in the screen information conversion process, Converts the data of the analysis result of the screen analysis process into a web page based on the web page, the web page, for GUI components detected by the screen analysis process that can be operated by the user, A hyperlink corresponding to a GUI component is included. In the operation execution processing, the operation corresponding to the hyperlink transmitted when the hyperlink is selected in order to cause the operation-side terminal to execute the predetermined processing. 39. The method according to claim 38, wherein data for identifying a GUI component and data for identifying an operation on the GUI component are used. 40. The remote control program according to claim 39. 41. The conversion rule registration process according to claim 32, wherein a process of registering a rule for converting data of an analysis result of the screen analysis process into text of an e-mail is executed. 39. The remote control program according to any one of items 39. 42. In the screen analysis processing, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected to extract video data of the display, and in the screen information transmission processing, 42. A processing for transmitting the video data extracted by the screen analysis processing to the operation-side terminal for displaying the image data on the operation-side terminal. The remote control program according to one. 43. In the screen analysis processing, when a moving image or a still image is displayed on the screen of the operation target device, the display is detected and video data of the display is extracted. 33. The apparatus according to claim 32, wherein the video data extracted by the screen analysis processing is converted into data suitable for transmission to the operation side terminal or display on the screen of the operation side terminal. 41. The remote control program according to any one of 41. 44. The device characteristics of the operation side terminal detected in the device characteristics detection processing include a unique ID, a model, a processing speed, a communication speed, a communication cost per unit data amount, and a unit time per unit time of the operation side terminal. Communication cost, data format that can be displayed on the screen, sound data format that can be played, format of executable programs and scripts, amount of data that can be received at a time, displayable colors, displayable fonts, screen resolution, The physical length of the screen in the vertical and horizontal directions and at least one of the number of characters in the vertical and horizontal directions that can be displayed in one screen are included in any one of claims 36 to 43. The remote control program according to 1.