WO1995022137A1

WO1995022137A1 - Methods and apparatus for remote monitoring and control of automated video data systems

Info

Publication number: WO1995022137A1
Application number: PCT/US1995/000180
Authority: WO
Inventors: Winfred G. Vanmourik; Robert J. Shillman
Original assignee: Cognex Corporation
Priority date: 1994-02-08
Filing date: 1995-01-03
Publication date: 1995-08-17
Also published as: AU1523695A

Abstract

The invention provides an improvement to a data processing system of the type that includes a video source (14) that generates a video signal in response to a control signal; a host controller (12) that generates that control signal in response to host instruction signals; and a user interface that generates a graphical interface signal, e.g., for display on a computer monitor, that represents the status of the host controller. The improvement is characterized by a video interface (36, 38) that modifies the graphical interface signal to incorporate an image representing at least a portion of that generated by the video source. A communication element (such as a pair of modems (42, 50)) transfers this modified graphical signal from the host controller (12) to a remote 'terminal', where that signal is displayed. The terminal can be, for example, a digital data processor, e.g., personal computer (26).

Description

METHODS AND APPARATUS FOR REMOTE MONITORING AND CONTROL OF AUTOMATED VIDEO DATA SYSTEMS

Technical Field

The present invention relates to methods and apparatus for data processing and, more particularly, to methods and apparatus for remote monitoring and control of automated video systems, e.g., computers that control video data systems. Background Art

Among the wide variety of uses of computers is the processing of video data. In machine vision systems, for example, computers are employed to analyze acquired video images to determine, e.g., the placement of objects. Computers are also employed, by way of further example, in multimedia systems to compile, edit and present sequences of stored video images. Regardless of the particular application, systems such as these typically include a video source, e.g., a camera for acquiring a real-time image or a video disk for storing previously acquired images, and a computer for controlling that video source.

In addition to the complexity inherent to some of the applications, such as vision processing, the additional hardware and software makes these systems inherently complex. Their users often require the assistance of the system developer or other consultant to adapt them for a specific purpose. For example, the operator of a vision system may need the advice of a consultant to modify a vision system to inspect a newly substituted part.

When not done in person, these consultations are typically by phone. Manufacturers usually provide service call numbers attended by knowledgeable support personnel for this purpose.

Although telephone support is often quite adequate, the user may be hindered by his or her inability to describe the specific problem to the consultant. The consultant must then make a range of suggestions, each directed to solving a different potential problem source, in hopes of pinpointing the source. While this is enough of a problem for traditional computer systems, it is doubly so for systems having both a computer and a controlled video source.

The art has responded by providing systems, such as the "Remote Vision Support Package" or "RVSP" of Automatix, Inc.,( now Acuity Imaging Inc.), that couple a customer's vision processing system to a remote support consultant's system. The RVSP system is particularly understood to copy a portion of the control software, along with a single frozen monochrome image of the customer's video data, from the customer system to the consultant's system. The consultant can then, apparently, manipulate the control software and video data on his or her (the consultant's) system, while offering the customer advice on how to modify that software.

Although systems such as RVSP afford the consultant limited additional information for pinpointing the problem source, the consultant still lacks the information and control they would be provided, e.g., by a personal visit to the customer site.

In view of the foregoing, an object of the invention is to provide improved methods and apparatus for data processing and, more particularly, improved methods and apparatus for remote monitoring and control of computers that control video data sources. Another object is to provide a mechanism for remote monitoring and control that gives a consultant as much access to information and control of a remote site as he or she would have in person.

Still another object is to provide improved methods and apparatus for remote instruction of the operator of a computer that controls a video data source.

Yet another object is to provide improved methods and apparatus for remote engineering and consulting support of vision processing systems.

These and other objects of the invention are evident in the sections that follow. Disclosure of the Invention

The aforementioned objects are obtained by the present invention which provides, in one aspect, an improvement on a data processing system of the type that includes a video source that generates a video signal in response to a control signal; a host controller that generates the control signal in response to host instruction signals; and a user interface that generates a graphical interface signal, e.g., for display on a computer monitor, that represents the status of the host controller.

The improvement is characterized by a video interface that modifies the graphical interface signal to incorporate an image representing at least a portion of that generated by the video source. A communication element (e.g., a pair of modems) transfers this modified graphical signal from the host controller to a remote "terminal," where that signal is displayed. The terminal can be, for example, a digital data processor, e.g., personal computer. In another aspect of the invention, the remote terminal includes an input device, such as a keyboard, trackball and /or mouse, that responds to the direction of a remote operator (e.g., a consultant) to generate remote instructions signals. Those signals are transmitted to the host controller, where they are treated as host instruction signals. In still another aspect, the invention provides an apparatus of the type described above, in which the host controller, itself, includes an input device -- again, for example, a keyboard, trackball and /or mouse — to accept host operator directives. The host controller can generate the control signal for the video source based on directives issued by the host operator and those issued by the remote operator, as well as in response to instructions issued by a computer program executing on the host controller. In still other aspects, the invention provides a data processing system, paralleling the host controller discussed above, that is diagnosable at a remote terminal. Still other aspects provide methods paralleling the operation of these apparatus. These and other aspects of the invention are evident in the drawings and in the detailed description which follows. Brief Description of the Drawings

The foregoing and other aspects of the invention may be more fully understood from the following description, when read together with the accompanying drawings in which like reference number indicate like parts in the several figures, and in which:

Figure 1 is a schematic block diagram of a preferred embodiment of the present invention connected to a video processing system; and

Figure 2 depicts the operation of a video processing system configured for use with the present invention. Best Mode for Carrying Out the Invention

Figure 1 is a schematic block diagram of a preferred data processing system 10 configured according to the present invention. The system 10 includes a host controller 12, a video processor system 14, and a remote computer 26. The host controller 12 includes a central processing unit (CPU) 32, a computer memory 34 (including video memory 34A), a video interface 36, a tv/video interface 38, a video processor interface 40, and a modem 42, connected as shown.

The video processing system 14 includes a camera 16, connected to the system 14 via the pathway 54, a monitor 18, connected to the system 14 via the pathway 56, and an optional input device 21, connected as shown.

The remote terminal 26 connects via transmission media 24 to the host controller 12. The remote terminal 26 can include a CPU 44, a computer memory 46, a video interface 48, and a modem 50, connected as shown. The remote terminal 26 optionally includes a remote monitor 28 and an input device 30, connected as shown.

The Video Processor 14 (VP) is preferably a stand-alone processing system sufficient for the dedicated processing of machine vision tool software and factory floor automation control equipment. The VP 14 stores a video processing application 60 (not shown) that controls the operation of the VP 14 in order to process images captured by the camera 16 and to generate processed video signals amenable for display on the monitor 18. In a preferred embodiment, the video processing application 60 is responsive to user-generated directives that can be entered via the input device 21 in response to data input menus displayed on monitor 18. These menus can be displayed separately or in combination with the processed video image. The input device 21 can include a conventional keyboard, mouse pointer, or trackball.

The VP 14 can be any conventional video processing apparatus including a computer-controlled machine vision apparatus of the type commercially available in the marketplace. A preferred such system is that sold by the assignee hereof under the COGNEX family of tradenames (e.g., the COGNEX 3100 and COGNEX 4400).

The host controller 12 connects to the VP 14 via pathway 52 and pathway 56. In one embodiment, the pathway 52 is a bi-directional, serial communications channel for passing data between the VP 14 and the host controller 12. The data on pathway 52 can include control signals transferred from the host controller 12 to the VP 14 for operating the VP 14. These control signals can be program source code that can be interpreted and executed by the VP 14. Additionally, the VP 14 can transfer to the host controller 12 data that represents the operational status of the VP 14.

The pathway 56 can be a video channel suitable for carrying an NTSC or PAL video signal. In one embodiment, the pathway 56 is constructed from coaxial cable and BNC connectors. The pathway 56 can couple at one end to a video coupler 58, that can be a BNC video splitter coupling and at the host controller 12 end to a BNC coupling, or other appropriate coupling including RCA phono connector or multipin DIN connectors. The pathway 56 can carry the video signal generated by the VP 14 for display on the monitor 18 and that can include the processed video image and a graphical image, such as user menu prompts.

The host controller 12 is preferably a commonly available PC running an MS Windows (version 3.1) operating system. The windows operating system generates a graphical interface that represents to the user the operating condition of the PC. The CPU 32 and the memory 34 can be the type of microprocessors and memory conventional to commercial PC systems. The CPU 32 of the host controller 12 responds to instructions stored in the memory 34, typically as application programs, or responds to input data typically entered through the input device 22 and under the control of a computer program. The input device 22 can include a conventional keyboard and mouse. The monitor 20 can be a standard computer monitor suitable for displaying the contents of the video memory 34A and preferably supports a VGA class video standard. The video interface 36 can be any commonly available video driver board suitable for transforming the contents of the video memory 34A into the video signal 37 that is suitable for display on the monitor 20.

The interface 40 can include a serial interface, a parallel interface or a hardwired bus interface or any other interface structure suitable for maintaining a bi-directional data pathway between the host controller 12 and the VP 14. In a preferred embodiment the interface 40 is a serial interface card connected into a bus slot of the host controller 12.

The tv/video interface 38 is connected through pathway 56 to receive the processed video signal generated by the VP 14. In a preferred embodiment, a video splitting coupler 58 connects via coaxial cable between the VP 14, the monitor 18 and the tv/video interface 38 to affect the simultaneous transmission of the same video signal 56 to both the monitor 18 and the tv/video interface 38. The tv/video interface 38 can be a video capture card designed to work with the Windows operating system running on the host controller 12. The tv/video interface 38 can connect at a backplane slot in the host controller 12 to interface the video signal 56 with the video memory 34A. The tv/video interface 38 can be constructed to accept a composite video signal or S-video in either the NTSC or PAL standard. The tv/video interface 38 receives the processed video signal via pathway 56 and generates a digital signal representative of the video signal. The digital signal is suitable for storing in the video memory 34 A and for processing by the video interface 36. The tv/video interface 38 can be a commonly available video capture card that can receive a motion video signal and that can generate and maintain a digital signal representative of the motion video signal. In one example the Captivator TM video capture card sold by Video-Logic is used to generate the digital video signal.

In operation, the digital video signal is displayed in a window generated under the Windows operating system. The displayed digital signal appears substantially as the image displayed on the VP monitor 18. In a preferred embodiment, the window displaying the digital video signal can be sized and located as any conventional window controlled by a Windows-type operating system. In this way the tv/video interface board 38 modifies the graphical interface signal generated by the Windows operating system, to include portions of the image displayed on the VP monitor 18. The image displayed in the window can be a live action video image of the image displayed on the vp /monitor 18, or can be a still frame. Additionally, in a preferred embodiment the live video can be displayed in the window, while the host controller 12 captures still images and stores the captured images as computer files, preferably with conventional picture file formats such as Bitmap files or Device Independent Bitmap (DIB) files.

The host controller 12 and the remote terminal 26 are coupled via a communication system that includes the two modems, 42 and 50, connected via the transmission media 24, and a communications control system 62 (not shown). The modem 42 can be connected to a slot in the backplane of the host controller 12, to a serial interface or to a parallel interface. In a preferred embodiment, the modem 42 is an external modem that connects through a parallel interface to the host controller 12. The modem 42 is preferably operated at 28.8Kbaud or higher and can use data compression to achieve an effective transmission rate of 115.2 Kbaud or higher. Preferably, the modem 50 and remote computer 26 are similarly configured, although it should be apparent to one skilled in the art that any configuration sufficient to support the 28.8 Kbaud rate can be employed in the present invention.

The transmission media 24 can be ordinary telephone lines, fiber optic lines, an ISDN service connection, a cellular service connection or other media suitable for supporting the operating baud rate.

In a preferred embodiment, the host controller 12 includes a communications control application 62 that interfaces with the Windows operating system. The communications application 62 includes a software interface that controls the operation of the modem 42 and a host connect interface that connects the remote computer 26 to the host controller 12. Preferably the communications application 62 is of the type such as PC ANYWHERE TM that couples the remote terminal 26 to control the operation of the host controller 12.

The remote terminal 26 is preferably a complete computer system and can be any commonly available PC running an MS Windows (version 3.1) operating system. The terminal 26 includes a display monitor 28 and preferably includes a keyboard and mouse as input device 30. Additionally, the remote terminal 26 can include an optional video processing apparatus (not shown). Preferably, the optional video processing apparatus is a duplicate of the video processing system 14 and a preferred one of such systems is that sold by the assignee hereof under the aforementioned COGNEX family of tradenames.

In a preferred embodiment, the remote terminal 26 includes a communications control application 62 that interfaces with the Windows operating system. The communications application 62 controls the operation of the modem 50 and connects the remote terminal 26 to the host controller 12. Preferably the communications application 62 is of the type such as PC ANYWHERE TM. The application 62 includes functionality for terminal emulation to achieve remote control of the host controller 12 through the remote terminal 26. The terminal emulation functionality reproduces the video display of host monitor 20 on the remote monitor 28 and allows input received at remote input device 30 to control operation of the host controller 12.

In a preferred embodiment, the communications application 62 further includes functionality for generating a communications channel between the host controller 12 and the remote terminal 26. The communication channel can be of the type supported by the PC AnywhereTM communications application that can create a split screen display on both the host monitor 20 and the remote monitor 28. One portion of the screen displays input entered through host device 22 and the other portion of the screen displays input entered through the remote device 30. In this way, messages can be interactively generated and transmitted between a user at the host controller 12 and on operator at the remote terminal 26. This allows an operator at the remote terminal to answer questions and offer instructions to the user of the host controller 12.

Additionally, in a preferred embodiment of the invention, the communications application 62 includes functionality for temporarily suspending the transmission of computer data between the host controller 12 and the remote terminal 26, and using transmission media 24 for communicating audio signals, preferably voice signals. The voice function can be of the type supported by the PC AnywhereTM communications application. With reference to Figure 2 the operation of a data processing system 10 configured according to the present invention can be explained. Figure 2 depicts the VP 14 with the camera 16 directed at a specimen 64 that is seated on an optional specimen table 66. The VP 14 connects via the pathways 52 and to the host controller 12. The host controller 12 connects via the modem 42, transmission media 24 and the modem 50 to the remote terminal 26.

As further depicted by Figure 2, an image of the specimen 64 is captured by the camera 16 and processed by the VP 14 according to the processing application 60 to produce the video signal on pathway 56. As described above, the video signal on pathway 56 can be displayed simultaneously on the monitor 18 and the monitor 20. In the illustrated embodiment, the video signal is displayed in a first window 68 on the monitor 20. By operation of the communications application 62, the display of the host controller 12 is reproduced on the display 28 of the remote terminal 26. The entire video display of monitor 20, including the video image displayed in window 68, is transmitted to the remote terminal 26 and reproduced as window 69. In this way, an operator at the remote computer 26 can view a representation of the signal carried via pathway 56 and generated by the VP 14. This image can include both processed video data, such as the image captured by camera 16, and graphical image data, such as graphical menus generated by VP 14 and displayed on monitor 18.

The input device 21 can demark the operator's location on the monitor 18. A position icon, e.g., a "I"-bar, is generated by the input device 21 to provide a visual aid in showing the position of interest. Preferably, the operator specifies that position of interest via an input device 21 which can be a keyboard, mouse or trackball.

In a one embodiment, the host controller 12 further includes an application development interface that is responsible for the activities relating to design of a processing application. The application development interface can encompass an interactive programming environment for developing a computer program of the type such as processing application 60. The interactive programming environment can provide a menu driven graphical interface that prompts the user during the development of the processing application.

A preferred system providing such an environment is described in co-pending U.S.S.N. 08/043,295, filed on April 6, 1993, and assigned to the assignee hereof, the teachings of which are incorporated herein by reference. Within that environment the user operating host controller 12 can develop processing applications and these processing applications can be run on the video processing system 14. In this way, the user can customize the video processing system 14 to a particular application.

During the development of a processing application 60, the interactive programming environment graphically displays on the display element 20 the code of the application being developed. As illustrated in Figure 2, the application development interface can be operated in a separate window 70. The window 70 is reproduced by communications application 62 on the remote monitor 28 as the window 72. In the windows 70 and 72, the processing application being developed can be displayed as human readable text and, more preferably, as source code in the format of a high-level computer language, e.g., FORTRAN or C.

In the preferred embodiment, the programming environment is an interactive program that graphically displays a list of permissible programming modifications. The programming modifications are syntactically correct — that is, they include additions or deletions that insure that the program contains proper language constructs. In a preferred embodiment, the permissible programming modifications include permissible additions, deletions and other modifications of the program. The additions, themselves, include programming statements, e.g., commands, declarations, subroutines and functions and function calls. The programming environment responds to an operator's selection of a modification to modify the program accordingly.

An operator's selections can be made via the input devices 22 that can include a keyboard, mouse or trackball. The position marker 74 is controlled by the input device 22 and operator selections are made as in a conventional event driven system that implements a familiar point and click interface to the operator. All selections offered by the graphical interface menus can be clicked, dragged, double clicked and otherwise manipulated with a pointing device 22 uniformly.

During operation of the present invention, the video image displayed on the VP monitor 18 is captured and displayed in an operating system window 68 of the host monitor 20. The window 68 is reproduced in window 69 at the remote computer display 28 for viewing and analysis by a remote operator. The video image displayed in windows 68 and 69 can be related to the processing application being developed in windows 72 and 70. Specifically, the position marker 74 can be manipulated with the remote input device 30 to mark select portions of the image displayed by the VP14 on the monitor 18.

The processing application that is being developed to process the video image can be displayed in a separate window 70 on the host monitor 20 and reproduced in the window 72 on the remote monitor 28. The operator of remote terminal 26 can analyze the code while viewing the video image produced by the code. The remote operator can enter data through the remote input devices 30 to operate the graphical programming environment operating on the host controller 12 to modify the processing application being developed on the host controller 12. In a preferred embodiment, the host video interface 36 is configured to generate a video signal 37 suitable for displaying video images according to the 800 X 600 X 256 video standard and the remote video interface 48 is configured to generate a video signal 39 suitable for displaying video images according to the 1024 X 768 X 256 video standard.

Described above are improved methods and apparatus for remotely monitoring and controlling automated video data systems, such as machine vision systems, by transferring to a remote terminal a modified graphical interface signal that includes an image representing at least a portion of that generated by the video data source and by communicating instruction signals between the remote terminal and the automated video data source.

Thus, for example, a consultant at the remote terminal can view a representation of the video image generated at the source while remotely instructing the operation of the video data source. Additionally, the consultant can remotely observe the instructions of a programmer operating the video data source to identify any programmer errors affecting the development of the processing application. Furthermore, the programmer and the consultant can jointly instruct the video data source.

In accordance with the above description, the invention attains the objects set forth.

It is further intended that all matter and the description and drawings be interpreted as illustrative and not in a limiting sense. That is, while various embodiments of the invention have been described in detail, other alterations which will be apparent to those skilled in the art are intended to be embraced within the spirit and scope of the invention.

In view of the foregoing, what is claimed is:

Claims

CLAI S :

1. A data processing system comprising

A. video source means for generating, in response to a control signal, a video signal of a type amenable for display on a monitor, and

B. host controller means, coupled to said video source means, for responding to host instruction signals for generating said control signal,

C. said host controller means including user interface means, coupled to said host controller , for generating a graphical interface signal representative of a response status of said host controller means to instruction signals, D. video interface means, coupled to said video source means and to said user interface means, for modifying said graphical interface signal to include representations of at least portions of said video signal, and

E. communications means, coupled to at least said user interface means and to a terminal means, for transmitting said modified graphical interface signal to said terminal means, and

F. said terminal means for at least receiving and displaying said modified graphical interface signal.

2. A data processing system according to claim 1, wherein

A. said terminal means includes remote user input means for generating remote instruction signals, and

B. said communications means further includes means for transmitting said remote instruction signals from said terminal means to said host controller means as host instruction signals.

3. A data processing system according to claim 2, wherein said remote user input means comprises means for responding to remote operator directives for generating said remote instruction signals.

4. A data processing system according to claim 3, wherein said remote user input means comprises any of a keyboard and a pointing device.

5. A data processing system according to claim 1 further characterized by host user input means, coupled to said host controller means, for generating said host instruction signals.

6. A data processing system according to claim 5 wherein said host controller means comprises means for generating said control signals as a combination of (i) said host instruction signals generated by said host user input means, and (ii) said host instructions signals generated as remote instruction signals and transmitted by said communications means to said host controller means as host instruction signals

7. A data processing system according to claim 6 wherein said host user input means comprises means for responding to host operator directives for generating said host instruction signals.

8. A data processing system according to claim 7 wherein said host user input means comprises any of a keyboard and a pointing device.

9. A data processing system according to claim 1 wherein said terminal means is disposed remotely from said host controller means.

10. A data processing system according to claim 9 wherein

A. each of said host controller means and said terminal means comprise a digital data processor, and

B. said communications means comprises remote-control-and-communications means for permitting digital data processors to at least communicate with one another.

11. A data processing system according to claim 10 wherein said communications means further comprises first and second modems, each coupled with a respective one of said host controller means and said terminal means, for transferring signals therebetween.

12. A data processing system according to claim 10, wherein said host controller means includes programmable controller means for generating said control signal based on an operator-generated program and said host instruction signals.

13. A data processing system diagnosable by a remote terminal means for at least receiving and displaying a diagnostic signal, said data processing system comprising A. video source means for generating, in response to a control signal, a video signal of a type amenable for display on a monitor,

B. host controller means, coupled to said video source means, for responding to host instruction signals for generating said control signal, said host controller means including user interface means, coupled to said host controller, for generating a graphical interface signal representative of a response status of said host controller means to instruction signals,

C. video interface means, coupled to said video source means and to said user interface means, for modifying said graphical interface signal to include representations of at least portions of said video signal, and

D. communications means, coupled to at least said user interface means and to a diagnostic terminal means, for transmitting said modified graphical interface signal to said terminal means as said diagnostic signal.

14. A data processing system according to claim 13, wherein said communications means further includes means for receiving, as host instruction signals, instruction signals generated by said terminal means.

15. A data processing system according to claim 14, comprising host user input means, coupled to said host controller means, for generating said host instruction signals.

16. A data processing system according to claim 15 wherein said host controller means comprises means for generating said control signals as a combination of (i) said host instruction signals generated by said host user input means, and (ii) said host instructions signals received by said communications means as generated by said terminal means.

17. A data processing system according to claim 16 wherein said host user input means comprises means for responding to host operator directives for generating said host instruction signals.

18. A data processing system according to claim 17 wherein said host user input means comprises any of a keyboard and a pointing device.

19. A data processing system according to claim 18, wherein

A. said host controller means comprises a digital data processor, and

B. said communications means comprises remote-control-and-communications means for permitting said host controller means to communicate with, and be controlled by, a remote digital data processor.

20. A data processing system according to claim 10 wherein said communications means further comprises first and second modems, each coupled with a respective one of said host controller means and said terminal means, for transferring signals therebetween.

21. A data processing system according to claim 13 wherein said host controller means includes programmable controller means for generating said control signal based on an operator-generated program and said host instruction signals.

22. A method of operating a data processing system of the type having video source means for generating, in response to a control signal, a video signal of a type amenable for display on a monitor, host controller means, coupled to said video source means, for responding to host instruction signals for generating said control signal, said host controller means including user interface means, coupled to said host controller, for generating a graphical interface signal representative of a response status of said host controller means to instruction signals, said method comprising the steps of

A. modifying said graphical interface signal to include representations of at least portions of said video signal,

B. transmitting said modified graphical interface signal to a remote site, and

C. receiving and displaying that modified graphical interface signal at the remote site.

23. A method according to claim 22, including the steps of

A. generating remote instruction signals at the remote site, and B. transmitting said remote instruction signals from the remote site to said host controller means as host instruction signals.

24. A method according to claim 23, further comprising the step of responding to remote operator directives for generating said remote instruction signals.

25. A method according to claim 24, further comprising the steps of inputting said remote operator directives with any of a keyboard and a pointing device.

26. A method according to claim 23, further comprising the step of responding to host operator directives for generating host instruction signals.

27. A method according to claim 26, further comprising the step of inputting said host operator directives with any of a keyboard and a pointing device.

28. A method according to claim 26, further comprising the step of generating said control signals as a function of (i) said host instruction signals generated in response to host operator directives, and (ii) said remote instruction signals and transmitted to said host controller means as host instruction signals

29. A method according to claim 22, further comprising the step of transmitting said signals between said host controller means and said remote site via remote communications media.

30. A method according to claim 29, further comprising the step of transmitting said signals between said host controller means and said remote site via one or more modems.

31. A method according to claim 22, further comprising the step of generating said control signal as a function of an operator-generated program and said host instruction signals.