US20060005160A1 - Image acquisition device - Google Patents
Image acquisition device Download PDFInfo
- Publication number
- US20060005160A1 US20060005160A1 US11/212,326 US21232605A US2006005160A1 US 20060005160 A1 US20060005160 A1 US 20060005160A1 US 21232605 A US21232605 A US 21232605A US 2006005160 A1 US2006005160 A1 US 2006005160A1
- Authority
- US
- United States
- Prior art keywords
- graphical program
- program
- image
- acquisition device
- image acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2294—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing by remote test
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/26—Functional testing
- G06F11/273—Tester hardware, i.e. output processing circuits
- G06F11/2733—Test interface between tester and unit under test
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/30—Circuit design
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/30—Circuit design
- G06F30/34—Circuit design for reconfigurable circuits, e.g. field programmable gate arrays [FPGA] or programmable logic devices [PLD]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/30—Circuit design
- G06F30/34—Circuit design for reconfigurable circuits, e.g. field programmable gate arrays [FPGA] or programmable logic devices [PLD]
- G06F30/343—Logical level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
Definitions
- the present invention relates to graphical programming, and in particular to a system and method for deploying a graphical program onto an image acquisition device.
- the present invention further relates to a system and method for configuring an image acquisition or machine vision device to perform image processing functions, wherein the instrument or device includes a programmable hardware element and/or a processor and memory.
- high level text-based programming languages have been used by programmers in writing applications programs.
- Many different high level programming languages exist including BASIC, C, FORTRAN, Pascal, COBOL, ADA, APL, etc.
- Programs written in these high level languages are translated to the machine language level by translators known as compilers or interpreters.
- compilers or interpreters The high level programming languages in this level, as well as the assembly language level, are referred to as text-based programming environments.
- Examples of fields in which computer systems are employed to interact with physical systems are the fields of instrumentation, process control, industrial automation, and simulation.
- Computer measurement and control of devices such as instruments or industrial automation hardware has become increasingly desirable in view of the increasing complexity and variety of instruments and devices available for use.
- U.S. Pat. Nos. 4,901,221; 4,914,568; 5,291,587; 5,301,301; and 5,301,336; among others, to Kodosky et al disclose a graphical system and method for modeling a process, i.e., a graphical programming environment which enables a user to easily and intuitively model a process.
- the graphical programming environment disclosed in Kodosky et al can be considered a higher and more intuitive way in which to interact with a computer.
- a graphically based programming environment can be represented at a level above text-based high level programming languages such as C, Basic, Java, etc.
- the method disclosed in Kodosky et al allows a user to construct a diagram using a block diagram editor.
- the block diagram may include a plurality of interconnected icons such that the diagram created graphically displays a procedure or method for accomplishing a certain result, such as manipulating one or more input variables and/or producing one or more output variables.
- data structures and/or program instructions may be automatically constructed which characterize an execution procedure that corresponds to the displayed procedure.
- the graphical program may be compiled or interpreted by a computer.
- the user may select various function nodes or icons that accomplish his desired result and connect the function nodes together.
- the function nodes may be connected in one or more of a data flow, control flow, and/or execution flow format.
- the function nodes may also be connected in a “signal flow” format, which is a subset of data flow.
- the function nodes may be connected between the terminals of the various user interface elements, e.g., between the respective controls and indicators.
- the user may create or assemble a graphical program, referred to as a block diagram, graphically representing the desired process.
- the assembled graphical program may be represented in the memory of the computer system as data structures and/or program instructions.
- the assembled graphical program i.e., these data structures, may then be compiled or interpreted to produce machine language that accomplishes the desired method or process as shown in the block diagram.
- Input data to a graphical program may be received from any of various sources, such as from a device, unit under test, a process being measured or controlled, another computer program, or from a file.
- a user may input data to a graphical program or virtual instrument using a graphical user interface, e.g., a front panel as described above.
- the input data may propagate through the data flow block diagram or graphical program and appear as changes on the output indicators.
- the front panel can be analogized to the front panel of an instrument.
- the front panel can be analogized to the MMI (Man Machine Interface) of a device.
- the user may adjust the controls on the front panel to affect the input and view the output on the respective indicators.
- the front panel may be used merely to view the input and output, or just the output, and the input may not be interactively manipulable by the user during program execution.
- graphical programming has become a powerful tool available to programmers.
- Graphical programming environments such as the National Instruments LabVIEW product have become very popular.
- Tools such as LabVIEW have greatly increased the productivity of programmers, and increasing numbers of programmers are using graphical programming environments to develop their software applications.
- graphical programming tools are being used for test and measurement, data acquisition, process control, man machine interface (MMI), supervisory control and data acquisition (SCADA) applications, simulation, image processing/machine vision applications, and motion control, among others.
- MMI man machine interface
- SCADA supervisory control and data acquisition
- One embodiment of the present invention comprises a computer-implemented system and method for deploying a graphical program onto an image acquisition device, such as an image acquisition board coupled to or included in a computer system, or a smart camera.
- an image acquisition device such as an image acquisition board coupled to or included in a computer system, or a smart camera.
- This provides the user the ability to develop or define desired functionality using graphical programming techniques, while enabling the resulting program to operate directly on an image acquisition device, such as a smart camera.
- the computer-implemented system and method may be used for configuring an image acquisition device or smart camera instrument to perform an image processing or machine vision function, where the image acquisition device (e.g., smart camera) includes at least one functional unit, such as a processor and memory, or a programmable hardware element.
- the user may first create a graphical program which performs or represents the desired functionality.
- the graphical program may comprise a block diagram which includes a plurality of interconnected nodes which visually indicate functionality of the graphical program.
- the plurality of nodes may be interconnected in one or more of a data flow, control flow, or execution flow format.
- the graphical program may include a single diagram or a hierarchy of subprograms or sub-diagrams.
- the user may place various constructs in portions of the graphical program to aid in conversion of these portions into hardware form.
- data structures and/or software code may be automatically created and stored in memory corresponding to the graphical program being created.
- the graphical program may implement an image processing function. More generally, in a measurement or instrumentation application, the graphical program may implement a measurement function.
- a portion of the graphical program may also implement a user interface, described below.
- the graphical program may be create on or stored on a computer system.
- the method may comprise coupling the image acquisition device to the computer system which stores the graphical program, and, where the image acquisition device includes a functional unit, deploying the graphical program onto the functional unit in the image acquisition device to configure the functional unit. After the deployment of the graphical program, the functional unit may be operable to implement the graphical program.
- the image acquisition device may be disconnected from the computer system after deploying the graphical program.
- the image acquisition device may also be coupled to the computer system over a network, and deploying the graphical program may include the computer system deploying the graphical program over the network to the image acquisition device.
- a computer-implemented method for configuring an image acquisition device (or smart camera) to perform an image processing function may include creating a graphical program on a computer system, as described above, where the graphical program implements the image processing function, and where the graphical program comprises a plurality of interconnected nodes which visually indicate functionality of the graphical program; deploying the graphical program on the image acquisition device, where the image acquisition device is coupled to or comprised in the computer system; the image acquisition device acquiring an image of an object; and the image acquisition device executing to perform the image processing function on the image.
- the image acquisition device may include a processor and a memory
- deploying the graphical program on the image acquisition device may include transferring the graphical program to the memory on the image acquisition device, where the image acquisition device executing to perform the image processing function on the image includes the processor in the image acquisition device executing the graphical program from the memory.
- deploying the graphical program on the image acquisition device may include generating an executable program, such as a machine language program, based on the graphical program, which implements at least a portion of the functionality of the graphical program; and transferring the executable program to the memory on the image acquisition device.
- the image acquisition device may include a programmable hardware element
- deploying the graphical program on the image acquisition device may include generating a hardware description based on the graphical program, which describes a hardware implementation of the graphical program, and configuring the programmable hardware element in the image acquisition device utilizing the hardware description.
- the programmable hardware element may implement a hardware implementation of the graphical program.
- the image acquisition device executing to perform the image processing function on the image may include the programmable hardware element in the image acquisition device executing to perform the image processing function on the image.
- the image acquisition device may include a processor and a memory and a programmable hardware element.
- deploying the graphical program on the image acquisition device may include transferring a first portion of the graphical program to the memory on the image acquisition device; generating a hardware description based on a second portion of the graphical program, which describes a hardware implementation of the graphical program; and configuring the programmable hardware element in the image acquisition device utilizing the hardware description.
- the programmable hardware element may implement a hardware implementation of the second portion of the graphical program.
- the image acquisition device executing to perform the image processing function on the image may include the processor in the image acquisition device executing the first portion of the graphical program from the memory and the programmable hardware element executing the second portion of the graphical program.
- the first and second portions of the graphical program may each include image processing portions.
- the first portion of the graphical program may include an image processing portion
- the second portion of the graphical program may include a control portion.
- the first portion of the graphical program may include the control portion
- the second portion of the graphical program may include the image processing portion.
- deploying the graphical program on the image acquisition device may include generating an executable program based on a first portion of the graphical program, and which implements functionality of the graphical program; transferring the executable program to the memory on the image acquisition device; generating a hardware description based on a second portion of the graphical program, where the hardware description describes a hardware implementation of the graphical program; and configuring the programmable hardware element in the image acquisition device utilizing the hardware description, where after being configured, the programmable hardware element implements a hardware implementation of the second portion of the graphical program.
- the image acquisition device executing to perform the image processing function on the image may include the processor in the image acquisition device executing the executable program from the memory and the programmable hardware element executing the second portion of the graphical program.
- the image acquisition device may include a first processor, a second processor, and at least one memory
- the image acquisition device executing to perform the image processing function on the image may include the first processor in the image acquisition device executing a first portion of the graphical program from the at least one memory and the second processor in the image acquisition device executing a second portion of the graphical program from the at least one memory.
- the first portion of the graphical program may include a first portion of the image processing function
- the second portion of the graphical program may include a second portion of the image processing function.
- a camera associated with the image acquisition device may acquire an image of an object.
- the image acquisition device or smart camera may be used in a machine vision inspection application, or a manufacturing assembly application.
- the at least one functional unit in the image acquisition device or smart camera may then receive the acquired image of the object.
- the at least one functional unit in the image acquisition device or smart camera may then execute to perform the image processing function (or machine vision function) on the image.
- the results of this image processing may then be used to make a decision in a machine vision inspection application or may be used to perform an operation in a manufacturing assembly application.
- the user interface portion may be presented on a display. A user or operator may use the displayed user interface portion to view or control the image processing function (or machine vision function).
- the image acquisition device may receive an image present signal, where the image acquisition device acquires the image of the object in response to receiving the image present signal.
- the image acquisition device may perform a control operation after executing to perform the image processing function on the image.
- the image acquisition device may generate a pass/fail indication after executing to perform the image processing function on the image.
- the image acquisition device may determine characteristics of the image after performing the image processing function, and then perform an operation based on the determined characteristics of the image.
- the image acquisition device may be coupled to a camera, and the image acquisition device acquiring an image of an object may include the camera acquiring the image of the object and the image acquisition device receiving and storing the image.
- the image acquisition device is a smart camera, the smart camera incorporates a camera as well as at least one functional unit.
- the graphical program includes a user interface portion
- the method may include presenting the user interface portion on a display and/or receiving user input to the user interface portion on the display to control the image acquisition device, during the functional unit in the image acquisition device executing to perform the image processing function on the image.
- the user interface portion may operate as a front panel for the image acquisition device.
- the method may further include compiling the user interface portion into executable code for execution by a processor and storing the executable code in a memory, and the processor executing the executable code from the memory to present the user interface portion on the display.
- the processor and memory may be included on the image acquisition device, and the processor in the image acquisition device may execute the executable code from the memory to present the user interface portion on the display during the image acquisition device executing to perform the image processing function on the image.
- the processor and the memory may be included on the computer system, and the computer system may execute the executable code from the memory to present the user interface portion on the display during the image acquisition device executing to perform the image processing function on the image.
- the method may operate to configure an image acquisition device, such as an image acquisition board or smart camera, with a graphical program to perform image processing or machine vision functions, where the device includes a functional unit, such as a programmable hardware element or processor/memory.
- an image acquisition device such as an image acquisition board or smart camera
- a graphical program to perform image processing or machine vision functions
- the device includes a functional unit, such as a programmable hardware element or processor/memory.
- FIG. 1 illustrates an image acquisition/machine vision system according to one embodiment of the invention
- FIG. 2A illustrates a computer system coupled to a smart camera according to one embodiment of the invention
- FIG. 2B illustrates a smart camera, according to one embodiment of the invention
- FIGS. 3A and 3B illustrate image processing/machine vision systems according to embodiments of the invention
- FIG. 4 is a block diagram of the computer system of FIGS. 1, 2A , and 3 A;
- FIGS. 5A, 5B and 5 C are block diagrams illustrating an interface card or device configured with a programmable hardware element and/or a processor and memory, according to various embodiments of the present invention
- FIGS. 6A-6C are block diagrams illustrating a smart camera configured with a programmable hardware element and/or a processor and memory, according to various embodiments of the invention.
- FIG. 7 is a flowchart diagram illustrating deployment of a graphical program on a device, according to one embodiment of the invention.
- FIG. 8A is a flowchart illustrating configuration of an image acquisition device or smart camera including a programmable hardware element, according to one embodiment of the invention.
- FIG. 8B is a flowchart illustrating configuration of an image acquisition device or smart camera including a processor and memory, according to one embodiment of the invention.
- FIG. 9 illustrates the conversion of a graphical program to hardware and software implementations
- FIG. 10 illustrates the generation of various types of hardware and software descriptions from a VDiagram tree
- FIG. 11 illustrates the conversion of a graphical program into a hardware description and the use of the hardware description to program an FPGA
- FIG. 12 illustrates the conversion of a graphical program into a software source code description and the compilation and linkage of the source code
- FIG. 13 is a flowchart diagram illustrating operation of one embodiment of the invention, including compiling a first portion of the graphical program into machine language and converting a second portion of the graphical program into a hardware implementation;
- FIG. 14 is a flowchart diagram illustrating creation of a graphical program according to the preferred embodiment.
- FIG. 1 Image Acquisition or Machine Vision System
- FIG. 1 illustrates a host computer system 102 coupled to an image acquisition device 134 .
- image acquisition device is intended to include any of various types of devices that are operable to acquire and/or store an image.
- An image acquisition device may also optionally be further operable to analyze or process the acquired or stored image. Examples of an image acquisition device include an image acquisition (or machine vision) card (also called a video capture board), a device external to a computer that operates similarly to an image acquisition card, a smart camera, a robot having machine vision, and other similar types of devices.
- image acquisition (or machine vision) card also called a video capture board
- a device external to a computer that operates similarly to an image acquisition card
- smart camera a smart camera
- robot having machine vision
- image processing function is also intended to include an image processing (or machine vision) algorithm that combines a sequence of two or more image processing functions or tools and/or decision operations that process an image in a desired way or which implement an image processing or machine vision application, such as part inspection, automated assembly, image analysis, pattern matching, edge detection, etc.
- FIG. 1 illustrates an exemplary image acquisition or machine vision system 100 .
- the image acquisition device 134 may in turn couple to or comprise a camera 132 .
- the image acquisition device 134 may include a functional unit for performing an image processing function as described below.
- the host computer 102 may comprise a CPU, a display screen, memory, and one or more input devices such as a mouse or keyboard as shown.
- the computer 102 may operate with the image acquisition device to analyze, measure or control a device or process 150 .
- the computer 102 may be used only to configure a functional unit in the image acquisition device.
- the term “functional unit” may include a processor and memory or a programmable hardware element.
- the term “functional unit” may include one or more processors and memories and/or one or more programmable hardware elements.
- the term “processor” is intended to include any of types of processors, CPUs, microcontrollers, or other devices capable of executing software instructions.
- programmable hardware element is intended to include various types of programmable hardware, reconfigurable hardware, programmable logic, or field-programmable devices (FPDs), such as one or more FPGAs (Field Programmable Gate Arrays), or one or more PLDs (Programmable Logic Devices), such as one or more Simple PLDs (SPLDs) or one or more Complex PLDs (CPLDs), or other types of programmable hardware.
- FPDs field Programmable Gate Arrays
- PLDs Programmable Logic Devices
- SPLDs Simple PLDs
- CPLDs Complex PLDs
- video device or camera 132 may be coupled to the computer 102 via the image acquisition device or card 134 .
- the camera 132 and/or image acquisition device 134 may couple to the computer 102 through a serial bus, a network, or through other means.
- the image acquisition system 100 may be used in an manufacturing assembly, test, measurement, and/or control application, among others.
- a unit under test (UUT) 150 is shown which may be positioned by a motion control device 136 (and interface card 138 ), and imaged and analyzed by the camera 132 and image acquisition device 134 . It is noted that in various other embodiments the UUT 150 may comprise a process or system to be measured and/or analyzed.
- the computer 102 may include a memory medium on which computer programs, e.g., graphical programs, according to the present invention may be stored.
- the term “memory medium” includes a non-volatile medium, e.g., a magnetic media or hard disk, or optical storage; a volatile medium, such as computer system memory, e.g., random access memory (RAM) such as DRAM, SRAM, EDO RAM, RAMBUS RAM, DR DRAM, etc.; or an installation medium, such as a CD-ROM or floppy disks 104 , on which the computer programs according to the present invention may be stored for loading into the computer system.
- RAM random access memory
- the term “memory medium” may also include other types of memory or combinations thereof.
- the memory medium may be comprised in the computer 102 where the programs are executed or may be located on a second computer which is coupled to the computer 102 through a network, such as a local area network (LAN), a wide area network (WAN), or the Internet.
- a network such as a local area network (LAN), a wide area network (WAN), or the Internet.
- the second computer operates to provide the program instructions through the network to the computer 102 for execution.
- the graphical program may be deployed by either one or more of: 1) converting the graphical program (or a portion thereof) into a hardware implementation and configuring the programmable hardware element with this hardware implementation, 2) transferring the graphical program (or a portion thereof) to a memory of the functional unit for execution by a processor (wherein the processor may execute a graphical program execution engine and optionally a real time operating system), or 3) compiling the graphical program (or a portion thereof) into an executable program and transferring the executable program to a memory of the functional unit for execution by a processor (wherein the processor may optionally execute a real time operating system).
- graphical program or “block diagram” is intended to include a program comprising graphical code, e.g., two or more nodes or icons interconnected in one or more of a data flow, control flow, or execution flow format, where the interconnected nodes or icons may visually indicates the functionality of the program.
- graphical program or “block diagram” are each intended to include a program comprising a plurality of interconnected nodes or icons which visually indicates the functionality of the program.
- a graphical program may comprise a block diagram and may also include a user interface portion or front panel portion. The user interface portion may be contained in the block diagram or may be contained in one or more separate panels or windows.
- a graphical program may be created using any of various types of systems which are used to develop or create graphical code or graphical programs, including LabVIEW, DASYLab, and DiaDem from National Instruments, Visual Designer from Intelligent Instrumentation, Agilent VEE (Visual Engineering Environment), Snap-Master by HEM Data Corporation, SoftWIRE from Measurement Computing, ObjectBench by SES (Scientific and Engineering Software), Simulink from the MathWorks, WiT from Coreco, Vision Program Manager from PPT Vision, Hypersignal, VisiDAQ, Vis Sim, and Khoros, among others.
- the system uses the LabVIEW graphical programming system available from National Instruments.
- FIGS. 2 A and 2 B Image Acquisition System Having a Smart Camera
- FIGS. 2A and 2B illustrate an image acquisition system with a smart camera.
- a smart camera may include a housing which encloses a portion or all of the smart camera components, or may be comprised on a frame which primarily provides structural support for the smart camera components.
- FIG. 2A Image Acquisition System with Smart Camera
- FIG. 2A illustrates an image acquisition system comprising computer system 102 coupled to a smart camera 190 .
- the smart camera is an example of an image acquisition device.
- the term “smart camera” is intended to include any of various types of devices that include a camera or other image sensor and a functional unit capable of being configured to perform an image processing function to analyze or process an acquired image.
- traditional computer vision is based on a camera/computer system in which the image processing or understanding algorithm is embedded in the computer 102
- the computational load of vision algorithms may be circumvented or mitigated by merging low level processing with the camera or sensor in a single module.
- a hardware architecture may be defined in a Hardware Description Language (e.g., VHDL), simulated and synthesized into digital structures that can then be configured in a Field Programmable Gate Array (FPGA).
- VHDL Hardware Description Language
- FPGA Field Programmable Gate Array
- the flexibility of an FPGA may be combined with a sensor for real time image processing.
- the flexibility of a processor and memory may be combined with a sensor for real time image processing.
- Examples of smart cameras include: NAVSYS Corporation's GI-EYE, which generates digital image data that are automatically tagged with geo-registration meta-data to indicate the precise position and attitude of the camera when the image was taken; Vision Components' GmbH Smart Machine Vision Cameras, which integrate a high-resolution Charge Coupled Device (CCD) sensor with a fast image-processing signal processor, and provide various interfaces to allow communication with the outside world; and Visual Inspection Systems' SMART cameras with on-board DSP capabilities, including frame grabbers and robot guidance systems, among others.
- CCD Charge Coupled Device
- the computer system 102 shown in FIG. 2A may include a memory medium.
- the memory medium may store a graphical programming development system for developing graphical programs.
- the graphical programming development system may be used to develop a graphical program that implements an image processing function.
- one or more of the nodes in the graphical program may implement or represent an image processing function.
- the memory medium may also store one or more computer programs which are executable to convert at least a portion of a graphical program into a form for configuring a functional unit comprised in the smart camera 190 .
- the smart camera 190 may include a functional unit, which may be a programmable hardware element (programmable or reconfigurable hardware), e.g., an FPGA, and/or may be a processor and memory.
- the functional unit in the smart camera 190 may be configured with a graphical program that implements the image processing function.
- the smart camera 190 may also comprise a camera coupled to the functional unit.
- the smart camera 190 may also include a memory (a memory medium) coupled to the camera that stores an acquired image. If the smart camera 190 includes an analog camera, the smart camera 190 may further include analog to digital (A/D) logic for converting analog image signals into a digital image for storage in the memory.
- A/D analog to digital
- the smart camera 190 may also optionally include timer/counter logic that may perform timing/counting operations, e.g., during operation of the functional unit.
- the smart camera may include a housing which encloses some or all of the components of the smart camera.
- FIGS. 6A-6D Various embodiments of the smart camera 190 are shown in FIGS. 6A-6D , described below.
- FIG. 2B Smart Camera
- FIG. 2B illustrates another embodiment of a smart camera 190 .
- the smart camera includes a camera, coupled to circuitry/logic (e.g., one or more circuit boards) for performing various image processing and/or acquisition functions.
- the circuitry/logic may include a functional unit, such as a programmable hardware element, e.g., an FPGA and/or a processor and memory.
- the functional unit in the smart camera 190 may be configured with a graphical program that implements the image processing function, and may also include a memory coupled to the camera for storing an acquired image.
- the smart camera 190 may further include analog to digital (A/D) logic for converting analog image signals into a digital image for storage in the memory.
- the smart camera 190 may also optionally include timer/counter logic that may perform timing/counting operations, e.g., during operation of the programmable hardware element.
- the smart camera 190 may also include various I/O ports for communicating with external devices, such as computer system 102 .
- the smart camera may include a frame or structure to support the components comprised in the smart camera.
- the smart camera may include a “PC-on-a-Card” which may provide part or all of the functionality of a personal computer.
- FIGS. 6A-6D Various embodiments of the smart camera 190 are shown in FIGS. 6A-6D , described below.
- FIGS. 3 A and 3 B Image Processing Systems
- FIGS. 3A and 3B illustrate image processing or machine vision systems according to various embodiments of the invention.
- the image processing system of FIG. 3A may comprise a computer 102 and a smart camera 190 , and may further include a motion control device 192 .
- the image processing system of FIG. 3B may comprise smart camera 190 and motion control device 192 , and may not include computer system 102 .
- the smart camera 190 may include a digital camera that acquires a digital video signal which comprises an image, or a sequence of images, or other data desired to be acquired.
- the smart camera 190 may instead include an analog camera that acquires an analog video signal, and the smart camera 190 may further include A/D converters for converting the analog video signal into a digital image.
- the smart camera 190 may include a functional unit configured according to a graphical program.
- the functional unit may be configured to perform an image processing function as represented by a graphical program.
- a graphical program may have been first created to perform the image processing function, such as with a graphical development environment on the computer system 102 , and the graphical program may then have been deployed onto the functional unit of the smart camera to implement the image processing function.
- the digital video signal or digital image may be provided to the functional unit in the smart camera 190 , wherein the image processing function is performed.
- the functional unit in the smart camera 190 may perform a portion or all of the image processing function, and the computer 102 may perform a portion of the image processing function.
- the functional unit in the smart camera 190 may perform the actual processing of the image to determine characteristics of the image, and the computer 102 may then perform an operation based on this result, such as rejecting a part from an assembly line, or logging the results to file.
- the functional unit in the smart camera 190 may perform the processing of the image to determine characteristics of the image, and may also optionally perform an operation based on this result, and the computer system 102 may execute software to provide a user interface for the system, e.g., the computer system 102 may execute a user interface portion of a graphical program, where the block diagram of the graphical program is used to configure the functional unit in the smart camera 190 .
- a first portion of the graphical program e.g., DSP functions requiring real time performance
- a second portion of the graphical program e.g., a user interface where real time performance is not required, may be executed by the computer system 102 .
- the functional unit in the smart camera 190 may perform all of the desired image processing function, including optionally performing an operation based on determined characteristics of the image, and hence the computer system 102 is not necessary during operation of the system.
- the smart camera 190 may include a processor and memory (in addition to the functional unit which executes the image processing function) which may execute the second portion of the graphical program, e.g., the user interface portion.
- the functional unit in the smart camera 190 may control the motion control device 192 .
- the motion control device 192 may also include a functional unit that has been configured with a graphical program.
- the functional unit in the motion control device 192 may be configured to perform a motion control function.
- a graphical program may have been first created that performs the motion control function, and the graphical program may then have been deployed onto the motion control device 192 as described herein.
- Examples of motion control functions include moving a part or object to be imaged by a camera, rejecting a part on an assembly line, or placing or affixing components on a part being assembled, or a robotics application, among others.
- FIG. 4 Computer Block Diagram
- FIG. 4 is an exemplary block diagram of the computer 102 of FIGS. 1, 2A , and 3 A.
- the elements of a computer not necessary to understand the operation of the present invention have been omitted for simplicity.
- the computer 102 may include at least one central processing unit (CPU) or processor 160 which is coupled to a processor or host bus 162 .
- the CPU 160 may be any of various types, including an x86 processor, a PowerPC processor, a CPU from the Motorola family of processors, a CPU from the SPARC family of RISC processors, as well as others.
- Main memory 166 is coupled to the host bus 162 by means of memory controller 164 .
- the main memory 166 stores a graphical programming system, and also stores software for deploying at least a portion of a graphical program onto an image acquisition device 134 . This software will be discussed in more detail below.
- the main memory 166 may also store operating system software, i.e., software for operation of the computer system, as well as one or more application programs, as is well known to those skilled in the art.
- the host bus 162 is coupled to an expansion or input/output bus 170 by means of a bus controller 168 or bus bridge logic.
- the expansion bus 170 is preferably the PCI (Peripheral Component Interconnect) expansion bus, although other bus types may be used.
- the expansion bus 170 may include slots for various devices, the examples shown including a motion control device 138 and an image acquisition device 134 , as described above in reference to FIG. 1 .
- the computer 102 may further comprise a video display subsystem 180 and hard drive 182 coupled to the expansion bus 170 , also shown.
- One or more of the interface cards or devices may be coupled to or comprised in the computer 102 and may comprise a functional unit, such as a programmable hardware element (FPGA) and/or processor and memory.
- the computer 102 may also include a network interface for coupling to a network, wherein the target device containing the functional unit may be coupled to the network.
- the computer 102 may be used for configuring a functional unit hardware in a target device over a network.
- FIGS. 5 A- 5 D Image Acquisition Device Hardware Diagram
- FIG. 5A is a block diagram illustrating a device, e.g., an interface card 134 A, configured with programmable hardware according to one embodiment. It is noted that FIG. 5A is exemplary only, and an interface card or device configured with programmable hardware according to the present invention may have various architectures or forms, as desired.
- the device may be internal or external to the computer 102 , and may be connected to the computer through a network, such as the Internet.
- the interface card 134 A illustrated in FIG. 5A may be the image acquisition device 134 or the motion control interface card 138 , shown in FIGS. 1-3 .
- the interface card 134 A may include an I/O connector 202 which is operable to send/receive signals.
- the I/O connector 202 may present analog and/or digital connections for receiving/providing analog or digital signals, such as image signals or pixel data.
- the I/O connector 202 may be adapted for coupling to an external camera.
- the interface card 134 A may also include data acquisition (DAQ) logic 204 .
- the data acquisition logic 204 may comprise analog to digital (A/D) converters, digital to analog (D/A) converters, timer counters (TC) and signal conditioning (SC) logic as shown.
- the DAQ logic 204 may provide data acquisition functionality of the image acquisition card 134 A.
- the DAQ logic 204 comprises 4 A/D converters, 4 D/A converters, 23 digital I/Os, a RTSI connector, and a TIO. This extra hardware is useful for signal processing and motion control applications.
- the functional unit e.g., programmable hardware element or FPGA 206 and/or processor 212 may access these resources directly, thereby enabling creation of very image processing applications, among others.
- the functional unit of the interface card 134 A may include a programmable hardware element 206 and a processor 212 and memory 214 .
- the programmable hardware 206 may comprise a field programmable gate array (FPGA) such as those available from Xilinx, Altera, etc.
- the programmable hardware element 206 may be coupled to the DAQ logic 204 and may also be coupled to a local bus interface 208 , described below.
- a graphical program can be created on the computer 102 , or on another computer in a networked system, and at least a portion of the graphical program can be converted into a hardware implementation form for execution in the FPGA 206 .
- the portion of the graphical program converted into a hardware implementation form is preferably a portion which requires fast and/or real time execution.
- the interface card 134 A may further include a dedicated on-board processor 212 and memory 214 .
- This enables a portion of the graphical program to be compiled into machine language for storage in the memory 214 and execution by the processor 212 .
- a portion of the graphical program may also (or instead) be transferred to the memory 214 in its native format for execution by the processor. This may be in addition to a portion of the graphical program being converted into a hardware implementation form in the FPGA 206 .
- the memory 214 may store a real time operating system (RTOS) for execution by the processor 212 . Where the graphical program executes in its native format on the device 134 A, the memory may also store a graphical program execution engine that is executed by the processor 212 .
- RTOS real time operating system
- a portion of the graphical program may be transferred and/or compiled for execution on the on-board processor 212 and executed locally on the interface card 134 A via the processor 212 and memory 214 , and a second portion of the graphical program may be translated or converted into a hardware executable format and downloaded to the FPGA 206 for hardware implementation.
- a first portion of a block diagram (that requires real time or fast execution) of a graphical program may be converted into a hardware executable format and downloaded to the FPGA 206 for hardware implementation, and a second portion of a block diagram (that may not require real time performance) may be stored in the memory 214 as program instructions and executed by the processor 212 , in either a compiled or interpreted manner.
- a portion or all of the block diagram portion of the graphical program may be converted into a hardware executable format and downloaded to the FPGA 206 for hardware implementation, and a user interface portion (or front panel portion) of the graphical program may be stored in the memory 214 as program instructions and executed by the processor 212 , in either a compiled or interpreted manner.
- the portion of the graphical program which requires the most real time or deterministic (reliable and consistent) performance may be executed directly in hardware for fast operation, and other parts of the block diagram, or the user interface portion, which may not require real time performance, may execute on the processor 212 .
- the processor may then send resulting signals to the video subsystem for display of the user interface on the computer display.
- the interface card 134 A may further include bus interface logic 216 and a control/data bus 218 .
- the interface card 134 A is a PCI bus-compliant interface card adapted for coupling to the PCI bus of the host computer 102 , or adapted for coupling to a PXI (PCI extensions for Instrumentation) bus.
- the bus interface logic 216 and the control/data bus 218 thus present a PCI or PXI interface.
- the interface card 134 A may also include local bus interface logic 208 .
- the local bus interface logic 208 may present a RTSI (Real Time System Integration) bus for routing timing and trigger signals between the interface card 134 A and one or more other devices or cards, such as a motion device.
- RTSI Real Time System Integration
- the interface card 134 A also includes a non-volatile memory 288 coupled to the programmable hardware element 206 , the processor 212 and the memory 214 .
- the non-volatile memory 288 may be operable to store the hardware description received from the host computer system to enable execution of the hardware description in the programmable hardware element 206 prior to or during booting of the computer system 102 .
- the non-volatile memory 288 may also store software used by the processor, such as a RTOS and/or a graphical program execution engine.
- the programmable hardware element 206 is not included on the interface card 134 B, and thus the functional unit of the device 134 B comprises only processor 212 and memory 214 .
- the graphical program may be transferred in its native format to the memory 214 and executed by the processor 212 using a graphical program execution engine and possibly a RTOS.
- the graphical program may be compiled into an executable program (e.g., machine language, a script, or an interpretable data structure) and transferred to the memory 214 for execution by processor 212 .
- the processor 212 is not included on the interface card 134 C, i.e., the functional unit of the device 134 C comprises the FPGA 206 and the memory 214 .
- the memory 214 may be used for storing FPGA state information.
- any supervisory control portion of the graphical program which is necessary or desired to execute on a programmable processor in software may be executed by the host CPU in the computer system 102 , and is not executed locally by a processor on the interface card 134 B.
- FIGS. 6 A- 6 C Block Diagrams of Smart Cameras with Programmable Hardware
- FIGS. 6A-6C are block diagrams of various embodiments of a smart camera 190 with programmable hardware. As may be seen, the various embodiments share a number of features and configurations with the various interface cards 134 described above with reference to FIGS. 5A-5C , specifically the use of various combinations of processor/memory elements and programmable hardware, e.g., FPGAs, to execute portions of a graphical program. It should be noted that these embodiments are meant to be illustrative only, and are not intended to limit the architecture, components, or form of the smart camera 190 .
- FIG. 6A illustrates one embodiment of a smart camera 190 which includes a camera 282 and a functional unit.
- the functional unit may comprise a programmable hardware element 206 , e.g., a field programmable gate array (FPGA) such as those available from Xilinx, Altera, etc, and may also comprise a processor 212 and memory 214 .
- FPGA field programmable gate array
- Each of the programmable hardware element 206 , processor 212 and memory 214 may be coupled to the camera 282 and/or to an image memory 284 .
- the smart camera 190 A may also include a non-volatile memory 288 coupled to the programmable hardware element 206 , the processor 212 , the memory 214 and the image memory 284 .
- the non-volatile memory 288 may be operable to store the hardware description and/or graphical program received from the host computer system to enable execution of the functional unit prior to or during booting of the computer system 102 .
- the smart camera 190 may include an I/O connector 202 which is operable to send and receive signals.
- the I/O connector 202 may present analog and/or digital connections for receiving/providing analog or digital signals.
- the I/O connector 202 may enable the smart camera to communicate with computer system 102 to receive a graphical program for performing image processing functions.
- a graphical program can be created on the computer 102 , or on another computer in a networked system, and at least a portion of the graphical program can be deployed onto a functional unit of the smart camera.
- the portion of the graphical program converted into a hardware implementation form is preferably a portion which requires fast and/or real-time execution.
- the smart camera 190 A includes a dedicated on-board processor 212 and memory 214 in addition to the programmable hardware element 206 .
- This enables a portion of the graphical program to be compiled into machine language for storage in the memory 214 and execution by the processor 212 (or to execute natively). This is in addition to a portion of the graphical program being converted into a hardware implementation form in the FPGA 206 .
- a portion of the graphical program may be compiled for execution on the on-board processor 212 and executed locally on the smart camera 190 A via the processor 212 and memory 214 , and a second portion of the graphical program may be translated or converted into a hardware executable format and uploaded to the FPGA 206 for hardware implementation.
- a first portion of a block diagram (that requires real time or fast execution) of a graphical program may be converted into a hardware executable format and downloaded to the FPGA 206 for hardware implementation, and a second portion of a block diagram (that may or may not require real time performance) may be stored in the memory 214 as program instructions and executed by the processor 212 , in either a compiled or interpreted manner.
- a portion or all of the block diagram portion of the graphical program may be converted into a hardware executable format and downloaded to the FPGA 206 for hardware implementation, and a user interface portion (or front panel portion) of the graphical program may be downloaded and stored in the memory 214 as program instructions and executed by the processor 212 , in either a compiled or interpreted manner.
- the portion of the graphical program which requires the most real time or deterministic (reliable and consistent) performance may be executed directly in hardware for fast operation, and other parts of the block diagram, or the user interface portion, which may not require real time performance, may execute on the processor 212 .
- the processor may then send resulting signals to a video subsystem (of, for example, the computer system 102 ) for display of the user interface on the computer display.
- the smart camera 190 A may include image memory 284 which couples to the programmable hardware 206 , the camera 282 , the processor 212 , memory 214 , bus interface 216 , and the control/data bus 218 .
- the image memory 284 may be operable to store a portion of an image, or one or more images received from the camera 282 .
- the image memory 284 may enable the programmable hardware 206 and/or the processor 212 to retrieve the one or more images, operate on them, and return the modified images to the image memory. Additionally, the images may be retrievable for transmission to external systems, such as the computer system 102 , via the I/O connector 202 , for example.
- the smart camera 190 A may further include bus interface logic 216 and a control/data bus 218 .
- the smart camera 190 A is a PCI bus-compliant interface card adapted for coupling to the PCI bus of the host computer 102 , or adapted for coupling to a PXI (PCI eXtensions for Instrumentation) bus.
- the bus interface logic 216 and the control/data bus 218 thus present a PCI or PXI interface.
- the smart camera 190 A may also include local bus interface logic 208 , as shown in FIG. 5A .
- the local bus interface logic 208 may present a RTSI (Real Time System Integration) bus for routing timing and trigger signals between the smart camera 190 A and one or more other devices or cards, such as other smart camera, motion devices, or other smart sensors.
- RTSI Real Time System Integration
- the smart camera 190 A may include data acquisition (DAQ) logic 204 , not shown, such as analog to digital (A/D) converters, digital to analog (D/A) converters, timer counters (TC) and signal conditioning (SC) logic, as shown in FIG. 5A .
- DAQ logic 204 may be useful for signal processing and motion control applications.
- the programmable hardware element or FPGA 206 may access these resources directly, thereby enabling creation of very powerful measurement, DSP and control applications, among others.
- the smart camera 190 A may further include analog to digital (A/D) logic (not shown) for converting analog image signals into a digital image for storage in the memory.
- A/D analog to digital
- A/D analog to digital
- FIG. 6B illustrates another embodiment of a smart camera 190 B, with a configuration similar to that of interface card 134 B, described above with reference to FIG. 5B .
- the smart camera 190 B includes the camera 282 , processor 212 , memory 214 , image memory 284 , control/data bus 218 , and bus interface 216 described above with reference to FIG. 6A , but does not include the programmable hardware (e.g., FPGA).
- the functional unit comprises only the processor 212 and memory 214 .
- a portion or all of a graphical program may be deployed to the processor/memory component 212 / 214 .
- timer/counter logic 286 may also be included on the smart camera.
- the timer/counter logic 286 may be coupled to the other components of the smart camera 190 B, and may be operable to provide high-performance timing and counting functions to the various elements of the smart camera 190 B.
- the timer/counter logic may be used by the processor 212 to accurately control image acquisition by the camera 282 , such as for a high speed assembly line, or for capturing images of a fast-moving object.
- FIG. 6C illustrates another embodiment of a smart camera 190 C, where the smart camera 190 C includes programmable hardware 206 and non-volatile memory 288 , but does not include the processor 212 and memory 214 .
- the functional unit comprises only the FPGA 206 .
- any supervisory control portion of the graphical program which is necessary or desired to execute on a programmable processor in software may be executed by the host CPU in the computer system 102 , and is not executed locally by processor 212 , i.e., on the smart camera 190 C.
- FIG. 6C illustrates another embodiment of a smart camera 190 C, where the smart camera 190 C includes programmable hardware 206 and non-volatile memory 288 , but does not include the processor 212 and memory 214 .
- the functional unit comprises only the FPGA 206 .
- any supervisory control portion of the graphical program which is necessary or desired to execute on a programmable processor in software may be executed by the host CPU in the computer system 102 , and
- the smart camera may include camera 282 , bus interface 216 , control/data bus 218 , image memory 284 , and timer/counter logic 286 , as described above.
- a graphical image processing program in the form of a hardware description, may be converted into a hardware executable format and downloaded to the programmable hardware element 206 for hardware implementation.
- the non-volatile memory 288 (or a separate volatile memory not shown) may be operable to store the hardware description received from the host computer system 102 to enable execution of the hardware description in the programmable hardware element 206 prior to or during booting of the computer system 102 .
- FIG. 7 Method for Deploying a Graphical Program on an Image Acquisition Device
- FIG. 7 illustrates one embodiment of a method for deploying a graphical program on an image acquisition device to perform an image processing function, where the image acquisition device includes a functional unit.
- a computer-implemented method is presented for configuring an image acquisition device to perform an image processing function using a graphical program. Note that the method shown in FIG. 7 may be used to configure any of the instruments or devices shown in FIGS. 1-6D . It should be noted that in various embodiments, various steps in the method may occur concurrently, or in a different order than shown. Additionally, some steps may be omitted, or additional steps performed which are not shown, as desired. As shown, this method may operate as follows.
- a graphical program may be created that implements an image processing function.
- the graphical program is created on the computer system 102 , although in other embodiments the graphical program may be created on a different computer system, for example, coupled to computer system 102 over a network.
- the graphical program may be created or assembled by the user with a graphical programming environment, such as the LabVIEW graphical programming development environment.
- the graphical program may be created in step 702 by the user specifying one or more functions or tasks, after which the specification may be analyzed and the graphical program generated automatically or programmatically from the specification.
- the user may use a prototyping environment to create a script or prototype representing an image processing algorithm, and then a graphical program may be manually or programmatically created based on this prototype. It is noted that other approaches may also be used to create the graphical program, either manually by the user, or programmatically by software.
- the graphical program may be deployed on the image acquisition device 134 .
- the image acquisition device 134 is coupled to the computer system 102 to facilitate the deployment of the graphical program.
- the image acquisition device 134 may be comprised in the computer system 102 .
- the image acquisition device 134 may be coupled to the computer system 102 through a network, such as the Internet, or may be coupled to the computer system 102 through wireless means.
- a human user may manually retrieve the graphical program from the computer 102 and load the graphical program onto the image acquisition device 134 .
- deploying the graphical program on the image acquisition device 134 may comprise storing the graphical program on (at least part of) the functional unit, or configuring the functional unit with all or part of the graphical program, such that after the deployment, the functional unit is operable to execute or implement the functionality of the graphical program.
- the image acquisition device 134 may acquire an image of an object, such as, for example, the UUT 150 , or an element or state of a system or process.
- the image is acquired via camera 132 , although in other embodiments the image may be acquired from an external system, such as a computer or another image acquisition device.
- the image acquisition device 134 may receive an image present signal, and acquire the image of the object in response to receiving the image present signal.
- the image acquisition device may be coupled to a camera, and the image acquisition device acquiring an image of the object may comprise the camera acquiring the image of the object and the image acquisition device receiving and storing the image.
- the image acquisition device may execute to perform the image processing function on the acquired image from 706 .
- the functional unit on the image acquisition device 134 may execute all or part of the graphical program.
- the image processing function may operate to analyze the image and detect, recognize, or characterize one or more features or characteristics of the object, such as position, orientation, color, or possible flaws or lack thereof, among others.
- the image acquisition device may perform a control operation after executing to perform the image processing function on the image.
- the image acquisition device may generate a pass/fail indication after executing to perform the image processing function on the image.
- the image acquisition device executing to perform the image processing function on the acquired image may comprise the image acquisition device determining characteristics of the image after performing the image processing function, and the image acquisition device 134 performing an operation based on the determined characteristics of the image.
- the image acquisition device may determine a characteristic of the image indicating that the imaged object is defective.
- the image acquisition device 134 may then send a signal, such as a pass/fail indication, to a motion control device which may remove the defective object from the assembly line.
- a signal such as a pass/fail indication
- the image acquisition device 134 may include processor 212 and memory 214 , as described above with reference to FIGS. 5B and 6B .
- the deployment of the graphical program on the image acquisition device 134 described in 704 may then comprise transferring the graphical program to the memory 214 on the image acquisition device 134 .
- deploying the graphical program on the image acquisition device 134 may include generating an executable program (a machine language program) based on the graphical program, which implements the functionality of the graphical program, and transferring the executable program to the memory 214 on the image acquisition device.
- the image acquisition device executing to perform the image processing function on the image in 708 may comprise the processor 212 in the image acquisition device 134 executing the executable program from the memory 214 , possibly by executing a real time operating system (RTOS) from the memory 214 .
- RTOS real time operating system
- the image acquisition device executing to perform the image processing function on the image may comprise the processor in the image acquisition device executing the executable program (in executable form) from the memory 214 .
- deploying the graphical program on the image acquisition device 134 may include transferring the graphical program in its native or original format to the memory 214 on the image acquisition device.
- the image acquisition device executing to perform the image processing function on the image in 708 may comprise the processor 212 in the image acquisition device 134 executing a graphical program execution engine, and possibly a real time operating system (RTOS) from the memory 214 .
- RTOS real time operating system
- the image acquisition device may include a programmable hardware element 206 , such as an FPGA, as described above with reference to FIGS. 5C and 6C .
- the deployment of the graphical program on the image acquisition device 134 described in 704 may then comprise generating a hardware description based on the graphical program, which describes a hardware implementation of the graphical program, and configuring the programmable hardware element 206 in the image acquisition device 134 utilizing the hardware description.
- Configuring the programmable hardware element 206 in the image acquisition device 134 utilizing the hardware description may comprise converting the hardware description, such as a VHDL file, into a netlist using available synthesis tools, and the compiling the netlist into a hardware program file (also called a software bit stream), which may be used to configure or program the programmable hardware element 206 .
- converting the hardware description such as a VHDL file
- a netlist using available synthesis tools
- a hardware program file also called a software bit stream
- the programmable hardware 206 After configuring the programmable hardware element 206 with the hardware description, the programmable hardware 206 implements a hardware implementation of the graphical program.
- the image acquisition device executing to perform the image processing function on the image in 708 may comprise the programmable hardware element 206 in the image acquisition device executing to perform the image processing function on the image.
- the image acquisition device may include the processor 212 and memory 214 and the programmable hardware element 206 , as described above with reference to FIGS. 5A, 6A , and 6 D.
- the deployment of the graphical program on the image acquisition device 134 described in 704 may then comprise transferring a first portion of the graphical program to the memory 214 on the image acquisition device 134 (either in native or compiled form), generating a hardware description based on a second portion of the graphical program describing a hardware implementation of the graphical program, and configuring the programmable hardware element in the image acquisition device utilizing the hardware description, after which the programmable hardware element implements a hardware implementation of the second portion of the graphical program.
- the image acquisition device executing to perform the image processing function on the image in 708 may comprise the processor 212 in the image acquisition device 134 executing the first portion of the graphical program from the memory 214 and the programmable hardware element 206 executing the second portion of the graphical program.
- the deployment may include generating an executable program based on the first portion of the graphical program, which implements the functionality of the first portion of the graphical program, and transferring the executable program to the memory 214 on the image acquisition device.
- the second portion may be configured on the programmable hardware element.
- the first portion of the graphical program may comprise an image processing portion
- the second portion of the graphical program may comprise a control portion
- the first portion of the graphical program may comprise a control portion
- the second portion of the graphical program may comprise an image processing portion
- the image acquisition device executing to perform the image processing function on the image in 708 may comprise the processor 212 in the image acquisition device 134 executing the executable program from the memory 214 to implement the first portion and the programmable hardware element 206 executing the second portion of the graphical program.
- the image acquisition device 134 may include a first processor 212 A, a second processor 212 B, and at least one memory 214 .
- the image acquisition device executing to perform the image processing function on the image may comprise the first processor 212 A in the image acquisition device 134 executing a first portion of the graphical program from the at least one memory 214 and the second processor 212 B in the image acquisition device 134 executing a second portion of the graphical program from the at least one memory 214 .
- both the first portion and the second portion of the graphical program are executable by a processor from memory.
- the first portion of the graphical program may comprise an image processing portion, and the second portion of the graphical program may comprise a control portion.
- the first portion of the graphical program may comprise a first portion of the image processing function, and the second portion of the graphical program may comprise a second portion of the image processing function.
- the functional unit may comprise two or more programmable hardware elements which operate as above.
- the image acquisition device may comprise a smart camera, such as described above with reference to FIG. 2B , and FIGS. 6 A-C.
- the image acquisition device may comprise an image acquisition board coupled to or comprised in a computer system, as shown in FIG. 2A , and FIGS. 5 A-C.
- FIGS. 8A and 8B More detailed embodiments of this method are presented below with reference to FIGS. 8A and 8B .
- FIG. 8A Configuring an Image Acquisition Device or Smart Camera Including Programmable Hardware to Perform an Image Processing Function
- FIG. 8A is an exemplary embodiment illustrating a method for configuring an image acquisition device 134 , such as smart camera 190 , with a graphical program to perform an image processing function, where the image acquisition device 134 (or smart camera 190 ) includes a functional unit, more specifically, a programmable hardware element 206 .
- the method shown in FIG. 8A is a more detailed version of the method of FIG. 7 described above, wherein the method is described with reference to an image acquisition device or smart camera. It should be noted that in various embodiments, various steps in the method may occur concurrently, or in a different order than shown. Additionally, some steps may be omitted, or additional steps performed which are not shown, as desired. As shown, this method may operate as follows.
- a graphical program may be created on the computer system 102 (or on a different computer system).
- the graphical program may be created or assembled by the user arranging on a display a plurality of nodes or icons and then interconnecting the nodes to create the graphical program.
- data structures and/or program instructions
- the nodes may be interconnected in one or more of a data flow, control flow, or execution flow format.
- the graphical program may thus comprise a plurality of interconnected nodes or icons which visually indicates the functionality of the program.
- the graphical program may comprise a block diagram and may also include a user interface portion or front panel portion. Where the graphical program includes a user interface portion, the user may assemble the user interface on the display. As one example, the user may use the LabVIEW graphical programming development environment to create the graphical program.
- the graphical program may be created in step 862 by the user creating or specifying a prototype, followed by automatic or programmatic creation of the graphical program from the prototype.
- This functionality is described in U.S. patent application Ser. No. 09/587,682 titled “System and Method for Automatically Generating a Graphical Program to Perform an Image Processing Algorithm”, incorporated by reference above.
- the graphical program may be created in other manners, either by the user or programmatically, as desired.
- the instrument is an image acquisition device 134 (e.g., smart camera 190 )
- the graphical program may implement an image processing function.
- the image acquisition device 134 may be coupled to the computer system 102 .
- the image acquisition device 134 e.g., smart camera 190
- the smart camera 190 may comprise a camera and a programmable hardware element 206 . It is noted that the image acquisition device may be connected to the computer system 102 before, during or after the graphical program is created.
- a hardware configuration based on the graphical program may be downloaded onto the programmable hardware element 206 in the image acquisition device 134 (e.g., smart camera 190 ) to configure the programmable hardware element 206 .
- the image acquisition device 134 is coupled to the computer system 102 over a network
- deploying the graphical program may comprise the computer system 102 deploying the graphical program over the network to the image acquisition device 134 .
- the hardware configuration corresponds to a hardware implementation of the graphical program.
- downloading the hardware configuration onto the programmable hardware element 206 may comprise the following steps: generating a hardware description based on the graphical program, where the hardware description describes a hardware implementation of the graphical program; converting the hardware description into a netlist; compiling the netlist format into a hardware program file; and downloading the hardware program file to the programmable hardware element to configure the programmable hardware element. These steps are discussed below.
- the programmable hardware element is configured with a hardware implementation of the graphical program.
- the image acquisition device e.g., smart camera 190
- the smart camera 190 may be optionally disconnected from the computer system 102 , and may possibly be deployed for operation.
- the smart camera 190 may be disconnected from the computer system 102 and deployed in a vision application, such as a manufacturing vision inspection application, a manufacturing assembly application, or other vision application.
- the image acquisition device is an image acquisition card or board that is designed to be located in a slot of the computer 102
- the image acquisition card or board 134 may optionally remain in the computer system 102 for use, or may be transferred to a different computer system.
- the method described above may be repetitively used to manufacture a plurality of image acquisition devices 134 (e.g., smart camera 190 ) for later sale.
- the method described above may be used to configure an image acquisition device 134 that has already been deployed, where the image acquisition device 134 is configured over a network, i.e., where the hardware configuration is downloaded onto the image acquisition device 134 (or other instrument) over a network.
- the image acquisition device 134 can be used in an application.
- the image acquisition device 134 or smart camera 190 may acquire an image of an object in step 870 , e.g., from a camera 132 , or the camera in smart camera 190 may acquire the image.
- the programmable hardware element 206 in the image acquisition device 134 may then execute in step 872 to perform the image processing function on the image.
- the programmable hardware element 206 in the image acquisition device 134 executes to perform the image processing function on the image
- this user interface portion may optionally be presented on a display during this time in step 874 .
- the code corresponding to the user interface portion may be executed by a processor 160 in the computer system 102 or by a processor 212 on the image acquisition device 134 .
- the user interface portion may operate as a front panel for the image acquisition device 134 .
- the user may optionally provide user input to the user interface portion on the display to control the image acquisition device 134 while the programmable hardware element 206 in the image acquisition device 134 executes to perform the image processing function on the image.
- the user interface portion may be compiled into executable code for execution by a processor ( 160 or 212 ) and stored in a memory ( 166 or 214 ) accessible by the processor.
- the processor may then execute the executable code (the user interface portion) from the memory to present the user interface portion on the display.
- the processor 212 in the image acquisition device may execute the executable code from the memory 214 to present the user interface portion on the display during the image acquisition device 134 (or smart camera 190 ) executing to perform the image processing function on the image.
- the processor 160 of the computer system 102 may execute the executable code from the memory 166 to present the user interface portion on the display during the image acquisition device 134 (or smart camera 190 ) executing to perform the image processing function on the image.
- FIG. 8B Configuring an Image Acquisition Device or Smart Camera to Perform an Image Processing Function
- FIG. 8B is an exemplary embodiment illustrating a method for configuring an image acquisition device 134 , such as smart camera 190 , with a graphical program to perform an image processing function, where the image acquisition device 134 (or smart camera 190 ) includes a functional unit, more specifically, a processor 212 and memory 214 .
- the method shown in FIG. 8B is another, more detailed version of the method of FIG. 7 described above, and is similar to the method described above with reference to FIG. 8A , differing from that method only in steps 866 and 872 .
- an abbreviated description is provided.
- various steps in the method may occur concurrently, or in a different order than shown. Additionally, some steps may be omitted, or additional steps performed which are not shown, as desired. As shown, this method may operate as follows.
- a graphical program may be created on the computer system 102 (or on a different computer system), as described above with reference to FIG. 8A , step 862 .
- step 864 the image acquisition device 134 (e.g., smart camera 190 ) may be coupled to the computer system 102 , such as over a network, communication cable(s), through wireless means, or by any other method of coupling, as described above with reference to FIG. 8A , step 864 .
- the image acquisition device 134 e.g., smart camera 190
- the computer system 102 such as over a network, communication cable(s), through wireless means, or by any other method of coupling, as described above with reference to FIG. 8A , step 864 .
- the graphical program may be downloaded to the memory 214 of the image acquisition device 134 (e.g., smart camera 190 ).
- downloading the graphical program onto the memory 214 may comprise the following steps: generating an executable program (an executable program) based on the graphical program which implements the functionality of the graphical program; and transferring the executable program to the memory 214 on the image acquisition device.
- downloading the graphical program onto the memory 214 may comprise downloading the graphical program (or executable program) onto the memory 214 over a network.
- the image acquisition device 134 is configured to perform the image processing function implemented by the graphical program.
- the image acquisition device 134 e.g., smart camera 190
- the image acquisition device 134 may be optionally disconnected from the computer system 102 , and may possibly be deployed for operation, as described above in step 868 of FIG. 8A .
- the image acquisition device 134 e.g., smart camera 190
- the image acquisition device 134 can be used in an application.
- the image acquisition device 134 or smart camera 190 may acquire an image of an object in step 870 , e.g., from a camera 132 , or the camera in smart camera 190 may acquire the image.
- the image acquisition device 134 may then execute to perform the image processing function on the image, as indicated in step 873 .
- this user interface portion may optionally be presented on a display during this time in step 874 , as described above in step 874 of FIG. 8A .
- the user may optionally provide user input to the user interface portion on the display to control the image acquisition device while the image acquisition device 134 (i.e., the processor 212 and memory 214 ) executes to perform the image processing function on the image.
- the image acquisition device 134 i.e., the processor 212 and memory 214
- the image acquisition device executing to perform the image processing function on the image in 708 may comprise the processor 212 in the image acquisition device 134 executing the graphical program from the memory 214 .
- the image acquisition device executing to perform the image processing function on the image may comprise the processor in the image acquisition device executing the executable program (in executable form) from the memory 214 .
- the processor 212 in the image acquisition device may execute the executable code from the memory 214 to present the user interface portion on the display during the image acquisition device 134 (or smart camera 190 ) executing to perform the image processing function on the image.
- the processor 212 may execute both portions of the graphical program concurrently, such as by threading or multi-tasking.
- the image acquisition device 134 may include multiple processors, e.g., 212 A and 212 B, in which case the first portion of the graphical program (such as the user interface portion) may be executed by processor 212 A from memory 214 (or a first memory 214 A), while the second portion (such as the image processing portion) may be executed by processor 212 B from memory 214 (or a second memory 214 B), or from image memory 284 .
- processors e.g., 212 A and 212 B
- the first portion of the graphical program such as the user interface portion
- the second portion such as the image processing portion
- the graphical program which implements the image processing function may already have been created, for example, by a third party.
- the method may then comprise coupling the image acquisition device 134 to a computer system 102 which stores the graphical program, where the image acquisition device comprises a functional unit; and deploying the graphical program onto the functional unit in the image acquisition device 134 to configure the functional unit; where after the deploying, the functional unit is operable to implement the graphical program.
- step 862 and possibly steps 868 forward, may be omitted from the methods shown in FIGS. 8A and 8B .
- FIG. 9 Block Diagram of Conversion of a Graphical Program Into Hardware and Software Descriptions
- FIG. 9 is a block diagram illustrating the conversion of a graphical program, such as a graphical image processing or machine vision program, into hardware and software descriptions.
- the graphical program 300 may comprise graphical code, such as interconnected function nodes or icons.
- the graphical code in the graphical program may use graphical data flow, graphical control flow and/or graphical execution flow constructs, as noted above.
- the graphical program On the display, the graphical program may be represented as interconnected icons or function nodes.
- the graphical program 300 may comprise data structures (or scripts or code) representing functional operations, data flow and/or control flow, and execution order.
- the data structures or scripts or code
- the data structures may be automatically created and stored in memory.
- the graphical program 300 may be created with various development tools.
- the graphical program may be created using the following development systems: LabVIEW, BridgeVIEW, DASYLab, Visual Designer, HP VEE (Visual Engineering Environment), Snap-Master, GFS DiaDem, ObjectBench, Simulink, WiT, Vision Program Manager, Hypersignal, VisiDAQ, Vis Sim, Truly Visual, and Khoros, among others.
- graphical program 300 is a LabVIEW graphical program or virtual instrument (VI).
- programs of the present invention may create a VDiagram tree 302 from the data structures of the graphical program 300 .
- the VDiagram tree 302 is an abstract hardware graph which represents at least a portion of the graphical program 300 .
- the graph may be organized in a way that facilitates the generation of specific types of descriptions by back end programs of the present invention.
- the graphical programming system automatically creates and stores a VDiagram tree 302 (abstract hardware graph) in response to a user's creation of a graphical program. In this instance, conversion from graphical program data structures to a VDiagram tree is not necessary.
- a hardware description 304 may be generated from the abstract hardware graph 302 by a back end program.
- the hardware description 304 may be in any of various hardware description languages such as VHDL, EDIF, and Verilog.
- the hardware description 304 comprises one or more VHDL files.
- a hardware netlist 306 may be generated from the hardware description using various synthesis tools.
- the term “netlist” comprises various intermediate hardware-specific description formats comprising information regarding the particular hardware elements required to implement a hardware design and the relationship among those elements.
- the hardware netlist 306 is an FPGA-specific netlist.
- the hardware netlist 306 is used to create or configure one or more functional hardware devices or hardware elements 308 which are configured to execute the portion of the graphical program 300 that is represented by the abstract hardware graph 302 .
- Hardware element 308 may comprise any of various devices.
- hardware 308 may comprise a programmable logic device (PLD) such as an FPGA or CPLD.
- PLD programmable logic device
- hardware 308 may comprise other types of hardware devices, such as a traditional circuit board which is created using the hardware netlist 306 .
- hardware 308 is an interface card comprising an FPGA, where the interface card is comprised in the computer system where the graphical program 300 is created.
- the hardware 308 may also be comprised in an external device connected to the computer system where the graphical program 300 is created.
- the hardware 308 may be connected to the computer over an external serial or parallel bus, or over a network, such as the Internet.
- software description source code 310 may also be generated from the abstract hardware graph 302 by a back end program.
- the source code 310 may be in various source code languages such as C, C++, Java, etc.
- Machine code 312 may be produced from the source code 310 using various source code compilers.
- Linked machine code 314 may be produced from the machine code 312 using various machine code linkers. The linked machine code 314 is executable to perform the operations of the portion of the graphical program 300 that is represented by the abstract hardware graph 302 .
- FIG. 10 Block Diagram of Generation of Hardware and Software Descriptions from a Vdiagram Tree
- FIG. 10 is a block diagram illustrating the generation of various types of hardware and software descriptions from a VDiagram tree.
- programs of the present invention create a VDiagram tree 302 from a graphical program 300 .
- the VDiagram tree 302 represents at least a portion of the graphical program 300 .
- Back end programs 330 generate hardware descriptions from the VDiagram tree 302 .
- Exemplary back end programs 330 A, 330 B, and 330 C are illustrated.
- Back end 330 A generates a VHDL hardware description comprising one or more VHDL files.
- Back end 330 B generates an EDIF hardware description comprising one or more EDIF files.
- Back end 330 C generates a C source code software description comprising one or more C files. It is also contemplated that other back ends may be used to generate hardware descriptions in other hardware and/or software programming languages.
- back end programs that may be present are not limited.
- one or more back end programs may be called automatically as part of a process initiated by a user to generate hardware/software descriptions for the graphical program 300 .
- the VDiagram tree 302 may be generated and saved to a file, and the user may call a back end program at a later time to generate a hardware/software description.
- appropriate synthesis tools or compilers may be called to convert a hardware/software description into another format such as an FPGA-specific netlist or compiled machine code.
- FIG. 11 Block Diagram of Conversion of a Graphical Program into a Hardware Description for an FPGA
- FIG. 11 illustrates the exportation of at least a portion of a graphical program 300 into a hardware description and the use of the hardware description to program an FPGA.
- the VDiagram tree 302 comprises information representing the graphical program 300 , including the functional operations of the program.
- the VDiagram tree comprises VDiagrams, each of which maintains a list of components. This list of components includes components which represent functional operations.
- a back end program converts the VDiagram tree 302 to a hardware description 304 .
- Back end programs may implement the functionality of the components in the VDiagram component lists using constructs of their respective description languages. For example, a VHDL back end may create VHDL code to implement a component that performs a particular mathematical algorithm such as an exponential calculation. However, in one embodiment, such functional components are simply referenced as library components.
- FIG. 11 illustrates one embodiment in which the VDiagram tree references one or more library components.
- One embodiment of the present invention comprises pre-compiled function blocks 342 which implement these library components for particular hardware devices such as FPGAs.
- Various FPGA netlist synthesis tools may be called to generate an FPGA netlist 340 from the hardware description 304 . These synthesis tools may incorporate the pre-compiled function blocks 342 into the FPGA netlist 340 . Also, as shown, the synthesis tools may utilize hardware target-specific information in creating the netlist. For example, the exact form that the FPGA netlist takes may depend on the particular type of FPGA that will use the netlist, since FPGAs differ in their available resources.
- An FPGA bit stream program file 346 may be generated from the FPGA netlist 340 using readily available synthesis tools. This FPGA program file may be uploaded to an FPGA 348 .
- the FPGA 348 may be comprised in a hardware device such as an interface board.
- the FPGA is able to execute the portion of the graphical program 300 that is exported to the hardware description 304 . If the entire graphical program is not exported to the hardware description, then a portion of the program may execute on the general purpose CPU of the computer system. This portion preferably comprises the supervisory control and display portion of the program. Details follow on how the execution of the FPGA portion is coordinated with the execution of the main CPU portion and how the external hardware resource requirements for the FPGA portion are managed.
- FIG. 12 Block Diagram of Conversion of a Graphical Program into a Software Source Code Description with Compilation and Linkage
- FIG. 12 illustrates the exportation of at least a portion of a graphical program 300 into a software source code description and the compilation and linkage of the source code.
- the graphical program data structures may be first converted to a VDiagram tree 302 and then to software description source code 310 .
- the VDiagram tree 302 references library components to represent various functional components of the graphical program. These library components may be implemented in libraries, class libraries, macro definitions, etc. 360 . As shown in FIG. 12 , these class libraries, etc. may be used to produce the machine code 312 from the source code 310 . Also, binary object libraries 362 may implement some functionality of the software description. These binary object libraries may be linked in with the machine code 312 is linked to produce the linked executable code 314 . Libraries 360 and 362 may also contain compiler-specific or platform-specific information necessary to produce executable code 314 . Linked code 314 may be executed to perform the operations of the portion of the graphical program that is exported to the software source code description 310 .
- FIG. 13 Conversion of a Graphical Program into a Hardware Implementation
- FIG. 13 is a flowchart diagram illustrating operation of one embodiment of the present invention.
- One embodiment of the present invention comprises a computer-implemented method for generating hardware and/or software implementations of graphical programs or graphical code. It is noted that various of the steps in the flowchart can occur concurrently or in different orders.
- One goal of the present invention is to provide a development environment that will seamlessly allow use of a graphical programming system to design applications for reconfigurable or programmable hardware.
- the graphical programming system is LabVIEW
- the present invention allows LabVIEW users to design applications in LabVIEW for reconfigurable hardware.
- the present invention solves this problem by allowing a user to convert their graphical program, e.g., a G program, into application-specific hardware such as a programmed FPGA.
- the hardware maintains the exact functionality of the graphical program while running at speeds far exceeding that of traditional general-purpose processor platforms.
- One current implementation of the present invention is a desktop or embedded PC that contains an FPGA-based and/or processor based card or board.
- the system appears as a conventional graphical programming system while providing a seamless interface to the reconfigurable hardware.
- the preferred embodiment of the invention referred to as “FPGA LabVIEW”, provides a seamless interface to an FPGA.
- FPGA LabVIEW appears from the outside to be exactly the same as the normal LabVIEW graphical program development system.
- FIG. 13 illustrates one embodiment of the translation process from a graphical program to a hardware description that corresponds to the graphical program.
- a graphical programming application that is being targeted for a hardware implementation is designed in the same way as an ordinary graphical programming application.
- the user first creates a graphical program, also sometimes referred to as a block diagram. As described above, a design may be entered and debugged in the traditional software-based manner.
- the graphical program comprises a graphical data flow diagram which specifies functionality of the program to be performed. This graphical data flow diagram is preferably directly compilable into machine language code for execution on a computer system.
- the present system in one embodiment is capable of bisecting a design into hardware portions and software portions.
- Steps 404 - 414 are an example implementation of step 866 of FIG. 8A .
- the user may optionally select a first portion of the graphical program for conversion to a hardware implementation.
- This first portion of the graphical program which is desired for hardware implementation preferably comprises portions of the graphical program, e.g., particular subprograms, which require a fast or deterministic implementation and/or are desired to execute in a stand-alone hardware unit.
- portions of the graphical program which are desired to have a faster or more deterministic execution are selected in step 404 and converted into the hardware implementation in steps 406 - 414 .
- a default case is that the entire block diagram portion of the graphical program is selected for hardware implementation.
- step 422 the remaining portions of the graphical program which were not selected in step 404 , if any, may be compiled into machine code for execution on a CPU, such as the host processor in the computer 102 or the processor 212 comprised on the interface card 134 .
- the first portion of the program selected in step 404 preferably excludes program portions involving supervisory control and display. This enables the supervisory control and display portions to execute on the host CPU, which is optimal for these elements of the program.
- the user may specify portions, e.g. subprograms, which are to be exported to the hardware description format for conversion into a hardware implementation.
- portions e.g. subprograms
- the user selects which modules or subprograms to export to the hardware implementation at the time when the conversion process is initiated.
- the entire graphical program may be selected for conversion to a hardware implementation, and thus step 422 may not be performed.
- the graphical program portion selected in step 404 may first be processed to create an abstract hardware graph called a VDiagram tree, described above, which serves as an intermediate data structure.
- the VDiagram tree may contain a complete hardware representation of the program, but is not specific to any hardware description language.
- the VDiagram tree may comprise data structures representing hardware signals that implement the data flow within the graphical program, as well as data structures representing hardware signals that are added to preserve the proper execution flow (enable signals).
- a back end program may be called to parse the VDiagram tree and generate a hardware description from it.
- the back end may translate the information contained in the VDiagram tree into a specific hardware description language.
- a VHDL back end may be called to generate a VHDL file or set of files describing the program.
- the hardware description comprises a high-level hardware description of function blocks, logic, inputs, and outputs which perform the operation indicated by the portion of the graphical program selected in step 404 .
- Back end programs may generate software source code descriptions as well as hardware description language descriptions.
- FIG. 10 illustrates a back end 330 A which uses the VDiagram tree to generate one or more VHDL files; back end 330 B which generates one or more EDIF files; and back end 330 C which generates one or more C files. These three back ends are representative only. Other back ends may generate other types of descriptions for the program. For example, a Verilog back end may generate a Verilog file for the program. Also, more than one back end may be called to generate different program descriptions. In the preferred embodiment, a VHDL back end generates a VHDL description which may then be compiled and used to program a programmable logic device such as an FPGA.
- the method may operate to convert the hardware description into an FPGA-specific netlist.
- the netlist describes the components required to be present in the hardware as well as their interconnections. Conversion of the hardware description into the FPGA-specific netlist is preferably performed by any of various types of commercially available synthesis tools, such as those available from Xilinx, Altera, etc.
- the converting step 410 may utilize one or more pre-compiled function blocks from a library of pre-compiled function blocks 342 .
- the hardware description created in step 408 includes a reference to a pre-compiled function block from the library 342 .
- the hardware implementations for all of the function blocks are included in the function library.
- the respective pre-compiled function blocks are simply inserted into the netlist in place of these references in step 410 .
- the preferred embodiment of the invention thus includes the library 342 of pre-compiled function blocks, also referred to as the component library, which are used in creating the netlist.
- the preferred embodiment also includes hardware target specific information 344 which is used by step 410 in converting the hardware description into a netlist which is specific to a certain type or class of FPGA.
- the method may operate to compile the netlist into an FPGA program file, also referred to as a software bit stream.
- the FPGA program file is a file that can be readily uploaded to program an FPGA.
- step 414 the method may operate to transfer the FPGA program file to the FPGA, to produce a programmed hardware equivalent to the graphical program.
- the portion of a graphical program referenced in step 404 is comprised as a hardware implementation in an FPGA or other programmable hardware element.
- the hardware description is passed transparently through the FPGA vendor's synthesis tools. Because the vendor's tools may take a considerable amount of time to process the design and generate a programming bitstream, it is recommended that this only be done after the design has been debugged using traditional software-compilation techniques.
- the present invention may run on PC computers equipped with an FPGA-based expansion card on the PCI bus.
- Embodiments of the FPGA-based expansion card were described with reference to FIGS. 5A-5C (and possibly 6 A- 6 D).
- the graphical programming system may upload the programming bitstream generated by the FPGA vendor's design tools into the FPGA on this board.
- the FPGA then may begin processing data, and the graphical programming system may coordinate data flow between the FPGA and the host CPU.
- steps 410 and 412 can be combined into a single step, as can steps 404 - 410 .
- the user simply selects a hardware export option and indicates the hardware target or destination, causing steps 404 - 414 to be automatically performed.
- FIG. 13 applies to the preferred embodiment in which the programmable hardware element is an FPGA. However, the same or similar steps may be applied to convert a graphical program into a hardware implementation for other types of programmable or (re)configurable hardware, such as a CPLD.
- FIG. 14 Creation of a Graphical Program
- FIG. 14 is a more detailed flowchart diagram of step 862 of FIGS. 8A and 13 , illustrating creation of a graphical program according to one embodiment of the invention.
- the user may arrange on a screen (i.e., a display device) a graphical program or block diagram. This may include the user placing and connecting, e.g., wiring, various icons or nodes on the display screen in order to configure a graphical program. More specifically, the user may select various function icons or other icons and place or drop the icons in a block diagram panel, and then connect or “wire up” the icons to assemble the graphical program.
- the user may also assemble a user interface, which may be referred to as a front panel, comprising controls and indicators which indicate or represent input/output to/from the graphical program.
- a graphical program targeted to measurement or automation applications may be referred to as a virtual instrument (VI).
- the graphical program or VI may have a hierarchy of sub-graphical programs or sub-VIs.
- the graphical programming system is the LabVIEW graphical programming system available from National Instruments.
- LabVIEW graphical programming system available from National Instruments.
- the method may operate in step 432 to develop and store a tree of data structures which represent the graphical program.
- the graphical programming system may operate to develop and store a tree of data structures which represent the graphical program. More specifically, as the user assembles each individual node and wire, the graphical programming system may operate to develop and store (or populate) a corresponding data structure in the tree of data structures which represents the individual portion of the graphical program that was assembled.
- steps 430 and 432 may be an iterative process which is repetitively performed as the user creates the graphical program.
- the graphical programming system may automatically develop and store VDiagram data structures in response to the user creating the graphical program.
- scripts, DLLs, or other code may be created in memory.
- the user may optionally place constructs in the graphical program which indicate respective portions of graphical code which are either to be compiled into machine code for execution by a CPU or converted to a hardware description for implementation in a programmable hardware device such as an FPGA.
- the various methods described above may operate to deploy a graphical program implementing a function, such as an image processing or machine vision function, onto a programmable or configurable device, such as an image acquisition device, e.g., an image acquisition board or smart camera, thereby enabling the device to perform the function.
- a graphical program implementing a function such as an image processing or machine vision function
- a programmable or configurable device such as an image acquisition device, e.g., an image acquisition board or smart camera
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Evolutionary Computation (AREA)
- Quality & Reliability (AREA)
- Stored Programmes (AREA)
- Studio Devices (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
A computer-implemented system and method for deploying a graphical program onto an image acquisition (IMAQ) device. The method may operate to configure an image acquisition (IMAQ) device to perform image processing or machine vision functions, wherein the device includes a programmable hardware element and/or a processor and memory. The method comprises first creating a graphical program which implements the image processing or machine vision function. A portion of the graphical program may be converted into a hardware implementation on a programmable hardware element, and a portion may optionally be compiled into machine code for execution by a CPU. The programmable hardware element is thus configured utilizing a hardware description and implements a hardware implementation of at least a portion of the graphical program. The CPU-executable code may be executed by a computer coupled to the IMAQ device, or by a processor/memory on the IMAQ device.
Description
- This application is a divisional application of U.S. patent application Ser. No. 09/949,783 titled “System and Method for Deploying a Graphical Program on an Image Acquisition Device”, filed Sep. 10, 2001, whose inventors are Kevin L. Schultz, Jeffrey L. Kodosky, Hugo Andrade, Brian Keith Odom and Cary Paul Butler,
-
- which is a continuation-in-part of U.S. patent application Ser. No. 09/891,571 titled “System and Method for Configuring an Instrument to Perform Measurement Functions Utilizing Conversion of Graphical Programs into Hardware Implementations” filed on Jun. 25, 2001, whose inventors are Jeffrey L. Kodosky, Hugo Andrade, Brian Keith Odom, Cary Paul Butler, and Kevin L. Schultz,
- which is a continuation-in-part of U.S. patent application Ser. No. 09/499,503 titled “System and Method for Configuring a Programmable Hardware Instrument to Perform Measurement Functions Utilizing Estimation of the Hardware Implementation and Management of Hardware Resources”, filed on Feb. 7, 2000, whose inventors are Jeffrey L. Kodosky, Hugo Andrade, Brian Keith Odom, Cary Paul Butler, and Andrew Mihal,
- which is a continuation-in-part of U.S. patent application Ser. No. 08/912,427 titled “System and Method for Configuring an Instrument to Perform Measurement Functions Utilizing Conversion of Graphical Programs into Hardware Implementations”, filed on Aug. 18, 1997, whose inventors are Jeffrey L. Kodosky, Hugo Andrade, Brian Keith Odom and Cary Paul Butler, which is now U.S. Pat. No. 6,219,628.
- The present invention relates to graphical programming, and in particular to a system and method for deploying a graphical program onto an image acquisition device. The present invention further relates to a system and method for configuring an image acquisition or machine vision device to perform image processing functions, wherein the instrument or device includes a programmable hardware element and/or a processor and memory.
- Traditionally, high level text-based programming languages have been used by programmers in writing applications programs. Many different high level programming languages exist, including BASIC, C, FORTRAN, Pascal, COBOL, ADA, APL, etc. Programs written in these high level languages are translated to the machine language level by translators known as compilers or interpreters. The high level programming languages in this level, as well as the assembly language level, are referred to as text-based programming environments.
- Increasingly computers are required to be used and programmed by those who are not highly trained in computer programming techniques. When traditional text-based programming environments are used, the user's programming skills and ability to interact with the computer system often become a limiting factor in the achievement of optimal utilization of the computer system.
- There are numerous subtle complexities which a user must master before he can efficiently program a computer system in a text-based environment. The task of programming a computer system to model or implement a process often is further complicated by the fact that a sequence of mathematical formulas, mathematical steps or other procedures customarily used to conceptually model a process often does not closely correspond to the traditional text-based programming techniques used to program a computer system to model such a process. In other words, the requirement that a user program in a text-based programming environment places a level of abstraction between the user's conceptualization of the solution and the implementation of a method that accomplishes this solution in a computer program. Thus, a user often must substantially master different skills in order to both conceptually model a system and then to program a computer to model that system. Since a user often is not fully proficient in techniques for programming a computer system in a text-based environment to implement his model, the efficiency with which the computer system can be utilized to perform such modeling often is reduced.
- Examples of fields in which computer systems are employed to interact with physical systems are the fields of instrumentation, process control, industrial automation, and simulation. Computer measurement and control of devices such as instruments or industrial automation hardware has become increasingly desirable in view of the increasing complexity and variety of instruments and devices available for use. However, due to the wide variety of possible testing and control situations and environments, and also the wide array of instruments or devices available, it is often necessary for a user to develop a custom program to control a desired system.
- As discussed above, computer programs used to control such systems traditionally had to be written in text-based programming languages such as, for example, assembly language, C, FORTRAN, BASIC, etc. Traditional users of these systems, however, often were not highly trained in programming techniques and, in addition, text-based programming languages were not sufficiently intuitive to allow users to use these languages without training. Therefore, implementation of such systems frequently required the involvement of a programmer to write software for control and analysis of instrumentation or industrial automation data. Thus, development and maintenance of the software elements in these systems often proved to be difficult.
- U.S. Pat. Nos. 4,901,221; 4,914,568; 5,291,587; 5,301,301; and 5,301,336; among others, to Kodosky et al disclose a graphical system and method for modeling a process, i.e., a graphical programming environment which enables a user to easily and intuitively model a process. The graphical programming environment disclosed in Kodosky et al can be considered a higher and more intuitive way in which to interact with a computer. A graphically based programming environment can be represented at a level above text-based high level programming languages such as C, Basic, Java, etc.
- The method disclosed in Kodosky et al allows a user to construct a diagram using a block diagram editor. The block diagram may include a plurality of interconnected icons such that the diagram created graphically displays a procedure or method for accomplishing a certain result, such as manipulating one or more input variables and/or producing one or more output variables. In response to the user constructing a diagram or graphical program using the block diagram editor, data structures and/or program instructions may be automatically constructed which characterize an execution procedure that corresponds to the displayed procedure. The graphical program may be compiled or interpreted by a computer.
- Therefore, Kodosky et al teaches a graphical programming environment wherein a user places or manipulates icons and interconnects or “wires up” the icons in a block diagram using a block diagram editor to create a graphical “program.” A graphical program for performing an instrumentation, measurement or automation function, such as measuring a Unit Under Test (UUT) or device, controlling or modeling instruments, controlling or measuring a system or process, or for modeling or simulating devices, may be referred to as a virtual instrument (VI). Thus, a user can create a computer program solely by using a graphically based programming environment. This graphically based programming environment may be used for creating virtual instrumentation systems, modeling processes, control, simulation, and numerical analysis, as well as for any type of general programming.
- A graphical program may have a graphical user interface. For example, in creating a graphical program, a user may create a front panel or user interface panel. The front panel may include various graphical user interface elements or front panel objects, such as user interface controls and/or indicators, that represent or display the respective input and/or output that will be used by the graphical program or VI, and may include other icons which represent devices being controlled. The front panel may be comprised in a single window of user interface elements, or may comprise a plurality of individual windows each having one or more user interface elements, wherein the individual windows may optionally be tiled together. When the controls and indicators are created in the front panel, corresponding icons or terminals may be automatically created in the block diagram by the block diagram editor. Alternatively, the user can place terminal icons in the block diagram which may cause the display of corresponding front panel objects in the front panel, either at edit time or later at run time. As another example, the front panel may comprise front panel objects, e.g., the GUI, embedded in the block diagram.
- During creation of the block diagram portion of the graphical program, the user may select various function nodes or icons that accomplish his desired result and connect the function nodes together. For example, the function nodes may be connected in one or more of a data flow, control flow, and/or execution flow format. The function nodes may also be connected in a “signal flow” format, which is a subset of data flow. The function nodes may be connected between the terminals of the various user interface elements, e.g., between the respective controls and indicators. Thus the user may create or assemble a graphical program, referred to as a block diagram, graphically representing the desired process. The assembled graphical program may be represented in the memory of the computer system as data structures and/or program instructions. The assembled graphical program, i.e., these data structures, may then be compiled or interpreted to produce machine language that accomplishes the desired method or process as shown in the block diagram.
- Input data to a graphical program may be received from any of various sources, such as from a device, unit under test, a process being measured or controlled, another computer program, or from a file. Also, a user may input data to a graphical program or virtual instrument using a graphical user interface, e.g., a front panel as described above. The input data may propagate through the data flow block diagram or graphical program and appear as changes on the output indicators. In an instrumentation application, the front panel can be analogized to the front panel of an instrument. In an industrial automation application the front panel can be analogized to the MMI (Man Machine Interface) of a device. The user may adjust the controls on the front panel to affect the input and view the output on the respective indicators. Alternatively, the front panel may be used merely to view the input and output, or just the output, and the input may not be interactively manipulable by the user during program execution.
- Thus, graphical programming has become a powerful tool available to programmers. Graphical programming environments such as the National Instruments LabVIEW product have become very popular. Tools such as LabVIEW have greatly increased the productivity of programmers, and increasing numbers of programmers are using graphical programming environments to develop their software applications. In particular, graphical programming tools are being used for test and measurement, data acquisition, process control, man machine interface (MMI), supervisory control and data acquisition (SCADA) applications, simulation, image processing/machine vision applications, and motion control, among others.
- A primary goal of graphical programming, including virtual instrumentation, is to provide the user the maximum amount of flexibility to create his/her own applications and/or define his/her own instrument functionality. In this regard, it is desirable to extend the level at which the user is able to program a device, e.g., extend the level at which a user of instrumentation or industrial automation hardware is able to program an instrument. The evolution of the levels at which the user has been able to program an instrument is essentially as follows.
-
- 1. User level software (LabVIEW, LabWindows CVI, Visual Basic, etc.)
- 2. Kernel level software
- 3. Auxiliary kernel level software (a second kernel running along side the main OS, e.g., InTime, VentureCom, etc.)
- 4. Embedded kernel level software
- 5. Hardware level software (FPGA)
- In general, going down the above list, the user is able to create software applications which provide a more deterministic real-time response. Currently, some programming development tools for instrumentation or industrial automation provide an interface at level 1 above. In general, most users are unable and/or not allowed to program at the kernel level or auxiliary kernel level.
- It would be highly desirable to provide the user with the ability to develop user level software which operates at the embedded kernel level and/or the hardware level. More particularly, it would be desirable to provide the user with the ability to develop high level software, such as graphical programs, which can then be readily deployed onto an image acquisition device, such as a smart camera. This would provide the user with the dual benefits of being able to program device functionality at the highest level possible (e.g., graphical programs), while also providing the ability to have the created program operate directly on an embedded processor or in hardware for increased speed and efficiency.
- One embodiment of the present invention comprises a computer-implemented system and method for deploying a graphical program onto an image acquisition device, such as an image acquisition board coupled to or included in a computer system, or a smart camera. This provides the user the ability to develop or define desired functionality using graphical programming techniques, while enabling the resulting program to operate directly on an image acquisition device, such as a smart camera. The computer-implemented system and method may be used for configuring an image acquisition device or smart camera instrument to perform an image processing or machine vision function, where the image acquisition device (e.g., smart camera) includes at least one functional unit, such as a processor and memory, or a programmable hardware element.
- The user may first create a graphical program which performs or represents the desired functionality. The graphical program may comprise a block diagram which includes a plurality of interconnected nodes which visually indicate functionality of the graphical program. The plurality of nodes may be interconnected in one or more of a data flow, control flow, or execution flow format. The graphical program may include a single diagram or a hierarchy of subprograms or sub-diagrams. In one embodiment, the user may place various constructs in portions of the graphical program to aid in conversion of these portions into hardware form. As the user creates or assembles the graphical program on the display, data structures and/or software code may be automatically created and stored in memory corresponding to the graphical program being created. In an image acquisition or analysis application, the graphical program may implement an image processing function. More generally, in a measurement or instrumentation application, the graphical program may implement a measurement function. A portion of the graphical program may also implement a user interface, described below.
- Thus, in one embodiment, the graphical program may be created or assembled by the user arranging on a display a plurality of nodes or icons and then interconnecting the nodes to create the graphical program. In response to the user assembling the graphical program, data structures (and/or program instructions) may be created and stored which represent the graphical program. The graphical program may thus comprise a plurality of interconnected nodes or icons which visually indicates the functionality of the program. As noted above, the graphical program may comprise a block diagram and may also include a user interface portion or front panel portion. Where the graphical program includes a user interface portion, the user may assemble the user interface on the display.
- In one embodiment, the graphical program may be create on or stored on a computer system. The method may comprise coupling the image acquisition device to the computer system which stores the graphical program, and, where the image acquisition device includes a functional unit, deploying the graphical program onto the functional unit in the image acquisition device to configure the functional unit. After the deployment of the graphical program, the functional unit may be operable to implement the graphical program. In one embodiment, the image acquisition device may be disconnected from the computer system after deploying the graphical program. The image acquisition device may also be coupled to the computer system over a network, and deploying the graphical program may include the computer system deploying the graphical program over the network to the image acquisition device.
- In one embodiment, a computer-implemented method for configuring an image acquisition device (or smart camera) to perform an image processing function may include creating a graphical program on a computer system, as described above, where the graphical program implements the image processing function, and where the graphical program comprises a plurality of interconnected nodes which visually indicate functionality of the graphical program; deploying the graphical program on the image acquisition device, where the image acquisition device is coupled to or comprised in the computer system; the image acquisition device acquiring an image of an object; and the image acquisition device executing to perform the image processing function on the image.
- In one embodiment, the image acquisition device may include a processor and a memory, and deploying the graphical program on the image acquisition device may include transferring the graphical program to the memory on the image acquisition device, where the image acquisition device executing to perform the image processing function on the image includes the processor in the image acquisition device executing the graphical program from the memory. In one embodiment, deploying the graphical program on the image acquisition device may include generating an executable program, such as a machine language program, based on the graphical program, which implements at least a portion of the functionality of the graphical program; and transferring the executable program to the memory on the image acquisition device. In one embodiment, compiling the graphical program into an executable program may comprise converting the graphical program first into a text-language program, and then compiling the text-language program into the executable program. The image acquisition device executing to perform the image processing function on the image may include the processor in the image acquisition device executing the executable program from the memory.
- In one embodiment, the image acquisition device may include a programmable hardware element, and deploying the graphical program on the image acquisition device may include generating a hardware description based on the graphical program, which describes a hardware implementation of the graphical program, and configuring the programmable hardware element in the image acquisition device utilizing the hardware description. After being configured with the hardware description the programmable hardware element may implement a hardware implementation of the graphical program. The image acquisition device executing to perform the image processing function on the image may include the programmable hardware element in the image acquisition device executing to perform the image processing function on the image.
- In one embodiment, the image acquisition device may include a processor and a memory and a programmable hardware element. In this embodiment, deploying the graphical program on the image acquisition device may include transferring a first portion of the graphical program to the memory on the image acquisition device; generating a hardware description based on a second portion of the graphical program, which describes a hardware implementation of the graphical program; and configuring the programmable hardware element in the image acquisition device utilizing the hardware description. After being configured, the programmable hardware element may implement a hardware implementation of the second portion of the graphical program.
- The image acquisition device executing to perform the image processing function on the image may include the processor in the image acquisition device executing the first portion of the graphical program from the memory and the programmable hardware element executing the second portion of the graphical program.
- The first and second portions of the graphical program may each include image processing portions. In some embodiments, the first portion of the graphical program may include an image processing portion, and the second portion of the graphical program may include a control portion. In other embodiments, the first portion of the graphical program may include the control portion, and the second portion of the graphical program may include the image processing portion.
- In one embodiment, deploying the graphical program on the image acquisition device may include generating an executable program based on a first portion of the graphical program, and which implements functionality of the graphical program; transferring the executable program to the memory on the image acquisition device; generating a hardware description based on a second portion of the graphical program, where the hardware description describes a hardware implementation of the graphical program; and configuring the programmable hardware element in the image acquisition device utilizing the hardware description, where after being configured, the programmable hardware element implements a hardware implementation of the second portion of the graphical program. The image acquisition device executing to perform the image processing function on the image may include the processor in the image acquisition device executing the executable program from the memory and the programmable hardware element executing the second portion of the graphical program.
- In one embodiment, the image acquisition device may include a first processor, a second processor, and at least one memory, and the image acquisition device executing to perform the image processing function on the image may include the first processor in the image acquisition device executing a first portion of the graphical program from the at least one memory and the second processor in the image acquisition device executing a second portion of the graphical program from the at least one memory. For example, the first portion of the graphical program may include a first portion of the image processing function, and the second portion of the graphical program may include a second portion of the image processing function.
- During execution, a camera associated with the image acquisition device, or comprised in the smart camera, may acquire an image of an object. For example, the image acquisition device or smart camera may be used in a machine vision inspection application, or a manufacturing assembly application. The at least one functional unit in the image acquisition device or smart camera may then receive the acquired image of the object. The at least one functional unit in the image acquisition device or smart camera may then execute to perform the image processing function (or machine vision function) on the image. The results of this image processing may then be used to make a decision in a machine vision inspection application or may be used to perform an operation in a manufacturing assembly application. During execution of the at least one functional unit in the image acquisition device or smart camera, the user interface portion may be presented on a display. A user or operator may use the displayed user interface portion to view or control the image processing function (or machine vision function).
- During operation, the image acquisition device may receive an image present signal, where the image acquisition device acquires the image of the object in response to receiving the image present signal. In another embodiment, the image acquisition device may perform a control operation after executing to perform the image processing function on the image. In yet another embodiment, the image acquisition device may generate a pass/fail indication after executing to perform the image processing function on the image. Alternatively, the image acquisition device may determine characteristics of the image after performing the image processing function, and then perform an operation based on the determined characteristics of the image.
- In one embodiment, the image acquisition device may be coupled to a camera, and the image acquisition device acquiring an image of an object may include the camera acquiring the image of the object and the image acquisition device receiving and storing the image. Where the image acquisition device is a smart camera, the smart camera incorporates a camera as well as at least one functional unit.
- In one embodiment the graphical program includes a user interface portion, and the method may include presenting the user interface portion on a display and/or receiving user input to the user interface portion on the display to control the image acquisition device, during the functional unit in the image acquisition device executing to perform the image processing function on the image. For example, the user interface portion may operate as a front panel for the image acquisition device.
- In one embodiment, the method may further include compiling the user interface portion into executable code for execution by a processor and storing the executable code in a memory, and the processor executing the executable code from the memory to present the user interface portion on the display. The processor and memory may be included on the image acquisition device, and the processor in the image acquisition device may execute the executable code from the memory to present the user interface portion on the display during the image acquisition device executing to perform the image processing function on the image. In another embodiment, the processor and the memory may be included on the computer system, and the computer system may execute the executable code from the memory to present the user interface portion on the display during the image acquisition device executing to perform the image processing function on the image.
- Thus the method may operate to configure an image acquisition device, such as an image acquisition board or smart camera, with a graphical program to perform image processing or machine vision functions, where the device includes a functional unit, such as a programmable hardware element or processor/memory.
- A better understanding of the present invention can be obtained when the following detailed description of the preferred embodiment is considered in conjunction with the following drawings, in which:
-
FIG. 1 illustrates an image acquisition/machine vision system according to one embodiment of the invention; -
FIG. 2A illustrates a computer system coupled to a smart camera according to one embodiment of the invention; -
FIG. 2B illustrates a smart camera, according to one embodiment of the invention; -
FIGS. 3A and 3B illustrate image processing/machine vision systems according to embodiments of the invention; -
FIG. 4 is a block diagram of the computer system ofFIGS. 1, 2A , and 3A; -
FIGS. 5A, 5B and 5C are block diagrams illustrating an interface card or device configured with a programmable hardware element and/or a processor and memory, according to various embodiments of the present invention; -
FIGS. 6A-6C are block diagrams illustrating a smart camera configured with a programmable hardware element and/or a processor and memory, according to various embodiments of the invention; -
FIG. 7 is a flowchart diagram illustrating deployment of a graphical program on a device, according to one embodiment of the invention; -
FIG. 8A is a flowchart illustrating configuration of an image acquisition device or smart camera including a programmable hardware element, according to one embodiment of the invention; -
FIG. 8B is a flowchart illustrating configuration of an image acquisition device or smart camera including a processor and memory, according to one embodiment of the invention; -
FIG. 9 illustrates the conversion of a graphical program to hardware and software implementations; -
FIG. 10 illustrates the generation of various types of hardware and software descriptions from a VDiagram tree; -
FIG. 11 illustrates the conversion of a graphical program into a hardware description and the use of the hardware description to program an FPGA; -
FIG. 12 illustrates the conversion of a graphical program into a software source code description and the compilation and linkage of the source code; -
FIG. 13 is a flowchart diagram illustrating operation of one embodiment of the invention, including compiling a first portion of the graphical program into machine language and converting a second portion of the graphical program into a hardware implementation; and -
FIG. 14 is a flowchart diagram illustrating creation of a graphical program according to the preferred embodiment. - While the invention is susceptible to various modifications and alternative forms specific embodiments are shown by way of example in the drawings and will herein be described in detail. It should be understood however, that drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed. But on the contrary the invention is to cover all modifications, equivalents and alternative following within the spirit and scope of the present invention as defined by the appended claims.
- The following U.S. Patents and patent applications are hereby incorporated by reference in their entirety as though fully and completely set forth herein.
- U.S. patent application Ser. No. 09/949,783 titled “System and Method for Deploying a Graphical Program on an Image Acquisition Device”, filed Sep. 10, 2001.
- U.S. Pat. No. 4,914,568 titled “Graphical System for Modeling a Process and Associated Method,” issued on Apr. 3, 1990.
- U.S. Pat. No. 6,219,628 titled “System and Method for Configuring an Instrument to Perform Measurement Functions Utilizing Conversion of Graphical Programs into Hardware Implementations”.
- U.S. Pat. No. 6,173,438 titled “Embedded Graphical Programming System” filed Aug. 18, 1997, whose inventors are Jeffrey L. Kodosky, Darshan Shah, Samson DeKey, and Steve Rogers.
- U.S. patent application Ser. No. 09/499,503 titled “System and Method for Configuring a Programmable Hardware Instrument to Perform Measurement Functions Utilizing Estimation of the Hardware Implementation and Management of Hardware Resources”, filed on Feb. 7, 2000, whose inventors are Jeffrey L. Kodosky, Hugo Andrade, Brian Keith Odom, Cary Paul Butler, and Andrew Mihal.
- U.S. patent application Ser. No. 09/891,571 titled “System and Method for Configuring an Instrument to Perform Measurement Functions Utilizing Conversion of Graphical Programs into Hardware Implementations” filed on Jun. 25, 2001, whose inventors are Jeffrey L. Kodosky, Hugo Andrade, Brian Keith Odom, Cary Paul Butler, and Kevin L. Schultz.
- U.S. patent application Ser. No. 09/745,023 titled “System and Method for Programmatically Generating a Graphical Program in Response to Program Information,” filed Dec. 20, 2000, whose inventors are Ram Kudukoli, Robert Dye, Paul F. Austin, Lothar Wenzel and Jeffrey L. Kodosky.
- U.S. patent application Ser. No. 09/595,003 titled “System and Method for Automatically Generating a Graphical Program to Implement a Prototype”, filed Jun. 13, 2000, whose inventors are Nicolas Vazquez, Jeffrey L. Kodosky, Ram Kudukoli, Kevin L. Schultz, Dinesh Nair, and Christophe Caltagirone.
- U.S. patent application Ser. No. 09/587,682 titled “System and Method for Automatically Generating a Graphical Program to Perform an Image Processing Algorithm”, filed on Jun. 5, 2000, whose inventors are Nicolas Vazquez, Jeffrey L. Kodosky, Ram Kudukoli, Kevin L. Schultz, Dinesh Nair and Christophe Caltagirone.
- The LabVIEW and BridgeVIEW graphical programming manuals, including the “G Programming Reference Manual”, available from National Instruments Corporation, are also hereby incorporated by reference in their entirety.
-
FIG. 1 —Image Acquisition or Machine Vision System -
FIG. 1 illustrates ahost computer system 102 coupled to animage acquisition device 134. As used herein, the term “image acquisition device” is intended to include any of various types of devices that are operable to acquire and/or store an image. An image acquisition device may also optionally be further operable to analyze or process the acquired or stored image. Examples of an image acquisition device include an image acquisition (or machine vision) card (also called a video capture board), a device external to a computer that operates similarly to an image acquisition card, a smart camera, a robot having machine vision, and other similar types of devices. - As used herein, the terms “image processing” and “machine vision” are used interchangeably to refer to the processing of images to extract useful information from the image or determine characteristics of the image (or to determine characteristics of one or more objects displayed in the image). The term “image processing” is used herein to refer to both “image processing” and “machine vision”, to the extent these terms have different meanings. The term “image processing function” includes tools such as edge detection, blob analysis, pattern matching, and other image processing functions. The term “image processing function” may also include an operation or decision that is performed in response to the information extracted or characteristics determined from the image. The term “image processing function” is also intended to include an image processing (or machine vision) algorithm that combines a sequence of two or more image processing functions or tools and/or decision operations that process an image in a desired way or which implement an image processing or machine vision application, such as part inspection, automated assembly, image analysis, pattern matching, edge detection, etc.
- Thus,
FIG. 1 illustrates an exemplary image acquisition ormachine vision system 100. AsFIG. 1 shows, theimage acquisition device 134 may in turn couple to or comprise acamera 132. Theimage acquisition device 134 may include a functional unit for performing an image processing function as described below. - The
host computer 102 may comprise a CPU, a display screen, memory, and one or more input devices such as a mouse or keyboard as shown. Thecomputer 102 may operate with the image acquisition device to analyze, measure or control a device orprocess 150. Alternatively, thecomputer 102 may be used only to configure a functional unit in the image acquisition device. - As used herein, the term “functional unit” may include a processor and memory or a programmable hardware element. The term “functional unit” may include one or more processors and memories and/or one or more programmable hardware elements. As used herein, the term “processor” is intended to include any of types of processors, CPUs, microcontrollers, or other devices capable of executing software instructions. As used herein, the term “programmable hardware element” is intended to include various types of programmable hardware, reconfigurable hardware, programmable logic, or field-programmable devices (FPDs), such as one or more FPGAs (Field Programmable Gate Arrays), or one or more PLDs (Programmable Logic Devices), such as one or more Simple PLDs (SPLDs) or one or more Complex PLDs (CPLDs), or other types of programmable hardware.
- As shown, video device or
camera 132 may be coupled to thecomputer 102 via the image acquisition device orcard 134. Thecamera 132 and/orimage acquisition device 134 may couple to thecomputer 102 through a serial bus, a network, or through other means. - The
image acquisition system 100 may be used in an manufacturing assembly, test, measurement, and/or control application, among others. For illustration purposes, a unit under test (UUT) 150 is shown which may be positioned by a motion control device 136 (and interface card 138), and imaged and analyzed by thecamera 132 andimage acquisition device 134. It is noted that in various other embodiments theUUT 150 may comprise a process or system to be measured and/or analyzed. - Referring again to
FIG. 1 , thecomputer 102 may include a memory medium on which computer programs, e.g., graphical programs, according to the present invention may be stored. As used herein, the term “memory medium” includes a non-volatile medium, e.g., a magnetic media or hard disk, or optical storage; a volatile medium, such as computer system memory, e.g., random access memory (RAM) such as DRAM, SRAM, EDO RAM, RAMBUS RAM, DR DRAM, etc.; or an installation medium, such as a CD-ROM orfloppy disks 104, on which the computer programs according to the present invention may be stored for loading into the computer system. The term “memory medium” may also include other types of memory or combinations thereof. - The memory medium may be comprised in the
computer 102 where the programs are executed or may be located on a second computer which is coupled to thecomputer 102 through a network, such as a local area network (LAN), a wide area network (WAN), or the Internet. In this instance, the second computer operates to provide the program instructions through the network to thecomputer 102 for execution. - The software programs of the present invention may be stored in a memory medium of the
respective computer 102, or in a memory medium of another computer, and executed by the CPU. The CPU executing code and data from the memory medium thus comprises a means for deploying a graphical program onto an image acquisition device, e.g., a smart camera, according to the steps described below. - The memory medium may store a graphical programming development system for developing graphical programs. The memory medium may also store one or more computer programs which are executable to deploy a graphical program, such as by converting at least a portion of a graphical program into a form for configuring or programming a programmable hardware element, by executing the graphical program natively on a processor, or by converting the graphical program to a different form for execution by a processor and memory. The
image acquisition device 134 inFIG. 1 may be controlled by or configured by graphical software programs which are deployed or downloaded to the functional unit on thedevice 134. - As described below, in one embodiment the graphical program may be deployed by either one or more of: 1) converting the graphical program (or a portion thereof) into a hardware implementation and configuring the programmable hardware element with this hardware implementation, 2) transferring the graphical program (or a portion thereof) to a memory of the functional unit for execution by a processor (wherein the processor may execute a graphical program execution engine and optionally a real time operating system), or 3) compiling the graphical program (or a portion thereof) into an executable program and transferring the executable program to a memory of the functional unit for execution by a processor (wherein the processor may optionally execute a real time operating system).
- In the present application, the term “graphical program” or “block diagram” is intended to include a program comprising graphical code, e.g., two or more nodes or icons interconnected in one or more of a data flow, control flow, or execution flow format, where the interconnected nodes or icons may visually indicates the functionality of the program. Thus the terms “graphical program” or “block diagram” are each intended to include a program comprising a plurality of interconnected nodes or icons which visually indicates the functionality of the program. A graphical program may comprise a block diagram and may also include a user interface portion or front panel portion. The user interface portion may be contained in the block diagram or may be contained in one or more separate panels or windows. A graphical program may be created using any of various types of systems which are used to develop or create graphical code or graphical programs, including LabVIEW, DASYLab, and DiaDem from National Instruments, Visual Designer from Intelligent Instrumentation, Agilent VEE (Visual Engineering Environment), Snap-Master by HEM Data Corporation, SoftWIRE from Measurement Computing, ObjectBench by SES (Scientific and Engineering Software), Simulink from the MathWorks, WiT from Coreco, Vision Program Manager from PPT Vision, Hypersignal, VisiDAQ, Vis Sim, and Khoros, among others. In the preferred embodiment, the system uses the LabVIEW graphical programming system available from National Instruments.
- FIGS. 2A and 2B—Image Acquisition System Having a Smart Camera
-
FIGS. 2A and 2B illustrate an image acquisition system with a smart camera. AsFIGS. 2A and 2B illustrate, a smart camera may include a housing which encloses a portion or all of the smart camera components, or may be comprised on a frame which primarily provides structural support for the smart camera components. -
FIG. 2A —Image Acquisition System with Smart Camera -
FIG. 2A illustrates an image acquisition system comprisingcomputer system 102 coupled to asmart camera 190. The smart camera is an example of an image acquisition device. As used herein, the term “smart camera” is intended to include any of various types of devices that include a camera or other image sensor and a functional unit capable of being configured to perform an image processing function to analyze or process an acquired image. For example, while traditional computer vision is based on a camera/computer system in which the image processing or understanding algorithm is embedded in thecomputer 102, the computational load of vision algorithms may be circumvented or mitigated by merging low level processing with the camera or sensor in a single module. For example, a hardware architecture may be defined in a Hardware Description Language (e.g., VHDL), simulated and synthesized into digital structures that can then be configured in a Field Programmable Gate Array (FPGA). In one application of a smart camera, the flexibility of an FPGA may be combined with a sensor for real time image processing. In another application of a smart camera, the flexibility of a processor and memory may be combined with a sensor for real time image processing. Examples of smart cameras include: NAVSYS Corporation's GI-EYE, which generates digital image data that are automatically tagged with geo-registration meta-data to indicate the precise position and attitude of the camera when the image was taken; Vision Components' GmbH Smart Machine Vision Cameras, which integrate a high-resolution Charge Coupled Device (CCD) sensor with a fast image-processing signal processor, and provide various interfaces to allow communication with the outside world; and Visual Inspection Systems' SMART cameras with on-board DSP capabilities, including frame grabbers and robot guidance systems, among others. - The
computer system 102 shown inFIG. 2A may include a memory medium. As noted above, the memory medium may store a graphical programming development system for developing graphical programs. The graphical programming development system may be used to develop a graphical program that implements an image processing function. In this example, one or more of the nodes in the graphical program may implement or represent an image processing function. - The memory medium may also store one or more computer programs which are executable to convert at least a portion of a graphical program into a form for configuring a functional unit comprised in the
smart camera 190. - As mentioned above, the
smart camera 190 may include a functional unit, which may be a programmable hardware element (programmable or reconfigurable hardware), e.g., an FPGA, and/or may be a processor and memory. The functional unit in thesmart camera 190 may be configured with a graphical program that implements the image processing function. Thesmart camera 190 may also comprise a camera coupled to the functional unit. Thesmart camera 190 may also include a memory (a memory medium) coupled to the camera that stores an acquired image. If thesmart camera 190 includes an analog camera, thesmart camera 190 may further include analog to digital (A/D) logic for converting analog image signals into a digital image for storage in the memory. Thesmart camera 190 may also optionally include timer/counter logic that may perform timing/counting operations, e.g., during operation of the functional unit. AsFIG. 2A shows, in one embodiment, the smart camera may include a housing which encloses some or all of the components of the smart camera. Various embodiments of thesmart camera 190 are shown inFIGS. 6A-6D , described below. -
FIG. 2B —Smart Camera -
FIG. 2B illustrates another embodiment of asmart camera 190. AsFIG. 2B shows, the smart camera includes a camera, coupled to circuitry/logic (e.g., one or more circuit boards) for performing various image processing and/or acquisition functions. As mentioned above, the circuitry/logic may include a functional unit, such as a programmable hardware element, e.g., an FPGA and/or a processor and memory. As also described above, the functional unit in thesmart camera 190 may be configured with a graphical program that implements the image processing function, and may also include a memory coupled to the camera for storing an acquired image. If the camera is an analog camera, thesmart camera 190 may further include analog to digital (A/D) logic for converting analog image signals into a digital image for storage in the memory. Thesmart camera 190 may also optionally include timer/counter logic that may perform timing/counting operations, e.g., during operation of the programmable hardware element. Thesmart camera 190 may also include various I/O ports for communicating with external devices, such ascomputer system 102. AsFIG. 2B shows, in one embodiment, the smart camera may include a frame or structure to support the components comprised in the smart camera. In one embodiment, the smart camera may include a “PC-on-a-Card” which may provide part or all of the functionality of a personal computer. Various embodiments of thesmart camera 190 are shown inFIGS. 6A-6D , described below. - FIGS. 3A and 3B—Image Processing Systems
-
FIGS. 3A and 3B illustrate image processing or machine vision systems according to various embodiments of the invention. The image processing system ofFIG. 3A may comprise acomputer 102 and asmart camera 190, and may further include amotion control device 192. The image processing system ofFIG. 3B may comprisesmart camera 190 andmotion control device 192, and may not includecomputer system 102. - The
smart camera 190 may include a digital camera that acquires a digital video signal which comprises an image, or a sequence of images, or other data desired to be acquired. In one embodiment, thesmart camera 190 may instead include an analog camera that acquires an analog video signal, and thesmart camera 190 may further include A/D converters for converting the analog video signal into a digital image. - The
smart camera 190 may include a functional unit configured according to a graphical program. For example, the functional unit may be configured to perform an image processing function as represented by a graphical program. Thus a graphical program may have been first created to perform the image processing function, such as with a graphical development environment on thecomputer system 102, and the graphical program may then have been deployed onto the functional unit of the smart camera to implement the image processing function. - In the machine vision system of
FIG. 3A , the digital video signal or digital image may be provided to the functional unit in thesmart camera 190, wherein the image processing function is performed. - In the embodiment of
FIG. 3A , the functional unit in thesmart camera 190 may perform a portion or all of the image processing function, and thecomputer 102 may perform a portion of the image processing function. For example, the functional unit in thesmart camera 190 may perform the actual processing of the image to determine characteristics of the image, and thecomputer 102 may then perform an operation based on this result, such as rejecting a part from an assembly line, or logging the results to file. As another example, the functional unit in thesmart camera 190 may perform the processing of the image to determine characteristics of the image, and may also optionally perform an operation based on this result, and thecomputer system 102 may execute software to provide a user interface for the system, e.g., thecomputer system 102 may execute a user interface portion of a graphical program, where the block diagram of the graphical program is used to configure the functional unit in thesmart camera 190. Thus, in one embodiment, a first portion of the graphical program, e.g., DSP functions requiring real time performance, may be executed by the smart camera, i.e., the functional unit, and a second portion of the graphical program, e.g., a user interface where real time performance is not required, may be executed by thecomputer system 102. - In the embodiment of
FIG. 3B , the functional unit in thesmart camera 190 may perform all of the desired image processing function, including optionally performing an operation based on determined characteristics of the image, and hence thecomputer system 102 is not necessary during operation of the system. In another embodiment, thesmart camera 190 may include a processor and memory (in addition to the functional unit which executes the image processing function) which may execute the second portion of the graphical program, e.g., the user interface portion. - In the embodiments of
FIGS. 3A and 3B , the functional unit in the smart camera 190 (or thecomputer system 102 inFIG. 3A ) may control themotion control device 192. In an alternate embodiment, themotion control device 192 may also include a functional unit that has been configured with a graphical program. The functional unit in themotion control device 192 may be configured to perform a motion control function. Thus a graphical program may have been first created that performs the motion control function, and the graphical program may then have been deployed onto themotion control device 192 as described herein. Examples of motion control functions include moving a part or object to be imaged by a camera, rejecting a part on an assembly line, or placing or affixing components on a part being assembled, or a robotics application, among others. -
FIG. 4 —Computer Block Diagram -
FIG. 4 is an exemplary block diagram of thecomputer 102 ofFIGS. 1, 2A , and 3A. The elements of a computer not necessary to understand the operation of the present invention have been omitted for simplicity. Thecomputer 102 may include at least one central processing unit (CPU) orprocessor 160 which is coupled to a processor or host bus 162. TheCPU 160 may be any of various types, including an x86 processor, a PowerPC processor, a CPU from the Motorola family of processors, a CPU from the SPARC family of RISC processors, as well as others.Main memory 166 is coupled to the host bus 162 by means ofmemory controller 164. Themain memory 166 stores a graphical programming system, and also stores software for deploying at least a portion of a graphical program onto animage acquisition device 134. This software will be discussed in more detail below. Themain memory 166 may also store operating system software, i.e., software for operation of the computer system, as well as one or more application programs, as is well known to those skilled in the art. - The host bus 162 is coupled to an expansion or input/output bus 170 by means of a bus controller 168 or bus bridge logic. The expansion bus 170 is preferably the PCI (Peripheral Component Interconnect) expansion bus, although other bus types may be used. The expansion bus 170 may include slots for various devices, the examples shown including a
motion control device 138 and animage acquisition device 134, as described above in reference toFIG. 1 . Thecomputer 102 may further comprise avideo display subsystem 180 andhard drive 182 coupled to the expansion bus 170, also shown. - One or more of the interface cards or devices (e.g., those shown in
FIGS. 1, 2A , 2B, 3A and/or 3B) may be coupled to or comprised in thecomputer 102 and may comprise a functional unit, such as a programmable hardware element (FPGA) and/or processor and memory. Thecomputer 102 may also include a network interface for coupling to a network, wherein the target device containing the functional unit may be coupled to the network. Thus thecomputer 102 may be used for configuring a functional unit hardware in a target device over a network. - FIGS. 5A-5D—Image Acquisition Device Hardware Diagram
-
FIG. 5A is a block diagram illustrating a device, e.g., aninterface card 134A, configured with programmable hardware according to one embodiment. It is noted thatFIG. 5A is exemplary only, and an interface card or device configured with programmable hardware according to the present invention may have various architectures or forms, as desired. For example, the device may be internal or external to thecomputer 102, and may be connected to the computer through a network, such as the Internet. Theinterface card 134A illustrated inFIG. 5A may be theimage acquisition device 134 or the motioncontrol interface card 138, shown inFIGS. 1-3 . - As shown in
FIG. 5A , theinterface card 134A may include an I/O connector 202 which is operable to send/receive signals. The I/O connector 202 may present analog and/or digital connections for receiving/providing analog or digital signals, such as image signals or pixel data. In one embodiment, the I/O connector 202 may be adapted for coupling to an external camera. - The
interface card 134A may also include data acquisition (DAQ)logic 204. As shown, thedata acquisition logic 204 may comprise analog to digital (A/D) converters, digital to analog (D/A) converters, timer counters (TC) and signal conditioning (SC) logic as shown. TheDAQ logic 204 may provide data acquisition functionality of theimage acquisition card 134A. In one embodiment, theDAQ logic 204 comprises 4 A/D converters, 4 D/A converters, 23 digital I/Os, a RTSI connector, and a TIO. This extra hardware is useful for signal processing and motion control applications. The functional unit, e.g., programmable hardware element orFPGA 206 and/orprocessor 212 may access these resources directly, thereby enabling creation of very image processing applications, among others. - In the embodiment of
FIG. 5A , the functional unit of theinterface card 134A may include aprogrammable hardware element 206 and aprocessor 212 andmemory 214. In one embodiment, theprogrammable hardware 206 may comprise a field programmable gate array (FPGA) such as those available from Xilinx, Altera, etc. Theprogrammable hardware element 206 may be coupled to theDAQ logic 204 and may also be coupled to a local bus interface 208, described below. Thus a graphical program can be created on thecomputer 102, or on another computer in a networked system, and at least a portion of the graphical program can be converted into a hardware implementation form for execution in theFPGA 206. The portion of the graphical program converted into a hardware implementation form is preferably a portion which requires fast and/or real time execution. - In the embodiment of
FIG. 5A , theinterface card 134A may further include a dedicated on-board processor 212 andmemory 214. This enables a portion of the graphical program to be compiled into machine language for storage in thememory 214 and execution by theprocessor 212. A portion of the graphical program may also (or instead) be transferred to thememory 214 in its native format for execution by the processor. This may be in addition to a portion of the graphical program being converted into a hardware implementation form in theFPGA 206. Thememory 214 may store a real time operating system (RTOS) for execution by theprocessor 212. Where the graphical program executes in its native format on thedevice 134A, the memory may also store a graphical program execution engine that is executed by theprocessor 212. - Thus, in one embodiment, after a graphical program has been created, a portion of the graphical program may be transferred and/or compiled for execution on the on-
board processor 212 and executed locally on theinterface card 134A via theprocessor 212 andmemory 214, and a second portion of the graphical program may be translated or converted into a hardware executable format and downloaded to theFPGA 206 for hardware implementation. - As mentioned above, as one example, a first portion of a block diagram (that requires real time or fast execution) of a graphical program may be converted into a hardware executable format and downloaded to the
FPGA 206 for hardware implementation, and a second portion of a block diagram (that may not require real time performance) may be stored in thememory 214 as program instructions and executed by theprocessor 212, in either a compiled or interpreted manner. As another example, a portion or all of the block diagram portion of the graphical program may be converted into a hardware executable format and downloaded to theFPGA 206 for hardware implementation, and a user interface portion (or front panel portion) of the graphical program may be stored in thememory 214 as program instructions and executed by theprocessor 212, in either a compiled or interpreted manner. Thus the portion of the graphical program which requires the most real time or deterministic (reliable and consistent) performance may be executed directly in hardware for fast operation, and other parts of the block diagram, or the user interface portion, which may not require real time performance, may execute on theprocessor 212. Where the processor executes the user interface portion, the processor may then send resulting signals to the video subsystem for display of the user interface on the computer display. - As shown, the
interface card 134A may further include bus interface logic 216 and a control/data bus 218. In one embodiment, theinterface card 134A is a PCI bus-compliant interface card adapted for coupling to the PCI bus of thehost computer 102, or adapted for coupling to a PXI (PCI extensions for Instrumentation) bus. The bus interface logic 216 and the control/data bus 218 thus present a PCI or PXI interface. - The
interface card 134A may also include local bus interface logic 208. In one embodiment, the local bus interface logic 208 may present a RTSI (Real Time System Integration) bus for routing timing and trigger signals between theinterface card 134A and one or more other devices or cards, such as a motion device. - In one embodiment, the
interface card 134A also includes anon-volatile memory 288 coupled to theprogrammable hardware element 206, theprocessor 212 and thememory 214. Thenon-volatile memory 288 may be operable to store the hardware description received from the host computer system to enable execution of the hardware description in theprogrammable hardware element 206 prior to or during booting of thecomputer system 102. Thenon-volatile memory 288 may also store software used by the processor, such as a RTOS and/or a graphical program execution engine. - In the embodiment of
FIG. 5B , theprogrammable hardware element 206 is not included on theinterface card 134B, and thus the functional unit of thedevice 134B comprisesonly processor 212 andmemory 214. Thus, in deploying the graphical program, a portion or all of the graphical program which is to be deployed may be transferred to thememory 214 for execution by theprocessor 212. The graphical program may be transferred in its native format to thememory 214 and executed by theprocessor 212 using a graphical program execution engine and possibly a RTOS. Alternatively, the graphical program may be compiled into an executable program (e.g., machine language, a script, or an interpretable data structure) and transferred to thememory 214 for execution byprocessor 212. - In the embodiment of
FIG. 5C , theprocessor 212 is not included on theinterface card 134C, i.e., the functional unit of thedevice 134C comprises theFPGA 206 and thememory 214. In this embodiment, thememory 214 may be used for storing FPGA state information. Thus in the embodiment ofFIG. 5C , any supervisory control portion of the graphical program which is necessary or desired to execute on a programmable processor in software may be executed by the host CPU in thecomputer system 102, and is not executed locally by a processor on theinterface card 134B. - FIGS. 6A-6C—Block Diagrams of Smart Cameras with Programmable Hardware
-
FIGS. 6A-6C are block diagrams of various embodiments of asmart camera 190 with programmable hardware. As may be seen, the various embodiments share a number of features and configurations with thevarious interface cards 134 described above with reference toFIGS. 5A-5C , specifically the use of various combinations of processor/memory elements and programmable hardware, e.g., FPGAs, to execute portions of a graphical program. It should be noted that these embodiments are meant to be illustrative only, and are not intended to limit the architecture, components, or form of thesmart camera 190. -
FIG. 6A illustrates one embodiment of asmart camera 190 which includes acamera 282 and a functional unit. In this embodiment, the functional unit may comprise aprogrammable hardware element 206, e.g., a field programmable gate array (FPGA) such as those available from Xilinx, Altera, etc, and may also comprise aprocessor 212 andmemory 214. Each of theprogrammable hardware element 206,processor 212 andmemory 214 may be coupled to thecamera 282 and/or to animage memory 284. - As shown, the
smart camera 190A may also include anon-volatile memory 288 coupled to theprogrammable hardware element 206, theprocessor 212, thememory 214 and theimage memory 284. Thenon-volatile memory 288 may be operable to store the hardware description and/or graphical program received from the host computer system to enable execution of the functional unit prior to or during booting of thecomputer system 102. - Although not shown, the
smart camera 190 may include an I/O connector 202 which is operable to send and receive signals. The I/O connector 202 may present analog and/or digital connections for receiving/providing analog or digital signals. For example the I/O connector 202 may enable the smart camera to communicate withcomputer system 102 to receive a graphical program for performing image processing functions. Thus a graphical program can be created on thecomputer 102, or on another computer in a networked system, and at least a portion of the graphical program can be deployed onto a functional unit of the smart camera. The portion of the graphical program converted into a hardware implementation form is preferably a portion which requires fast and/or real-time execution. - As noted above, the
smart camera 190A includes a dedicated on-board processor 212 andmemory 214 in addition to theprogrammable hardware element 206. This enables a portion of the graphical program to be compiled into machine language for storage in thememory 214 and execution by the processor 212 (or to execute natively). This is in addition to a portion of the graphical program being converted into a hardware implementation form in theFPGA 206. Thus, in one embodiment, after a graphical program has been created, a portion of the graphical program may be compiled for execution on the on-board processor 212 and executed locally on thesmart camera 190A via theprocessor 212 andmemory 214, and a second portion of the graphical program may be translated or converted into a hardware executable format and uploaded to theFPGA 206 for hardware implementation. - For example, in a manner similar to that described above with reference to
FIG. 5A , a first portion of a block diagram (that requires real time or fast execution) of a graphical program, e.g., a graphical image processing program, may be converted into a hardware executable format and downloaded to theFPGA 206 for hardware implementation, and a second portion of a block diagram (that may or may not require real time performance) may be stored in thememory 214 as program instructions and executed by theprocessor 212, in either a compiled or interpreted manner. As another example, a portion or all of the block diagram portion of the graphical program may be converted into a hardware executable format and downloaded to theFPGA 206 for hardware implementation, and a user interface portion (or front panel portion) of the graphical program may be downloaded and stored in thememory 214 as program instructions and executed by theprocessor 212, in either a compiled or interpreted manner. Thus the portion of the graphical program which requires the most real time or deterministic (reliable and consistent) performance may be executed directly in hardware for fast operation, and other parts of the block diagram, or the user interface portion, which may not require real time performance, may execute on theprocessor 212. Where the processor executes the user interface portion, the processor may then send resulting signals to a video subsystem (of, for example, the computer system 102) for display of the user interface on the computer display. - As noted above the
smart camera 190A may includeimage memory 284 which couples to theprogrammable hardware 206, thecamera 282, theprocessor 212,memory 214, bus interface 216, and the control/data bus 218. Theimage memory 284 may be operable to store a portion of an image, or one or more images received from thecamera 282. Theimage memory 284 may enable theprogrammable hardware 206 and/or theprocessor 212 to retrieve the one or more images, operate on them, and return the modified images to the image memory. Additionally, the images may be retrievable for transmission to external systems, such as thecomputer system 102, via the I/O connector 202, for example. - As shown, the
smart camera 190A may further include bus interface logic 216 and a control/data bus 218. In one embodiment, thesmart camera 190A is a PCI bus-compliant interface card adapted for coupling to the PCI bus of thehost computer 102, or adapted for coupling to a PXI (PCI eXtensions for Instrumentation) bus. The bus interface logic 216 and the control/data bus 218 thus present a PCI or PXI interface. - Although not shown, in one embodiment, the
smart camera 190A may also include local bus interface logic 208, as shown inFIG. 5A . In one embodiment, the local bus interface logic 208 may present a RTSI (Real Time System Integration) bus for routing timing and trigger signals between thesmart camera 190A and one or more other devices or cards, such as other smart camera, motion devices, or other smart sensors. - In yet another embodiment, the
smart camera 190A may include data acquisition (DAQ)logic 204, not shown, such as analog to digital (A/D) converters, digital to analog (D/A) converters, timer counters (TC) and signal conditioning (SC) logic, as shown inFIG. 5A . TheDAQ logic 204 may be useful for signal processing and motion control applications. The programmable hardware element orFPGA 206 may access these resources directly, thereby enabling creation of very powerful measurement, DSP and control applications, among others. For example, if thecamera 282 is an analog camera, thesmart camera 190A may further include analog to digital (A/D) logic (not shown) for converting analog image signals into a digital image for storage in the memory. -
FIG. 6B illustrates another embodiment of asmart camera 190B, with a configuration similar to that ofinterface card 134B, described above with reference toFIG. 5B . AsFIG. 6B shows, in this embodiment, thesmart camera 190B includes thecamera 282,processor 212,memory 214,image memory 284, control/data bus 218, and bus interface 216 described above with reference toFIG. 6A , but does not include the programmable hardware (e.g., FPGA). Thus, in this embodiment the functional unit comprises only theprocessor 212 andmemory 214. Thus a portion or all of a graphical program may be deployed to the processor/memory component 212/214. AsFIG. 6B also shows, timer/counter logic 286 may also be included on the smart camera. The timer/counter logic 286 may be coupled to the other components of thesmart camera 190B, and may be operable to provide high-performance timing and counting functions to the various elements of thesmart camera 190B. For example, in one embodiment, the timer/counter logic may be used by theprocessor 212 to accurately control image acquisition by thecamera 282, such as for a high speed assembly line, or for capturing images of a fast-moving object. -
FIG. 6C illustrates another embodiment of asmart camera 190C, where thesmart camera 190C includesprogrammable hardware 206 andnon-volatile memory 288, but does not include theprocessor 212 andmemory 214. Thus in the embodiment ofFIG. 6C the functional unit comprises only theFPGA 206. Thus, any supervisory control portion of the graphical program which is necessary or desired to execute on a programmable processor in software may be executed by the host CPU in thecomputer system 102, and is not executed locally byprocessor 212, i.e., on thesmart camera 190C. AsFIG. 6C also shows, the smart camera may includecamera 282, bus interface 216, control/data bus 218,image memory 284, and timer/counter logic 286, as described above. Thus, a graphical image processing program, in the form of a hardware description, may be converted into a hardware executable format and downloaded to theprogrammable hardware element 206 for hardware implementation. Note that, as described above, the non-volatile memory 288 (or a separate volatile memory not shown) may be operable to store the hardware description received from thehost computer system 102 to enable execution of the hardware description in theprogrammable hardware element 206 prior to or during booting of thecomputer system 102. -
FIG. 7 —Method for Deploying a Graphical Program on an Image Acquisition Device -
FIG. 7 illustrates one embodiment of a method for deploying a graphical program on an image acquisition device to perform an image processing function, where the image acquisition device includes a functional unit. Said another way, a computer-implemented method is presented for configuring an image acquisition device to perform an image processing function using a graphical program. Note that the method shown inFIG. 7 may be used to configure any of the instruments or devices shown inFIGS. 1-6D . It should be noted that in various embodiments, various steps in the method may occur concurrently, or in a different order than shown. Additionally, some steps may be omitted, or additional steps performed which are not shown, as desired. As shown, this method may operate as follows. - In step 702 a graphical program may be created that implements an image processing function. In the preferred embodiment, the graphical program is created on the
computer system 102, although in other embodiments the graphical program may be created on a different computer system, for example, coupled tocomputer system 102 over a network. The graphical program may be created or assembled by the user with a graphical programming environment, such as the LabVIEW graphical programming development environment. In an alternate embodiment, the graphical program may be created instep 702 by the user specifying one or more functions or tasks, after which the specification may be analyzed and the graphical program generated automatically or programmatically from the specification. For example, the user may use a prototyping environment to create a script or prototype representing an image processing algorithm, and then a graphical program may be manually or programmatically created based on this prototype. It is noted that other approaches may also be used to create the graphical program, either manually by the user, or programmatically by software. - In
step 704 the graphical program may be deployed on theimage acquisition device 134. Note that in the preferred embodiment, theimage acquisition device 134 is coupled to thecomputer system 102 to facilitate the deployment of the graphical program. In one embodiment, theimage acquisition device 134 may be comprised in thecomputer system 102. In other embodiments theimage acquisition device 134 may be coupled to thecomputer system 102 through a network, such as the Internet, or may be coupled to thecomputer system 102 through wireless means. In another embodiment, a human user may manually retrieve the graphical program from thecomputer 102 and load the graphical program onto theimage acquisition device 134. As described in more detail below, deploying the graphical program on theimage acquisition device 134 may comprise storing the graphical program on (at least part of) the functional unit, or configuring the functional unit with all or part of the graphical program, such that after the deployment, the functional unit is operable to execute or implement the functionality of the graphical program. - In
step 706, theimage acquisition device 134 may acquire an image of an object, such as, for example, theUUT 150, or an element or state of a system or process. In the preferred embodiment, the image is acquired viacamera 132, although in other embodiments the image may be acquired from an external system, such as a computer or another image acquisition device. In one embodiment, theimage acquisition device 134 may receive an image present signal, and acquire the image of the object in response to receiving the image present signal. In one embodiment, the image acquisition device may be coupled to a camera, and the image acquisition device acquiring an image of the object may comprise the camera acquiring the image of the object and the image acquisition device receiving and storing the image. - Finally, in step 708, the image acquisition device may execute to perform the image processing function on the acquired image from 706. In other words, the functional unit on the
image acquisition device 134 may execute all or part of the graphical program. For example, the image processing function may operate to analyze the image and detect, recognize, or characterize one or more features or characteristics of the object, such as position, orientation, color, or possible flaws or lack thereof, among others. In one embodiment, the image acquisition device may perform a control operation after executing to perform the image processing function on the image. In another embodiment, the image acquisition device may generate a pass/fail indication after executing to perform the image processing function on the image. In another embodiment, the image acquisition device executing to perform the image processing function on the acquired image may comprise the image acquisition device determining characteristics of the image after performing the image processing function, and theimage acquisition device 134 performing an operation based on the determined characteristics of the image. - For example, in a quality control system in an automated assembly plant, in performing the image processing function on the acquired image, the image acquisition device may determine a characteristic of the image indicating that the imaged object is defective. The
image acquisition device 134 may then send a signal, such as a pass/fail indication, to a motion control device which may remove the defective object from the assembly line. An example of such a system is described above with reference toFIGS. 3A and 3B . - In one embodiment, the
image acquisition device 134 may includeprocessor 212 andmemory 214, as described above with reference toFIGS. 5B and 6B . The deployment of the graphical program on theimage acquisition device 134 described in 704 may then comprise transferring the graphical program to thememory 214 on theimage acquisition device 134. - In one embodiment, deploying the graphical program on the
image acquisition device 134 may include generating an executable program (a machine language program) based on the graphical program, which implements the functionality of the graphical program, and transferring the executable program to thememory 214 on the image acquisition device. In this embodiment, the image acquisition device executing to perform the image processing function on the image in 708 may comprise theprocessor 212 in theimage acquisition device 134 executing the executable program from thememory 214, possibly by executing a real time operating system (RTOS) from thememory 214. In the case where the executable program is generated from the graphical program, the image acquisition device executing to perform the image processing function on the image may comprise the processor in the image acquisition device executing the executable program (in executable form) from thememory 214. - In another embodiment, deploying the graphical program on the
image acquisition device 134 may include transferring the graphical program in its native or original format to thememory 214 on the image acquisition device. In this embodiment, the image acquisition device executing to perform the image processing function on the image in 708 may comprise theprocessor 212 in theimage acquisition device 134 executing a graphical program execution engine, and possibly a real time operating system (RTOS) from thememory 214. - In another embodiment, the image acquisition device may include a
programmable hardware element 206, such as an FPGA, as described above with reference toFIGS. 5C and 6C . The deployment of the graphical program on theimage acquisition device 134 described in 704 may then comprise generating a hardware description based on the graphical program, which describes a hardware implementation of the graphical program, and configuring theprogrammable hardware element 206 in theimage acquisition device 134 utilizing the hardware description. Configuring theprogrammable hardware element 206 in theimage acquisition device 134 utilizing the hardware description may comprise converting the hardware description, such as a VHDL file, into a netlist using available synthesis tools, and the compiling the netlist into a hardware program file (also called a software bit stream), which may be used to configure or program theprogrammable hardware element 206. - After configuring the
programmable hardware element 206 with the hardware description, theprogrammable hardware 206 implements a hardware implementation of the graphical program. In this embodiment, the image acquisition device executing to perform the image processing function on the image in 708 may comprise theprogrammable hardware element 206 in the image acquisition device executing to perform the image processing function on the image. - In yet another embodiment, the image acquisition device may include the
processor 212 andmemory 214 and theprogrammable hardware element 206, as described above with reference toFIGS. 5A, 6A , and 6D. The deployment of the graphical program on theimage acquisition device 134 described in 704 may then comprise transferring a first portion of the graphical program to thememory 214 on the image acquisition device 134 (either in native or compiled form), generating a hardware description based on a second portion of the graphical program describing a hardware implementation of the graphical program, and configuring the programmable hardware element in the image acquisition device utilizing the hardware description, after which the programmable hardware element implements a hardware implementation of the second portion of the graphical program. This embodiment of the invention may be referred to as a “dual mode” embodiment. In this embodiment, the image acquisition device executing to perform the image processing function on the image in 708 may comprise theprocessor 212 in theimage acquisition device 134 executing the first portion of the graphical program from thememory 214 and theprogrammable hardware element 206 executing the second portion of the graphical program. - In an exemplary dual mode embodiment, the deployment may include generating an executable program based on the first portion of the graphical program, which implements the functionality of the first portion of the graphical program, and transferring the executable program to the
memory 214 on the image acquisition device. The second portion may be configured on the programmable hardware element. - In one embodiment, the first portion of the graphical program may comprise an image processing portion, and the second portion of the graphical program may comprise a control portion. In another embodiment, the first portion of the graphical program may comprise a control portion, and the second portion of the graphical program may comprise an image processing portion. In this embodiment, the image acquisition device executing to perform the image processing function on the image in 708 may comprise the
processor 212 in theimage acquisition device 134 executing the executable program from thememory 214 to implement the first portion and theprogrammable hardware element 206 executing the second portion of the graphical program. - In yet another embodiment of the present invention, the
image acquisition device 134 may include a first processor 212A, a second processor 212B, and at least onememory 214. In this case, the image acquisition device executing to perform the image processing function on the image may comprise the first processor 212A in theimage acquisition device 134 executing a first portion of the graphical program from the at least onememory 214 and the second processor 212B in theimage acquisition device 134 executing a second portion of the graphical program from the at least onememory 214. Note that in this case, both the first portion and the second portion of the graphical program are executable by a processor from memory. For example, the first portion of the graphical program may comprise an image processing portion, and the second portion of the graphical program may comprise a control portion. In another embodiment, the first portion of the graphical program may comprise a first portion of the image processing function, and the second portion of the graphical program may comprise a second portion of the image processing function. Alternatively, the functional unit may comprise two or more programmable hardware elements which operate as above. - In one embodiment, the image acquisition device may comprise a smart camera, such as described above with reference to
FIG. 2B , and FIGS. 6A-C. In another embodiment, the image acquisition device may comprise an image acquisition board coupled to or comprised in a computer system, as shown inFIG. 2A , and FIGS. 5A-C. - More detailed embodiments of this method are presented below with reference to
FIGS. 8A and 8B . -
FIG. 8A —Configuring an Image Acquisition Device or Smart Camera Including Programmable Hardware to Perform an Image Processing Function -
FIG. 8A is an exemplary embodiment illustrating a method for configuring animage acquisition device 134, such assmart camera 190, with a graphical program to perform an image processing function, where the image acquisition device 134 (or smart camera 190) includes a functional unit, more specifically, aprogrammable hardware element 206. The method shown inFIG. 8A is a more detailed version of the method ofFIG. 7 described above, wherein the method is described with reference to an image acquisition device or smart camera. It should be noted that in various embodiments, various steps in the method may occur concurrently, or in a different order than shown. Additionally, some steps may be omitted, or additional steps performed which are not shown, as desired. As shown, this method may operate as follows. - First, in step 862 a graphical program may be created on the computer system 102 (or on a different computer system). The graphical program may be created or assembled by the user arranging on a display a plurality of nodes or icons and then interconnecting the nodes to create the graphical program. In response to the user assembling the graphical program, data structures (and/or program instructions) may be created and stored which represent the graphical program. The nodes may be interconnected in one or more of a data flow, control flow, or execution flow format. The graphical program may thus comprise a plurality of interconnected nodes or icons which visually indicates the functionality of the program. As noted above, the graphical program may comprise a block diagram and may also include a user interface portion or front panel portion. Where the graphical program includes a user interface portion, the user may assemble the user interface on the display. As one example, the user may use the LabVIEW graphical programming development environment to create the graphical program.
- In an alternate embodiment, the graphical program may be created in
step 862 by the user creating or specifying a prototype, followed by automatic or programmatic creation of the graphical program from the prototype. This functionality is described in U.S. patent application Ser. No. 09/587,682 titled “System and Method for Automatically Generating a Graphical Program to Perform an Image Processing Algorithm”, incorporated by reference above. The graphical program may be created in other manners, either by the user or programmatically, as desired. In the present example where the instrument is an image acquisition device 134 (e.g., smart camera 190), the graphical program may implement an image processing function. - In
step 864 the image acquisition device 134 (e.g., smart camera 190) may be coupled to thecomputer system 102. The image acquisition device 134 (e.g., smart camera 190) may be coupled to thecomputer system 102 over a network, via communication cable(s), through wireless means, or by any other method of coupling, as desired. As noted above, thesmart camera 190 may comprise a camera and aprogrammable hardware element 206. It is noted that the image acquisition device may be connected to thecomputer system 102 before, during or after the graphical program is created. - In step 866 a hardware configuration based on the graphical program may be downloaded onto the
programmable hardware element 206 in the image acquisition device 134 (e.g., smart camera 190) to configure theprogrammable hardware element 206. For example, in the embodiment where theimage acquisition device 134 is coupled to thecomputer system 102 over a network, deploying the graphical program may comprise thecomputer system 102 deploying the graphical program over the network to theimage acquisition device 134. The hardware configuration corresponds to a hardware implementation of the graphical program. In one embodiment, downloading the hardware configuration onto theprogrammable hardware element 206 may comprise the following steps: generating a hardware description based on the graphical program, where the hardware description describes a hardware implementation of the graphical program; converting the hardware description into a netlist; compiling the netlist format into a hardware program file; and downloading the hardware program file to the programmable hardware element to configure the programmable hardware element. These steps are discussed below. - After the downloading step is performed, the programmable hardware element is configured with a hardware implementation of the graphical program. At this point, in
step 868 the image acquisition device (e.g., smart camera 190) may be optionally disconnected from thecomputer system 102, and may possibly be deployed for operation. For example, in the case ofsmart camera 190, thesmart camera 190 may be disconnected from thecomputer system 102 and deployed in a vision application, such as a manufacturing vision inspection application, a manufacturing assembly application, or other vision application. If the image acquisition device is an image acquisition card or board that is designed to be located in a slot of thecomputer 102, the image acquisition card orboard 134 may optionally remain in thecomputer system 102 for use, or may be transferred to a different computer system. Alternatively, the method described above may be repetitively used to manufacture a plurality of image acquisition devices 134 (e.g., smart camera 190) for later sale. As another alternative, the method described above may be used to configure animage acquisition device 134 that has already been deployed, where theimage acquisition device 134 is configured over a network, i.e., where the hardware configuration is downloaded onto the image acquisition device 134 (or other instrument) over a network. - After the
programmable hardware element 206 in the image acquisition device 134 (e.g., smart camera 190) is configured with a hardware implementation of the graphical program, theimage acquisition device 134 can be used in an application. Thus theimage acquisition device 134 orsmart camera 190 may acquire an image of an object instep 870, e.g., from acamera 132, or the camera insmart camera 190 may acquire the image. Theprogrammable hardware element 206 in theimage acquisition device 134 may then execute instep 872 to perform the image processing function on the image. - While the
programmable hardware element 206 in theimage acquisition device 134 executes to perform the image processing function on the image, if the graphical program includes a user interface portion, this user interface portion may optionally be presented on a display during this time instep 874. The code corresponding to the user interface portion may be executed by aprocessor 160 in thecomputer system 102 or by aprocessor 212 on theimage acquisition device 134. The user interface portion may operate as a front panel for theimage acquisition device 134. The user may optionally provide user input to the user interface portion on the display to control theimage acquisition device 134 while theprogrammable hardware element 206 in theimage acquisition device 134 executes to perform the image processing function on the image. For example, the user interface portion may be compiled into executable code for execution by a processor (160 or 212) and stored in a memory (166 or 214) accessible by the processor. The processor may then execute the executable code (the user interface portion) from the memory to present the user interface portion on the display. - In an embodiment where the image acquisition device 134 (or smart camera 190) includes the
processor 212 and thememory 214, theprocessor 212 in the image acquisition device may execute the executable code from thememory 214 to present the user interface portion on the display during the image acquisition device 134 (or smart camera 190) executing to perform the image processing function on the image. - In another embodiment, the
processor 160 of thecomputer system 102 may execute the executable code from thememory 166 to present the user interface portion on the display during the image acquisition device 134 (or smart camera 190) executing to perform the image processing function on the image. -
FIG. 8B —Configuring an Image Acquisition Device or Smart Camera to Perform an Image Processing Function -
FIG. 8B is an exemplary embodiment illustrating a method for configuring animage acquisition device 134, such assmart camera 190, with a graphical program to perform an image processing function, where the image acquisition device 134 (or smart camera 190) includes a functional unit, more specifically, aprocessor 212 andmemory 214. The method shown inFIG. 8B is another, more detailed version of the method ofFIG. 7 described above, and is similar to the method described above with reference toFIG. 8A , differing from that method only insteps 866 and 872. Thus, in describing this embodiment of the method, where the steps are the same as described above, an abbreviated description is provided. As noted above, in various embodiments, various steps in the method may occur concurrently, or in a different order than shown. Additionally, some steps may be omitted, or additional steps performed which are not shown, as desired. As shown, this method may operate as follows. - First, in step 862 a graphical program may be created on the computer system 102 (or on a different computer system), as described above with reference to
FIG. 8A ,step 862. - In
step 864 the image acquisition device 134 (e.g., smart camera 190) may be coupled to thecomputer system 102, such as over a network, communication cable(s), through wireless means, or by any other method of coupling, as described above with reference toFIG. 8A ,step 864. - In
step 867 the graphical program may be downloaded to thememory 214 of the image acquisition device 134 (e.g., smart camera 190). As mentioned above, downloading the graphical program onto thememory 214 may comprise the following steps: generating an executable program (an executable program) based on the graphical program which implements the functionality of the graphical program; and transferring the executable program to thememory 214 on the image acquisition device. As also mentioned above, downloading the graphical program onto thememory 214 may comprise downloading the graphical program (or executable program) onto thememory 214 over a network. - After the downloading step is performed, the
image acquisition device 134 is configured to perform the image processing function implemented by the graphical program. At this point, instep 868 the image acquisition device 134 (e.g., smart camera 190) may be optionally disconnected from thecomputer system 102, and may possibly be deployed for operation, as described above instep 868 ofFIG. 8A . - After the image acquisition device 134 (e.g., smart camera 190) is configured with the graphical program, i.e., after the graphical program is loaded into the
memory 214, theimage acquisition device 134 can be used in an application. Thus theimage acquisition device 134 orsmart camera 190 may acquire an image of an object instep 870, e.g., from acamera 132, or the camera insmart camera 190 may acquire the image. Theimage acquisition device 134 may then execute to perform the image processing function on the image, as indicated in step 873. - While the image acquisition device executes to perform the image processing function on the image, if the graphical program includes a user interface portion, this user interface portion may optionally be presented on a display during this time in
step 874, as described above instep 874 ofFIG. 8A . - In one embodiment, the user may optionally provide user input to the user interface portion on the display to control the image acquisition device while the image acquisition device 134 (i.e., the
processor 212 and memory 214) executes to perform the image processing function on the image. - In one embodiment, the image acquisition device executing to perform the image processing function on the image in 708 may comprise the
processor 212 in theimage acquisition device 134 executing the graphical program from thememory 214. In the embodiment where the executable program is generated from the graphical program, the image acquisition device executing to perform the image processing function on the image may comprise the processor in the image acquisition device executing the executable program (in executable form) from thememory 214. - In one embodiment, the
processor 212 in the image acquisition device may execute the executable code from thememory 214 to present the user interface portion on the display during the image acquisition device 134 (or smart camera 190) executing to perform the image processing function on the image. In other words, theprocessor 212 may execute both portions of the graphical program concurrently, such as by threading or multi-tasking. In another embodiment, the image acquisition device 134 (or smart camera 190) may include multiple processors, e.g., 212A and 212B, in which case the first portion of the graphical program (such as the user interface portion) may be executed by processor 212A from memory 214 (or a first memory 214A), while the second portion (such as the image processing portion) may be executed by processor 212B from memory 214 (or a second memory 214B), or fromimage memory 284. - In one embodiment, the graphical program which implements the image processing function may already have been created, for example, by a third party. The method may then comprise coupling the
image acquisition device 134 to acomputer system 102 which stores the graphical program, where the image acquisition device comprises a functional unit; and deploying the graphical program onto the functional unit in theimage acquisition device 134 to configure the functional unit; where after the deploying, the functional unit is operable to implement the graphical program. In other words, step 862, and possibly steps 868 forward, may be omitted from the methods shown inFIGS. 8A and 8B . -
FIG. 9 —Block Diagram of Conversion of a Graphical Program Into Hardware and Software Descriptions -
FIG. 9 is a block diagram illustrating the conversion of a graphical program, such as a graphical image processing or machine vision program, into hardware and software descriptions. Thegraphical program 300 may comprise graphical code, such as interconnected function nodes or icons. The graphical code in the graphical program may use graphical data flow, graphical control flow and/or graphical execution flow constructs, as noted above. On the display, the graphical program may be represented as interconnected icons or function nodes. In the memory of the computer system, thegraphical program 300 may comprise data structures (or scripts or code) representing functional operations, data flow and/or control flow, and execution order. As the user assembles the graphical program on the display, e.g., by selecting, arranging, and connecting various icons or function nodes on the display, the data structures (or scripts or code) may be automatically created and stored in memory. - The
graphical program 300 may be created with various development tools. For example, the graphical program may be created using the following development systems: LabVIEW, BridgeVIEW, DASYLab, Visual Designer, HP VEE (Visual Engineering Environment), Snap-Master, GFS DiaDem, ObjectBench, Simulink, WiT, Vision Program Manager, Hypersignal, VisiDAQ, Vis Sim, Truly Visual, and Khoros, among others. In the preferred embodiment,graphical program 300 is a LabVIEW graphical program or virtual instrument (VI). - In one embodiment, programs of the present invention may create a
VDiagram tree 302 from the data structures of thegraphical program 300. TheVDiagram tree 302 is an abstract hardware graph which represents at least a portion of thegraphical program 300. The graph may be organized in a way that facilitates the generation of specific types of descriptions by back end programs of the present invention. In one embodiment, the graphical programming system automatically creates and stores a VDiagram tree 302 (abstract hardware graph) in response to a user's creation of a graphical program. In this instance, conversion from graphical program data structures to a VDiagram tree is not necessary. - A
hardware description 304 may be generated from theabstract hardware graph 302 by a back end program. Thehardware description 304 may be in any of various hardware description languages such as VHDL, EDIF, and Verilog. In the preferred embodiment, thehardware description 304 comprises one or more VHDL files. Ahardware netlist 306 may be generated from the hardware description using various synthesis tools. As noted above, the term “netlist” comprises various intermediate hardware-specific description formats comprising information regarding the particular hardware elements required to implement a hardware design and the relationship among those elements. In the preferred embodiment, thehardware netlist 306 is an FPGA-specific netlist. Thehardware netlist 306 is used to create or configure one or more functional hardware devices orhardware elements 308 which are configured to execute the portion of thegraphical program 300 that is represented by theabstract hardware graph 302. -
Hardware element 308 may comprise any of various devices. For example,hardware 308 may comprise a programmable logic device (PLD) such as an FPGA or CPLD. However,hardware 308 may comprise other types of hardware devices, such as a traditional circuit board which is created using thehardware netlist 306. In the preferred embodiment,hardware 308 is an interface card comprising an FPGA, where the interface card is comprised in the computer system where thegraphical program 300 is created. Thehardware 308 may also be comprised in an external device connected to the computer system where thegraphical program 300 is created. Thehardware 308 may be connected to the computer over an external serial or parallel bus, or over a network, such as the Internet. - As shown in
FIG. 9 , softwaredescription source code 310 may also be generated from theabstract hardware graph 302 by a back end program. Thesource code 310 may be in various source code languages such as C, C++, Java, etc.Machine code 312 may be produced from thesource code 310 using various source code compilers. Linkedmachine code 314 may be produced from themachine code 312 using various machine code linkers. The linkedmachine code 314 is executable to perform the operations of the portion of thegraphical program 300 that is represented by theabstract hardware graph 302. -
FIG. 10 —Block Diagram of Generation of Hardware and Software Descriptions from a Vdiagram Tree -
FIG. 10 is a block diagram illustrating the generation of various types of hardware and software descriptions from a VDiagram tree. As described forFIG. 9 , programs of the present invention create aVDiagram tree 302 from agraphical program 300. TheVDiagram tree 302 represents at least a portion of thegraphical program 300. Back end programs 330 generate hardware descriptions from theVDiagram tree 302. Exemplaryback end programs Back end 330A generates a VHDL hardware description comprising one or more VHDL files.Back end 330B generates an EDIF hardware description comprising one or more EDIF files.Back end 330C generates a C source code software description comprising one or more C files. It is also contemplated that other back ends may be used to generate hardware descriptions in other hardware and/or software programming languages. - The number and type of back end programs that may be present are not limited. In the preferred embodiment, one or more back end programs may be called automatically as part of a process initiated by a user to generate hardware/software descriptions for the
graphical program 300. In another embodiment, theVDiagram tree 302 may be generated and saved to a file, and the user may call a back end program at a later time to generate a hardware/software description. - As described above for
FIG. 9 , appropriate synthesis tools or compilers may be called to convert a hardware/software description into another format such as an FPGA-specific netlist or compiled machine code. -
FIG. 11 —Block Diagram of Conversion of a Graphical Program into a Hardware Description for an FPGA -
FIG. 11 illustrates the exportation of at least a portion of agraphical program 300 into a hardware description and the use of the hardware description to program an FPGA. As described above forFIG. 10 , theVDiagram tree 302 comprises information representing thegraphical program 300, including the functional operations of the program. As described in detail below, the VDiagram tree comprises VDiagrams, each of which maintains a list of components. This list of components includes components which represent functional operations. - A back end program converts the
VDiagram tree 302 to ahardware description 304. Back end programs may implement the functionality of the components in the VDiagram component lists using constructs of their respective description languages. For example, a VHDL back end may create VHDL code to implement a component that performs a particular mathematical algorithm such as an exponential calculation. However, in one embodiment, such functional components are simply referenced as library components. -
FIG. 11 illustrates one embodiment in which the VDiagram tree references one or more library components. One embodiment of the present invention comprises pre-compiled function blocks 342 which implement these library components for particular hardware devices such as FPGAs. Various FPGA netlist synthesis tools may be called to generate anFPGA netlist 340 from thehardware description 304. These synthesis tools may incorporate the pre-compiled function blocks 342 into theFPGA netlist 340. Also, as shown, the synthesis tools may utilize hardware target-specific information in creating the netlist. For example, the exact form that the FPGA netlist takes may depend on the particular type of FPGA that will use the netlist, since FPGAs differ in their available resources. - An FPGA bit
stream program file 346 may be generated from theFPGA netlist 340 using readily available synthesis tools. This FPGA program file may be uploaded to anFPGA 348. TheFPGA 348 may be comprised in a hardware device such as an interface board. After being programmed with theprogram file 346, the FPGA is able to execute the portion of thegraphical program 300 that is exported to thehardware description 304. If the entire graphical program is not exported to the hardware description, then a portion of the program may execute on the general purpose CPU of the computer system. This portion preferably comprises the supervisory control and display portion of the program. Details follow on how the execution of the FPGA portion is coordinated with the execution of the main CPU portion and how the external hardware resource requirements for the FPGA portion are managed. -
FIG. 12 —Block Diagram of Conversion of a Graphical Program into a Software Source Code Description with Compilation and Linkage -
FIG. 12 illustrates the exportation of at least a portion of agraphical program 300 into a software source code description and the compilation and linkage of the source code. As shown, the graphical program data structures may be first converted to aVDiagram tree 302 and then to softwaredescription source code 310. - As described above for
FIG. 10 , in the preferred embodiment theVDiagram tree 302 references library components to represent various functional components of the graphical program. These library components may be implemented in libraries, class libraries, macro definitions, etc. 360. As shown inFIG. 12 , these class libraries, etc. may be used to produce themachine code 312 from thesource code 310. Also,binary object libraries 362 may implement some functionality of the software description. These binary object libraries may be linked in with themachine code 312 is linked to produce the linkedexecutable code 314.Libraries 360 and 362 may also contain compiler-specific or platform-specific information necessary to produceexecutable code 314.Linked code 314 may be executed to perform the operations of the portion of the graphical program that is exported to the softwaresource code description 310. -
FIG. 13 —Conversion of a Graphical Program into a Hardware Implementation -
FIG. 13 is a flowchart diagram illustrating operation of one embodiment of the present invention. One embodiment of the present invention comprises a computer-implemented method for generating hardware and/or software implementations of graphical programs or graphical code. It is noted that various of the steps in the flowchart can occur concurrently or in different orders. - One goal of the present invention is to provide a development environment that will seamlessly allow use of a graphical programming system to design applications for reconfigurable or programmable hardware. In the preferred embodiment where the graphical programming system is LabVIEW, the present invention allows LabVIEW users to design applications in LabVIEW for reconfigurable hardware.
- Many applications, such as signal processing and real-time motion control, are easily implemented in a graphical programming language, such as the LabVIEW G language. However, in some instances traditional software compilation methods cannot produce an application that is fast enough to meet a user's needs. The present invention solves this problem by allowing a user to convert their graphical program, e.g., a G program, into application-specific hardware such as a programmed FPGA. The hardware maintains the exact functionality of the graphical program while running at speeds far exceeding that of traditional general-purpose processor platforms. One current implementation of the present invention is a desktop or embedded PC that contains an FPGA-based and/or processor based card or board.
- In one embodiment, the system appears as a conventional graphical programming system while providing a seamless interface to the reconfigurable hardware. For example, the preferred embodiment of the invention, referred to as “FPGA LabVIEW”, provides a seamless interface to an FPGA. FPGA LabVIEW appears from the outside to be exactly the same as the normal LabVIEW graphical program development system.
-
FIG. 13 illustrates one embodiment of the translation process from a graphical program to a hardware description that corresponds to the graphical program. A graphical programming application that is being targeted for a hardware implementation is designed in the same way as an ordinary graphical programming application. As shown, instep 862 the user first creates a graphical program, also sometimes referred to as a block diagram. As described above, a design may be entered and debugged in the traditional software-based manner. In one embodiment, the graphical program comprises a graphical data flow diagram which specifies functionality of the program to be performed. This graphical data flow diagram is preferably directly compilable into machine language code for execution on a computer system. - When the design is finalized, the user can instruct the system to compile the design for the FPGA hardware. Unfortunately, some graphical programming constructs may not be efficiently implemented in FPGA hardware. For example, file I/O is a task that is usually better left to the general-purpose host processor. The present system in one embodiment is capable of bisecting a design into hardware portions and software portions.
- Steps 404-414 are an example implementation of step 866 of
FIG. 8A . - In
step 404, the user may optionally select a first portion of the graphical program for conversion to a hardware implementation. This first portion of the graphical program which is desired for hardware implementation preferably comprises portions of the graphical program, e.g., particular subprograms, which require a fast or deterministic implementation and/or are desired to execute in a stand-alone hardware unit. In general, portions of the graphical program which are desired to have a faster or more deterministic execution are selected instep 404 and converted into the hardware implementation in steps 406-414. A default case is that the entire block diagram portion of the graphical program is selected for hardware implementation. - In
step 422 the remaining portions of the graphical program which were not selected instep 404, if any, may be compiled into machine code for execution on a CPU, such as the host processor in thecomputer 102 or theprocessor 212 comprised on theinterface card 134. The first portion of the program selected instep 404 preferably excludes program portions involving supervisory control and display. This enables the supervisory control and display portions to execute on the host CPU, which is optimal for these elements of the program. - In one embodiment, during creation of the graphical program in
step 862 the user may specify portions, e.g. subprograms, which are to be exported to the hardware description format for conversion into a hardware implementation. In another embodiment the user selects which modules or subprograms to export to the hardware implementation at the time when the conversion process is initiated. In another embodiment, the entire graphical program may be selected for conversion to a hardware implementation, and thus step 422 may not be performed. - In
step 406 the graphical program portion selected instep 404 may first be processed to create an abstract hardware graph called a VDiagram tree, described above, which serves as an intermediate data structure. The VDiagram tree may contain a complete hardware representation of the program, but is not specific to any hardware description language. For example, the VDiagram tree may comprise data structures representing hardware signals that implement the data flow within the graphical program, as well as data structures representing hardware signals that are added to preserve the proper execution flow (enable signals). - In
step 408, a back end program may be called to parse the VDiagram tree and generate a hardware description from it. The back end may translate the information contained in the VDiagram tree into a specific hardware description language. For example, a VHDL back end may be called to generate a VHDL file or set of files describing the program. The hardware description comprises a high-level hardware description of function blocks, logic, inputs, and outputs which perform the operation indicated by the portion of the graphical program selected instep 404. - Various types of back end programs may be present. Back end programs may generate software source code descriptions as well as hardware description language descriptions. For example,
FIG. 10 illustrates aback end 330A which uses the VDiagram tree to generate one or more VHDL files;back end 330B which generates one or more EDIF files; andback end 330C which generates one or more C files. These three back ends are representative only. Other back ends may generate other types of descriptions for the program. For example, a Verilog back end may generate a Verilog file for the program. Also, more than one back end may be called to generate different program descriptions. In the preferred embodiment, a VHDL back end generates a VHDL description which may then be compiled and used to program a programmable logic device such as an FPGA. - In
step 410 the method may operate to convert the hardware description into an FPGA-specific netlist. The netlist describes the components required to be present in the hardware as well as their interconnections. Conversion of the hardware description into the FPGA-specific netlist is preferably performed by any of various types of commercially available synthesis tools, such as those available from Xilinx, Altera, etc. - In one embodiment, the converting
step 410 may utilize one or more pre-compiled function blocks from a library of pre-compiled function blocks 342. Thus, for certain function blocks which are difficult to compile, or less efficient to compile, from a hardware description into a netlist format, the hardware description created instep 408 includes a reference to a pre-compiled function block from thelibrary 342. Alternatively, hardware implementations for all of the function blocks are included in the function library. The respective pre-compiled function blocks are simply inserted into the netlist in place of these references instep 410. The preferred embodiment of the invention thus includes thelibrary 342 of pre-compiled function blocks, also referred to as the component library, which are used in creating the netlist. The preferred embodiment also includes hardware targetspecific information 344 which is used bystep 410 in converting the hardware description into a netlist which is specific to a certain type or class of FPGA. - In
step 412 the method may operate to compile the netlist into an FPGA program file, also referred to as a software bit stream. The FPGA program file is a file that can be readily uploaded to program an FPGA. - After the netlist has been compiled into an FPGA program file in
step 412, then instep 414 the method may operate to transfer the FPGA program file to the FPGA, to produce a programmed hardware equivalent to the graphical program. Thus, upon completion ofstep 414, the portion of a graphical program referenced instep 404 is comprised as a hardware implementation in an FPGA or other programmable hardware element. - In the preferred embodiment, the hardware description is passed transparently through the FPGA vendor's synthesis tools. Because the vendor's tools may take a considerable amount of time to process the design and generate a programming bitstream, it is recommended that this only be done after the design has been debugged using traditional software-compilation techniques.
- As described above, the present invention may run on PC computers equipped with an FPGA-based expansion card on the PCI bus. Embodiments of the FPGA-based expansion card were described with reference to
FIGS. 5A-5C (and possibly 6A-6D). The graphical programming system may upload the programming bitstream generated by the FPGA vendor's design tools into the FPGA on this board. The FPGA then may begin processing data, and the graphical programming system may coordinate data flow between the FPGA and the host CPU. - It is noted that various of the above steps can be combined and/or can be made to appear invisible to the user. For example, steps 410 and 412 can be combined into a single step, as can steps 404-410. In the preferred embodiment, after the user creates the graphical program in step 402, the user simply selects a hardware export option and indicates the hardware target or destination, causing steps 404-414 to be automatically performed.
-
FIG. 13 applies to the preferred embodiment in which the programmable hardware element is an FPGA. However, the same or similar steps may be applied to convert a graphical program into a hardware implementation for other types of programmable or (re)configurable hardware, such as a CPLD. -
FIG. 14 —Creation of a Graphical Program -
FIG. 14 is a more detailed flowchart diagram ofstep 862 ofFIGS. 8A and 13 , illustrating creation of a graphical program according to one embodiment of the invention. As shown, instep 430 the user may arrange on a screen (i.e., a display device) a graphical program or block diagram. This may include the user placing and connecting, e.g., wiring, various icons or nodes on the display screen in order to configure a graphical program. More specifically, the user may select various function icons or other icons and place or drop the icons in a block diagram panel, and then connect or “wire up” the icons to assemble the graphical program. The user may also assemble a user interface, which may be referred to as a front panel, comprising controls and indicators which indicate or represent input/output to/from the graphical program. A graphical program targeted to measurement or automation applications may be referred to as a virtual instrument (VI). The graphical program or VI may have a hierarchy of sub-graphical programs or sub-VIs. - In the preferred embodiment, the graphical programming system is the LabVIEW graphical programming system available from National Instruments. For more information on creating a graphical program in the LabVIEW graphical programming system, please refer to the LabVIEW system available from National Instruments as well as the above patent applications incorporated by reference.
- In response to the user arranging on the screen a graphical program, the method may operate in step 432 to develop and store a tree of data structures which represent the graphical program. Thus, as the user places and arranges on the screen function nodes, structure nodes, input/output terminals, and connections or wires, etc., the graphical programming system may operate to develop and store a tree of data structures which represent the graphical program. More specifically, as the user assembles each individual node and wire, the graphical programming system may operate to develop and store (or populate) a corresponding data structure in the tree of data structures which represents the individual portion of the graphical program that was assembled. Thus, steps 430 and 432 may be an iterative process which is repetitively performed as the user creates the graphical program. In one embodiment, the graphical programming system may automatically develop and store VDiagram data structures in response to the user creating the graphical program.
- In an alternate embodiment, as the user places, arranges and interconnects nodes on the display, scripts, DLLs, or other code may be created in memory.
- In one embodiment, the user may optionally place constructs in the graphical program which indicate respective portions of graphical code which are either to be compiled into machine code for execution by a CPU or converted to a hardware description for implementation in a programmable hardware device such as an FPGA.
- Thus, the various methods described above may operate to deploy a graphical program implementing a function, such as an image processing or machine vision function, onto a programmable or configurable device, such as an image acquisition device, e.g., an image acquisition board or smart camera, thereby enabling the device to perform the function.
- Although the system and method of the present invention has been described in connection with the preferred embodiment, it is not intended to be limited to the specific form set forth herein, but on the contrary, it is intended to cover such alternatives, modifications, and equivalents, as can be reasonably included within the spirit and scope of the invention as defined by the appended claims.
Claims (12)
1. An image acquisition device, comprising:
an input for receiving an image of an object;
a memory coupled to the input for storing data corresponding to the image of the object, wherein the memory further stores a real time operating system, wherein the memory also stores an executable program, wherein the executable program implements an image processing function, wherein the executable program is created based on a graphical program, and wherein the graphical program comprises a plurality of interconnected nodes which visually indicate functionality of the graphical program; and
a processor coupled to the memory, wherein the processor is operable to execute the executable program to perform the image processing function on an acquired image, wherein the processor is further operable to execute the real time operating system in executing the executable program.
2. The image acquisition device of claim 1 , wherein, to create the executable program based on the graphical program, the graphical program is compiled to generate the executable program.
3. The image acquisition device of claim 1 , wherein, to create the executable program based on the graphical program, the graphical program is converted to a text-based program, and the text-based program compiled to generate the executable program.
4. The image acquisition device of claim 1 ,
wherein the memory further stores a graphical program execution engine; and
wherein the processor is operable to execute the graphical program execution engine in executing the executable program.
5. The image acquisition device of claim 1 , further comprising:
analog to digital conversion logic coupled to the input and to the processor for performing analog to digital conversion logic on an acquired analog image to produce a digital image.
6. The image acquisition device of claim 1 , further comprising:
timer/counter logic, wherein the timer/counter logic performs timing and/or counting operations while the processor in the image acquisition device executes to perform the image processing function on the image.
7. A smart camera, comprising:
a housing;
a sensor comprised in the housing, wherein the sensor is operable to acquire an image of an object;
a memory, coupled to the sensor and comprised in the housing, wherein the memory is operable to store data corresponding to the image of the object, wherein the memory stores a real time operating system, wherein the memory also stores an executable program implementing an image processing function, wherein the executable program is created based on a graphical program, and wherein the graphical program comprises a plurality of interconnected nodes which visually indicate functionality of the graphical program; and
a processor coupled to the memory, wherein the processor is operable to execute the executable program to perform the image processing function on an acquired image;
wherein the processor is operable to execute the real time operating system in executing the executable program.
8. The smart camera of claim 7 , wherein, to create the executable program based on the graphical program, the graphical program is compiled to generate the executable program.
9. The image acquisition device of claim 7 , wherein, to create the executable program based on the graphical program, the graphical program is converted to a text-based program, and the text-based program compiled to generate the executable program.
10. The image acquisition device of claim 7 ,
wherein the memory further stores a graphical program execution engine; and
wherein the processor is operable to execute the graphical program execution engine in executing the executable program.
11. The smart camera of claim 7 , further comprising:
analog to digital conversion logic, coupled to the input and to the processor and comprised in the housing, wherein the analog to digital conversion logic is operable to perform analog to digital conversion on an acquired analog image to produce a digital image.
12. The smart camera of claim 7 , further comprising:
timer/counter logic, coupled to the processor and comprised in the housing, wherein the timer/counter logic performs timing and/or counting operations while the processor in the smart camera executes to perform the image processing function on the image.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/212,326 US20060005160A1 (en) | 1997-08-18 | 2005-08-26 | Image acquisition device |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/912,427 US6219628B1 (en) | 1997-08-18 | 1997-08-18 | System and method for configuring an instrument to perform measurement functions utilizing conversion of graphical programs into hardware implementations |
US09/499,503 US6608638B1 (en) | 2000-02-07 | 2000-02-07 | System and method for configuring a programmable hardware instrument to perform measurement functions utilizing estimation of the hardware implentation and management of hardware resources |
US09/891,571 US6784903B2 (en) | 1997-08-18 | 2001-06-25 | System and method for configuring an instrument to perform measurement functions utilizing conversion of graphical programs into hardware implementations |
US09/949,783 US6971066B2 (en) | 1997-08-18 | 2001-09-10 | System and method for deploying a graphical program on an image acquisition device |
US11/212,326 US20060005160A1 (en) | 1997-08-18 | 2005-08-26 | Image acquisition device |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/949,783 Division US6971066B2 (en) | 1997-08-18 | 2001-09-10 | System and method for deploying a graphical program on an image acquisition device |
Publications (1)
Publication Number | Publication Date |
---|---|
US20060005160A1 true US20060005160A1 (en) | 2006-01-05 |
Family
ID=35515504
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/949,783 Expired - Lifetime US6971066B2 (en) | 1997-08-18 | 2001-09-10 | System and method for deploying a graphical program on an image acquisition device |
US11/212,326 Abandoned US20060005160A1 (en) | 1997-08-18 | 2005-08-26 | Image acquisition device |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/949,783 Expired - Lifetime US6971066B2 (en) | 1997-08-18 | 2001-09-10 | System and method for deploying a graphical program on an image acquisition device |
Country Status (1)
Country | Link |
---|---|
US (2) | US6971066B2 (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050132098A1 (en) * | 2003-12-11 | 2005-06-16 | Canon Kabushiki Kaisha | Relay apparatus for relaying communication from CPU to peripheral device |
US20080058969A1 (en) * | 2006-09-01 | 2008-03-06 | Fisher-Rosemount Systems, Inc. | Graphical Programming Language Object Editing and Reporting Tool |
US20100194548A1 (en) * | 2009-01-30 | 2010-08-05 | Tm Industrials, Llc | Downtime monitoring apparatus and method |
WO2013029008A1 (en) * | 2011-08-25 | 2013-02-28 | Cornell University | Retinal encoder for machine vision |
US9123093B1 (en) | 2008-08-29 | 2015-09-01 | Cognex Corporation | Vision inspection programming method and apparatus |
CN105511887A (en) * | 2015-12-28 | 2016-04-20 | 鲍敏祺 | Hardware visualization programming framework |
US20160263379A1 (en) * | 2010-02-26 | 2016-09-15 | Cornell University | Retina prosthesis |
US9925373B2 (en) | 2010-09-10 | 2018-03-27 | Cornell University | Neurological prosthesis |
CN108245099A (en) * | 2018-01-15 | 2018-07-06 | 深圳市沃特沃德股份有限公司 | Robot moving method and device |
US10515269B2 (en) | 2015-04-20 | 2019-12-24 | Cornell University | Machine vision with dimensional data reduction |
CN112707506A (en) * | 2020-12-28 | 2021-04-27 | 天津凯英科技发展股份有限公司 | Sewage treatment method and device and electronic equipment |
Families Citing this family (52)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH10334212A (en) * | 1997-04-01 | 1998-12-18 | Fuji Photo Film Co Ltd | System for printing image from image file with additional information |
US6971066B2 (en) * | 1997-08-18 | 2005-11-29 | National Instruments Corporation | System and method for deploying a graphical program on an image acquisition device |
US7937665B1 (en) | 2000-06-13 | 2011-05-03 | National Instruments Corporation | System and method for automatically generating a graphical program to implement a prototype |
US8640027B2 (en) * | 2000-06-13 | 2014-01-28 | National Instruments Corporation | System and method for configuring a hardware device to execute a prototype |
US7961201B1 (en) | 2000-12-21 | 2011-06-14 | Cognex Corporation | Method and apparatus for producing graphical machine vision content for distribution via a network |
US7383536B1 (en) * | 2000-12-29 | 2008-06-03 | Petry John | Remote machine vision application program development method |
US7962898B1 (en) | 2000-12-29 | 2011-06-14 | Cognex Corporation | Optimized distribution of machine vision processing |
US6876368B2 (en) * | 2001-08-14 | 2005-04-05 | National Instruments Corporation | System and method for deploying a graphical program to a PDA device |
US7076322B2 (en) * | 2002-03-29 | 2006-07-11 | National Instruments Corporation | System and method for satisfying move constraints when performing a motion control sequence |
US7139979B2 (en) * | 2002-06-10 | 2006-11-21 | National Instruments Corporation | Displaying operations in an application using a graphical programming representation |
US8074201B2 (en) * | 2002-07-10 | 2011-12-06 | National Instruments Corporation | Deployment and execution of a program on an embedded device |
US7340737B2 (en) | 2002-07-03 | 2008-03-04 | National Instruments Corporation | Wireless deployment / distributed execution of graphical programs to smart sensors |
WO2004023325A1 (en) * | 2002-09-04 | 2004-03-18 | Mentor Graphics (Holdings) Ltd. | Polymorphic computational system and method |
US20040233237A1 (en) * | 2003-01-24 | 2004-11-25 | Andreas Randow | Development environment for DSP |
US7159204B2 (en) * | 2003-01-28 | 2007-01-02 | Altera Corporation | System and method for design entry and synthesis in programmable logic devices |
US7647562B2 (en) * | 2003-04-03 | 2010-01-12 | National Instruments Corporation | Deployment and execution of a graphical program on an embedded device from a PDA |
US20050203872A1 (en) * | 2004-03-05 | 2005-09-15 | Kwong Kwan John M. | Method and apparatus making, operating and using media parsers to mark, read, and unmark instances of media formats supporting one, two and multi-dimensional instances and data streams |
US7587080B1 (en) * | 2004-11-04 | 2009-09-08 | Rockwell Automation Technologies, Inc. | Image retention user interface |
WO2006052435A2 (en) * | 2004-11-04 | 2006-05-18 | Rockwell Automation Technologies, Inc. | Image retention user interface |
JP2006140935A (en) * | 2004-11-15 | 2006-06-01 | Watec Co Ltd | Small-sized ccd camera, and controller for parameter setting |
US7392489B1 (en) | 2005-01-20 | 2008-06-24 | Altera Corporation | Methods and apparatus for implementing application specific processors |
US7613858B1 (en) * | 2005-01-24 | 2009-11-03 | Altera Corporation | Implementing signal processing cores as application specific processors |
US20060210169A1 (en) * | 2005-03-03 | 2006-09-21 | General Dynamics Advanced Information Systems, Inc. | Apparatus and method for simulated sensor imagery using fast geometric transformations |
DE102005034168A1 (en) * | 2005-07-21 | 2007-02-01 | Siemens Ag | Method for operating and monitoring a control device, corresponding HMI device, control device and machine with such a control device and uses of the method and data storage media |
AU2008286957B2 (en) * | 2007-08-10 | 2012-11-01 | Smiths Medical Asd, Inc. | System for controlling medical devices |
JP5147448B2 (en) * | 2008-02-21 | 2013-02-20 | 株式会社日立ハイテクノロジーズ | Image processing apparatus for semiconductor appearance inspection apparatus, semiconductor appearance inspection apparatus, and image processing method |
US8175725B2 (en) * | 2008-09-08 | 2012-05-08 | National Instruments Corporation | Adapting move constraints for position based moves in a trajectory generator |
US9569695B2 (en) | 2012-04-24 | 2017-02-14 | Stmicroelectronics S.R.L. | Adaptive search window control for visual search |
US9298438B2 (en) * | 2012-06-20 | 2016-03-29 | Microsoft Technology Licensing, Llc | Profiling application code to identify code portions for FPGA implementation |
US9424019B2 (en) | 2012-06-20 | 2016-08-23 | Microsoft Technology Licensing, Llc | Updating hardware libraries for use by applications on a computer system with an FPGA coprocessor |
US10909137B2 (en) | 2014-10-06 | 2021-02-02 | Fisher-Rosemount Systems, Inc. | Streaming data for analytics in process control systems |
US9558220B2 (en) | 2013-03-04 | 2017-01-31 | Fisher-Rosemount Systems, Inc. | Big data in process control systems |
US10866952B2 (en) | 2013-03-04 | 2020-12-15 | Fisher-Rosemount Systems, Inc. | Source-independent queries in distributed industrial system |
US10386827B2 (en) | 2013-03-04 | 2019-08-20 | Fisher-Rosemount Systems, Inc. | Distributed industrial performance monitoring and analytics platform |
US10678225B2 (en) | 2013-03-04 | 2020-06-09 | Fisher-Rosemount Systems, Inc. | Data analytic services for distributed industrial performance monitoring |
US10282676B2 (en) | 2014-10-06 | 2019-05-07 | Fisher-Rosemount Systems, Inc. | Automatic signal processing-based learning in a process plant |
US10649424B2 (en) | 2013-03-04 | 2020-05-12 | Fisher-Rosemount Systems, Inc. | Distributed industrial performance monitoring and analytics |
US9665088B2 (en) | 2014-01-31 | 2017-05-30 | Fisher-Rosemount Systems, Inc. | Managing big data in process control systems |
US10649449B2 (en) | 2013-03-04 | 2020-05-12 | Fisher-Rosemount Systems, Inc. | Distributed industrial performance monitoring and analytics |
US10223327B2 (en) | 2013-03-14 | 2019-03-05 | Fisher-Rosemount Systems, Inc. | Collecting and delivering data to a big data machine in a process control system |
US11573672B2 (en) | 2013-03-15 | 2023-02-07 | Fisher-Rosemount Systems, Inc. | Method for initiating or resuming a mobile control session in a process plant |
EP2973242B1 (en) | 2013-03-15 | 2020-12-23 | Fisher-Rosemount Systems, Inc. | Modelling and adjustment of process plants |
US10235477B2 (en) * | 2014-07-31 | 2019-03-19 | National Instruments Corporation | Prototyping an image processing algorithm and emulating or simulating execution on a hardware accelerator to estimate resource usage or performance |
US10168691B2 (en) | 2014-10-06 | 2019-01-01 | Fisher-Rosemount Systems, Inc. | Data pipeline for process control system analytics |
CN104394137B (en) * | 2014-11-18 | 2017-07-21 | 小米科技有限责任公司 | A kind of method and device of prompting voice call |
CN106470307A (en) * | 2015-08-18 | 2017-03-01 | 泰科电子(上海)有限公司 | Programmable machine sighting device |
US10503483B2 (en) | 2016-02-12 | 2019-12-10 | Fisher-Rosemount Systems, Inc. | Rule builder in a process control network |
US10318904B2 (en) | 2016-05-06 | 2019-06-11 | General Electric Company | Computing system to control the use of physical state attainment of assets to meet temporal performance criteria |
US10691304B1 (en) * | 2018-10-22 | 2020-06-23 | Tableau Software, Inc. | Data preparation user interface with conglomerate heterogeneous process flow elements |
US11250032B1 (en) | 2018-10-22 | 2022-02-15 | Tableau Software, Inc. | Data preparation user interface with conditional remapping of data values |
EP4172757A1 (en) * | 2020-06-29 | 2023-05-03 | Aurora Labs Ltd | Efficient controller data generation and extraction |
US12032994B1 (en) | 2021-10-18 | 2024-07-09 | Tableau Software, LLC | Linking outputs for automatic execution of tasks |
Citations (62)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4811052A (en) * | 1985-08-08 | 1989-03-07 | Canon Kabushiki Kaisha | Control device for control of multi-function control units in an image processing apparatus |
US4941182A (en) * | 1987-07-29 | 1990-07-10 | Phoenix Software Development Co. | Vision system and method for automated painting equipment |
US5168441A (en) * | 1990-05-30 | 1992-12-01 | Allen-Bradley Company, Inc. | Methods for set up and programming of machine and process controllers |
US5309556A (en) * | 1986-03-07 | 1994-05-03 | Hewlett-Packard Company | Method for using interactive computer graphics to control electronic instruments |
US5321603A (en) * | 1992-12-15 | 1994-06-14 | Allen-Bradley Company, Inc. | Programming apparatus for an industrial controller using two-dimensional graphic behavior profiles |
US5392207A (en) * | 1993-08-20 | 1995-02-21 | Allen-Bradley Company, Inc. | Programmable motion controller with graphical programming aid |
US5497498A (en) * | 1992-11-05 | 1996-03-05 | Giga Operations Corporation | Video processing module using a second programmable logic device which reconfigures a first programmable logic device for data transformation |
US5504917A (en) * | 1986-04-14 | 1996-04-02 | National Instruments Corporation | Method and apparatus for providing picture generation and control features in a graphical data flow environment |
US5535342A (en) * | 1992-11-05 | 1996-07-09 | Giga Operations Corporation | Pld connector for module having configuration of either first PLD or second PLD and reconfigurable bus for communication of two different bus protocols |
US5541849A (en) * | 1990-04-06 | 1996-07-30 | Lsi Logic Corporation | Method and system for creating and validating low level description of electronic design from higher level, behavior-oriented description, including estimation and comparison of timing parameters |
US5583749A (en) * | 1994-11-30 | 1996-12-10 | Altera Corporation | Baseboard and daughtercard apparatus for reconfigurable computing systems |
US5603043A (en) * | 1992-11-05 | 1997-02-11 | Giga Operations Corporation | System for compiling algorithmic language source code for implementation in programmable hardware |
US5631974A (en) * | 1990-08-31 | 1997-05-20 | Canon Research Centre Europe, Ltd. | Image processing |
US5638299A (en) * | 1995-06-22 | 1997-06-10 | Miller; Keith | Light weight, self-contained programmable data-acquisition system |
US5652875A (en) * | 1993-09-27 | 1997-07-29 | Giga Operations Corporation | Implementation of a selected instruction set CPU in programmable hardware |
US5684980A (en) * | 1992-07-29 | 1997-11-04 | Virtual Computer Corporation | FPGA virtual computer for executing a sequence of program instructions by successively reconfiguring a group of FPGA in response to those instructions |
US5732277A (en) * | 1986-10-24 | 1998-03-24 | National Instruments Corporation | Graphical system for modelling a process and associated method |
US5737235A (en) * | 1995-05-02 | 1998-04-07 | Xilinx Inc | FPGA with parallel and serial user interfaces |
US5742504A (en) * | 1995-11-06 | 1998-04-21 | Medar, Inc. | Method and system for quickly developing application software for use in a machine vision system |
US5754227A (en) * | 1994-09-28 | 1998-05-19 | Ricoh Company, Ltd. | Digital electronic camera having an external input/output interface through which the camera is monitored and controlled |
US5838968A (en) * | 1996-03-01 | 1998-11-17 | Chromatic Research, Inc. | System and method for dynamic resource management across tasks in real-time operating systems |
US5844606A (en) * | 1994-03-03 | 1998-12-01 | Fuji Photo Film Co., Ltd. | Videocamera having a multiconnector connectable to a variety of accessories |
US5847953A (en) * | 1996-09-23 | 1998-12-08 | National Instruments Corporation | System and method for performing class checking of objects in a graphical data flow program |
US5933638A (en) * | 1996-04-04 | 1999-08-03 | Cencik; Peter | Method and apparatus for creating application computer programs using an objects evaluation matrix |
US5990906A (en) * | 1997-06-25 | 1999-11-23 | National Instruments Corporation | Undo feature for a graphical programming system |
US6006039A (en) * | 1996-02-13 | 1999-12-21 | Fotonation, Inc. | Method and apparatus for configuring a camera through external means |
US6028611A (en) * | 1996-08-29 | 2000-02-22 | Apple Computer, Inc. | Modular digital image processing via an image processing chain |
US6064409A (en) * | 1993-09-22 | 2000-05-16 | National Instruments Corporation | System and method for providing audio probe and debugging features in a graphical data flow program |
US6157394A (en) * | 1996-08-29 | 2000-12-05 | Apple Computer, Inc. | Flexible digital image processing via an image processing chain with modular image processors |
US6173438B1 (en) * | 1997-08-18 | 2001-01-09 | National Instruments Corporation | Embedded graphical programming system |
US6177950B1 (en) * | 1996-01-17 | 2001-01-23 | Avt Audio Visual | Multifunctional portable telephone |
US6219628B1 (en) * | 1997-08-18 | 2001-04-17 | National Instruments Corporation | System and method for configuring an instrument to perform measurement functions utilizing conversion of graphical programs into hardware implementations |
US6272245B1 (en) * | 1998-01-23 | 2001-08-07 | Seiko Epson Corporation | Apparatus and method for pattern recognition |
US20010013890A1 (en) * | 1998-04-29 | 2001-08-16 | Chandrasekhar Narayanaswami | Camera for use with personal digital assistants with high speed communication link |
US6282462B1 (en) * | 1996-06-28 | 2001-08-28 | Metrovideo Inc. | Image acquisition system |
US6298474B1 (en) * | 1999-04-30 | 2001-10-02 | Intergral Vision, Inc. | Method and system for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system and computer-readable storage medium having a program for executing the method |
US6345130B1 (en) * | 1996-08-28 | 2002-02-05 | Ralip International Ab | Method and arrangement for ensuring quality during scanning/copying of images/documents |
US20020054125A1 (en) * | 1996-01-25 | 2002-05-09 | Oliver Salomon | Graphical user interface for the programming of programmable controllers |
US20020071051A1 (en) * | 1996-06-27 | 2002-06-13 | Osamu Ikeda | Electronic imaging apparatus having opening that receives plural function cards and having plural connector terminals |
US6437826B1 (en) * | 1998-07-06 | 2002-08-20 | Conexant Systems, Inc. | Digital video teleconferencing camera system having a base |
US20020140824A1 (en) * | 2001-04-02 | 2002-10-03 | Christoff Jordan C. | System and method for processing low illumination image data |
US20020186302A1 (en) * | 1999-09-03 | 2002-12-12 | Veijo Pulkinnen | Camera control in a process control system |
US6496361B2 (en) * | 1998-11-16 | 2002-12-17 | Acer Incorporated | Embedded CMOS camera in a laptop computer |
US6504575B1 (en) * | 1998-02-27 | 2003-01-07 | Flashpoint Technology, Inc. | Method and system for displaying overlay bars in a digital imaging device |
US6546207B2 (en) * | 2000-02-07 | 2003-04-08 | Pentax Corporation | Camera capable of inputting data and selectively displaying image |
US6587140B2 (en) * | 1997-10-23 | 2003-07-01 | Eastman Kodak Company | System and method for using a single intelligence circuit in both a digital camera and printer |
US6597346B1 (en) * | 1998-10-02 | 2003-07-22 | Honeywell Inc. | Hand held computer with see-through display |
US6628198B2 (en) * | 2001-02-15 | 2003-09-30 | International Business Machines Corporation | Security system for preventing a personal computer from being stolen or used by unauthorized people |
US6680749B1 (en) * | 1997-05-05 | 2004-01-20 | Flashpoint Technology, Inc. | Method and system for integrating an application user interface with a digital camera user interface |
US6795651B2 (en) * | 1998-11-09 | 2004-09-21 | Silverbrook Research Pty. Ltd. | Camera with detachable printer |
US20040183914A1 (en) * | 1997-07-15 | 2004-09-23 | Kia Silverbrook | Portable hand held camera |
US6832920B2 (en) * | 1993-11-12 | 2004-12-21 | Intel Corporation | Compliant communications connectors |
US6842652B2 (en) * | 2002-02-22 | 2005-01-11 | Concord Camera Corp. | Image capture device |
US6873367B1 (en) * | 1999-06-03 | 2005-03-29 | Olympus Optical Co., Ltd. | Electronic camera system |
US6886168B2 (en) * | 2000-01-20 | 2005-04-26 | Q3Dm, Llc | Method and system for extensible data processing |
US6888566B2 (en) * | 1999-12-14 | 2005-05-03 | Canon Kabushiki Kaisha | Method and apparatus for uniform lineal motion blur estimation using multiple exposures |
US6923377B2 (en) * | 1994-04-29 | 2005-08-02 | Psc Inc. | PCMCIA interface card for coupling input devices such as barcode scanning engines to personal digital assistants and palmtop computers |
US6930709B1 (en) * | 1997-12-04 | 2005-08-16 | Pentax Of America, Inc. | Integrated internet/intranet camera |
US6971066B2 (en) * | 1997-08-18 | 2005-11-29 | National Instruments Corporation | System and method for deploying a graphical program on an image acquisition device |
US6992711B2 (en) * | 1999-12-17 | 2006-01-31 | Minolta Co., Ltd. | Digital camera with a plurality of media for recording captured images |
US7142731B1 (en) * | 1999-02-17 | 2006-11-28 | Nec Corporation | Image processing system |
US7327396B2 (en) * | 2002-04-10 | 2008-02-05 | National Instruments Corporation | Smart camera with a plurality of slots for modular expansion capability through a variety of function modules connected to the smart camera |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1994015311A1 (en) | 1992-12-28 | 1994-07-07 | Xilinx, Inc. | Method for entering state flow diagrams using schematic editor programs |
-
2001
- 2001-09-10 US US09/949,783 patent/US6971066B2/en not_active Expired - Lifetime
-
2005
- 2005-08-26 US US11/212,326 patent/US20060005160A1/en not_active Abandoned
Patent Citations (64)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4811052A (en) * | 1985-08-08 | 1989-03-07 | Canon Kabushiki Kaisha | Control device for control of multi-function control units in an image processing apparatus |
US5309556A (en) * | 1986-03-07 | 1994-05-03 | Hewlett-Packard Company | Method for using interactive computer graphics to control electronic instruments |
US5504917A (en) * | 1986-04-14 | 1996-04-02 | National Instruments Corporation | Method and apparatus for providing picture generation and control features in a graphical data flow environment |
US5732277A (en) * | 1986-10-24 | 1998-03-24 | National Instruments Corporation | Graphical system for modelling a process and associated method |
US4941182A (en) * | 1987-07-29 | 1990-07-10 | Phoenix Software Development Co. | Vision system and method for automated painting equipment |
US5541849A (en) * | 1990-04-06 | 1996-07-30 | Lsi Logic Corporation | Method and system for creating and validating low level description of electronic design from higher level, behavior-oriented description, including estimation and comparison of timing parameters |
US5168441A (en) * | 1990-05-30 | 1992-12-01 | Allen-Bradley Company, Inc. | Methods for set up and programming of machine and process controllers |
US5631974A (en) * | 1990-08-31 | 1997-05-20 | Canon Research Centre Europe, Ltd. | Image processing |
US5684980A (en) * | 1992-07-29 | 1997-11-04 | Virtual Computer Corporation | FPGA virtual computer for executing a sequence of program instructions by successively reconfiguring a group of FPGA in response to those instructions |
US5535342A (en) * | 1992-11-05 | 1996-07-09 | Giga Operations Corporation | Pld connector for module having configuration of either first PLD or second PLD and reconfigurable bus for communication of two different bus protocols |
US5603043A (en) * | 1992-11-05 | 1997-02-11 | Giga Operations Corporation | System for compiling algorithmic language source code for implementation in programmable hardware |
US5497498A (en) * | 1992-11-05 | 1996-03-05 | Giga Operations Corporation | Video processing module using a second programmable logic device which reconfigures a first programmable logic device for data transformation |
US5321603A (en) * | 1992-12-15 | 1994-06-14 | Allen-Bradley Company, Inc. | Programming apparatus for an industrial controller using two-dimensional graphic behavior profiles |
US5392207A (en) * | 1993-08-20 | 1995-02-21 | Allen-Bradley Company, Inc. | Programmable motion controller with graphical programming aid |
US6064409A (en) * | 1993-09-22 | 2000-05-16 | National Instruments Corporation | System and method for providing audio probe and debugging features in a graphical data flow program |
US5652875A (en) * | 1993-09-27 | 1997-07-29 | Giga Operations Corporation | Implementation of a selected instruction set CPU in programmable hardware |
US6832920B2 (en) * | 1993-11-12 | 2004-12-21 | Intel Corporation | Compliant communications connectors |
US5844606A (en) * | 1994-03-03 | 1998-12-01 | Fuji Photo Film Co., Ltd. | Videocamera having a multiconnector connectable to a variety of accessories |
US6923377B2 (en) * | 1994-04-29 | 2005-08-02 | Psc Inc. | PCMCIA interface card for coupling input devices such as barcode scanning engines to personal digital assistants and palmtop computers |
US5754227A (en) * | 1994-09-28 | 1998-05-19 | Ricoh Company, Ltd. | Digital electronic camera having an external input/output interface through which the camera is monitored and controlled |
US5583749A (en) * | 1994-11-30 | 1996-12-10 | Altera Corporation | Baseboard and daughtercard apparatus for reconfigurable computing systems |
US5737235A (en) * | 1995-05-02 | 1998-04-07 | Xilinx Inc | FPGA with parallel and serial user interfaces |
US5638299A (en) * | 1995-06-22 | 1997-06-10 | Miller; Keith | Light weight, self-contained programmable data-acquisition system |
US5742504A (en) * | 1995-11-06 | 1998-04-21 | Medar, Inc. | Method and system for quickly developing application software for use in a machine vision system |
US6177950B1 (en) * | 1996-01-17 | 2001-01-23 | Avt Audio Visual | Multifunctional portable telephone |
US20020054125A1 (en) * | 1996-01-25 | 2002-05-09 | Oliver Salomon | Graphical user interface for the programming of programmable controllers |
US6006039A (en) * | 1996-02-13 | 1999-12-21 | Fotonation, Inc. | Method and apparatus for configuring a camera through external means |
US5838968A (en) * | 1996-03-01 | 1998-11-17 | Chromatic Research, Inc. | System and method for dynamic resource management across tasks in real-time operating systems |
US5933638A (en) * | 1996-04-04 | 1999-08-03 | Cencik; Peter | Method and apparatus for creating application computer programs using an objects evaluation matrix |
US20020071051A1 (en) * | 1996-06-27 | 2002-06-13 | Osamu Ikeda | Electronic imaging apparatus having opening that receives plural function cards and having plural connector terminals |
US6282462B1 (en) * | 1996-06-28 | 2001-08-28 | Metrovideo Inc. | Image acquisition system |
US6345130B1 (en) * | 1996-08-28 | 2002-02-05 | Ralip International Ab | Method and arrangement for ensuring quality during scanning/copying of images/documents |
US6028611A (en) * | 1996-08-29 | 2000-02-22 | Apple Computer, Inc. | Modular digital image processing via an image processing chain |
US6157394A (en) * | 1996-08-29 | 2000-12-05 | Apple Computer, Inc. | Flexible digital image processing via an image processing chain with modular image processors |
US5847953A (en) * | 1996-09-23 | 1998-12-08 | National Instruments Corporation | System and method for performing class checking of objects in a graphical data flow program |
US6680749B1 (en) * | 1997-05-05 | 2004-01-20 | Flashpoint Technology, Inc. | Method and system for integrating an application user interface with a digital camera user interface |
US5990906A (en) * | 1997-06-25 | 1999-11-23 | National Instruments Corporation | Undo feature for a graphical programming system |
US7453492B2 (en) * | 1997-07-15 | 2008-11-18 | Silverbrook Research Pty Ltd | Portable hand held camera |
US20040183914A1 (en) * | 1997-07-15 | 2004-09-23 | Kia Silverbrook | Portable hand held camera |
US6971066B2 (en) * | 1997-08-18 | 2005-11-29 | National Instruments Corporation | System and method for deploying a graphical program on an image acquisition device |
US6219628B1 (en) * | 1997-08-18 | 2001-04-17 | National Instruments Corporation | System and method for configuring an instrument to perform measurement functions utilizing conversion of graphical programs into hardware implementations |
US6173438B1 (en) * | 1997-08-18 | 2001-01-09 | National Instruments Corporation | Embedded graphical programming system |
US6587140B2 (en) * | 1997-10-23 | 2003-07-01 | Eastman Kodak Company | System and method for using a single intelligence circuit in both a digital camera and printer |
US6930709B1 (en) * | 1997-12-04 | 2005-08-16 | Pentax Of America, Inc. | Integrated internet/intranet camera |
US6272245B1 (en) * | 1998-01-23 | 2001-08-07 | Seiko Epson Corporation | Apparatus and method for pattern recognition |
US6504575B1 (en) * | 1998-02-27 | 2003-01-07 | Flashpoint Technology, Inc. | Method and system for displaying overlay bars in a digital imaging device |
US20010013890A1 (en) * | 1998-04-29 | 2001-08-16 | Chandrasekhar Narayanaswami | Camera for use with personal digital assistants with high speed communication link |
US6657654B2 (en) * | 1998-04-29 | 2003-12-02 | International Business Machines Corporation | Camera for use with personal digital assistants with high speed communication link |
US6437826B1 (en) * | 1998-07-06 | 2002-08-20 | Conexant Systems, Inc. | Digital video teleconferencing camera system having a base |
US6597346B1 (en) * | 1998-10-02 | 2003-07-22 | Honeywell Inc. | Hand held computer with see-through display |
US6795651B2 (en) * | 1998-11-09 | 2004-09-21 | Silverbrook Research Pty. Ltd. | Camera with detachable printer |
US6496361B2 (en) * | 1998-11-16 | 2002-12-17 | Acer Incorporated | Embedded CMOS camera in a laptop computer |
US7142731B1 (en) * | 1999-02-17 | 2006-11-28 | Nec Corporation | Image processing system |
US6298474B1 (en) * | 1999-04-30 | 2001-10-02 | Intergral Vision, Inc. | Method and system for interactively developing a graphical control-flow structure and associated application software for use in a machine vision system and computer-readable storage medium having a program for executing the method |
US6873367B1 (en) * | 1999-06-03 | 2005-03-29 | Olympus Optical Co., Ltd. | Electronic camera system |
US20020186302A1 (en) * | 1999-09-03 | 2002-12-12 | Veijo Pulkinnen | Camera control in a process control system |
US6888566B2 (en) * | 1999-12-14 | 2005-05-03 | Canon Kabushiki Kaisha | Method and apparatus for uniform lineal motion blur estimation using multiple exposures |
US6992711B2 (en) * | 1999-12-17 | 2006-01-31 | Minolta Co., Ltd. | Digital camera with a plurality of media for recording captured images |
US6886168B2 (en) * | 2000-01-20 | 2005-04-26 | Q3Dm, Llc | Method and system for extensible data processing |
US6546207B2 (en) * | 2000-02-07 | 2003-04-08 | Pentax Corporation | Camera capable of inputting data and selectively displaying image |
US6628198B2 (en) * | 2001-02-15 | 2003-09-30 | International Business Machines Corporation | Security system for preventing a personal computer from being stolen or used by unauthorized people |
US20020140824A1 (en) * | 2001-04-02 | 2002-10-03 | Christoff Jordan C. | System and method for processing low illumination image data |
US6842652B2 (en) * | 2002-02-22 | 2005-01-11 | Concord Camera Corp. | Image capture device |
US7327396B2 (en) * | 2002-04-10 | 2008-02-05 | National Instruments Corporation | Smart camera with a plurality of slots for modular expansion capability through a variety of function modules connected to the smart camera |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7328288B2 (en) * | 2003-12-11 | 2008-02-05 | Canon Kabushiki Kaisha | Relay apparatus for relaying communication from CPU to peripheral device |
US20050132098A1 (en) * | 2003-12-11 | 2005-06-16 | Canon Kabushiki Kaisha | Relay apparatus for relaying communication from CPU to peripheral device |
US20080058969A1 (en) * | 2006-09-01 | 2008-03-06 | Fisher-Rosemount Systems, Inc. | Graphical Programming Language Object Editing and Reporting Tool |
US7668608B2 (en) | 2006-09-01 | 2010-02-23 | Fisher-Rosemount Systems, Inc. | Graphical programming language object editing and reporting tool |
US9123093B1 (en) | 2008-08-29 | 2015-09-01 | Cognex Corporation | Vision inspection programming method and apparatus |
US20100194548A1 (en) * | 2009-01-30 | 2010-08-05 | Tm Industrials, Llc | Downtime monitoring apparatus and method |
US10561841B2 (en) * | 2010-02-26 | 2020-02-18 | Cornell University | Retina prosthesis |
US20190201693A1 (en) * | 2010-02-26 | 2019-07-04 | Cornell University | Retina prosthesis |
US10039921B2 (en) * | 2010-02-26 | 2018-08-07 | Cornell University | Retina prosthesis |
US20160263379A1 (en) * | 2010-02-26 | 2016-09-15 | Cornell University | Retina prosthesis |
US9925373B2 (en) | 2010-09-10 | 2018-03-27 | Cornell University | Neurological prosthesis |
US9547804B2 (en) * | 2011-08-25 | 2017-01-17 | Cornell University | Retinal encoder for machine vision |
US20170255837A1 (en) * | 2011-08-25 | 2017-09-07 | Cornell University | Retinal encoder for machine vision |
US10303970B2 (en) * | 2011-08-25 | 2019-05-28 | Cornell University | Retinal encoder for machine vision |
US20140355861A1 (en) * | 2011-08-25 | 2014-12-04 | Cornell University | Retinal encoder for machine vision |
WO2013029008A1 (en) * | 2011-08-25 | 2013-02-28 | Cornell University | Retinal encoder for machine vision |
US10769483B2 (en) * | 2011-08-25 | 2020-09-08 | Cornell University | Retinal encoder for machine vision |
US11640681B2 (en) * | 2011-08-25 | 2023-05-02 | Cornell University | Retinal encoder for machine vision |
US10515269B2 (en) | 2015-04-20 | 2019-12-24 | Cornell University | Machine vision with dimensional data reduction |
US11430263B2 (en) | 2015-04-20 | 2022-08-30 | Cornell University | Machine vision with dimensional data reduction |
CN105511887A (en) * | 2015-12-28 | 2016-04-20 | 鲍敏祺 | Hardware visualization programming framework |
CN108245099A (en) * | 2018-01-15 | 2018-07-06 | 深圳市沃特沃德股份有限公司 | Robot moving method and device |
CN112707506A (en) * | 2020-12-28 | 2021-04-27 | 天津凯英科技发展股份有限公司 | Sewage treatment method and device and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
US20020055947A1 (en) | 2002-05-09 |
US6971066B2 (en) | 2005-11-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6971066B2 (en) | System and method for deploying a graphical program on an image acquisition device | |
US6784903B2 (en) | System and method for configuring an instrument to perform measurement functions utilizing conversion of graphical programs into hardware implementations | |
US7290244B2 (en) | System and method for configuring a reconfigurable system | |
US7085670B2 (en) | Reconfigurable measurement system utilizing a programmable hardware element and fixed hardware resources | |
US7606950B2 (en) | Graphical programs with direct memory access FIFO for controller/FPGA communications | |
US8640027B2 (en) | System and method for configuring a hardware device to execute a prototype | |
US6584601B1 (en) | System and method for converting graphical programs into hardware implementations which utilize probe insertion | |
US6608638B1 (en) | System and method for configuring a programmable hardware instrument to perform measurement functions utilizing estimation of the hardware implentation and management of hardware resources | |
US7743362B2 (en) | Automatic generation of application domain specific graphical programs | |
US7340693B2 (en) | System for designing re-programmable digital hardware platforms | |
US8448135B2 (en) | Race structure for a graphical program | |
US7725877B2 (en) | Graphical program which includes an I/O node for hardware abstraction | |
US8943469B2 (en) | Type generic graphical programming | |
US7316009B2 (en) | Emulation of a programmable hardware element | |
US6219628B1 (en) | System and method for configuring an instrument to perform measurement functions utilizing conversion of graphical programs into hardware implementations | |
US8078980B2 (en) | User defined wire appearance indicating communication functionality in a graphical programming environment | |
US7533347B2 (en) | Creating a graphical user interface for selected parameters of a graphical program | |
WO2007047909A2 (en) | Graphical programs with fifo structure for controller with fpga communications | |
US7725874B2 (en) | Combination structure nodes for a graphical program | |
US8074177B2 (en) | User defined wire appearance indicating data type in a graphical programming environment | |
US20060036799A1 (en) | Multi-platform development and execution of graphical programs | |
US7293112B2 (en) | Graphical program node for displaying acquired images | |
US7647578B2 (en) | Programmatic creation and management of tasks in a graphical program | |
AU2003264083B2 (en) | Object-oriented design method for the time-effective and cost-effective development of production-grade embedded systems based on a standardized system architecture |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |