CA2387441A1 - Image-synchronized multichannel biomedical data acquisition system - Google Patents
Image-synchronized multichannel biomedical data acquisition system Download PDFInfo
- Publication number
- CA2387441A1 CA2387441A1 CA002387441A CA2387441A CA2387441A1 CA 2387441 A1 CA2387441 A1 CA 2387441A1 CA 002387441 A CA002387441 A CA 002387441A CA 2387441 A CA2387441 A CA 2387441A CA 2387441 A1 CA2387441 A1 CA 2387441A1
- Authority
- CA
- Canada
- Prior art keywords
- video
- data
- frame
- computer
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
Abstract
A new video-synchronized multi-channel data acquisition system has been developed for frequency-domain muscle fatigue research. The system records storage-intensive video images onto a video tape, and simultaneously acquires biomedical data and video time codes onto a computer hard disk to achieve high-speed recording a long duration. A video time-code-bridge-file was created by the computer to synchronize the biomedical data with the recorded video frames in realtime. The two-column time-code-bridge-file matches each video frame-start with the corresponding index number of the acquired data. With the bridge file, the system is able to automatically search and output a frame of the acquired multi-channel data correspondent to a given video frame, and also able to search and display a video frame correspondent to a given frame of acquired multi-channel data. Currently, this system is capable of recording 30 minutes of video-synchronized multi-channel biological data with the summed data rate of 2.16 Mbit/sec and synchronization accuracy of 0.22 mS.
Description
IMAGE-SYNCHRONIZED MULTICHANNEL
BIOMEDICAL DATA ACQUISITION SYSTEM
FIELD OF THE INVENTION
The present invention is directed to event synchronized data acquisition systems, and more particularly to video synchronized multi channel data acquisition for the analysis of biomedical information.
BACKGROUND OF THE INVENTION
Video-synchronized data acquisition is finding more applica-tions in biomedical engineering research. Some of these applications require to a video-synchronized data-acquisition system with multi-data channels and high-frequency bandwidth, and also maintain long acquisition duration.
One example of high-frequency application is frequency-domain muscle-fatigue analysis, for which a mufti-channel data acquisition system with a high raw-data signal frequency bandwidth of up to 3 kHz per channel is s5 recommended by D. G. Gerleman and T.M. Cook, "Instrumentation," in Selected Topics in Surface Electromyography for Use in the Occupational Setting:
Expert Perspectives, U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control, National Institute for Occupa-tional Safety and Health, pp. 44-68,1992. A number of schemes have been 2o published to achieve video-synchronized data acquisition, such as, D. M.
Gaskill, "Techniques for synchronizing thermal array chart recorders to video," 28th International Telemetering Conference (Proceedings)-ITC/USA/92, Vol. 28, pp. 61-64,1992; M. Vannier et al., "Time and motion studies of medical imaging workstations," SPIE, Vol.1653, Image Capture, 25 Formatting, and Display, pp. 274-280,1992; T. Y Yen and R. G. Radwin, "A
video-based system for acquiring biomechanical data synchronized with arbitrary events and activities," IEEE Transactions on Biomedical Engineering, Vol. 42, pp. 944-948,1995; M. Gacia et al., "Characterization of near-bed coherent structures in turbulent open channel flow using synchronized high-speed video and hot-film measurements," Experiments in Fluids, Vol. 19, pp.
16-28,1995; T. Engstrom and P. Medbo, "Data collection and analysis of manual work using video recording and personal computer techniques,"
International Journal of Industrial Ergonomics,19 (1997), pp. 291-298,1997.
Some of the schemes consist of simple event analyses stamped with video time codes, e.g., M. Vannier et al., "Time and motion studies of medical imaging workstations," SPIE, Vol.1653, Image Capture, Formatting, and Display, pp. 274-280,1992; T. Engstrom and P. Medbo, "Data collection and analysis of manual work using video recording and personal computer to techniques," International Journal of Industrial Ergonomics,19 (1997), pp.
291-298,1997. Some other sophisticated schemes have achieved multi-channel data acquisition. Yen and Radwin utilized an audio-track digital recording method to record up to 32 channels of 8-bit data onto video tape audio tracks, and to record corresponding video images onto the video track of the same tape. See, T. Y Yen and R. G. Radwin, "A video-based system for acquiring biomechanical data synchronized with arbitrary events and activities," IEEE Transactions on Biomedical Engineering, Vol. 42, pp. 944-948, 1995. This method naturally synchronizes the video frames with the acquired data for extensive recording duration. But, because of the 2o frequency-bandwidth limit of the audio tracks (20 Hz - 20 kHz), the scanning frequency of the recorded data is limited to 60 Hz per channel. Higher-scanning-rate multi-channel data acquisition was realized by using a high-speed video camera (1000 frames/sec) which outputs a synchronization signal in each video frame to trigger a computer data acquisition sequence.
See, M. Gacia et al., "Characterization of near-bed coherent structures in turbulent open channel flow using synchronized high-speed video and hot-film measurements," Experiments in Fluids, Vol.19, pp. 16-28,1995. The maximum scanning frequency of this method may reach 1000 Hz per channel for 20 seconds of 1000 frames per second video recording. To 3o acquire high frequency multi-channel data for longer duration, Gaskill _2_ suggested a data acquisition system using a digital audio tape recorder or a thermal chart recorder controlled by video time codes or vertical-video-sync pulses output from the video recording system.
This literature reports a new computerized data-acquisition system which extends Gaskill's suggestion by using a computer synchroni-zation program to realize high-frequency long-duration video-synchronized biomedical-data acquisition. It would be desirable therefore to provide a broad band mufti-channel video-synchronized data acquisition system which correlates video signal information with numerous data leads io associated with biomedical signals and the like. Additionally, it would be advantageous to provide for the acquisition of frequency domain signals which may be correlated with video signals acquired by the system. The system records the storage-intensive video images onto a video tape, and simultaneously acquires biomedical data and video time codes onto a computer hard disk. A time-code-bridge-file is created by the computer to synchronize the acquired biomedical data on the hard drive with the recorded human images on the video tape. Without manipulating video signals in the system computer, this separated recording method boosts the data acquisition speed, enables the realtime data synchronization and long 2o recording duration. The video time-code frame-start bits were selected as the synchronization reference. With this reference, the synchronization error can be theoretically controlled within one SMPTE time-code bit duration (0.417 mS).
SUMMARY OF THE INVENTION
A video-synchronized mufti-channel data acquisition system is provided for frequency domain analysis of biomedical signals acquired with video signals, and particularly for correlating analysis with muscle fatigue research. The disclosed system records storage-intensive video images onto a video tape, and also provides for the data acquisition of information signals with associated video time codes on a personal computer hard disk to achieve high speed recording over a long duration. A video time code-bridge file has been provided for use by the computer to synchronize the biomedical data with the recorded video frame in real time.
Briefly summarized, the present invention relates to a data acquisition system, in which an apparatus such as a video tape recorder is provided for recording video frames from an input video signal. A
biomedical instrument generates an input biomedical data stream to a system computer providing at least one digital input-output port and to interface circuitry for collecting the input video time code signal and for collecting the input biomedical data stream. A bridge file is provided with the system computer for synchronizing the video frames from the input video signal with the input biomedical data stream.
The appended claims set forth the features of the present invention with particularity. The invention and its advantages may be best understood from the following detailed description taken in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG.1 shows a video-synchronized biomedical data acquisition 2o system in accordance with the invention;
FIG. 2 illustrates in block diagram form the program flow of the data acquisition system of FIG. 1;
FIG. 3 illustrates a longitudinal time code output decoding procedure in accordance with the invention;
FIG. 4 shows the structure of the program files of the data acquisition system correlating acquired data file information with elements per video frame;
FIG. 5 shows the structure of the interface board shown in the system diagram of FIG.1;
FIGS. 6a and 6b illustrate the instrumentation setup for synchronizing extracted video information with biomedical signal acquisition in accordance with the invention; and FIG. 7 illustrates an instrumental setup for data and video acquisition by one computer;
FIG. 8 illustrates block diagrams for data and video acquisition by one computer;
FIG. 9 illustrates program file structures for data and video acquisition by one computer;
so FIG.10 illustrates an instrumental setup for data and video acquisition by separate computers;
FIG.11 illustrates block diagrams for data and video acquisition by separate computers; and FIG. 12 illustrates two time-code-bridge-files for data and video acquisition by separate computers.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
As shown in FIG.1, a computerized video-synchronized data acquisition system 10 is shown a video camera 12 to output NTSC color video signals 14, a digital VTR 16 with a SMPTE longitudinal time code (LTC) output 22, a TV monitor 18, a mufti-channel biomedical-signal-measuring instrument 20 with parallel digital output 24, a signal-interface board 26 to condition the incoming (22,24) and outgoing (28,30) data and control the direction of data flow, a 266-MHz Pentium II microprocessor-based system PC computer 32 with a 32-bit digital I/O board, and an analysis PC computer 34 if the system needs to run the instrument's own analysis software. LabVIEW graphical programming software (National Instruments, Austin, TX) was used to program the data acquisition, processing, storage and replay, and VTR control.
The system block diagrams are shown in FIG. 2. During video-synchronized data acquisition (FIG. 2 (a)), the VTR records NTSC human activity images and outputs a 1-bit LTC stream to the interface board. The LTC stream is bi-phase modulated by frame numbers and sync words, as shown in FIG. 3(a). Such techniques for modulations with frame numbers and sync word have been demonstrated in D. M. Huber, Audio production technigues for video. White Plains, NY and London: Knowledge Industry Publications, Inc., pp. 93-100,1987.
FIG. 2 represents three flow charts, i.e., video synchronized to data acquisition 36 (2a); data search 38 (2b); and video search 40 (2c).
The video synchronization data acquisition 36 is illustrated in block diagram form in 2(a), and further broken down into hardware component 54 and software component 56. The hardware component 54 illustrates video recording 42 and data recording 44 which may be provided simultaneously i5 to provide longitudinal time code (LTC) output to record human activity with video imaging, and additionally provide a biomedical data stream from a data recording instrument 44, which is provided as a mufti-channel electromyography (EMG) data acquisition instrument. An interface board 46 corresponding to the interface board 26 of FIG.1 operates in a data 2o acquisition mode to receive the LTC and data signals from the video recording and data recording blocks 42 and 44 respectively. Accordingly, the interface board in its data acquisition mode 46 provides output signals to display biomedical data on an analysis computer monitor 48, and an output signal to a data acquisition board 50. The data acquisition at block 50 25 provides write data to the system computer buffer at block 52. Next, a software loop is provided in the software section 56, which obtains data from the system computer buffer in a read data buffer block 58. An acquired data file is written to the hard disk of the system computer at block 60, and an LTC decode is performed at block 62. Block 64 is provided to correct LTC
3o errors, and interpolation of the LTC signal at block 66 is used with the bridge file elements in a block 68 for forming bridge file elements. The bridge file elements from block 68 are provided as appended elements to the bridge file on the hard disk at block 70.
At FIG. 2(c), a data search 38 is illustrated in which a replay of the video tape is performed at 72, from which selected video frames at 74 are presented to the interface board 26 in its acquisition mode at block 76. A
decode of the LTC selected frame at 78 along with a search of the bridge file 80 is used to retrieve data from the acquired data file at block 82. The interface board 26 then provided in a replay mode at block 84 facilitates the io replay and output of data to an analysis computer at block 86.
At FIG. 2(c) the video search 40 is illustrated, wherein the interface board 26 in its replay mode at block 88 replays a data segment 90 from which a selected data frame is provided at block 92. At block 94 the selected data frame is used to determine LTC of the selected frame. An RS 232 interface is facilitated at block 96 which is used with a program loop for reading the video tape LTC starting at block 98. Block 100 is used to compare the LTC difference and block 102 calculates forward or reverse time. Block 104 converts the calculated time to forward or reverse on the vide tape recorder 16. A loop decision block at 106 determines whether the 2o video frame selected is correct or forward or reverse video frames must be used, and then if the video frame is correct at decision block 106, a display of the video frame is provided on the TV monitor at block 108.
FIGS. 3(a), 3(b), 3(c), and 3(d) illustrate the LTC decoding procedure from a data stream 110 obtained from the video recording. In FIG. 4(a), the acquired data file 112 is illustrated as several elements per video frame (EPVF) arranged as 32-bit integer data elements provided with a video frame number and an index number at the beginning of each frame.
FIG. 4(b) shows the interpolated time code bridge file which may be used to correlate the video frame number with the LTC and data file index columns.
Simultaneously, the biomedical instrument measures human biomedical signals, and outputs up to 31 bits of data/sync signals to the interface board. The interface board in turn buffers the total of 32-bit data/ sync/ LTC signals, and outputs them to the 32-bit digital I/ O board (National Instrument AT-DIO-32F). The interface board also directs the 31-bit data/sync signals to the data analysis computer for realtime data display. The digital I/ O board scans the 32-bit signal stream at a scanning frequency of up to 72 kHz, converts the stream into 32-bit integers, and writes the integers into the system computer buffer. Whenever the buffer is to half full, the program retrieves all the integers from the buffer, and appends them to an acquired-data-file on the hard disk (see FIG. 4(a) for acquired-data-file structure). In the meantime, the program retrieves the LTC bit stream from the integers, and sends the LTC stream to the decoding subroutine for LTC decoding. During decoding, the program determines ~5 the index number of the acquired-data-file at each video frame-start, and translates each 80-bit LTC into a unique 8-digit frame number integer as shown in FIGS. 3(b), 3(c) and 3(d). After decoding, the program forms a time-code-bridge-file with a two-column array. As shown in FIG. 4(b), the LTC column of the array contains the 8-digit frame-number integers of all 2o recorded video frames, and the data-file-index column contains the corresponding index numbers of the acquired-data-file at frame-starts. In this way, each row of the array represents a unique video frame and its corresponding acquired-data-file index number at the frame-start. With this time-code-bridge-file, the computer is able to search for any data segment 25 correspondent to a given video frame, or search for any video frame correspondent to a given data segment.
The interface board has two flow-control modes: data-acquisition mode and data-replay mode that are shown in FIGS. 5(a) and 5(b), respectively. The main task of the interface board is to control the data 3o flow direction. FIGS. 5(a) and 5(b) illustrate the interface board 26 in data _g_ acquisition mode and data replay modes, respectively. As shown in FIG. 5(a), the data acquisition mode configuration of the interface board 26 facilitates the receipt of LTC from the video tape recorder 16, as well as the biomedical data stream from the biomedical instrument 20, which signals are buffered in a tristate digital buffer of the data acquisition board 26, which provides buffered output signals to the digital I/O board which provides an input-output port for the system computer 32. Additionally, a buffered output of data/sync signals are provided to the analysis computer 34 . In FIG. 5(b), the interface board 26 configured in the data replay mode 1o receives control data from the system computer 32, which provides data signals to the analysis computer 34 to replay the acquired data to the analysis computer 34.
The flow-control mode is selected by the program which outputs a control bit from the digital I/O board to the interface board. The interface board buffers the incoming and outgoing data and converts the polarity of the LTC stream from ~10 V to 0-5 V. In data-acquisition mode, the 31-bit data/ sync signals from the biomedical instrument, and the corresponding 1-bit LTC stream from the VTR are input to the interface board. The interface board directs the total 32-bit data/sync/LTC signals 2o to the data acquisition board for data acquisition, and directs the 31-bit data/sync signals to the data analysis computer for realtime display. In data-replay mode, the program outputs the desired 31-bit data/ sync signal segment through the I/O board to the interface board. The interface board directs these 31-bit data only to the data analysis computer for further data analysis.
After every scan, the computer appends a 32-bit integer, which contains data/sync/LTC signals, as a file element to the acquired-data-file on the hard disk. As shown in FIG. 4(a), the acquired-data-file is a one-column array containing a large number of elements that equals the 3o scanning frequency X seconds of recording. Every video frame of the acquired-data-file contains a number of elements which equals scanning frequency = 29.97 frames/second.
The SMPTE LTC output from the VTR are bi-phase modulated, as shown in FIG. 3(a). When the recorded signal pulse shifts up or down only at the extremes of the period for a signal bit (417 ~S), the pulse is coded as a binary "0". A binary "1" is coded for a bit when a pulse shift occurs halfway through the bit period, as shown by Huber. The first decoding step was to demodulate the bi-phase modulated LTCs to "0"s and "1"s, as shown in FIG. 3(b). The second step is to determine the frame-starts. During decoding, the program continuously compares the demodulated 0-1 sequence with the 16-bit "sync word" pattern shown as the second row in FIG. 3(b). Whenever the pattern of any 16-bit segment of the 0-1 sequence matches that of the "sync word," the bit after this segment would be determined as the frame-start bit. In the last step, the program converts 64 LTC bits from each frame-start bit into a unique eight-digit LTC integer, as shown in FIGS. 3(c) and 3(d).
During data acquisition, the program retrieves all 32-bit integers from the buffer whenever the buffer is half full, writes the data to the hard disk, decodes the LTCs and forms the time-code-bridge-file all in 2o realtime. Even with a 266-MHz Pentium II microprocessor-based personal computer, the program cannot run fast enough for realtime decoding all LTCs (29.97 LTCs/second). The program decodes only one set of frame-start and LTC integer for every 15 video frames, and uniformly interpolates the remaining frame-start/LTC-integer values between current and last correctly decoded frame-starts. Since the video tape speed of a digital VTR
is highly stable, the timing of the interpolated video frame-starts can be very accurate.
LTC decoding errors can be caused by a VTR coding error, noise, electromagnetic interference and overflow of the computer buffer.
3o The program has an LTC decoding error correction subroutine to detect decoding errors. The subroutine scrutinizes every decoded LTC to determine whether the decoded frame-start is within a predicted tolerance range (1/80 of a video frame, set by this program) and whether the decoded LTC integer matches the predicted number. If it is not, the program automatically discards this erroneous LTC.
After each LTC decoding, the program appends the correctly decoded frame-start and LTC integer to a compact-time-code-bridge-file. In turn, the program uniformly interpolates the frame-starts and LTC integers between the current and the last correctly decoded frame-starts, then io appends the decoded and interpolated values to a two-column interpolated-time-code-bridge-file that contains the frame-starts and LTC integers of all recorded video frames (number of rows = 29.97 frames/second X seconds of recording), as shown in FIG. 4(b). Like the interpolated bridge file, the compact bridge file is also a two-column array except that it contains only decoded values. Its size is 1/15 of that of the interpolated bridge file when no decoding error occurs, and will be smaller if decoding errors occur and the consequent LTC decodings are discarded. When the program uses the bridge file for data segment or video frame search, it first searches the whole compact bridge file to determine an estimated file index number range, then 2o searches only the portion of the interpolated bridge file within the estimated range to determine the exact index number of the acquired-data-file. This search procedure would reduce the data-searching time in long-duration data recording when the interpolated bridge file becomes very large.
The flow chart for data search is shown in FIG. 2(b). When an interesting video frame is determined and frozen on the TV monitor during a video replay, the program turns the interface board to acquisition mode, acquires and decodes the frozen LTC. Then, the program searches the compact/ interpolated time-code files for the corresponding frame-start index number of the acquired-data-file. Obtaining the expected 3o corresponding index number, the program turns the interface board to the data-replay mode, retrieves the expected data segment from the acquired-data-file, displays it on the computer monitor and outputs it to the data analysis computer for further data analysis using the data analysis software.
As shown in the video-search flow chart in FIG. 2(c), the interface board is switched to data-replay mode during video frame search.
First, the program uses the compact/interpolated-time-code-bridge-files to retrieve a desired length of data from the acquired-data-file, replays the data frame-by-frame on the system and analysis computer monitors, and tracks the corresponding LTCs. When an interesting frame of replayed data is 1o determined and frozen on the computer monitor, the program retrieves the corresponding LTC and controls the VTR through an RS232 serial bus to search for the expected video frame. First, the program reads the LTC of the current video-tape position through the RS232 interface and determines the difference between the video-tape LTC and the frozen-data LTC. Second, s5 the program calculates the "Forward" or "Rewind" running time from the current video-tape position to the expected frame. The program then controls the VTR to forward or rewind the video tape to that frame. When the VTR stops, the program finally examines the LTC from the VTR to determine whether the current LTC matches the expected frame number. If 2o it is not, the program will repeat the search procedure.
In a synchronization accuracy test, an accuracy testing program compared the frame numbers, frame-starts and frame lengths of the acquired biomedical data with those of the recorded video time codes frame-by-frame, and calculated the mean and standard deviation of the 25 whole acquisition synchronization errors. As shown in FIG. 6, the video vertical-sync pulses from a video camera (Panasonic AJ-D200P) and the corresponding LTCs were acquired by the system computer through the data channels as simulated dual-channel biomedical data. After data acquisition, the acquired LTC frame numbers of the simulated data would 3o be compared with the corresponding frame numbers of the acquired LTC
from the VTR to determine frame-number synchronization error; and the acquired vertical-sync pulses of the simulated data would be compared with the corresponding frame-starts of the acquired LTC from the VTR to determine synchronization errors in frame-start and frame length. This synchronization test was reliable because the two signal references, video vertical-sync pulses and the corresponding LTC frame-starts, are consistently synchronized.
With reference to FIGS. 6(a) and 6(b), the instrumentation setup is illustrated for synchronization accuracy testing, in which the video camera 12 provides a video output signal to a time code generator 122, from which vertical sync pulse extraction is provided at block 124. Additionally, a time code amplifier 126 and a video tape recorder 128 generate output signals to the interface board 26, which is configured in its data acquisition mode at block 130. A digital input-output board provided in an input mode s5 132 facilitates the synchronization as accuracy test data to the system computer 32, configured as a control computer. FIG. 6(b) further illustrates the vertical sync pulse extractor 124, illustrating the use of a monostable mufti-vibrator 134 for pulse detection and extraction of the vertical sync pulse signals from the video output of the time coder generator 122.
2o In the test setup (FIG. 6(a)), the LTC was generated by an LTC
generator (Horita TG-50) and synchronized with the video signals from a video camera (Panasonic AJ-D200P). The odd-field vertical-sync pulses were extracted from the video signals (FIG. 6(b)), and input to one of the 31 data/sync channels (Bit 15) of the digital I/O board via the interface board 25 as the simulated data for frame-start and frame-length comparisons. The LTC stream was input both to the data-channel Bit 16 via the time-code amplifier and interface board, and to the "LTC In" of the I/O board via the VTR (Panasonic AJ-D750P) and interface board. The signal/ data latencies through the VTR and interface board would be counted in the 3o synchronization test.
After data acquisition, the accuracy testing program searched the bridge file for the acquired-data-file index at each frame-start, in order to retrieve the corresponding simulated data from the acquired-data-file.
When the program obtained the simulated data, it first decoded the frame number of the acquired LTC from Bit 16, and determined the difference between the frame number of the LTC from Bit 16 and the frame number in the bridge file. Second, the program compared the timing difference between the rising edge of the odd-field vertical-sync pulse from Bit 15 and the corresponding frame-start in the bridge file. Finally, the program io compared the duration difference between odd-field vertical-sync pulse period from Bit 15 and the corresponding frame-start period in the bridge file. The mean, standard deviation, and maximum and minimum values of these differences were then calculated.
The use of the time-code-bridge-files enables this video-s5 synchronized data acquisition system to perform data acquisition and subsequent video synchronization. The use of LTC decoding interpolation enables the system to perform the video synchronization with acquired data in realtime. Currently, the maximum scanning frequency for realtime video-synchronized 32-bit data acquisition (31-bit data/ sync signals, and 1 bit LTC
2o stream) is 72 kHz without interruption. The maximum summed bit rate is 2.16 Mbits/ second. This 32-bit data acquisition system can be flexibly configured. As the system is configured as a 30-channel-data plus one sync-signal channel with 12 bits of resolution, the maximum scanning frequency for each channel would be 6 kHz. At 72 kHz of the maximum scanning 25 frequency, the maximum recording duration without data overflow is 30 minutes, which records 506.70 Mbytes of information onto the hard disk:
506.25 Mbytes for the acquired-data-file, 421.5 kbytes for the interpolated-time-code-bridge-file and 28.1 kbytes for the compact-time-code-bridge-file.
The program effectively decoded the LTC stream in realtime with typically 3o two decoding errors in 30 minutes of acquisition at 72 kHz. The decoding errors would be even fewer if a lower-scanning frequency were used in data acquisition. The program detects and discards all erroneously decoded LTCs to ensure that the two time-code-bridge-files are always error free.
Table 1 shows a system comparison of this new method (bridge file based) with other published data.
In the timing accuracy test,1800 seconds of video signals and corresponding simulated-biomedical signals were acquired at a scanning frequency of 72 kHz using the testing setup in FIGS. 6(a) and 6(b). The accuracy testing program examined the acquired-data-file and the interpolated-time-code-bridge-file, frame by frame. The frame numbers of the acquired video images and the corresponding simulated-biomedical data were examined to be totally synchronized without any detected frame-number errors. In the frame-start timing examination, the average frame-start of the recorded video images was slightly ahead of those of the s5 simulated-biomedical data. In the frame-length examination, the average frame length of the recorded video images was slightly shorter than that of the simulated biomedical data. The mean total synchronization error ( ~ frame-start error ~ + ~ frame-length error ~ ) was 0.216 mS (0.647% of a frame) and the maximum error was 0.653 mS (1.957% of a frame). The 2o maximum latency of the interface board (10.0 nS on rising edges and 18.0 nS
on falling edges for biomedical data, and 1.24 ~,S on rising edges and 0.36 ~.S
on falling edges for LTCs), was counted in the above synchronization errors.
166.80 mS of the VTR latency and 79.6 ~S of the inherent phase difference between frame-starts and the odd-filed vertical-sync pulses were already 25 compensated for in the accuracy testing program. The synchronization errors also counted in the timing errors caused by the VTR latency fluctuation, and by the vertical-sync pulse/frame-start phase difference fluctuation. The statistical test data of the new system along with other available synchronization-error test data are listed in Table 2 for comparison.
Table 1 Comparison of Video-Synchronized Data Acquisition Systems System Bridge File FSK Video Hi. Speed Camera Frame Rate 29.97 29.97 500 (Frame/ sec) ype of Input Parallel Serial Un-specified Signal Summed Data Rate2160 15.36 Un-specified (kbit/ sec) to umber of Channels30 32 2 Scanning Freq. 6,000 60 500 / Channel (Hz) Resolution (bit)12 8 Un-specified Max. Recording 30 >_ 120 0.33 s5 Duration (min) Table 2 Comparison of the System Synchronization Tests System Bridge FSK Video File Replications53835 11848 (frames) 2o Sync. ErrorsFrame Frame- Frame Total Total (mS) Number start Length Mean 0 -0.216 +0.003 0.219 16.6 Std. Dev. 0 0.124 0.062 0.186 0.1 Maximum 0 +0.653 +0.399 1.052 16.6 25 Minimum 0 0.000 +0.003 0.003 15.1 This time-code-bridge-file method could also be applied to synchronize the biomedical data acquired on a computer hard disk and corresponding video frames acquired on the same or separate hard disk.
The bridge file would be a three-column array, with a column of recorded video frame numbers, and two columns of the index numbers of the acquired video-signal file and data file, at the beginning of the corresponding video frames.
The weight of the interface board is less than 1 kg. Along with a laptop computer, a PCMCIA data acquisition card, and a compact VCR
with LTC output, the whole system would be portable for field data collection.
The system accurately synchronizes the acquired biomedical data recorded on the computer hard disk with the corresponding human 1o activity images recorded on a video tape in realtime. With 2.16 Mbit/sec of summed data rate and less than 1.1 mS of the maximum data-synchronization error, this data acquisition system is adequate for frequency-domain muscle-fatigue research, and would be a useful tool for human hazard exposure assessment, rehabilitation monitoring, and athlete s5 monitoring in sport physiological programs. While present preferred embodiments of the mufti-channel, video synchronized EMG data acquisition system has been illustrated, it should be appreciated by those skilled in the art that modifications of the foregoing preferred embodiments may be made in various aspects. The present invention is set forth with 2o particularity in the appended claims. It is deemed that the spirit and scope of the invention encompasses such modifications and alterations to the preferred embodiments as would be apparent to one of ordinary skill in the art and familiar with the data acquisition teachings of the present application.
25 Synchronization of data acquired by a computer with the video images acquired by the same computer Instrumental Setup As shown in Fig. 7, this computerized video-synchronized data acquisition system consists of a video camera to output NTSC color video signals, a time-code generator with SMPTE longitudinal time code (LTC) output synchronized with the video signals from the video camera, a TV
monitor, a mufti-channel biomedical-signal-measuring instrument with parallel digital output, a signal-interface board to condition the incoming and outgoing data and control the direction of data flow, a system computer connected with a 32-bit digital I/O board and a realtime video capture board, and an analysis computer if the system needs to run the instrument's own analysis software.
Formation of Acquired-Data-File and Acquired-Video-File to The system block diagrams are shown in Fig. 8. During video-synchronized data acquisition (Fig. 8 (a)), the video signals from the video camera are sent through the time-code generator to synchronize the LTC
stream generated by the time-code generator. Then the video signals are sent to the TV monitor for display and to the video capture board for ~5 realtime video-frame capture by the system computer. The LTC from the time-code generator is sent to the interface board (the interface board is in Data Acquisition Mode), and to the video-capture board. Simultaneously, the biomedical instrument measures human biomedical signals, and outputs up to 31 bits of data/ sync signals to the interface board.
2o To form an acquired-data-file, the interface board buffers the total of 32-bit data/sync/LTC signals, and outputs them to the 32-bit digital I/O board. (The interface board also directs the 31-bit data/sync signals to the analysis computer for realtime data display.) The digital I/O board scans the 32-bit signal stream at a scanning frequency of up to 72 kHz, 25 converts every 32-bit signal sample into a 32-bit integer, and writes these data/sync/LTC integers into a data buffer. Every time the buffer is half full, the system computer program retrieves all the integers from the buffer, and appends them to an acquired-data-file on the hard disk (Fig. 9 (a)).
-ls-To form an acquired-video-file, simultaneously, the system computer controls the video capture board to capture video stream in realtime along with the LTC stream, converts the video/LTC signals to mufti-bit integers, and writes these video/LTC integers to a video buffer.
When the buffer is half full, the program retrieves all video/LTC integers from the buffer, and appends them to an acquired-video-file on the same hard disk (Fig. 9 (b)).
Decoding of Frame-Starts and Frame-Numbers In realtime or offline, depending on the computer signal-processing speed, the program retrieves the LTC bit stream from the data/LTC integers, and sends the LTC stream to the decoding subroutine for LTC decoding. During decoding, the program determines the index number of the acquired-data-file at each video frame-start, and translates the corresponding 80-bit LTC into a unique 8-digit frame number integer as shown in Fig. 9.
In the meantime, as with the data/LTC decoding, the program also retrieves the LTC bit stream from the video/LTC integers, and sends the LTC stream to the decoding subroutine for LTC decoding. During decoding, the program determines the index number of the acquired-video-2o file at each video frame-start, and translates the corresponding 80-bit LTC
into a unique 8-digit frame number integer (Fig. 9).
Formation of Time-Code-Bridge-File After decoding, the program forms a time-code-bridge-file with a three-column array. As shown in Fig. 9 (c), the LTC column of the array contains the 8-digit frame-number integers of all recorded video frames. The data-file-index column contains the corresponding index numbers of the acquired-data-file at frame-starts, and the video-file-index column contains the corresponding index numbers of the acquired-video-file at frame-starts. In this way, each row of the array represents a unique video frame, and its corresponding acquired-data-file index number and acquired-video-file index number at the frame-start. With this time-code-bridge-file, the computer is able to search for any data segment correspondent to a given video frame, or search for any video frame correspondent to a given data segment.
Similar Hardware and Software Implementations to those in Manuscript The description of the interface board, LTC decoding procedure, interpolated video frame decoding, LTC decoding error to correction, compact and interpolated time-code-bridge-files are the same as those under the respective subtitles in Hardware and Software Implementation section in the manuscript. The interpolation is not very necessarily needed if the decoding procedure is conducted offline.
Search for Data Segment r5 The flow chart for data search is shown in Fig. 8 (b). Given the desired begin and end frame numbers of video to be displayed, the program uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)), display the video images on the system computer monitor, frame by frame, and track the corresponding 2o LTCs. When an interesting video frame is determined and frozen on the monitor during a video replay, the program tracks the LTC of the frozen frame. Then, the program searches the time-code-bridge-file for the corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)).
Obtaining the expected corresponding index number, the program retrieves 25 the corresponding data frame, or multiple data frames related to the corresponding data frame, from the acquired-data-file, and turns the interface board to Data Replay Mode. Finally, the program displays the data on the system computer monitor, and/ or outputs it to the analysis computer for further data analysis using the data analysis software.
Search for Video Frame As shown in the video-search flow chart in Fig. 8 (c), the interface board is switched to Data Replay Mode during video frame search.
Given desired begin and end frame numbers of data to be displayed, the program uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)]), replay the data segment-by-segment (one segment contains one or multiple frames, determined by io the user) on the system computer and/or analysis computer monitors, and track the corresponding LTCs. When one or multiple interesting frames of replayed data are determined and frozen on the computer monitor, the program tracks the corresponding frozen LTCs, searches the time-code-bridge-file for the corresponding frame-start index numbers of the acquired-video-file (Fig. 9 (b)). Obtaining the expected corresponding index numbers, the program retrieves the expected video frames from the acquired-video-file, displays the corresponding video images and LTCs, frame by frame, on the computer monitor.
Synchronization of data acquired by a computer with the video images 2o acquired by a separate computer Instrumental Setup As shown in Fig.10, this computerized video-synchronized data acquisition system consists of a video camera to output NTSC color video signals, a time-code generator with SMPTE longitudinal time code (LTC) output synchronized with the video signals from the video camera, a TV monitor, a multi-channel biomedical-signal-measuring instrument with parallel digital output, a signal-interface board to condition the incoming and outgoing data and control the direction of data flow, a system computer connected with a 32-bit digital I/O board, a video computer connected with a realtime video capture board, and an analysis computer if the system needs to run the instrument's own analysis software.
Formation of Acquired-Data-File and Acquired-Video-File The system block diagrams are shown in Fig.11. During video-synchronized data acquisition (Fig.11 (a)), the video signals from the video camera are sent through the time-code generator to synchronize the LTC stream generated by the time-code generator. Then the video signals are sent to the TV monitor for display and to the video capture board for to realtime video-frame capture by the video computer. The LTC from the time-code generator is sent to the interface board (the interface board is in Data Acquisition Mode), and to the video-capture board connected with the video computer. Simultaneously, the biomedical instrument measures human biomedical signals, and outputs up to 31 bits of data/ sync signals to s5 the interface board.
To form an acquired-data-file, the interface board buffers the total of 32-bit data/sync/LTC signals, and outputs them to the 32-bit digital I/ O board connected with the system computer. (The interface board also directs the 31-bit data/sync signals to the analysis computer for realtime 2o data display.) The digital I/O board scans the 32-bit signal stream at a scanning frequency of up to 72 kHz, converts every 32-bit signal sample into a 32-bit integer, and writes these data/sync/LTC integers into a data buffer.
When the buffer is half full, the system computer program retrieves all the integers from the buffer, and appends them to an acquired-data-file on the 25 system computer hard disk (Fig. 9 (a)).
To form an acquired-video-file, simultaneously, the video computer controls the video capture board to capture video stream in realtime along with the LTC stream, converts the video/ LTC signals to multi-bit integers, and writes these video/LTC integers to a video buffer.
When the buffer is half full, the program retrieves all video/LTC integers from the buffer, and appends them to an acquired-video-file on the video computer hard disk (Fig. 9 (b)).
Decoding of Frame-Starts and Frame-Numbers In realtime or offline, depending on the computer signal-processing speed, the system computer program retrieves the LTC bit stream from the data/LTC integers, and sends the LTC stream to the decoding subroutine for L TC decoding. During decoding, the system computer program determines the index number of the acquired-data-file at so each video frame-start, and translates the corresponding 80-bit LTC into a unique 8-digit frame number integer as shown in Fig. 9.
In the meantime, as with the data/ LTC decoding, the video computer program retrieves the LTC bit stream from the video/LTC
integers, and sends the LTC stream to the decoding subroutine for LTC
s5 decoding. During decoding, the video computer program determines the index number of the acquired-video-file at each video frame-start, and translates the corresponding 80-bit LTC into a unique 8-digit frame number integer (Fig. 9).
Formation of Time-Code-Bridge-File 2o The system may use two different time-code-bridge-file configurations. After decoding, the system forms one or two time-code-bridge-files depending on the bridge file configuration.
Configuration 1. The system forms a three-column time-code-bridge-file which may be on either the system computer hard disk or the 25 video computer hard disk. As shown in Fig. 9 (c), the LTC column of the array contains the 8-digit frame-number integers of all recorded video frames. The data-file-index column contains the corresponding index numbers of the acquired-data-file at frame-starts, and the video-file-index column contains the corresponding index numbers of the acquired-video-file at frame-starts. In this way, each row of the array represents a unique video frame, and its corresponding acquired-data-file index number and acquired-video-file index number at the frame-start. With this time-code-bridge-file, the computer is able to search for any data segment correspondent to any given video frame, or search for any video frame correspondent to any given data segment.
Configuration 2. The system forms a two-column data-time-code-bridge-file on the system computer hard disk (Fig.12 (a)), and a two-lo column video-time-code-bridge-file on the video computer hard disk (Fig.12 (b)). As in Configuration 1, each row of the data-time-code-bridge-file and the corresponding row of the video-time-code-bridge-file represent a unique video frame, and its corresponding acquired-data-file index number and acquired-video-file index number at the frame-start. The two time-code-i5 bridge-files communicate between the two computers to synchronize the data with video.
Similar Hardware and Software Implementations to those in Manuscript The description of the interface board, LTC decoding procedure, interpolated video frame decoding, LTC decoding error 2o correction, compact and interpolated time-code-bridge-files are the same as those under the respective subtitles in Hardware and Software Implementation section in the manuscript. The interpolation is not necessarily needed if the decoding procedure is conducted offline.
Search for Data Segment 25 Configuration 1. The flow chart for data search is shown in Fig.11 (b). Given the desired begin and end frame numbers of video to be displayed, the system uses the three-column time-code-bridge-file (Fig. 9 (c)) to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)) in the video computer, display the video images on the video computer monitor, frame by frame, and track the corresponding LTCs. When an interesting video frame is determined and frozen on the monitor during a video replay, the system tracks the LTC of the frozen frame. Then, the system searches the time-code-bridge-file for the corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)). Obtaining the expected corresponding index number, the system retrieves the corresponding data frame, or multiple data frames related to the corresponding data frame, from the acquired-data-file in the system computer, and turns the interface board io to Data Replay Mode. Finally, the system computer program displays the data on the system computer monitor, and/ or outputs it to the analysis computer for further data analysis using the data analysis software.
Configuration 2. The flow chart for data search is also shown in Fig.11 (b). Given the desired begin and end frame numbers of video to be displayed, the system uses the video-time-code-bridge-file (Fig.12 (b)) to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)) in the video computer, display the video images on the video computer monitor, frame by frame, and track the corresponding LTCs. When an interesting video frame is determined and frozen on the video computer 2o monitor during a video replay, the system tracks the LTC of the frozen frame. Then, the system searches the data-time-code-bridge-file (Fig.12 (a)) for the corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)) in the system computer. Obtaining the expected corresponding index number, the system retrieves the corresponding data frame, or multiple data frames related to the corresponding data frame, from the acquired-data-file in the system computer, and turns the interface board to Data Replay Mode. Finally, the system computer program displays the data on the system computer monitor, and/ or outputs it to the analysis computer for further data analysis using the data analysis software.
Search for Video Frame Configuration 1. As shown in the video-search flow chart in Fig. 11 (c), the interface board is switched to Data Replay Mode during video frame search. Given the desired begin and end frame numbers of data to be displayed, the system uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)) in the system computer, replays the data segment-by-segment (one segment contains one or multiple frames, determined by the user) on the system computer and/ or analysis computer monitors, and track the corresponding LTCs. When one so or multiple interesting frames of replayed data are determined and frozen on the system computer monitor, the system tracks the corresponding frozen LTCs, searches the time-code-bridge-file for the corresponding frame-start index numbers of the acquired-video-file (Fig. 9 (b)) in the video computer. Obtaining the expected corresponding index numbers, the s5 system retrieves the expected video frames from the acquired-video-file in the video computer, displays the corresponding video images and LTCs, frame by frame, on the video computer monitor.
Configuration 2. As shown in the video-search flow chart in Fig.11 (c), the interface board is switched to Data Replay Mode during video 2o frame search. Given the desired begin and end frame numbers of data to be displayed, the system uses the data-time-code-bridge-file (Fig.12 (a)) to retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)) in the system computer, replay the data segment-by-segment (one segment contains one or multiple frames, determined by the user) on the system 25 computer and/ or analysis computer monitors, and track the corresponding LTCs. When one or multiple interesting frames of replayed data are determined and frozen on the system computer monitor, the system tracks the corresponding frozen LTCs, searches the video-time-code-bridge-file (Fig.12 (b)) for the corresponding frame-start index numbers of the 3o acquired-video-file (Fig. 9 (b)) in the video computer. Obtaining the expected corresponding index numbers, the system retrieves the expected video frames from the acquired-video-file in the video computer and displays the corresponding video images and LTCs, frame by frame, on the video computer monitor.
BIOMEDICAL DATA ACQUISITION SYSTEM
FIELD OF THE INVENTION
The present invention is directed to event synchronized data acquisition systems, and more particularly to video synchronized multi channel data acquisition for the analysis of biomedical information.
BACKGROUND OF THE INVENTION
Video-synchronized data acquisition is finding more applica-tions in biomedical engineering research. Some of these applications require to a video-synchronized data-acquisition system with multi-data channels and high-frequency bandwidth, and also maintain long acquisition duration.
One example of high-frequency application is frequency-domain muscle-fatigue analysis, for which a mufti-channel data acquisition system with a high raw-data signal frequency bandwidth of up to 3 kHz per channel is s5 recommended by D. G. Gerleman and T.M. Cook, "Instrumentation," in Selected Topics in Surface Electromyography for Use in the Occupational Setting:
Expert Perspectives, U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control, National Institute for Occupa-tional Safety and Health, pp. 44-68,1992. A number of schemes have been 2o published to achieve video-synchronized data acquisition, such as, D. M.
Gaskill, "Techniques for synchronizing thermal array chart recorders to video," 28th International Telemetering Conference (Proceedings)-ITC/USA/92, Vol. 28, pp. 61-64,1992; M. Vannier et al., "Time and motion studies of medical imaging workstations," SPIE, Vol.1653, Image Capture, 25 Formatting, and Display, pp. 274-280,1992; T. Y Yen and R. G. Radwin, "A
video-based system for acquiring biomechanical data synchronized with arbitrary events and activities," IEEE Transactions on Biomedical Engineering, Vol. 42, pp. 944-948,1995; M. Gacia et al., "Characterization of near-bed coherent structures in turbulent open channel flow using synchronized high-speed video and hot-film measurements," Experiments in Fluids, Vol. 19, pp.
16-28,1995; T. Engstrom and P. Medbo, "Data collection and analysis of manual work using video recording and personal computer techniques,"
International Journal of Industrial Ergonomics,19 (1997), pp. 291-298,1997.
Some of the schemes consist of simple event analyses stamped with video time codes, e.g., M. Vannier et al., "Time and motion studies of medical imaging workstations," SPIE, Vol.1653, Image Capture, Formatting, and Display, pp. 274-280,1992; T. Engstrom and P. Medbo, "Data collection and analysis of manual work using video recording and personal computer to techniques," International Journal of Industrial Ergonomics,19 (1997), pp.
291-298,1997. Some other sophisticated schemes have achieved multi-channel data acquisition. Yen and Radwin utilized an audio-track digital recording method to record up to 32 channels of 8-bit data onto video tape audio tracks, and to record corresponding video images onto the video track of the same tape. See, T. Y Yen and R. G. Radwin, "A video-based system for acquiring biomechanical data synchronized with arbitrary events and activities," IEEE Transactions on Biomedical Engineering, Vol. 42, pp. 944-948, 1995. This method naturally synchronizes the video frames with the acquired data for extensive recording duration. But, because of the 2o frequency-bandwidth limit of the audio tracks (20 Hz - 20 kHz), the scanning frequency of the recorded data is limited to 60 Hz per channel. Higher-scanning-rate multi-channel data acquisition was realized by using a high-speed video camera (1000 frames/sec) which outputs a synchronization signal in each video frame to trigger a computer data acquisition sequence.
See, M. Gacia et al., "Characterization of near-bed coherent structures in turbulent open channel flow using synchronized high-speed video and hot-film measurements," Experiments in Fluids, Vol.19, pp. 16-28,1995. The maximum scanning frequency of this method may reach 1000 Hz per channel for 20 seconds of 1000 frames per second video recording. To 3o acquire high frequency multi-channel data for longer duration, Gaskill _2_ suggested a data acquisition system using a digital audio tape recorder or a thermal chart recorder controlled by video time codes or vertical-video-sync pulses output from the video recording system.
This literature reports a new computerized data-acquisition system which extends Gaskill's suggestion by using a computer synchroni-zation program to realize high-frequency long-duration video-synchronized biomedical-data acquisition. It would be desirable therefore to provide a broad band mufti-channel video-synchronized data acquisition system which correlates video signal information with numerous data leads io associated with biomedical signals and the like. Additionally, it would be advantageous to provide for the acquisition of frequency domain signals which may be correlated with video signals acquired by the system. The system records the storage-intensive video images onto a video tape, and simultaneously acquires biomedical data and video time codes onto a computer hard disk. A time-code-bridge-file is created by the computer to synchronize the acquired biomedical data on the hard drive with the recorded human images on the video tape. Without manipulating video signals in the system computer, this separated recording method boosts the data acquisition speed, enables the realtime data synchronization and long 2o recording duration. The video time-code frame-start bits were selected as the synchronization reference. With this reference, the synchronization error can be theoretically controlled within one SMPTE time-code bit duration (0.417 mS).
SUMMARY OF THE INVENTION
A video-synchronized mufti-channel data acquisition system is provided for frequency domain analysis of biomedical signals acquired with video signals, and particularly for correlating analysis with muscle fatigue research. The disclosed system records storage-intensive video images onto a video tape, and also provides for the data acquisition of information signals with associated video time codes on a personal computer hard disk to achieve high speed recording over a long duration. A video time code-bridge file has been provided for use by the computer to synchronize the biomedical data with the recorded video frame in real time.
Briefly summarized, the present invention relates to a data acquisition system, in which an apparatus such as a video tape recorder is provided for recording video frames from an input video signal. A
biomedical instrument generates an input biomedical data stream to a system computer providing at least one digital input-output port and to interface circuitry for collecting the input video time code signal and for collecting the input biomedical data stream. A bridge file is provided with the system computer for synchronizing the video frames from the input video signal with the input biomedical data stream.
The appended claims set forth the features of the present invention with particularity. The invention and its advantages may be best understood from the following detailed description taken in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG.1 shows a video-synchronized biomedical data acquisition 2o system in accordance with the invention;
FIG. 2 illustrates in block diagram form the program flow of the data acquisition system of FIG. 1;
FIG. 3 illustrates a longitudinal time code output decoding procedure in accordance with the invention;
FIG. 4 shows the structure of the program files of the data acquisition system correlating acquired data file information with elements per video frame;
FIG. 5 shows the structure of the interface board shown in the system diagram of FIG.1;
FIGS. 6a and 6b illustrate the instrumentation setup for synchronizing extracted video information with biomedical signal acquisition in accordance with the invention; and FIG. 7 illustrates an instrumental setup for data and video acquisition by one computer;
FIG. 8 illustrates block diagrams for data and video acquisition by one computer;
FIG. 9 illustrates program file structures for data and video acquisition by one computer;
so FIG.10 illustrates an instrumental setup for data and video acquisition by separate computers;
FIG.11 illustrates block diagrams for data and video acquisition by separate computers; and FIG. 12 illustrates two time-code-bridge-files for data and video acquisition by separate computers.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
As shown in FIG.1, a computerized video-synchronized data acquisition system 10 is shown a video camera 12 to output NTSC color video signals 14, a digital VTR 16 with a SMPTE longitudinal time code (LTC) output 22, a TV monitor 18, a mufti-channel biomedical-signal-measuring instrument 20 with parallel digital output 24, a signal-interface board 26 to condition the incoming (22,24) and outgoing (28,30) data and control the direction of data flow, a 266-MHz Pentium II microprocessor-based system PC computer 32 with a 32-bit digital I/O board, and an analysis PC computer 34 if the system needs to run the instrument's own analysis software. LabVIEW graphical programming software (National Instruments, Austin, TX) was used to program the data acquisition, processing, storage and replay, and VTR control.
The system block diagrams are shown in FIG. 2. During video-synchronized data acquisition (FIG. 2 (a)), the VTR records NTSC human activity images and outputs a 1-bit LTC stream to the interface board. The LTC stream is bi-phase modulated by frame numbers and sync words, as shown in FIG. 3(a). Such techniques for modulations with frame numbers and sync word have been demonstrated in D. M. Huber, Audio production technigues for video. White Plains, NY and London: Knowledge Industry Publications, Inc., pp. 93-100,1987.
FIG. 2 represents three flow charts, i.e., video synchronized to data acquisition 36 (2a); data search 38 (2b); and video search 40 (2c).
The video synchronization data acquisition 36 is illustrated in block diagram form in 2(a), and further broken down into hardware component 54 and software component 56. The hardware component 54 illustrates video recording 42 and data recording 44 which may be provided simultaneously i5 to provide longitudinal time code (LTC) output to record human activity with video imaging, and additionally provide a biomedical data stream from a data recording instrument 44, which is provided as a mufti-channel electromyography (EMG) data acquisition instrument. An interface board 46 corresponding to the interface board 26 of FIG.1 operates in a data 2o acquisition mode to receive the LTC and data signals from the video recording and data recording blocks 42 and 44 respectively. Accordingly, the interface board in its data acquisition mode 46 provides output signals to display biomedical data on an analysis computer monitor 48, and an output signal to a data acquisition board 50. The data acquisition at block 50 25 provides write data to the system computer buffer at block 52. Next, a software loop is provided in the software section 56, which obtains data from the system computer buffer in a read data buffer block 58. An acquired data file is written to the hard disk of the system computer at block 60, and an LTC decode is performed at block 62. Block 64 is provided to correct LTC
3o errors, and interpolation of the LTC signal at block 66 is used with the bridge file elements in a block 68 for forming bridge file elements. The bridge file elements from block 68 are provided as appended elements to the bridge file on the hard disk at block 70.
At FIG. 2(c), a data search 38 is illustrated in which a replay of the video tape is performed at 72, from which selected video frames at 74 are presented to the interface board 26 in its acquisition mode at block 76. A
decode of the LTC selected frame at 78 along with a search of the bridge file 80 is used to retrieve data from the acquired data file at block 82. The interface board 26 then provided in a replay mode at block 84 facilitates the io replay and output of data to an analysis computer at block 86.
At FIG. 2(c) the video search 40 is illustrated, wherein the interface board 26 in its replay mode at block 88 replays a data segment 90 from which a selected data frame is provided at block 92. At block 94 the selected data frame is used to determine LTC of the selected frame. An RS 232 interface is facilitated at block 96 which is used with a program loop for reading the video tape LTC starting at block 98. Block 100 is used to compare the LTC difference and block 102 calculates forward or reverse time. Block 104 converts the calculated time to forward or reverse on the vide tape recorder 16. A loop decision block at 106 determines whether the 2o video frame selected is correct or forward or reverse video frames must be used, and then if the video frame is correct at decision block 106, a display of the video frame is provided on the TV monitor at block 108.
FIGS. 3(a), 3(b), 3(c), and 3(d) illustrate the LTC decoding procedure from a data stream 110 obtained from the video recording. In FIG. 4(a), the acquired data file 112 is illustrated as several elements per video frame (EPVF) arranged as 32-bit integer data elements provided with a video frame number and an index number at the beginning of each frame.
FIG. 4(b) shows the interpolated time code bridge file which may be used to correlate the video frame number with the LTC and data file index columns.
Simultaneously, the biomedical instrument measures human biomedical signals, and outputs up to 31 bits of data/sync signals to the interface board. The interface board in turn buffers the total of 32-bit data/ sync/ LTC signals, and outputs them to the 32-bit digital I/ O board (National Instrument AT-DIO-32F). The interface board also directs the 31-bit data/sync signals to the data analysis computer for realtime data display. The digital I/ O board scans the 32-bit signal stream at a scanning frequency of up to 72 kHz, converts the stream into 32-bit integers, and writes the integers into the system computer buffer. Whenever the buffer is to half full, the program retrieves all the integers from the buffer, and appends them to an acquired-data-file on the hard disk (see FIG. 4(a) for acquired-data-file structure). In the meantime, the program retrieves the LTC bit stream from the integers, and sends the LTC stream to the decoding subroutine for LTC decoding. During decoding, the program determines ~5 the index number of the acquired-data-file at each video frame-start, and translates each 80-bit LTC into a unique 8-digit frame number integer as shown in FIGS. 3(b), 3(c) and 3(d). After decoding, the program forms a time-code-bridge-file with a two-column array. As shown in FIG. 4(b), the LTC column of the array contains the 8-digit frame-number integers of all 2o recorded video frames, and the data-file-index column contains the corresponding index numbers of the acquired-data-file at frame-starts. In this way, each row of the array represents a unique video frame and its corresponding acquired-data-file index number at the frame-start. With this time-code-bridge-file, the computer is able to search for any data segment 25 correspondent to a given video frame, or search for any video frame correspondent to a given data segment.
The interface board has two flow-control modes: data-acquisition mode and data-replay mode that are shown in FIGS. 5(a) and 5(b), respectively. The main task of the interface board is to control the data 3o flow direction. FIGS. 5(a) and 5(b) illustrate the interface board 26 in data _g_ acquisition mode and data replay modes, respectively. As shown in FIG. 5(a), the data acquisition mode configuration of the interface board 26 facilitates the receipt of LTC from the video tape recorder 16, as well as the biomedical data stream from the biomedical instrument 20, which signals are buffered in a tristate digital buffer of the data acquisition board 26, which provides buffered output signals to the digital I/O board which provides an input-output port for the system computer 32. Additionally, a buffered output of data/sync signals are provided to the analysis computer 34 . In FIG. 5(b), the interface board 26 configured in the data replay mode 1o receives control data from the system computer 32, which provides data signals to the analysis computer 34 to replay the acquired data to the analysis computer 34.
The flow-control mode is selected by the program which outputs a control bit from the digital I/O board to the interface board. The interface board buffers the incoming and outgoing data and converts the polarity of the LTC stream from ~10 V to 0-5 V. In data-acquisition mode, the 31-bit data/ sync signals from the biomedical instrument, and the corresponding 1-bit LTC stream from the VTR are input to the interface board. The interface board directs the total 32-bit data/sync/LTC signals 2o to the data acquisition board for data acquisition, and directs the 31-bit data/sync signals to the data analysis computer for realtime display. In data-replay mode, the program outputs the desired 31-bit data/ sync signal segment through the I/O board to the interface board. The interface board directs these 31-bit data only to the data analysis computer for further data analysis.
After every scan, the computer appends a 32-bit integer, which contains data/sync/LTC signals, as a file element to the acquired-data-file on the hard disk. As shown in FIG. 4(a), the acquired-data-file is a one-column array containing a large number of elements that equals the 3o scanning frequency X seconds of recording. Every video frame of the acquired-data-file contains a number of elements which equals scanning frequency = 29.97 frames/second.
The SMPTE LTC output from the VTR are bi-phase modulated, as shown in FIG. 3(a). When the recorded signal pulse shifts up or down only at the extremes of the period for a signal bit (417 ~S), the pulse is coded as a binary "0". A binary "1" is coded for a bit when a pulse shift occurs halfway through the bit period, as shown by Huber. The first decoding step was to demodulate the bi-phase modulated LTCs to "0"s and "1"s, as shown in FIG. 3(b). The second step is to determine the frame-starts. During decoding, the program continuously compares the demodulated 0-1 sequence with the 16-bit "sync word" pattern shown as the second row in FIG. 3(b). Whenever the pattern of any 16-bit segment of the 0-1 sequence matches that of the "sync word," the bit after this segment would be determined as the frame-start bit. In the last step, the program converts 64 LTC bits from each frame-start bit into a unique eight-digit LTC integer, as shown in FIGS. 3(c) and 3(d).
During data acquisition, the program retrieves all 32-bit integers from the buffer whenever the buffer is half full, writes the data to the hard disk, decodes the LTCs and forms the time-code-bridge-file all in 2o realtime. Even with a 266-MHz Pentium II microprocessor-based personal computer, the program cannot run fast enough for realtime decoding all LTCs (29.97 LTCs/second). The program decodes only one set of frame-start and LTC integer for every 15 video frames, and uniformly interpolates the remaining frame-start/LTC-integer values between current and last correctly decoded frame-starts. Since the video tape speed of a digital VTR
is highly stable, the timing of the interpolated video frame-starts can be very accurate.
LTC decoding errors can be caused by a VTR coding error, noise, electromagnetic interference and overflow of the computer buffer.
3o The program has an LTC decoding error correction subroutine to detect decoding errors. The subroutine scrutinizes every decoded LTC to determine whether the decoded frame-start is within a predicted tolerance range (1/80 of a video frame, set by this program) and whether the decoded LTC integer matches the predicted number. If it is not, the program automatically discards this erroneous LTC.
After each LTC decoding, the program appends the correctly decoded frame-start and LTC integer to a compact-time-code-bridge-file. In turn, the program uniformly interpolates the frame-starts and LTC integers between the current and the last correctly decoded frame-starts, then io appends the decoded and interpolated values to a two-column interpolated-time-code-bridge-file that contains the frame-starts and LTC integers of all recorded video frames (number of rows = 29.97 frames/second X seconds of recording), as shown in FIG. 4(b). Like the interpolated bridge file, the compact bridge file is also a two-column array except that it contains only decoded values. Its size is 1/15 of that of the interpolated bridge file when no decoding error occurs, and will be smaller if decoding errors occur and the consequent LTC decodings are discarded. When the program uses the bridge file for data segment or video frame search, it first searches the whole compact bridge file to determine an estimated file index number range, then 2o searches only the portion of the interpolated bridge file within the estimated range to determine the exact index number of the acquired-data-file. This search procedure would reduce the data-searching time in long-duration data recording when the interpolated bridge file becomes very large.
The flow chart for data search is shown in FIG. 2(b). When an interesting video frame is determined and frozen on the TV monitor during a video replay, the program turns the interface board to acquisition mode, acquires and decodes the frozen LTC. Then, the program searches the compact/ interpolated time-code files for the corresponding frame-start index number of the acquired-data-file. Obtaining the expected 3o corresponding index number, the program turns the interface board to the data-replay mode, retrieves the expected data segment from the acquired-data-file, displays it on the computer monitor and outputs it to the data analysis computer for further data analysis using the data analysis software.
As shown in the video-search flow chart in FIG. 2(c), the interface board is switched to data-replay mode during video frame search.
First, the program uses the compact/interpolated-time-code-bridge-files to retrieve a desired length of data from the acquired-data-file, replays the data frame-by-frame on the system and analysis computer monitors, and tracks the corresponding LTCs. When an interesting frame of replayed data is 1o determined and frozen on the computer monitor, the program retrieves the corresponding LTC and controls the VTR through an RS232 serial bus to search for the expected video frame. First, the program reads the LTC of the current video-tape position through the RS232 interface and determines the difference between the video-tape LTC and the frozen-data LTC. Second, s5 the program calculates the "Forward" or "Rewind" running time from the current video-tape position to the expected frame. The program then controls the VTR to forward or rewind the video tape to that frame. When the VTR stops, the program finally examines the LTC from the VTR to determine whether the current LTC matches the expected frame number. If 2o it is not, the program will repeat the search procedure.
In a synchronization accuracy test, an accuracy testing program compared the frame numbers, frame-starts and frame lengths of the acquired biomedical data with those of the recorded video time codes frame-by-frame, and calculated the mean and standard deviation of the 25 whole acquisition synchronization errors. As shown in FIG. 6, the video vertical-sync pulses from a video camera (Panasonic AJ-D200P) and the corresponding LTCs were acquired by the system computer through the data channels as simulated dual-channel biomedical data. After data acquisition, the acquired LTC frame numbers of the simulated data would 3o be compared with the corresponding frame numbers of the acquired LTC
from the VTR to determine frame-number synchronization error; and the acquired vertical-sync pulses of the simulated data would be compared with the corresponding frame-starts of the acquired LTC from the VTR to determine synchronization errors in frame-start and frame length. This synchronization test was reliable because the two signal references, video vertical-sync pulses and the corresponding LTC frame-starts, are consistently synchronized.
With reference to FIGS. 6(a) and 6(b), the instrumentation setup is illustrated for synchronization accuracy testing, in which the video camera 12 provides a video output signal to a time code generator 122, from which vertical sync pulse extraction is provided at block 124. Additionally, a time code amplifier 126 and a video tape recorder 128 generate output signals to the interface board 26, which is configured in its data acquisition mode at block 130. A digital input-output board provided in an input mode s5 132 facilitates the synchronization as accuracy test data to the system computer 32, configured as a control computer. FIG. 6(b) further illustrates the vertical sync pulse extractor 124, illustrating the use of a monostable mufti-vibrator 134 for pulse detection and extraction of the vertical sync pulse signals from the video output of the time coder generator 122.
2o In the test setup (FIG. 6(a)), the LTC was generated by an LTC
generator (Horita TG-50) and synchronized with the video signals from a video camera (Panasonic AJ-D200P). The odd-field vertical-sync pulses were extracted from the video signals (FIG. 6(b)), and input to one of the 31 data/sync channels (Bit 15) of the digital I/O board via the interface board 25 as the simulated data for frame-start and frame-length comparisons. The LTC stream was input both to the data-channel Bit 16 via the time-code amplifier and interface board, and to the "LTC In" of the I/O board via the VTR (Panasonic AJ-D750P) and interface board. The signal/ data latencies through the VTR and interface board would be counted in the 3o synchronization test.
After data acquisition, the accuracy testing program searched the bridge file for the acquired-data-file index at each frame-start, in order to retrieve the corresponding simulated data from the acquired-data-file.
When the program obtained the simulated data, it first decoded the frame number of the acquired LTC from Bit 16, and determined the difference between the frame number of the LTC from Bit 16 and the frame number in the bridge file. Second, the program compared the timing difference between the rising edge of the odd-field vertical-sync pulse from Bit 15 and the corresponding frame-start in the bridge file. Finally, the program io compared the duration difference between odd-field vertical-sync pulse period from Bit 15 and the corresponding frame-start period in the bridge file. The mean, standard deviation, and maximum and minimum values of these differences were then calculated.
The use of the time-code-bridge-files enables this video-s5 synchronized data acquisition system to perform data acquisition and subsequent video synchronization. The use of LTC decoding interpolation enables the system to perform the video synchronization with acquired data in realtime. Currently, the maximum scanning frequency for realtime video-synchronized 32-bit data acquisition (31-bit data/ sync signals, and 1 bit LTC
2o stream) is 72 kHz without interruption. The maximum summed bit rate is 2.16 Mbits/ second. This 32-bit data acquisition system can be flexibly configured. As the system is configured as a 30-channel-data plus one sync-signal channel with 12 bits of resolution, the maximum scanning frequency for each channel would be 6 kHz. At 72 kHz of the maximum scanning 25 frequency, the maximum recording duration without data overflow is 30 minutes, which records 506.70 Mbytes of information onto the hard disk:
506.25 Mbytes for the acquired-data-file, 421.5 kbytes for the interpolated-time-code-bridge-file and 28.1 kbytes for the compact-time-code-bridge-file.
The program effectively decoded the LTC stream in realtime with typically 3o two decoding errors in 30 minutes of acquisition at 72 kHz. The decoding errors would be even fewer if a lower-scanning frequency were used in data acquisition. The program detects and discards all erroneously decoded LTCs to ensure that the two time-code-bridge-files are always error free.
Table 1 shows a system comparison of this new method (bridge file based) with other published data.
In the timing accuracy test,1800 seconds of video signals and corresponding simulated-biomedical signals were acquired at a scanning frequency of 72 kHz using the testing setup in FIGS. 6(a) and 6(b). The accuracy testing program examined the acquired-data-file and the interpolated-time-code-bridge-file, frame by frame. The frame numbers of the acquired video images and the corresponding simulated-biomedical data were examined to be totally synchronized without any detected frame-number errors. In the frame-start timing examination, the average frame-start of the recorded video images was slightly ahead of those of the s5 simulated-biomedical data. In the frame-length examination, the average frame length of the recorded video images was slightly shorter than that of the simulated biomedical data. The mean total synchronization error ( ~ frame-start error ~ + ~ frame-length error ~ ) was 0.216 mS (0.647% of a frame) and the maximum error was 0.653 mS (1.957% of a frame). The 2o maximum latency of the interface board (10.0 nS on rising edges and 18.0 nS
on falling edges for biomedical data, and 1.24 ~,S on rising edges and 0.36 ~.S
on falling edges for LTCs), was counted in the above synchronization errors.
166.80 mS of the VTR latency and 79.6 ~S of the inherent phase difference between frame-starts and the odd-filed vertical-sync pulses were already 25 compensated for in the accuracy testing program. The synchronization errors also counted in the timing errors caused by the VTR latency fluctuation, and by the vertical-sync pulse/frame-start phase difference fluctuation. The statistical test data of the new system along with other available synchronization-error test data are listed in Table 2 for comparison.
Table 1 Comparison of Video-Synchronized Data Acquisition Systems System Bridge File FSK Video Hi. Speed Camera Frame Rate 29.97 29.97 500 (Frame/ sec) ype of Input Parallel Serial Un-specified Signal Summed Data Rate2160 15.36 Un-specified (kbit/ sec) to umber of Channels30 32 2 Scanning Freq. 6,000 60 500 / Channel (Hz) Resolution (bit)12 8 Un-specified Max. Recording 30 >_ 120 0.33 s5 Duration (min) Table 2 Comparison of the System Synchronization Tests System Bridge FSK Video File Replications53835 11848 (frames) 2o Sync. ErrorsFrame Frame- Frame Total Total (mS) Number start Length Mean 0 -0.216 +0.003 0.219 16.6 Std. Dev. 0 0.124 0.062 0.186 0.1 Maximum 0 +0.653 +0.399 1.052 16.6 25 Minimum 0 0.000 +0.003 0.003 15.1 This time-code-bridge-file method could also be applied to synchronize the biomedical data acquired on a computer hard disk and corresponding video frames acquired on the same or separate hard disk.
The bridge file would be a three-column array, with a column of recorded video frame numbers, and two columns of the index numbers of the acquired video-signal file and data file, at the beginning of the corresponding video frames.
The weight of the interface board is less than 1 kg. Along with a laptop computer, a PCMCIA data acquisition card, and a compact VCR
with LTC output, the whole system would be portable for field data collection.
The system accurately synchronizes the acquired biomedical data recorded on the computer hard disk with the corresponding human 1o activity images recorded on a video tape in realtime. With 2.16 Mbit/sec of summed data rate and less than 1.1 mS of the maximum data-synchronization error, this data acquisition system is adequate for frequency-domain muscle-fatigue research, and would be a useful tool for human hazard exposure assessment, rehabilitation monitoring, and athlete s5 monitoring in sport physiological programs. While present preferred embodiments of the mufti-channel, video synchronized EMG data acquisition system has been illustrated, it should be appreciated by those skilled in the art that modifications of the foregoing preferred embodiments may be made in various aspects. The present invention is set forth with 2o particularity in the appended claims. It is deemed that the spirit and scope of the invention encompasses such modifications and alterations to the preferred embodiments as would be apparent to one of ordinary skill in the art and familiar with the data acquisition teachings of the present application.
25 Synchronization of data acquired by a computer with the video images acquired by the same computer Instrumental Setup As shown in Fig. 7, this computerized video-synchronized data acquisition system consists of a video camera to output NTSC color video signals, a time-code generator with SMPTE longitudinal time code (LTC) output synchronized with the video signals from the video camera, a TV
monitor, a mufti-channel biomedical-signal-measuring instrument with parallel digital output, a signal-interface board to condition the incoming and outgoing data and control the direction of data flow, a system computer connected with a 32-bit digital I/O board and a realtime video capture board, and an analysis computer if the system needs to run the instrument's own analysis software.
Formation of Acquired-Data-File and Acquired-Video-File to The system block diagrams are shown in Fig. 8. During video-synchronized data acquisition (Fig. 8 (a)), the video signals from the video camera are sent through the time-code generator to synchronize the LTC
stream generated by the time-code generator. Then the video signals are sent to the TV monitor for display and to the video capture board for ~5 realtime video-frame capture by the system computer. The LTC from the time-code generator is sent to the interface board (the interface board is in Data Acquisition Mode), and to the video-capture board. Simultaneously, the biomedical instrument measures human biomedical signals, and outputs up to 31 bits of data/ sync signals to the interface board.
2o To form an acquired-data-file, the interface board buffers the total of 32-bit data/sync/LTC signals, and outputs them to the 32-bit digital I/O board. (The interface board also directs the 31-bit data/sync signals to the analysis computer for realtime data display.) The digital I/O board scans the 32-bit signal stream at a scanning frequency of up to 72 kHz, 25 converts every 32-bit signal sample into a 32-bit integer, and writes these data/sync/LTC integers into a data buffer. Every time the buffer is half full, the system computer program retrieves all the integers from the buffer, and appends them to an acquired-data-file on the hard disk (Fig. 9 (a)).
-ls-To form an acquired-video-file, simultaneously, the system computer controls the video capture board to capture video stream in realtime along with the LTC stream, converts the video/LTC signals to mufti-bit integers, and writes these video/LTC integers to a video buffer.
When the buffer is half full, the program retrieves all video/LTC integers from the buffer, and appends them to an acquired-video-file on the same hard disk (Fig. 9 (b)).
Decoding of Frame-Starts and Frame-Numbers In realtime or offline, depending on the computer signal-processing speed, the program retrieves the LTC bit stream from the data/LTC integers, and sends the LTC stream to the decoding subroutine for LTC decoding. During decoding, the program determines the index number of the acquired-data-file at each video frame-start, and translates the corresponding 80-bit LTC into a unique 8-digit frame number integer as shown in Fig. 9.
In the meantime, as with the data/LTC decoding, the program also retrieves the LTC bit stream from the video/LTC integers, and sends the LTC stream to the decoding subroutine for LTC decoding. During decoding, the program determines the index number of the acquired-video-2o file at each video frame-start, and translates the corresponding 80-bit LTC
into a unique 8-digit frame number integer (Fig. 9).
Formation of Time-Code-Bridge-File After decoding, the program forms a time-code-bridge-file with a three-column array. As shown in Fig. 9 (c), the LTC column of the array contains the 8-digit frame-number integers of all recorded video frames. The data-file-index column contains the corresponding index numbers of the acquired-data-file at frame-starts, and the video-file-index column contains the corresponding index numbers of the acquired-video-file at frame-starts. In this way, each row of the array represents a unique video frame, and its corresponding acquired-data-file index number and acquired-video-file index number at the frame-start. With this time-code-bridge-file, the computer is able to search for any data segment correspondent to a given video frame, or search for any video frame correspondent to a given data segment.
Similar Hardware and Software Implementations to those in Manuscript The description of the interface board, LTC decoding procedure, interpolated video frame decoding, LTC decoding error to correction, compact and interpolated time-code-bridge-files are the same as those under the respective subtitles in Hardware and Software Implementation section in the manuscript. The interpolation is not very necessarily needed if the decoding procedure is conducted offline.
Search for Data Segment r5 The flow chart for data search is shown in Fig. 8 (b). Given the desired begin and end frame numbers of video to be displayed, the program uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)), display the video images on the system computer monitor, frame by frame, and track the corresponding 2o LTCs. When an interesting video frame is determined and frozen on the monitor during a video replay, the program tracks the LTC of the frozen frame. Then, the program searches the time-code-bridge-file for the corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)).
Obtaining the expected corresponding index number, the program retrieves 25 the corresponding data frame, or multiple data frames related to the corresponding data frame, from the acquired-data-file, and turns the interface board to Data Replay Mode. Finally, the program displays the data on the system computer monitor, and/ or outputs it to the analysis computer for further data analysis using the data analysis software.
Search for Video Frame As shown in the video-search flow chart in Fig. 8 (c), the interface board is switched to Data Replay Mode during video frame search.
Given desired begin and end frame numbers of data to be displayed, the program uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)]), replay the data segment-by-segment (one segment contains one or multiple frames, determined by io the user) on the system computer and/or analysis computer monitors, and track the corresponding LTCs. When one or multiple interesting frames of replayed data are determined and frozen on the computer monitor, the program tracks the corresponding frozen LTCs, searches the time-code-bridge-file for the corresponding frame-start index numbers of the acquired-video-file (Fig. 9 (b)). Obtaining the expected corresponding index numbers, the program retrieves the expected video frames from the acquired-video-file, displays the corresponding video images and LTCs, frame by frame, on the computer monitor.
Synchronization of data acquired by a computer with the video images 2o acquired by a separate computer Instrumental Setup As shown in Fig.10, this computerized video-synchronized data acquisition system consists of a video camera to output NTSC color video signals, a time-code generator with SMPTE longitudinal time code (LTC) output synchronized with the video signals from the video camera, a TV monitor, a multi-channel biomedical-signal-measuring instrument with parallel digital output, a signal-interface board to condition the incoming and outgoing data and control the direction of data flow, a system computer connected with a 32-bit digital I/O board, a video computer connected with a realtime video capture board, and an analysis computer if the system needs to run the instrument's own analysis software.
Formation of Acquired-Data-File and Acquired-Video-File The system block diagrams are shown in Fig.11. During video-synchronized data acquisition (Fig.11 (a)), the video signals from the video camera are sent through the time-code generator to synchronize the LTC stream generated by the time-code generator. Then the video signals are sent to the TV monitor for display and to the video capture board for to realtime video-frame capture by the video computer. The LTC from the time-code generator is sent to the interface board (the interface board is in Data Acquisition Mode), and to the video-capture board connected with the video computer. Simultaneously, the biomedical instrument measures human biomedical signals, and outputs up to 31 bits of data/ sync signals to s5 the interface board.
To form an acquired-data-file, the interface board buffers the total of 32-bit data/sync/LTC signals, and outputs them to the 32-bit digital I/ O board connected with the system computer. (The interface board also directs the 31-bit data/sync signals to the analysis computer for realtime 2o data display.) The digital I/O board scans the 32-bit signal stream at a scanning frequency of up to 72 kHz, converts every 32-bit signal sample into a 32-bit integer, and writes these data/sync/LTC integers into a data buffer.
When the buffer is half full, the system computer program retrieves all the integers from the buffer, and appends them to an acquired-data-file on the 25 system computer hard disk (Fig. 9 (a)).
To form an acquired-video-file, simultaneously, the video computer controls the video capture board to capture video stream in realtime along with the LTC stream, converts the video/ LTC signals to multi-bit integers, and writes these video/LTC integers to a video buffer.
When the buffer is half full, the program retrieves all video/LTC integers from the buffer, and appends them to an acquired-video-file on the video computer hard disk (Fig. 9 (b)).
Decoding of Frame-Starts and Frame-Numbers In realtime or offline, depending on the computer signal-processing speed, the system computer program retrieves the LTC bit stream from the data/LTC integers, and sends the LTC stream to the decoding subroutine for L TC decoding. During decoding, the system computer program determines the index number of the acquired-data-file at so each video frame-start, and translates the corresponding 80-bit LTC into a unique 8-digit frame number integer as shown in Fig. 9.
In the meantime, as with the data/ LTC decoding, the video computer program retrieves the LTC bit stream from the video/LTC
integers, and sends the LTC stream to the decoding subroutine for LTC
s5 decoding. During decoding, the video computer program determines the index number of the acquired-video-file at each video frame-start, and translates the corresponding 80-bit LTC into a unique 8-digit frame number integer (Fig. 9).
Formation of Time-Code-Bridge-File 2o The system may use two different time-code-bridge-file configurations. After decoding, the system forms one or two time-code-bridge-files depending on the bridge file configuration.
Configuration 1. The system forms a three-column time-code-bridge-file which may be on either the system computer hard disk or the 25 video computer hard disk. As shown in Fig. 9 (c), the LTC column of the array contains the 8-digit frame-number integers of all recorded video frames. The data-file-index column contains the corresponding index numbers of the acquired-data-file at frame-starts, and the video-file-index column contains the corresponding index numbers of the acquired-video-file at frame-starts. In this way, each row of the array represents a unique video frame, and its corresponding acquired-data-file index number and acquired-video-file index number at the frame-start. With this time-code-bridge-file, the computer is able to search for any data segment correspondent to any given video frame, or search for any video frame correspondent to any given data segment.
Configuration 2. The system forms a two-column data-time-code-bridge-file on the system computer hard disk (Fig.12 (a)), and a two-lo column video-time-code-bridge-file on the video computer hard disk (Fig.12 (b)). As in Configuration 1, each row of the data-time-code-bridge-file and the corresponding row of the video-time-code-bridge-file represent a unique video frame, and its corresponding acquired-data-file index number and acquired-video-file index number at the frame-start. The two time-code-i5 bridge-files communicate between the two computers to synchronize the data with video.
Similar Hardware and Software Implementations to those in Manuscript The description of the interface board, LTC decoding procedure, interpolated video frame decoding, LTC decoding error 2o correction, compact and interpolated time-code-bridge-files are the same as those under the respective subtitles in Hardware and Software Implementation section in the manuscript. The interpolation is not necessarily needed if the decoding procedure is conducted offline.
Search for Data Segment 25 Configuration 1. The flow chart for data search is shown in Fig.11 (b). Given the desired begin and end frame numbers of video to be displayed, the system uses the three-column time-code-bridge-file (Fig. 9 (c)) to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)) in the video computer, display the video images on the video computer monitor, frame by frame, and track the corresponding LTCs. When an interesting video frame is determined and frozen on the monitor during a video replay, the system tracks the LTC of the frozen frame. Then, the system searches the time-code-bridge-file for the corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)). Obtaining the expected corresponding index number, the system retrieves the corresponding data frame, or multiple data frames related to the corresponding data frame, from the acquired-data-file in the system computer, and turns the interface board io to Data Replay Mode. Finally, the system computer program displays the data on the system computer monitor, and/ or outputs it to the analysis computer for further data analysis using the data analysis software.
Configuration 2. The flow chart for data search is also shown in Fig.11 (b). Given the desired begin and end frame numbers of video to be displayed, the system uses the video-time-code-bridge-file (Fig.12 (b)) to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)) in the video computer, display the video images on the video computer monitor, frame by frame, and track the corresponding LTCs. When an interesting video frame is determined and frozen on the video computer 2o monitor during a video replay, the system tracks the LTC of the frozen frame. Then, the system searches the data-time-code-bridge-file (Fig.12 (a)) for the corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)) in the system computer. Obtaining the expected corresponding index number, the system retrieves the corresponding data frame, or multiple data frames related to the corresponding data frame, from the acquired-data-file in the system computer, and turns the interface board to Data Replay Mode. Finally, the system computer program displays the data on the system computer monitor, and/ or outputs it to the analysis computer for further data analysis using the data analysis software.
Search for Video Frame Configuration 1. As shown in the video-search flow chart in Fig. 11 (c), the interface board is switched to Data Replay Mode during video frame search. Given the desired begin and end frame numbers of data to be displayed, the system uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)) in the system computer, replays the data segment-by-segment (one segment contains one or multiple frames, determined by the user) on the system computer and/ or analysis computer monitors, and track the corresponding LTCs. When one so or multiple interesting frames of replayed data are determined and frozen on the system computer monitor, the system tracks the corresponding frozen LTCs, searches the time-code-bridge-file for the corresponding frame-start index numbers of the acquired-video-file (Fig. 9 (b)) in the video computer. Obtaining the expected corresponding index numbers, the s5 system retrieves the expected video frames from the acquired-video-file in the video computer, displays the corresponding video images and LTCs, frame by frame, on the video computer monitor.
Configuration 2. As shown in the video-search flow chart in Fig.11 (c), the interface board is switched to Data Replay Mode during video 2o frame search. Given the desired begin and end frame numbers of data to be displayed, the system uses the data-time-code-bridge-file (Fig.12 (a)) to retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)) in the system computer, replay the data segment-by-segment (one segment contains one or multiple frames, determined by the user) on the system 25 computer and/ or analysis computer monitors, and track the corresponding LTCs. When one or multiple interesting frames of replayed data are determined and frozen on the system computer monitor, the system tracks the corresponding frozen LTCs, searches the video-time-code-bridge-file (Fig.12 (b)) for the corresponding frame-start index numbers of the 3o acquired-video-file (Fig. 9 (b)) in the video computer. Obtaining the expected corresponding index numbers, the system retrieves the expected video frames from the acquired-video-file in the video computer and displays the corresponding video images and LTCs, frame by frame, on the video computer monitor.
Claims (16)
1. A data acquisition system, comprising:
an apparatus for recording video frames from an input video signal;
a biomedical instrument for generating an input biomedical data stream;
a system computer providing at least one digital input-output port and interface circuitry for collecting the input video signal and for collecting the input biomedical data stream; and a bridge file provided on said system computer for synchronizing the video frames from the input video signal with the input biomedical data stream.
an apparatus for recording video frames from an input video signal;
a biomedical instrument for generating an input biomedical data stream;
a system computer providing at least one digital input-output port and interface circuitry for collecting the input video signal and for collecting the input biomedical data stream; and a bridge file provided on said system computer for synchronizing the video frames from the input video signal with the input biomedical data stream.
2. A system as recited in claim 1 comprising a time-code generator for generating time-codes relating to the video signal for the bridge file on said system computer.
3. A system as recited in claim 1, wherein said system computer comprises a personal computer comprising an interface board and a video capture board.
4. A system as recited in claim 1 comprising a video camera, and wherein said apparatus for recording comprises a video tape recorder coupled to said video camera.
5. A system as recited in claim 1 comprising an analysis computer for receiving biomedical data via the interface circuitry.
6. A system as recited in claim 3 further comprising an analysis computer coupled to the interface board.
7. A system as recited in claim 1, wherein the biomedical instrument comprises a mufti-channel biomedical data acquisition instrument.
8. A system as recited in claim 7, wherein the multi-channel data acquisition biomedical instrument comprises a 16-channel electromyography (EMG) data acquisition instrument.
9. A data acquisition method comprising:
recording video frames from an input video signal;
generating an input biomedical data stream;
providing at least one digital input-output port with a system computer and interface circuitry;
collecting the input video signal at the interface circuitry of the system computer;
collecting the input biomedical data stream at the interface circuitry of the system computer; and providing a bridge file on the system computer for synchronizing video frames of the collected video signal with the collected input biomedical data stream.
recording video frames from an input video signal;
generating an input biomedical data stream;
providing at least one digital input-output port with a system computer and interface circuitry;
collecting the input video signal at the interface circuitry of the system computer;
collecting the input biomedical data stream at the interface circuitry of the system computer; and providing a bridge file on the system computer for synchronizing video frames of the collected video signal with the collected input biomedical data stream.
10. A method as recited in claim 9 comprising the set of generating time codes relating to the video signal for the bridge file on the system computer.
11. A method as recited in claim 9 comprising the set of providing an analysis computer coupled with the interface circuitry of the system computer for receiving the biomedical data stream.
12. A data acquisition system, comprising:
means for recording video frames from an input video signal;
means for generating an input biomedical data stream;
means for providing at least one digital input-output port with a system computer and interface circuitry;
means for collecting the input video signal at the interface circuitry of the system computer;
means for collecting the input biomedical data stream at the interface circuitry of the system computer; and means for providing a bridge file on the system computer for synchronizing video frames of the collected video signal with the collected biomedical data stream.
means for recording video frames from an input video signal;
means for generating an input biomedical data stream;
means for providing at least one digital input-output port with a system computer and interface circuitry;
means for collecting the input video signal at the interface circuitry of the system computer;
means for collecting the input biomedical data stream at the interface circuitry of the system computer; and means for providing a bridge file on the system computer for synchronizing video frames of the collected video signal with the collected biomedical data stream.
13. A system as recited in claim 12 comprising means for analyzing the collected biomedical data stream.
14. A system as recited in claim 12 comprising means for generating time-codes relating to the video signal for the bridge file on the system computer.
15. A system as recited in claim 12, wherein said means for recording video frames comprises a longitudinal time code (LTC) input video signal.
16. A system as recited in claim 15, wherein the longitudinal time code (LTC) input video signal and the input biomedical data stream are acquired simultaneously with said means for collecting the input video signal and said means for collecting the input biomedical data stream at the system computer for generating time-codes relating to the video signal for the bridge file on the system computer.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US15879499P | 1999-10-12 | 1999-10-12 | |
| US60/158,794 | 1999-10-12 | ||
| PCT/US2000/028170 WO2001027855A2 (en) | 1999-10-12 | 2000-10-12 | Image-synchronized multichannel biomedical data acquisition system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2387441A1 true CA2387441A1 (en) | 2001-04-19 |
Family
ID=22569738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002387441A Abandoned CA2387441A1 (en) | 1999-10-12 | 2000-10-12 | Image-synchronized multichannel biomedical data acquisition system |
Country Status (4)
| Country | Link |
|---|---|
| EP (1) | EP1281153A2 (en) |
| AU (1) | AU779045B2 (en) |
| CA (1) | CA2387441A1 (en) |
| WO (1) | WO2001027855A2 (en) |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2161921A1 (en) * | 2008-09-05 | 2010-03-10 | Thomson Licensing | Timer circuit for a video camera |
| US9575560B2 (en) | 2014-06-03 | 2017-02-21 | Google Inc. | Radar-based gesture-recognition through a wearable device |
| US9811164B2 (en) | 2014-08-07 | 2017-11-07 | Google Inc. | Radar-based gesture sensing and data transmission |
| US9921660B2 (en) | 2014-08-07 | 2018-03-20 | Google Llc | Radar-based gesture recognition |
| US9588625B2 (en) | 2014-08-15 | 2017-03-07 | Google Inc. | Interactive textiles |
| US10268321B2 (en) | 2014-08-15 | 2019-04-23 | Google Llc | Interactive textiles within hard objects |
| US11169988B2 (en) | 2014-08-22 | 2021-11-09 | Google Llc | Radar recognition-aided search |
| US9778749B2 (en) | 2014-08-22 | 2017-10-03 | Google Inc. | Occluded gesture recognition |
| US9600080B2 (en) | 2014-10-02 | 2017-03-21 | Google Inc. | Non-line-of-sight radar-based gesture recognition |
| US10064582B2 (en) | 2015-01-19 | 2018-09-04 | Google Llc | Noninvasive determination of cardiac health and other functional states and trends for human physiological systems |
| US10016162B1 (en) | 2015-03-23 | 2018-07-10 | Google Llc | In-ear health monitoring |
| US9983747B2 (en) | 2015-03-26 | 2018-05-29 | Google Llc | Two-layer interactive textiles |
| US9848780B1 (en) | 2015-04-08 | 2017-12-26 | Google Inc. | Assessing cardiovascular function using an optical sensor |
| CN111880650A (en) | 2015-04-30 | 2020-11-03 | 谷歌有限责任公司 | Gesture recognition based on wide field radar |
| WO2016176600A1 (en) | 2015-04-30 | 2016-11-03 | Google Inc. | Rf-based micro-motion tracking for gesture tracking and recognition |
| EP3289433A1 (en) | 2015-04-30 | 2018-03-07 | Google LLC | Type-agnostic rf signal representations |
| US10080528B2 (en) | 2015-05-19 | 2018-09-25 | Google Llc | Optical central venous pressure measurement |
| US9693592B2 (en) | 2015-05-27 | 2017-07-04 | Google Inc. | Attaching electronic components to interactive textiles |
| US10088908B1 (en) | 2015-05-27 | 2018-10-02 | Google Llc | Gesture detection and interactions |
| US10376195B1 (en) | 2015-06-04 | 2019-08-13 | Google Llc | Automated nursing assessment |
| US10817065B1 (en) | 2015-10-06 | 2020-10-27 | Google Llc | Gesture recognition using multiple antenna |
| WO2017079484A1 (en) | 2015-11-04 | 2017-05-11 | Google Inc. | Connectors for connecting electronics embedded in garments to external devices |
| US10492302B2 (en) | 2016-05-03 | 2019-11-26 | Google Llc | Connecting an electronic component to an interactive textile |
| US10175781B2 (en) | 2016-05-16 | 2019-01-08 | Google Llc | Interactive object with multiple electronics modules |
| US10579150B2 (en) | 2016-12-05 | 2020-03-03 | Google Llc | Concurrent detection of absolute distance and relative movement for sensing action gestures |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09501275A (en) * | 1993-04-16 | 1997-02-04 | アヴィッド・テクノロジー・インコーポレーテッド | Method and apparatus for synchronizing an image data stream with a corresponding audio data stream |
| US5517251A (en) * | 1994-04-28 | 1996-05-14 | The Regents Of The University Of California | Acquisition of video images simultaneously with analog signals |
| WO1996002188A1 (en) * | 1994-07-14 | 1996-02-01 | Infovision, Inc. | Video and analog monitoring and diagnosis system |
-
2000
- 2000-10-12 WO PCT/US2000/028170 patent/WO2001027855A2/en active IP Right Grant
- 2000-10-12 CA CA002387441A patent/CA2387441A1/en not_active Abandoned
- 2000-10-12 EP EP00972094A patent/EP1281153A2/en not_active Withdrawn
- 2000-10-12 AU AU10808/01A patent/AU779045B2/en not_active Ceased
Also Published As
| Publication number | Publication date |
|---|---|
| WO2001027855A3 (en) | 2002-11-07 |
| EP1281153A2 (en) | 2003-02-05 |
| AU1080801A (en) | 2001-04-23 |
| WO2001027855A2 (en) | 2001-04-19 |
| AU779045B2 (en) | 2005-01-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU779045B2 (en) | Image-synchronized multichannel biomedical data acquisition system | |
| AU693192B2 (en) | Acquisition of video images simultaneously with analog signals | |
| Yen et al. | A video-based system for acquiring biomechanical data synchronized with arbitrary events and activities | |
| US5791342A (en) | Medical data transmission system | |
| US5617871A (en) | Spread spectrum telemetry of physiological signals | |
| JP2596420B2 (en) | Method for recording video analog signal in digital format and method for reproducing the same | |
| US4703355A (en) | Audio to video timing equalizer method and apparatus | |
| US5205295A (en) | Method and apparatus for holter recorder with high resolution signal averaging capability for late potential analysis | |
| USRE33535E (en) | Audio to video timing equalizer method and apparatus | |
| US6792304B1 (en) | Mass communication assessment system | |
| BR0106721A (en) | Method and apparatus for processing available content | |
| GB2288942A (en) | Digital video tape recorder | |
| US6654633B2 (en) | Mobile neurological signal data acquisition system and method | |
| WO1994011995A1 (en) | Video logging system and method thereof | |
| Zeng et al. | A new video-synchronized multichannel biomedical data acquisition system | |
| EP0215604A2 (en) | Video and analogue data recording system | |
| JPS5816675B2 (en) | Jiyouhoushingoukikirokuuchi | |
| DeMaso et al. | Progress in Holter technology | |
| Kamp et al. | Improved telemetric EEG monitoring in epileptic patients | |
| Gaskill | Techniques for synchronizing thermal array chart recorders to video | |
| AU755867B2 (en) | Mass communication assessment system | |
| Cain | Forensic Examination of CCTV Digital VTR Surveillance Recording Equipment. | |
| JPS6149583A (en) | Video signal recording method | |
| JPH03268737A (en) | Living body monitoring recorder | |
| JPH04146722A (en) | Image regeneration processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Discontinued |