WO2008036081A2

WO2008036081A2 - High-speed camera

Info

Publication number: WO2008036081A2
Application number: PCT/US2006/036318
Authority: WO
Inventors: Mark John Riches
Original assignee: Drs Data And Imaging Systems, Inc.
Priority date: 2006-09-15
Filing date: 2006-09-15
Publication date: 2008-03-27
Also published as: WO2008036081A3

Abstract

A digital video camera automatically switches from line-scan to area-scan modes or vice versa on-the-fly during video image acquisition. When a predetermined condition occurs, the camera automatically switches between line-scan and area-scan mode. Upon occurrence of a second condition, the camera may automatically switch back to the previous mode.

Description

High-Speed Camera

Field of the Invention

The present invention generally relates to digital cameras, and more particularly to a digital camera that automatically switches between area-scan and line-scan modes.

Background of the Invention

Current high-speed digital video cameras are typically either area-scan cameras or line-scan cameras. Area-scan cameras may also be known as framing cameras, or other terms. Line-scan cameras may also be known as streaking cameras or digital-streaking cameras, or other terms. The cameras include an image sensor that receives the image. These image sensors may be, for example, CCD or CMOS-type image sensors. In an area-scan camera, the image sensor includes_^ an array of imaging pixels that may be formed in a grid or frame. For example, a typical image sensor may include a 1280 x 1024 grid of pixels.

An area-scan camera acquires video images by taking snap shots of the entire grid or a portion of the grid at a particular frame rate, such as 500 frames per second (fps.) The image data, which is read out line by line (1280 pixels per line) of the frame, is digitized and transferred to a data store, such as an external memory. Typically, the external memory will be large enough to allow for several seconds of recording (i.e., several gigabytes.) With these conventional high-speed area-scan cameras, the format of the frame may be altered. For example, the number of vertical lines per frame may be reduced, e.g., from 1280 x 1024 at 500 fps to 1280 x 512 at 1000 fps and the like.

A line-scan camera's image sensor may include a single line of pixels (e.g., 1280 x 1.) Alternatively, the image sensor may be similar to an area-scan camera's image sensor with multiple rows and columns, however only one or a few lines may be used to image at a time.

Conventional cameras may also be configured to operate in both area-scan mode and line-scan mode, however not at the same time. To change modes, the user manually selects the desired mode. To record video, in either area-scan mode or line-scan mode, the camera may be configured via an external trigger source to record continuously to a memory or trigger/stop/start upon demand (pre/mid/post triggering). In area-scan mode, a larger field of view is obtained at a point in time, however, the temporal resolution is typically lower than for line-scan mode. For example, in area-scan mode, a camera may obtain a frame of 1280 x 1024 pixels at 500 fps. When the conventional camera is manually switched to line-scan mode, the camera obtains a line of pixels at 500,000 lines per second. Tin^'s provides better temporal resolution. Since the width of the field may be a single pixel in line-scan mode, one can zoom in on the object during image acquisition, without needing to keep other objects within the same field of view. This is advantageous, for example, in syncho-ballistic studies, wherein the line-scan mode provides a larger syncho-ballistic pseudo two-dimensional image. However, the resulting image is typically stretched (i.e., distorted) compared to an image acquired using area-scan mode.

When using an area-scan camera for video acquisition, what appears in the recorded image is limited to the camera's field of view. Therefore, objects outside the field of view will not be recorded. This can be disadvantageous, for example, when recording a number of object moving in succession, wherein some of the object may have entered the field of view while others have not yet reached that area. In this case, all of the objects will not appears at the same time in the recorded image.

Figures 1 and 2 show an example of this conventional case. A recorded image is captured by an image sensor 102 having an array of 1280 vertical x 1024 horizontal pixels. While a projectile 104 and one of its driver bands 106 is in the field of view, the camera records a frame. A second driver band 108 and debris 110, which are traveling behind the projectile 104 and first driver band 106, will not appear in the recorded frame. As shown in Figure 2, the recorded frame 202 includes an image of the projectile 204 and an image of the first driver band 206. Similarly, if the event is recorded with a line-scan camera, objects that have not yet reached the active line will not be recorded.

Therefore, it can be beneficial to use a camera, which is able to operate in both area-scan mode and line-scan mode. When in line-scan mode, a higher temporal resolution may be achieved. However, it could be advantageous to switch to area-scan mode to obtain an undistorted image. Conventional cameras may allow a switch between modes, however, no conventional cameras perform the switch automatically. A user intervention is required to manually switch between modes. This is impractical in high-speed imaging, in which objects move extremely fast. A user is not be able to manually switch modes fast enough to capture desired objects in an opposite mode during high-speed imaging.

Summary of the Invention A camera consistent with the present invention automatically switches between line-scan and area-scan modes on-the-fly during operation. For example, the camera may be operating in line-scan mode and, upon a trigger, switches immediately (or via a programmed delay) to a single (or multiple) burst of conventional high-speed frames. After the frames have been captured, the camera can continue in area-scan mode if required, or return to and record in line-scan mode. The camera can switch from area-scan mode to line-scan mode or vice versa.

The trigger that effects transition from one mode to the other may be caused by a variety of factors, such as but not limited to a user input, lapse of a predefined period of time, acquisition of a predetermined number of frames, a change in value at one or more of the pixels, and the like. For example, the trigger may start when there is a light value change in a certain percentage (e.g., 20% of the pixels) of the central 500 pixels.

In accordance with systems consistent with the present invention, a camera capable of -operating iniine-scan mode and area-scan mode is provided. The camera comprises a processor that automatically switches an operating mode of the camera from line-scan mode to area-scan mode or vice versa during video image acquisition responsive to an occurrence of a predetermined condition.

In accordance with systems consistent with the present invention, a camera capable of operating in line-scan mode and area-scan mode is provided. The camera comprises means for automatically switching an operating mode of the camera from line-scan mode to area-scan mode or vice versa during video image acquisition responsive to an occurrence of a predetermined condition.

In accordance with methods consistent with the present invention, a method for operating a camera capable of operating in line-scan mode and area-scan mode is provided. The method comprises the steps of: determining whether a predetermined condition has occurred; and automatically switching an operating mode of the camera from line-scan mode to area-scan mode or vice versa during video image acquisition responsive to an occurrence of a predetermined condition.

Other apparatus, methods, features and advantages of the present invention will be or will become apparent to one having skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be within the scope of the present invention, and be protected by the accompanying drawings. Brief Description of the Drawings

Figure 1 depicts an example of acquiring an image using a conventional camera system. Figure 2 depicts an image captured using the conventional system of Figure 1. Figure 3 depicts an illustrative camera system consistent with the present invention. Figure 4 is a block diagram of the camera system of Figure 3. Figure 5 is a block diagram illustrating components of the camera head. Figure 6 is a data flow diagram for image data in the camera head. Figure 7 is a data flow diagram for the camera head field programmable gate array. Figure 8 is a block diagram of the data acquisition card.

Figure 9 is a data flow diagram for the data acquisition card field programmable gate array. Figure 10 is a control flow diagram for the data acquisition card. Figure 11 is a flow diagram of illustrative steps performed by the trigger program. Figure 12 is a data flow diagram for synchronization and frame rate generation in the data acquisition card.

Figure 13 is a data flow diagram for triggering logic in the data acquisition card.

Detailed Description of the Invention

Reference will now be made in detail to an implementation in accordance with methods, systems, and articles of manufacture consistent with the present invention as illustrated in the accompanying drawings. The same reference numbers may be used throughout the drawings and the following description to refer to the same or like parts.

A camera consistent with the present invention automatically switches between line-scan and area-scan modes on-the-fly during operation. For example, the camera may be operating in line-scan mode and, upon a trigger, switches immediately (or via a programmed delay) to a single (or multiple) burst of conventional high-speed frames. After the frames have been captured, the camera can continue in area-scan mode if required, or return to and record in line-scan mode.

Figure 3 depicts an illustrative camera system 300 consistent with the present invention. Camera system 300 includes a camera head 302. In the preferred embodiment, the digital camera head may be a DRS Data and Imaging Systems (DRS-DIS) RDT camera, but is not limited thereto. The DRS-DIS RDT camera includes a Photobit MVl 3 CMOS image sensor, which may receive 1280 vertical x 1024 horizontal pixels at, for example, 500 fps with a global shutter. The camera may also be rotated to 1024 vertical x 1280 horizontal pixels, or the digital image can be rotated after acquisition. One having skill in the art will appreciate that a different camera head may be used.

In the illustrative example, the camera is a color camera, however, a black and white camera may also be used. In an embodiment, one or more color filters may be placed over the different lines to achieve a selectable spectral response.

Camera head 302 connects to a data acquisition card 304 in a data processing system 306 via a communication cable 308. As will be described in more detail below, the data acquisition card receives image data from the camera head, may process the image data, and stores the image data in a memory. Further, the data acquisition card sends control signals, such as a trigger signal, to the camera head. In the preferred embodiment, the data acquisition card includes a DRS-DIS RDT PCI board and may include one or more Memory Daughter Boards, however, the camera system consistent with the present with the present invention is not limited thereto.

One having skill in the art will appreciate that a camera system consistent with the present invention is not limited to the configuration shown in Figure 1. For example, the functionality of the various components shown in Figure 1 may be included in a single-piece camera, which includes a camera head and the data processing functionality of the data processing system and data acquisition card. Further, one having skill in the art will appreciate that each program and logic described herein may be implemented hardware, software, or a combination of hardware and software.

Figure 4 depicts a block diagram of an illustrative data processing system 306. Data processing system 306 comprises a central processing unit (CPU) 402, a memory 410, a secondary storage device 408, a display device 404, and an input/output device 406. The data processing system may also include, for example, a keyboard and a mouse (not shown). The memory includes an operating system 420 (e.g., Windows, Solaris, Linux, or the like) and a camera program 422. The camera program is executed by the CPU under the control of the operating system 420. As will be described in more detail below, the camera program can receive user input for configuring when to issue triggers to the camera head. Image data from the camera head may be stored in an image data file 430 in the secondary storage.

The data processing system communicates via its bus 440 with the data acquisition card. Also, as shown, the data acquisition card communicates with the camera head via communication cable 308.

Figure 5 is a block diagram of components of the camera head. The camera head includes an image sensor 502 as discussed above. In the illustrative example, the image sensor is a Photobit MV13 CMOS image sensor, which may receive 1280 vertical x 1024 horizontal pixels at, for example, 500 fps with a global shutter.

Figure 6 depicts the image sensor in more detail. The image sensor includes a pixel array 602, such as a 1280 x 1024 pixel array. One having skill in the art will appreciate that the pixel array may have a different number of pixels. As known in the art, the image sensor acquires image data when light contacts the pixels of the pixel array. Analog image data is read out from the pixel array at a particular frame rate, such as at 500 fps. hi the illustrative example, when operating in line-scan mode, all lines are exposed (similar to area-scan mode.) However, the camera reads out one or more chosen lines of interest and then repeats exposure. When multiple lines are read out in line-scan mode, sensitivity can be increased by binning pixels, however, this may also result in lower resolution.

Analog image data from even-number pixel columns is received at a first analog-to-digital converter 606. The first analog-to-digital converter converts the analog image data to a digital signal and sends the digital image data to a first memory buffer 608. The digital image data 610 for the even-number pixel columns is then transmitted from the first memory buffer to a field programmable gate array 506. Analog image data from odd-number pixel columns is received at a second analog-to- digital converter 612. The second analog-to-digital converter converts the analog image data to a digital signal and sends the digital image data to a second memory buffer 614. The digital image data 616 for the even-number pixel columns is then transmitted from the first memory buffer to field programmable gate array 506.

Referring back to Figure 5, the digital image data for the even and odd-number pixel columns 504 are received by the field programmable gate array (FPGA) 506. FPGA 506 controls timing of the image sensor and formats data for transmission to the data acquisition card. Control signals 508, such as a trigger signal, are transmitted to the image sensor, hi the illustrative example, FPGA 506 includes 6 036318

volatile memory, therefore a configuration memory 510 provides startup settings to FPGA 506 after power-on. FPGA 506 may also obtain predetermined settings from an EEPROM 512.

FPGA 506 send processed digital image data 516 to the data acquisition card via a parallel-to- serial converter 518. A serial data stream of digital image data 520 is transmitted via the communication cable from the camera head to the data acquisition card. FPGA 506 also transmits status data 528, such as power status, to the data acquisition card. The status data 528 may pass through a coupler 530 prior to transmission as coupled status data 532 to the data acquisition card.

FPGA 506 receives control data 526 from the data acquisition card via a serial-to-parallel converter 524, which converts a serial data transmission 522 to a parallel data transmission 526. Control data may include, for example, a signal to open the shutter or a signal to close the shutter.

The image sensor may receive a biasing signal from a digital-to-analog converter 514. Digital- to-analog converter 514 receives control signals from FPGA 506 and a 5V voltage signal 536. The 5V voltage signal is also supplied to a low dropout regulator 534, which provides various voltage signals 538 and 540 to devices in the camera head.

Figure 7 depicts a data flow diagram of illustrative signal compression that may be used in the camera head FPGA 506. 10-bit digital image data 702 is supplied to compression logic circuits 704, which compresses the 10-bit data to 8-bit data 706. The 8-bit data is supplied to a two-line buffer 708, which buffers the data prior to output 710 to the data acquisition card. The two-line buffer may perform horizontal image flip or insert a horizontal ramp test signal 714 for test purposes. The compression logic circuits may perform a vertical image flip by reversing the order of readout from the image sensor. 8-bit test data 712 and a vertical ramp signal 716 may also be supplied to the compression logic circuits for test purposes.

The compression logic circuits may compress the 10-bit digital image data in a variety of manners. For example, prior to compression, the 10-bit digital image data may be saturated by passing the image data through a saturation logic circuit, hi some illustrative examples, the compression logic circuits may output one or more of the following illustrative compressed signals: compressed 10 bits to 8 bits, compressed bottom 9 bits to 8 bits with saturation, top 8 bits with or without saturation, mid 8 bits with or without saturation, bottom 8 bits with or without saturation, a test input, application of a compression algorithm, and the like. Image signal data compression is known in the art and will not be described in more detail herein. Image data signals from the camera head are received by the data acquisition card. Figure 8 shows components of the illustrative data acquisition card in more detail. The digital image data from the camera head is received by a field programmable gate array (FPGA) 802. FPGA 802 formats the data for storage, such as in an on-board memory 810 or in the secondary storage. FPGA 802 receives parallel image data 808, which has been converted from serial data 804 by a serial-to-parallel converter 806. When outputting control data 812 to the camera head, a parallel-to-serial converter 814 converts the control data to a serial control signal 816. As discussed above, the control data include be for example, a signal to open the shutter or a signal to close the shutter, and the like. FPGA 802 also receives a camera head status signal 822, which has been coupled by a coupler 820 from an original status signal 818. hi the illustrative example, FPGA 802 may obtain predetermined settings from an EEPROM 824.

A processor 826 controls the data acquisition card, including FPGA 802. For example, the processor provides signals for controlling synchronization and frame rate, triggering, and transmission of image data toτnemory 810 or to the secondary storage. The processor may send or receive data from an on-board flash memory 828.

FPGA 802 communicates with on-board memory 810 via a memory connection 830 and with the data processing system via a data acquisition card interface bus 832. Further, FPGA 802 can receive control signals from external sources, such as push buttons and the like, via a control connection 834.

Figure 9 is a data flow diagram that depicts image data flow from FPGA 802 to on-board memory 810 or to the data processing system. Image data from the camera head 902 and 904 is received at FIFO buffers 906 and 908, respectively. The FIFO buffers are regulated by clock signal inputs 910 and 912, respectively. A synchronization circuit 914 combines the respective signals FIFO output signals 916 and 918 and outputs a the combined image data signal 920. The combined image data signal 920 is received by a FIFO buffer 922, which buffers the data prior to sending it to one or more of the on-board memory 810 or to an auto exposure logic circuit 924. FPGA can also send the image data to the data processing system, for example for storage in the secondary memory. To do so, the image data is buffered in buffer 926 and FIFO buffer 928 prior to transmission to the bus.

Figure 10 is a control flow diagram depicting illustrative control and status signals within the camera system. As shown, the data acquisition card processor 826 controls timing and functionality of FPGA 802 via control signals 1004. FPGA 802 provides status signals back to the processor via status signals 1008. Further, processor 826 can issue control signals 1006 to the camera head via FPGA 802. Processor 826 also sends control signals 1010 to the buffers 1002 (e.g., buffers 922, 926, and 928) for delivery of image data to storage or to the data processing system. The buffers return status information 1012 to the processor.

One of the functions that processor 926 performs is to control switching between area-scan mode and line-scan mode. Figure 11 is a flow diagram that depicts illustrative steps performed by the processor for changing modes. In the illustrative example, the steps are performed by a program in a memory of the processor. First, the program obtains the condition at which the camera should be changed from area-scan mode to line-scan mode or vice versa (step 1102). The condition may be one or more of a variety of conditions. For example, the condition may be a pushbutton input from the user, lapse of a predefined period of time, acquisition of a predetermined number of frames, a change in value at one or more of the pixels, and the like.

If the processor determines that the condition has been met (step 1104), then the processor outputs a control signal to change scan mode (step 1106). In an illustrative example, the condition for mode change from line scan to area scan is when one or more of the line pixels deviates in brightness beyond a predetermined threshold. The processor monitors the image received from the camera head. If the processor detects that the requisite number of pixels have changed in brightness beyond the predetermined threshold (e.g., based on data received from the exposure logic), then the processor instructs FPGA 802 to switch to area-scan mode. The active line may be user selectable.

Referring back to the example of Figures 1 and 2, the camera system consistent with the present invention overcomes the disadvantages of conventional cameras by providing this automatic change from one mode to the other. As the nose of the object crosses the active imaging line, the line- scan signal will change, that is if we assume a bright background and a dark projectile then the signal will dim. Detecting this change, the processor instructs the camera to switch to area-scan mode to take one or more conventional still images of the object before returning to line-scan mode to capture objects that may be delayed or traveling behind the object and out-of-frame of the still image.

Therefore, the camera captures a still image (or several if required) of the object of interest but still continues to capture unexpected 'events' like debris which would otherwise be missed. Then, the processor may instruct the camera to return to line-scan mode, for example upon detection of another condition or after a particular number of frames captured, or the camera may continue in area-scan mode. The return to the previous mode may be automatic or responsive to a manual input received from the user. The camera can be configured via a trigger source to record continuously in looped external memory or trigger/stop/start anywhere upon demand (e.g., pre/mid/post triggering.)

The camera's ability to automatically switch between modes is beneficial for many applications, such as filming projectiles, shock waves, production lines, newsprint, rotating objects, and other propagating objects.

Figure 12 is a data flow diagram depicting illustrative signals in the data acquisition card for controlling synchronization and frame rate generation. A divider 1202 receives control signals 1204 from the processor and a frequency signal 1206. The frequency signal in the illustrative example is 24 MHz. The divider outputs a reduced-frequency signal 1208 to a multiplexer 1210. The multiplexer receives a source selection signal 1212 from the processor, which tells the multiplexer which signals to pass through. In the example, the multiplexer may receive a sync-in signal 1212 and output a sync-out signal 1214.

The sync-out signal 1214 is divided by a divider 1216, which outputs a desired frame rate 1218. The frame rate is determined based on control signals from the processor, including control signal 1220 into divider 1216. The sync-out signal 1214 is also inputted to a counter 1222. The counter 1222 outputs a sync-out overflow 1224 to the processor.

The sync-out signal 1214 is also used by a triggering logic of the FPGA 802 to set a trigger out signal. Figure 13 is a data flow diagram depicting illustrative signals in FPGA 802 for triggering logic. A multiplexer 1302 receives a trigger in signal 1304 from a source. The trigger in signal may be supplied, for example, from the processor responsive to a user pushbutton input, detection of a change in the image, and the like. The multiplexer 1302 also receives control signals 1306 from the processor. The multiplexer 1302 outputs signal 1308 to a flip-flop 1310 and a second multiplexer 1312.

The flip-flop 1310 also receives the sync-out signal 1214 and outputs a flip-flop output signal 1314. The second multiplexer 1312 multiplexes signal 1308 and the flip-flop output signal 1314 to latch the trigger out signal 1316 on the falling edge of the sync-out signal 1214. The second multiplexer 1312 also receives control signals 1318 from the processor.

An edge detector and multiplexer 1322 receives the trigger out signal 1316 and outputs the trigger signal 1324. As is known in the art, the trigger signal may be applied, for example, to count frames and capture data. The edge detector and multiplexer 1322 also receives control signals 1326 from the processor. The foregoing description of an implementation of the invention has been presented for purposes of illustration and description. It is not exhaustive and does not limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practicing the invention. For example, the described implementation includes software but the present implementation may be implemented as a combination of hardware and software or hardware alone. The invention may be implemented with both object-oriented and non- object-oriented programming systems. The scope of the invention is defined by the claims and their equivalents.

Claims

Claims What is claimed is:

1. A camera capable of operating in line-scan mode and area-scan mode, the camera comprising: a processor that automatically switches an operating mode of the camera from line-scan mode to area-scan mode or vice versa during video image acquisition responsive to an occurrence of a predetermined condition.

2. The camera of claim 1, further comprising: a camera head; a data processing system electrically coupled to the camera head, the data processing system including the processor.

3. The camera of claim 1 , wherein the processor returns the camera to a previous operating mode responsive to an occurrence of a second predetermined condition.

4. The camera of claim 3, wherein the second predetermined condition is selected from the group consisting of: a received user input, a lapse of a predetermined time period, the camera's acquisition of a predetermined number of frames, and a change in value at one or more of pixels in a sensing device of the camera.

5. The camera of claim 1 , wherein the predetermined condition is selected from the group consisting of: a received user input, a lapse of a predetermined time period, the camera's acquisition of a predetermined number of frames, and a change in value at one or more of pixels in a sensing device of the camera.

6. The camera of claim 1, wherein the camera switches between modes immediately.

7. The camera of claim 1, wherein the camera switches between modes after a predetermined time period.

8. The camera of claim 1, wherein the camera switches modes before triggering.

9. The camera of claim 1, wherein the camera switches modes during triggering.

10. The camera of claim 1 , wherein the camera switches modes after triggering.

11. A camera capable of operating in line-scan mode and area-scan mode, the camera comprising-: - means for automatically switching an operating mode of the camera from line- scan mode to area-scan mode or vice versa during video image acquisition responsive to an occurrence of a predetermined condition.

12. A method for operating a camera capable of operating in line-scan mode and area-scan mode, the method comprising the steps of: determining whether a predetermined condition has occurred; and automatically switching an operating mode of the camera from line-scan mode to area-scan mode or vice versa during video image acquisition responsive to an occurrence of a predetermined condition.

13. The method of claim 12, further comprising the step of: returning the camera to a previous operating mode responsive to an occurrence of a second predetermined condition.

14. The method of claim 13 , wherein the second predetermined condition is selected from the group consisting of: a received user input, a lapse of a predetermined time period, the camera's acquisition of a predetermined number of frames, and a change in value at one or more of pixels in a sensing device of the camera.

15. The method of claim 12, wherein the predetermined condition is selected from the group consisting of: a received user input, a lapse of a predetermined time period, the camera's acquisition of a predetermined number of frames, and a change in value at one or more of pixels in a sensing device of the camera.

16. The method of claim 12, wherein the camera switches between modes immediately.

17. The method of claim 12, wherein the camera switches between modes after a predetermined time period.

18. The method of claim 12, wherein the camera switches modes before triggering.

19. The method of claim 12, wherein the camera switches modes during triggering.

20. The method of claim 12, wherein the camera switches modes after triggering.