JP2006148862A - High resolution network camera with massively parallel implementation of image processing, compression and network server - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a network video camera capable of delivering high-resolution digital images, comprising over a million of pixels or more per image, at full motion video frame rate. <P>SOLUTION: The camera disclosed in the present invention comprises: a high-resolution image sensor 101; a computer network interface 117 for transmission of image data streams originated by said image sensor; and a multi-stage pipelined digital image processor capable of processing and compression of image data at the output rate of said image sensor, wherein said image processor comprises an image processing pipeline formed by multiple distinct stages, wherein most of said stages perform distinctly different image processing operations and wherein each said stage has an output latch or buffer 116, wherein image data propagate from stage to stage and wherein each of said stages performs its operation on image data concurrently and synchronously with most of other said stages performing their respective operations. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  Currently used video surveillance and surveillance systems are based on NTSC / PAL / SECAM analog video cameras with camera resolutions that are more strictly determined by the corresponding video standards. Thus, to see the fine details of the scene, the NTSC surveillance system must rely on expensive optical zoom and mechanical pan (camera rotation) and tilt. However, the higher the optical zoom, the more unavoidably the camera's field of view becomes narrower, and the operator needs to make a choice about the level of detail and the size of the area under surveillance. In addition, actuators that respond to mechanical pan, tilt, and zoom are typically slow compared to the camera frame rate. This makes it difficult, if not impossible, to zoom to a fast moving target such as a moving car license plate.

  For NTSC-based video, network cameras are not limited to a specific resolution and frame rate, but are primarily limited by the computational resources on the camera and the available network bandwidth. Network video cameras rely on packet-oriented digital image transmission and are not limited to any particular image resolution and frame rate. This leads to the development of a high-resolution video surveillance system that is generally superior to existing NTSC systems, providing video rate multiplex format capabilities and instantaneous pan, tilt and zoom that were not previously available for video surveillance. .

  However, to generate multi-megapixel images at video rates, network cameras must be able to perform image processing, compression and network transmission with much higher data bandwidth than normal NTSC cameras. Conventional methods that allow image processing, compression, and network protocols to be performed by a general-purpose DSP or microprocessor greatly limit the overall camera bandwidth that requires a trade-off between camera frame rate and resolution.

  The subject of the present invention is a network video camera capable of providing high resolution images having over 1 million pixels per image at video rate. The camera disclosed in the present invention includes a high-resolution image sensor operable at a video frame rate, an image buffer memory, and a network interface for transmitting an image of the camera in an off state. In a preferred embodiment, the camera disclosed in the present invention comprises an ASIC or one or more field programmable gate arrays (FPGAs) that operate under the control of a low-cost microprocessor, FPGAs constitute a large parallel image processing pipeline that performs time-limited processing on image pixels, and the flow of image pixels is such that each pipeline stage operates in parallel with all or most of the other pipeline stages. Processed by successive stages of the pipeline, the processor controls the operation of the image processing pipeline, relatively slow operations related to image pipeline and network initialization operations, automatic exposure and gain control, white balance and Protocol level network interface processing While row, maintaining the register space that constitutes the user interface to the camera.

  In a preferred embodiment, the present invention comprises a computer located and connected to the receiving side of a network interface, the computer acquiring image processing and images from one or more disclosed cameras of the present invention. All or part of the acquired software from the camera in response to a remote user request, running the acquisition software configured to display the image on a monitor with decompression and storing the image in a digital archive Configured to send images.

  The subject of the present invention is a network video camera that can provide high-resolution images with over 1 million pixels per image at full operating video frame rate. The camera disclosed in the present invention comprises a high resolution image sensor operable at a video frame rate, and a computer network interface for transmitting an image data stream generated by the image sensor of the off-state camera. Further comprising a multi-stage digital image processor capable of processing and compressing image data at a plurality of output rates and operating under the control of a low-cost microprocessor, the image processor being an image processing pipe formed by multiple individual stages. And most of the stages individually perform different image processing operations, each stage having an output latch or buffer to receive input data from one or more other stage output latches or buffers, and each stage Performs each process And other stages most parallel to and in synchronization executing the processing on the image data that. FIG. 1 shows a block diagram of a high resolution network camera with large parallel execution of image processing, compression and network interface.

  In a preferred embodiment, the camera disclosed in the present invention comprises one or more ASICs or field-pluggable gate arrays (FPGAs) operating under microprocessor control, where the ASIC or FPGA is the pixel output of the image sensor. Most are configured to implement a parallel image processing pipeline that performs time-limited processing on image pixels at a rate equal to the rate, and each of the image pixels is in another stage of the pipeline. Under the influence of successive stages of an image processing pipeline having pipeline stages that operate in parallel with all or many of the above, the microprocessor controls the operation of the image processing pipeline, and the image pipeline and network initialization operations , Relatively slow motion related to automatic exposure and gain control, And executes the network interface processing site balance and protocol level, maintaining the register space that constitutes the user interface to the camera.

  In the preferred embodiment of the present invention, a mostly parallel image processing pipeline interpolates a pixel array of one color per pixel into an image stream of three colors per pixel, and RGB correction with a 3 × 3 matrix of color correction. Color correction representing representation multiplexing, aperture correction representing application of a two-dimensional sharpening filter to an image, conversion from RGB to YUV for image representation suitable for JPEG or other image compression, gamma representing exponential image transformation An automatic exposure and gain adjustment (AE) and automatic white balance measurement (AWB) engine that collects image brightness and color statistics required for automatic exposure (AE) and automatic white balance (AWB) algorithms, along with corrections, processed image stream Buffering to one or more image frame buffers, image buffer memory accelerator controller, pie Line of image compression module, and an output control logic that provides an interface between one or more network packets memory buffer and image compression module and the network packet buffer.

  In another embodiment of the invention, the image processing pipeline implements additional operations including two-dimensional pixel defect correction, compensation of image distortion introduced by the optical system, image color saturation control and histogram equivalent processing. .

  In one embodiment of the invention, pixel defect correction is performed by replacing one defective pixel of a pixel with a pixel that directly surrounds the defective pixel. In a later embodiment, the pixels of all the images have the luminance values of the pixels processed at that time (symmetrical pixels) as the luminance values of eight pixels (surrounding pixels) directly surrounding the pixels of the same color. If a symmetric pixel has a luminance value that is higher or lower than the luminance values of all surrounding pixels, a replacement is performed, which replaces the corresponding color component of the pixel replacing the color component of all symmetric pixels. This pixel replacement is one of the surrounding pixels, and the replacement pixel has at least a different luminance value than the symmetric pixel.

  In one preferred embodiment of the present invention, once the frame is finished and the AE and AWB measurement engines accumulate image brightness and color statistics (preferred criteria for measurement are disclosed separately by the applicant). The microprocessor connected to the image processing pipeline determines a new color channel gain and color correction matrix along with an AE operation to determine a new gain setting for the pre-ADC amplifier along with a new value for the optical integration time. AWB calculation is performed for this purpose. Once such values are obtained, they are loaded into the image sensor and color pipeline registers to be applied to the image in the next frame.

  In the preferred embodiment of the present invention, two full (full) frame memories are used between image processing (to YUV) and image compression, and the processed but uncompressed image is stored in its memory buffer. The The memory buffer is accessed so that one buffer is filled with new uncompressed data while being read from the other buffers to the first stage of the image compression module. This arrangement basically allows the image readout speed from the camera to be independent of the image sensor pixel output and frame rate.

  In another embodiment of the invention, two full frame memories are used at the beginning of the image processing pipeline, and the unprocessed image sensor output is stored in this buffer, with only one color component per pixel. As a result, the demand for the size of the image buffer memory can be reduced. In this embodiment, repeated reading of the same image or image segment from the image buffer memory requires iterative processing of the image throughout the image processing pipeline.

  In yet another embodiment of the invention, the required size of the image buffer memory is further reduced by the memory placement at the end of the image compression module. In this embodiment, the camera does not support parallel output of multiple image windows generated from the same image frame.

  In one embodiment of the present invention, at least three full frame memories are used, new images are stored in one memory buffer, and two previously processed images are stored in two other memory buffers. Both image buffers containing stored and processed images can read the image to the output stage of the image processing pipeline, and buffers containing image frames already output from the camera can be used to store new image frames. it can. In this embodiment, the maximum possible camera frame rate is realized as an access to the input, and the frame buffer of the output image need not be synchronized.

  In the preferred embodiment of the present invention, the camera is capable of outputting multiple image formats, including full resolution images, reduced resolution with an image sub-window representing a rectangular area of the entire image. A thinned image is included. This is realized by reading a specific area of the image frame memory buffer. In the preferred embodiment of the present invention, one of the camera operating modes supports parallel output of multiple image formats. In the preferred embodiment, this parallel output includes the output of the entire field of the image with the scene thinned out in parallel with the output of the full resolution sub-window of the image, providing parallel utilization of the zoomed and wide-angle video streams. To do. The video stream time sandwiching means of different image formats are output and this sandwiching is performed either on a frame-by-frame basis or on a packet-by-packet basis. In the preferred embodiment, frame-by-frame pinching is achieved by reading data from the image memory buffer more than once during the frame time of the imager and outputting different image formats one after another from the camera. .

  In the preferred embodiment of the present invention, the JPEG compression pipelined configuration comprises stages that perform MCU formation, forward discrete cosine transform (FDCT), zigzag scanning (scanning), quantization and entropy coding. Including. In the preferred embodiment of the present invention, the two-dimensional FDCT is implemented as two paths through the one-dimensional FDCT transform, taking advantage of the fact that the FDCT is a separable transform.

  In yet another embodiment of the invention, other image compression methods such as JPEG2000 or MPEG-2 are substituted for the underlying JPEG.

  In the preferred embodiment, the operation of the image compression segment of the image processing pipeline is controlled by output control logic and a microprocessor in the camera, and uncompressed images are read from the image buffer memory and available network packet buffers. Processed by the pipeline's image compression stage at the rate required to fill the image, the microprocessor reads the image buffer memory and compression engine to read and compress enough image data for the next network packet to be formed. On the other hand, when there is no image request from the outside, the microprocessor deactivates the image buffer memory and the image compression pipeline in the same manner as when there is no empty network packet buffer.

  In the preferred embodiment of the present invention, a modified version of Trivial File Transfer Protocol (TFTP described in RFC783) is created as the first mode of image transmission, and the TFTP protocol header is formed in the transmit packet buffer by the microprocessor. The data field of the TFTP packet, that is, the image data, is formed by the output control logic of the image processing pipeline together with the corresponding checksum. FIG. 2 shows a flowchart of a high bandwidth camera network interface based on a modified TFTP protocol.

  In the preferred embodiment of the present invention, the output stage of the image processing pipeline comprises more than two network packet buffers, one buffer used for storing compressed data coming from the image compression module and other packets. The buffer contains packets that are sent simultaneously to the Media Access Control (MAC) Ethernet interface, and one or more packet buffers contain the next packet to be sent. This multi-buffer configuration ensures that there are packets that can always be used for transmission during the transmission of image frames, maximizes the use of available network bandwidth, and allows packets to be used for retransmission when a network error occurs. Guarantee that it remains.

  In the preferred embodiment of the present invention, the camera comprises a microprocessor and PHY hardware for interfacing with an image processing pipeline and Ethernet MAC, and is configured to support several network protocols. . In the preferred embodiment, a minimal set of protocols consisting of UDP, TFTP, ARP, IP and ICMP protocols is supported to reduce overall configuration complexity. In other embodiments of the invention, TCP / IP and DHCP protocols are also supported.

  In the preferred embodiment of the present invention, the microprocessor interprets incoming data packets and assembles responses to specific requests (eg, register access, ARP, ICMP, etc.), and network pipeline buffer in the image pipeline. It is configured to match the loading and reading of the data. In the preferred embodiment of the present invention, the microprocessor is a low cost microprocessor having a data processing throughput substantially lower than the throughput of the image processing pipeline.

  In the preferred embodiment of the present invention, the microprocessor is configured to receive and process image transmission requests, and the microprocessor periodically pools the network interface hardware for the validity of newly arrived packets. Or the microprocessor is interrupted by the MAC hardware once a new packet arrives on the network interface. In the preferred embodiment of the present invention, the characteristics of the incoming request are determined by the “file name” field of the TFTP request, and the file name, along with other request parameters including image size, resolution, and compression quality, Specify whether it relates to an image or register operation and whether the request is for a new image or a different part of a previously transmitted image.

  In the preferred embodiment of the present invention, if it is determined that the incoming network packet contains a request for image transmission, the microprocessor responds to the auto-negotiation parameter, otherwise the checksum of the next packet is Determines that it was generated by the checksum module of the image processing pipeline, generates a TFTP packet header, loads it into the currently available transmit packet buffer, and compresses the rest of the buffer into compressed image data The image processing pipeline is controlled so that it can be generated, and a transmission command is issued to the MAC hardware.

  Immediately after the first image packet is assembled, the microprocessor checks the end-of-frame flag (EOF) of the image buffer access control logic, and if not, reads the checksum to form the header of the next packet. Control the image processing pipeline to load the packet into the transmit buffer, issue a transmit command to the MAC hardware, and immediately repeat the same operation for one or more packets.

  Accordingly, one of the subject matter of the present invention is that the camera is configured to incorporate a modified TFTP protocol, and that the differences with respect to applicable standards relate to the handling of packet retransmissions in the occurrence of packet acknowledgments and network errors. is there. Such variations of the TFTP protocol are designed to easily increase the throughput of image distribution across the network, reducing or eliminating the impact of round trip packet delays on network interface throughput.

  In the preferred embodiment of the present invention, when the first image packet is transmitted, the microprocessor configures and controls the MAC hardware to transmit one or more packets without waiting for the arrival of an acknowledgement. .

  In the preferred embodiment, the camera microprocessor transmits two or more initial image packets without waiting for the arrival of an acknowledge and initiates a state waiting for the arrival of the acknowledge of the first transmitted packet. In this embodiment, if a packet sent by the end of the time limit period has not been acknowledged, the microprocessor will issue a packet resend command to the MAC hardware to resend the unacknowledged packet. It is configured. If there is no time limit, the microprocessor controls the MAC hardware to send the next buffered packet immediately after receiving the acknowledge, and fills the packet buffer containing the acknowledged packet with new image data. In this way, the image processing pipeline is controlled.

  In the preferred embodiment of the present invention, if an initial image packet is transmitted and all transmit buffers include the packet header of the previously transmitted packet, the microprocessor will only view the fields in the packet header that are different from the previous packet. It is configured to form new packets by updating, and most of the information in the IP, UDP and TFTP headers (source and destination IP and MAC addresses, UDP port, IP header checksum, etc.) Remains unchanged. In this embodiment, the fields of the image packet header that are changed from one packet to the next include the UDP checksum (including the new checksum) and the TFTP block number.

  The preferred embodiment modified TFTP protocol dramatically increases the throughput of the camera network interface by sending packets prior to the expected acknowledgment arrival to reduce or eliminate acknowledgment latency. Although this differs from the RFC 783 protocol specification of FTP, compatibility with clients that comply with TFTP is maintained as long as transmitted packets do not arrive inappropriately. In a local area network where the camera uses a predetermined large TFTP packet (greater than 1522 bytes), there is almost no inappropriate packet arrival. However, in the preferred embodiment of the present invention, the receiving TFTP client on the network interface is configured to reorder arrivals from the camera packet to guarantee an increasing order of the TFTP block numbers.

  In the preferred embodiment of the present invention, the camera microprocessor is configured to monitor an end-of-frame (EOF) flag set with image buffer access logic. The microprocessor is configured to load the EOF packet in the next available packet buffer once the EOF flag is set. In the preferred embodiment of the present invention, this EOF packet is shorter than the other image data packets and contains no data or includes a camera status indicator in the TFTP data field, and the reduced packet size is the last packet in the image transmission. means. When the last image packet is acknowledged upon receipt, the microprocessor prepares a new request by reconfiguring the network connection, the initialization of the TFTP block number, and the UDP source and destination ports.

  In the preferred embodiment, register writes and reads are also accomplished via the TFTP protocol, where the register number and value are embedded in the data field of the register access request, and the special form of the request is the TFTP “file name” field. Specified by. In another embodiment of the invention, the TCP / IP protocol is used for register and image access.

  In a preferred embodiment, the present invention comprises a computer provided on the receiving side of a network interface and connected to a monitor, the computer performing the functions of a local video server, image processing and one disclosed in the present invention. It is configured to operate acquisition software that is configured to acquire images from the above cameras, displays the image on the monitor with reverse compression, stores the image in a digital record, and allows the remote user to In response to the request, the entire image or partial image obtained from the camera is transmitted to the remote user. FIG. 3 shows a block diagram of a video system based on a high resolution network camera.

FIG. 1 shows a block diagram of a high resolution network camera with large parallel execution of image processing, compression and network server. FIG. 2 shows a flowchart of a high bandwidth camera network interface based on a modified TFTP protocol. FIG. 3 shows a block diagram of a video system based on a high resolution network camera.

Claims (48)

  A multi-stage capable of processing and compressing image data at a high resolution image sensor, a computer network interface for transmitting an image data stream generated by the image sensor of an off-state camera, and an output rate of the image sensor A pipelined digital image processor, the image processor comprising an image processing pipeline formed by multiple separate stages, most of the stages performing different image processing operations separately, Each stage has an output latch or buffer to receive input data from the output latch or buffer of one or more other stages, image data propagates from stage to stage, and each stage performs its own operations In parallel and synchronized with most stages Network camera to perform operations on data.   The network camera according to claim 1, further comprising a microprocessor or a CPU.   The microprocessor is configured to control the operation of the image processing pipeline, the control for initializing the pipeline, communicating with a pipeline measurement engine, and controlling data flow through a pipeline multiplexer. The network camera according to claim 2, comprising setting of the pipeline register and setting of pipeline parameters used in automatic white balance, automatic exposure and gain control operations.   The microprocessor includes processing of statistics accumulated by the measurement engine of the image processing pipeline, and calculation of automatic white balance (AWB) parameters along with exposure time and gain setting (AE) calculations. The network camera according to claim 3, wherein the network camera is also configured to perform an operation.   The microprocessor is configured to support network communication with an external camera operator, and the microprocessor communicates with the camera MAC hardware to interpret incoming network packets and form a network packet header. And wherein the microprocessor relies on output logic of an image processing pipeline for partial packet checksum operation and loading of image data in the data field of an outgoing network packet. The network camera described in 1.   The image processing pipeline comprises stages configured to perform color interpolation, image sharpening, color separation, gamma correction and color space conversion, wherein the color interpolation converts an image of one color per pixel to 3 per pixel. Configured to convert to a color image, the color interpolation is configured to perform multiplication of red, green and blue (RGB) pixel components by a 3 × 3 color correction matrix, and the image sharpening is Configured to apply a two-dimensional high-pass filter to the image, wherein the color space transform is a representation of the color components of the image as YUV, YCrCb or other color difference representation suitable for input to an image compression stage of the image processing pipeline. The network camera of claim 1, wherein the network camera is configured to convert to   The image processing pipeline includes a measurement engine configured to store image brightness and color statistics, the measurement engine being provided with means for transmitting the statistics to a previous microprocessor. The network camera described in 1.   7. The image processing pipeline comprises a stage configured to perform two-dimensional pixel defect correction, compensation for image distortion introduced by an optical system, color saturation control, and image histogram equalization. The network camera described.   The network camera according to claim 8, wherein the two-dimensional pixel defect correction is performed by replacing a defective pixel with one of non-defective pixels of a pixel directly surrounding the defective pixel.   For all pixels, the luminance value of the pixel (target pixel) processed at that time is compared with the luminance value of eight pixels (surrounding pixels) directly surrounding the target pixel with the same color, If the luminance value is greater than or less than the luminance value of the surrounding pixels, the replacement is performed, and the replacement is performed by replacing all the color components of the target pixel with the corresponding color components of the replacement pixel, and The network camera according to claim 9, wherein the replacement pixel is one of surrounding pixels, and the replacement pixel has a luminance value that is at least different in size from the luminance value of the target pixel.   The network camera of claim 1, wherein the image processing pipeline comprises a stage configured to perform image compression.   The image compression is based on JPEG compression, and the image processing pipeline includes formation of a minimum coded unit (MCU), a Forward Discrete Cosine Transform (FDCT), a zigzag transformation, a quantum The network camera of claim 11, comprising a stage configured to perform localization and entropy coding.   The network camera according to claim 11, wherein the image compression is JPEG2000 or MPEG-2 compression. A multi-stage capable of processing and compressing image data at a high resolution image sensor, a computer network interface for transmitting an image data stream generated by the image sensor of an off-state camera, and an output rate of the image sensor A pipelined digital image processor, the image processor comprising an image processing pipeline formed by multiple separate stages, most of the stages performing different image processing operations separately, Each stage has an output latch or buffer to receive input data from the output latch or buffer of one or more other stages, image data propagates from stage to stage, and each stage performs its own operations In parallel and synchronized with most stages Perform the operation on the data,
The network camera is capable of outputting a sequence of images of different sizes and resolutions in parallel, and includes one or more image memory buffers and an image memory buffer access controller, and the access controller stores images in the memory buffer. Configured to store and read stored images from the memory buffer, the memory access controller is further configured to reduce image resolution and size by performing image decimation and image windowing Network camera.   The camera comprises at least two image memory buffers, and the memory buffer access controller stores an image frame entering one of the memory buffers and reads a frame or a partial frame from the other memory buffer. The image is stored in the image buffer at a rate not smaller than the pixel output rate of the image sensor, and the image is read from other image buffers in the previous period at a rate determined by the output bandwidth of the network interface. When the memory access controller accesses the image buffer and completes reading of an image from one image buffer, the memory access controller uses the image buffer to store an incoming image frame, and previously uses it to store an image frame. The image buffer that has been Network camera according to claim 14 which is adapted to be used for reading the image frame to be.   The camera has at least three image memory buffers, one of which can be used to store incoming image frames, and the other image buffer can be used to read image frames output from the camera. 15. The network of claim 14, wherein the third image buffer holds the next image frame output from the camera, and once the image buffer is read, it is used to store a new incoming image frame. camera.   The image memory buffer access controller is configured to support parallel output of multiple image formats, and the image access controller forms an output image frame by reading the same image stored in the memory buffer more than once. The network camera of claim 14, wherein the image access controller generates different output frame formats by reading different segments of the stored image, and image frames of different formats are output one after another from the camera.   The memory access controller first reads and decimates all image frames stored in the image buffer, and the access controller subsequently reads a specific rectangular area of the same image to obtain a reduced resolution full field image. 15. The network camera according to claim 14, wherein the camera output format is continuously switched every frame between rectangular image windows of full resolution specified by the user.   The network camera according to claim 14, further comprising a microprocessor or a CPU.   The microprocessor is configured to control the operation of the image processing pipeline, the control for initializing the pipeline, communicating with a pipeline measurement engine, and controlling data flow through a pipeline multiplexer. The network camera according to claim 19, further comprising setting pipeline parameters used in pipeline register settings and automatic white balance, automatic exposure, and gain control operations.   The microprocessor includes processing of statistics accumulated by the measurement engine of the image processing pipeline, and calculation of automatic white balance (AWB) parameters along with exposure time and gain setting (AE) calculations. The network camera according to claim 20, wherein the network camera is also configured to perform an operation.   The microprocessor is configured to support network communication with an external camera operator, and the microprocessor communicates with the camera MAC hardware to interpret incoming network packets and form a network packet header. And wherein the microprocessor relies on output logic of an image processing pipeline for partial packet checksum operation and loading of image data in the data field of an outgoing network packet. The network camera described in 1.   The image processing pipeline comprises stages configured to perform color interpolation, image sharpening, color separation, gamma correction and color space conversion, wherein the color interpolation converts an image of one color per pixel to 3 per pixel. Configured to convert to a color image, the color interpolation is configured to perform multiplication of red, green and blue (RGB) pixel components by a 3 × 3 color correction matrix, and the image sharpening is Configured to apply a two-dimensional high-pass filter to the image, wherein the color space transform is a representation of the color components of the image as YUV, YCrCb or other color difference representation suitable for input to an image compression stage of the image processing pipeline. The network camera of claim 14, wherein the network camera is configured to convert to   20. The image processing pipeline comprises a measurement engine configured to store image luminance and color statistics, the measurement engine being provided with means for transmitting the statistics to the microprocessor. The network camera described in 1.   25. The image processing pipeline comprises a stage configured to perform two-dimensional pixel defect correction, compensation for image distortion introduced by an optical system, color saturation control, and image histogram equalization. The network camera described.   26. The network camera according to claim 25, wherein the two-dimensional pixel defect correction is performed by replacing a defective pixel with one of non-defective pixels of a pixel that directly surrounds the defective pixel.   For all pixels, the luminance value of the pixel (target pixel) processed at that time is compared with the luminance value of eight pixels (surrounding pixels) directly surrounding the target pixel with the same color, If the luminance value is greater than or less than the luminance value of the surrounding pixels, the replacement is performed, and the replacement is performed by replacing all the color components of the target pixel with the corresponding color components of the replacement pixel, and 27. The network camera according to claim 26, wherein the replacement pixel is one of surrounding pixels, and the replacement pixel has a luminance value that is at least different in size from the luminance value of the target pixel.   The network camera of claim 19, wherein the image processing pipeline comprises a stage configured to perform image compression.   The image compression is based on JPEG compression, and the image processing pipeline includes formation of a minimum coded unit (MCU), a Forward Discrete Cosine Transform (FDCT), a zigzag transformation, a quantum 30. The network camera of claim 28, further comprising a stage configured to perform localization and entropy coding.   The network camera according to claim 28, wherein the image compression is JPEG2000 or MPEG-2 compression. A multi-stage capable of processing and compressing image data at a high resolution image sensor, a computer network interface for transmitting an image data stream generated by the image sensor of an off-state camera, and an output rate of the image sensor A pipelined digital image processor, the image processor comprising an image processing pipeline formed by multiple separate stages, most of the stages performing different image processing operations separately, Each stage has an output latch or buffer to receive input data from the output latch or buffer of one or more other stages, image data propagates from stage to stage, and each stage performs its own operations In parallel and synchronized with most stages Perform the operation on the data,
The network camera includes a microprocessor, two or more network packet buffers, means for data transmission between the microprocessor and the network packet buffer, and data transmission between the image processing pipeline and the network packet buffer. And a network camera.   The network camera according to claim 31, wherein the network interface comprises an Ethernet media access control interface (MAC), and the camera comprises a communication means between the microprocessor and the media access control interface.   32. The network camera according to claim 31, wherein the microprocessor is a low-cost microprocessor having a data processing capability one digit or more smaller than that of the image processing pipeline.   The microprocessor is configured to receive and interpret incoming network data packets and assemble output packets in response to specific incoming network packets, the incoming data packets comprising image transmission requests, camera registers 35. The network camera of claim 32, comprising access requests to, address resolution protocol (ARP) requests, Internet Control Message Protocol (ICMP) pin requests, and other requests defined by related network standards.   The microprocessor is configured to respond to an external request for image transmission by forming a packet header for outgoing network packets, the header responding to a request for a computer network protocol used for image transmission 35. The network camera of claim 32, wherein the network processing pipeline is configured to control the image processing pipeline to load the header into the network packet buffer and load image data into the data field of an outgoing network packet.   The network camera of claim 32, wherein the camera supports a minimal set of computer network protocols comprising UDP, TFTP, ARP, IP and ICMP, the protocols being defined in respective standards.   37. The network camera of claim 36, wherein the camera also supports TCP and DHCP computer network protocols, the protocols being defined in their respective standards.   The camera comprises three or more output network packet buffers, one buffer is used for storing data arriving from the image processing pipeline, and the other buffer is used for transmission or transmitted to MAC hardware. 35. The network camera of claim 32, wherein the network camera includes packets and the one or more buffers hold packets in sequence following the packets currently available for transmission.   The microprocessor is configured to deactivate the image processing pipeline if all available network packets are full, and the microprocessor is just enough to fill the available network packet buffer. 32. The network camera of claim 31, wherein the network camera is configured to activate the image processing pipeline until data is processed or end of frame (EOF) is detected.   The camera is configured to transmit an image by means of a modified Trivial File Transfer Protocol (TFTP), and unlike the standard TFTP configuration (RFC783), two or more initial image data packets are sent from a remote client. Sequentially transmitted without waiting for the corresponding acknowledge to be received by the camera, the arriving acknowledge is examined by the microprocessor only after the first packet has been transmitted, and the microprocessor has already transmitted 32. The network camera of claim 31, wherein the network camera is configured to trigger a start of transmission of an additional data packet in response to receipt by an acknowledgment camera for one of the packets transmitted.   The microprocessor is configured to wait for the arrival of the packet acknowledge for a predetermined time limit, and the microprocessor transmits a previously transmitted network packet if the time limit elapses prior to the arrival of the acknowledgement. 41. The network camera of claim 40, configured to initiate retransmission of data stored in one of the buffers.   41. The output packet of claim 40, wherein all output packets are numbered in the order of initial transmission, and the receiving network client is arranged to reorder packets that arrive in order of increasing packet number. Network camera.   32. A network camera according to claim 31, wherein the resolution, size, compression quality and other parameters of the incoming image request are specified in the "file name" field of the TFTP along with the numerical value of the camera register access request.   A camera system comprising one or more network cameras, the network camera comprising a high-resolution image sensor and a computer network interface for transmission of an image data stream generated by the image sensor of an off-state camera A multi-stage pipelined digital image processor capable of processing and compressing image data at the output rate of the image sensor, the image processor being an image processing pipeline formed by multiple separate stages Most of the stages separately perform different image processing operations, each stage having an output latch or buffer to receive input data from the output latch or buffer of one or more other stages, Data from stage to stage Each stage performs operations on the image data in parallel and in synchronization with most other stages that perform the respective operations, receives images from the camera, and sends them to the camera by computer network means. A camera system further comprising a connected computer system.   The computer also includes a hard disk drive and a monitor, the computer acquires images from one or more of the cameras, compresses and displays them on the monitor, and digitally or partially displays the images on the hard disk drive. 45. The apparatus of claim 44, wherein the apparatus is configured to record and store and respond to an image transmission request from a remote user by sending a full or partial image acquired from the camera to the remote user. The camera system described.   The computer is configured to obtain an image stream comprising images of different formats from all or part of the camera, one form of the image stream being a full resolution partial image or image sandwiched between frames 45. The camera system of claim 44, comprising a thinned full field image having a window.   The camera system according to claim 46, wherein the computer is configured to simultaneously output the images of the different formats to the monitor.   48. The camera system of claim 47, wherein the computer outputs a full-field image to the monitor in parallel with a zoomed output or partial window of the image.

Images