Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/38—Transmitter circuitry for the transmission of television signals according to analogue transmission standards
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/76—Television signal recording
  • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
  • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
  • H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/21—Intermediate information storage
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/21—Intermediate information storage
  • H04N1/2104—Intermediate information storage for one or a few pictures
  • H04N1/2112—Intermediate information storage for one or a few pictures using still video cameras
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/21—Intermediate information storage
  • H04N1/2104—Intermediate information storage for one or a few pictures
  • H04N1/2112—Intermediate information storage for one or a few pictures using still video cameras
  • H04N1/2116—Picture signal recording combined with imagewise recording, e.g. photographic recording
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N1/32101—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N1/32128—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/80—Camera processing pipelines; Components thereof
  • H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
  • H04N25/44—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/76—Television signal recording
  • H04N5/907—Television signal recording using static stores, e.g. storage tubes or semiconductor memories

Definitions

• the invention relates to processing of digital images, and, in particular, to processing large digital images with a high pixel resolution in a still image mode. Further, the invention relates to an apparatus and a system performing processing of digital images.
• Digital cameras include image sensors to capture digital images and are generally operated in still image mode or live view mode. Different techniques may be used to produce still images and live view images. While it is important to produce still images with a high quality, images for live view or preview may be generated with less but still acceptable image quality.
• Image sensors typically detect a light intensity per pixel.
• color filters may be bonded to the substrate of the image sensor which allocate a certain color to a certain pixel, and each pixel detects the light intensity for the specific color.
• a typical pattern for a color filter used in digital cameras is a Bayer filter.
• a Bayer filter contains alternating rows of red and green filters and blue and green filters, wherein each row contains alternating red and green filters and blue and green filters, respectively.
• Green filters preferentially allow green light photons to pass to the detector (e.g., a photodiode) of the respective pixel of the image sensor.
• red and blue light photons that arrive at the green filter are not transmitted by the filter and, therefore, not detected by the respective detector.
• red filters preferentially allow red light photons and blue filters allow only blue light photons to pass to the respective detector.
• Each image frame has an image pixel resolution that depends on a pixel resolution of the image sensor and generally describes the detail of information that a digital image contains. In other words, by increasing the image pixel resolution, the level of detail in the image can be increased.
• the pixel resolution of an image frame is typically described by a number of pixel columns (image width) and a number of pixel rows (image height) which result in a number of total pixels that need to be processed by the specialized image processor.
• interpolating or demosaicing may be performed by specialized image signal processors (ISPs) but also by general purpose processors (CPUs) which execute image processing software programs.
• ISPs image signal processors
• CPUs general purpose processors
• a method for processing digital images in which an image frame is received from an image sensor at an image pre-processing apparatus.
• the image frame has a frame pixel resolution substantially equal to a sensor pixel resolution of the image sensor.
• a frame pixel resolution that is substantially equal to the sensor pixel resolution in this disclosure may include frame pixel resolutions that are equal to the sensor pixel resolution and frame pixel resolutions that slightly deviate from the sensor pixel resolution.
• substantially equal means that the frame pixel resolution can be slightly smaller than the sensor pixel resolution because some image frame rows and/or image frame columns are omitted from the full sensor pixel resolution.
• the image frame is divided into at least two image subframes to be sequentially processed by an image signal processor. Thereafter, the at least two image subframes are consecutively processed by the image signal processor.
• an image pre-processing apparatus which includes an image data receiver configured to receive an image frame from an image sensor, the image frame having a frame pixel resolution substantially equal to a sensor pixel resolution of the image sensor.
• the image pre-processing apparatus further includes a data interface, an imager data interface, and a direct memory access (DMA) controller in communication with the image data receiver, the data interface, and the imager data interface.
• DMA direct memory access
• the DMA controller is configured to store the image frame to a buffer memory via the data interface, to receive first and second image subframes being generated from the image frame by dividing the image frame into the first and second image subframes to be processed by an image signal processor, and to consecutively transmit the first and second subframes to the image signal processor via the imager data interface.
• Each of the first and the second image subframes have a subframe pixel area that is smaller than the sensor pixel resolution and a region in which the first and second image subframes overlap with each other.
• the subframe pixel resolution is predetermined by a processing capacity of the image signal processor.
• a camera system for processing digital images includes an image sensor, an image pre-processing apparatus in communication with the image sensor, an image processing apparatus in communication with the pre-processing apparatus and including a buffer memory and an image signal processor; and a display connected to the image processing apparatus.
• the image pre-processing apparatus includes an image data receiver configured to receive an image frame from the image sensor, the image frame having a frame pixel resolution substantially equal to a sensor pixel resolution of the image sensor, a data interface and an imager data interface, and a direct memory access (DMA) controller in communication with the image data receiver, the data interface, and the imager data interface.
• the DMA controller is configured to store the image frame to the buffer memory via the data interface, to receive first and second subframes being generated from the image frame, and to consecutively transmit the first and second subframes to the image signal processor via the imager data interface.
• an image pre-processing apparatus allows the use of pre-manufactured image processing apparatuses that have ISPs thereon with processing capabilities that would, without the operations performed by the image pre- processing apparatus, only be capable of processing downsized preview image frames, but not high resolution still image frames.
• FIG. 1 shows a schematic illustration of a camera system for processing digital images according to an exemplary embodiment of the invention
• FIG. 2 shows a schematic illustration of an image pre-processing apparatus according to an exemplary embodiment of the invention
• FIG. 3 shows another detailed schematic illustration of the camera system for processing digital images according to an exemplary embodiment of the invention
• FIG. 4A shows an illustration of an image frame generated by an image sensor
• FIG. 4B shows an illustration of a first image subframe generated from the image frame of FIG. 4 A
• FIG. 4C shows an illustration of a second image subframe generated from the image frame of FIG. 4 A
• FIG. 5 is a flow chart depicting operations performed by the camera system to process an image frame according to an exemplary embodiment of the invention
• FIG. 6 is a flow chart depicting operations performed by the camera system to generate a still image frame according to an exemplary embodiment of the invention
• FIG. 7 shows a schematic illustration of an image pre-processing apparatus according to a second exemplary embodiment of the invention.
• FIG. 8 shows a schematic illustration of a camera system for processing digital images according to a third exemplary embodiment of the invention.
• FIG. 9 shows a detailed schematic illustration of the camera system for processing digital images according to the second exemplary embodiment of the invention.
• FIG. 10 shows a detailed schematic illustration of the camera system for processing digital images according to the third exemplary embodiment of the invention.
• FIG. 11 shows a detailed schematic illustration of the camera system for processing digital images according to a fourth exemplary embodiment of the invention.
• FIG. 1 shows a camera system 100 for processing digital images according to an exemplary embodiment of the invention.
• the camera system 100 includes an image sensor 110, an image pre-processing apparatus 120, and an image processing apparatus 130 to which a display 140 and a storage medium 150 are connected.
• the image sensor 110 can be operated in a live view mode and in a still capture mode. In both modes, the full active area of the image sensor 110 is utilized, and an image frame is generated and outputted by the image sensor 110 to the image pre-processing apparatus 120. As described in more detail below, for previewing live view or still capture preview, the image frame is downsized by the image pre-processing apparatus 120 to enable a high frame rate. For still image capture, however, all lines and columns of the image frame are processed without downsizing.
• the display 140 may include an electronic view finder (EVF) that is connected to the image processing apparatus 130 via an MIPI display serial interface (MIPI DSI) (not shown) specified by the Mobile Industry Processor Interface (MIPI) Alliance, but is not limited thereto.
• EMF electronic view finder
• MIPI DSI MIPI display serial interface
• the display 140 may also include a back display of the digital camera (not shown) that is also connected to the image processing apparatus 130 via an MIPI.
• the storage medium 150 is a non-transitory computer readable storage medium, for example, a solid-state drive (SSD), but is not limited thereto. Any other non-transitory computer readable storage medium can be also utilized as the storage medium 150.
• FIG. 2 shows a schematic illustration of an image pre-processing apparatus 120.
• the image pre-processing apparatus 120 includes an image data receiver 210, a direct memory access (DMA) controller 220, a data interface 230 and an imager data interface 240 required for still image processing.
• the image data receiver 210 is configured to receive an image frame from the image sensor 110.
• the image frame has a frame pixel resolution that is substantially equal to the sensor pixel resolution of the image sensor. In other words, the image frame includes image information of all sensor pixels of the active area of the image sensor.
• the data interface 230 connects the image pre-processing apparatus 120 to a buffer memory 250 in the image processing apparatus 130.
• the imager data interface 240 connects the pre- processing apparatus 120 to an ISP 260.
• the DMA controller 220 is in communication with the image data receiver 210, the data interface 230, and the imager data interface 240, and is configured to store the image frame to the buffer memory 250 via the data interface 230, to receive first and second image subframes from the buffer memory 250 via the data interface 230, and to consecutively transmit the first and second subframes to the ISP 260 via the imager data interface
• FIG. 3 shows another more detailed schematic illustration of a camera system 300 for processing digital images according to an exemplary embodiment of the present invention.
• the camera system 300 includes an image pre- processing apparatus 310 to which image sensor 110 is connected, and an image processing apparatus 330.
• the image processing apparatus 330 is in communication with image pre- processing apparatus 310, display 140, and storage medium 150.
• the image processing apparatus 330 in the exemplary embodiment of the camera system 300 shown in FIG. 3 has a system on chip (SoC) architecture and integrates all components necessary to process an image frame received from an image sensor to generate digital images that can be displayed, printed or stored.
• image processing apparatus 330 includes image processor 342 which may be implemented, for example, as a digital signal processor (DSP) or a graphics processing unit (GPU).
• Image processing apparatus 330 further includes a first ISP 338 and data transceiver 332 configured to receive and transmit image frames to be stored in still image pre-buffer 334.
• an image data receiver 336 is provided which is configured to receive image subframes to be processed by the ISP 338.
• the image subframes processed by the ISP 338 are stored in still image post-buffer 340.
• a display controller 352 is provided which performs operations to allow the image frame captured by the image sensor 110 to be visible on the entire display 140.
• the display controller 352 is connected to the display 140 via display data transmitter 354.
• a storage controller 356 and a storage interface 358 are provided to store still image frames in a graphics image format or image frames in a raw image format in the storage medium 150.
• the image pre-processing apparatus 310 includes a data transceiver 312 and a first imager data transmitter 314.
• Data transceiver 312 and data transceiver 332 form a data interface between the image pre-processing apparatus 310 and the image processing apparatus 330.
• the data interface 312, 332 may be a high-speed serial computer expansion bus standard interface, such as a Peripheral Component Interconnect Express (PCIe) standard interface, but is not limited thereto.
• PCIe Peripheral Component Interconnect Express
• the imager data transmitter 314 together with the imager data receiver 336 form another interface (i.e., an imager data interface) between the image pre- processing apparatus 310 and the image processing apparatus 330.
• Data transceiver 312 and imager data transmitter 314 are controlled by receive DMA (RX-DMA) controller 316 and transmit DMA (TX-DMA) controller 318.
• RX-DMA controller 316 is in communication with imager data transmitter 314 via first in first out (FIFO) buffer 320.
• Image pre-processing apparatus 310 also includes image data receiver 322 and pixel processor 324 which is in communication with TX- DMA controller 318 via FIFO buffer 326.
• the first and second imager data interfaces 314, 336 and 346, 348 according to the exemplary embodiment shown in FIG. 3 are Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) image data interfaces.
• MIPI Mobile Industry Processor Interface
• CSI Camera Serial Interface
• the image pre-processing apparatus 310 is implemented as a field-programmable gate array (FPGA). However, the image pre-processing apparatus 310 may also be implemented as an application-specific integrated circuit (ASIC).
• FPGA field-programmable gate array
• ASIC application-specific integrated circuit
• Image pre-processing apparatus 310 further includes image downsizer 328 that is connected via FIFO buffer 344 to a second imager data transmitter 346 that forms together with a second imager data receiver 348 a second imager data interface.
• the imager data receiver 348 is connected to a second ISP 350.
• Both imager data interfaces 314, 336 and 346, 348 can be utilized in still capture mode.
• the ISPs 338 and 350 are configured upon entering the still capture mode and do not change between live view states and capture states. Still image processing is performed by image pre- processing apparatus 310, ISP 338, and image processor 342.
• image sensor 110 is operated in different modes, i.e., in live view mode and in a still capture mode.
• the first imager data interface 314, 336 together with ISP 338 are only used when still image frames are captured.
• the second imager data interface 346, 348 together with ISP 348 are only utilized for live view and capture preview.
• the full active area of the image sensor is used.
• the sensor pixels are corrected by pixel processor 324 in both modes.
• lines may be skipped to enable a high frame rate.
• the image sensor 110 may output only one quarter of the lines, but all columns. Since the imager data interface 346, 348 may support only a limited data rate, after pixel corrections, the frames may be horizontally resized 4: 1 by the image downsizer 328 to accommodate the limited data rate.
• the image frames in live view are processed by ISP 350 and then asymmetrically scaled to the correct resolution for the display 140.
• a typical aspect ratio of the image sensor 110 is 3:2, whereas a typical aspect ratio of the display 140 is 16:9. If, for example, the aspect ratio of the image sensor 110 is 3:2 and the aspect ratio of the display 140 is 16:9, black bars may be added at the display controller 352 so that the entire image frame received from the image sensor 110 is visible on the display 140.
• Still capture mode all lines and columns of the image sensor 110 are read out.
• pixel correction is performed by pixel processor 324 the same way as for live view mode.
• row skipping is also done by image downsizer 328.
• the format of the image data transmitted via imager data interface 346, 348 in still capture mode is the same as in live view mode and no reconfiguration of ISP 350 is required.
• Still image capture frames are displayed as they are captured by the image sensor 110 at a much lower frame rate than the live view image frames.
• live view image frames and still image capture frames usually have different integration times, and captures may be flash illuminated, there may be exposure and white balance changes between the image path flowing through ISP 350 and the image path flowing through ISP 338.
• Image statistics are collected in ISP 350 and the statistical data is used to determine exposure, white balance, and focus corrections for both live view by ISP 350 and subsequent processing of still image frames by ISP 336.
• Imager data interface 314, 336 and ISP 338 are only used when still image frames are captured.
• full resolution frames are output from the image pre- processing apparatus via data transceivers 312 and 332 to the image processing apparatus 330 and stored in a memory area of still-image pre-buffer 334.
• pre-ISP processing is performed by the image processor 342.
• pre-ISP processed image frame may be stored in storage medium 150 by storage controller 356 in a raw image format, for example in a digital negative (DNG) format.
• DNG digital negative
• the ISP 338 Since the ISP 338 has a processing capacity that cannot accommodate the full sensor width, the ISP 338 is not capable of processing the entire image frame generated by the image sensor 110 during still capture mode, the image frame needs to be processed in image subframes, i.e., in tiles or portions of half the image width.
• the image frame is divided by the image processor 342 into first and second image subframes to be sequentially processed by the ISP 338.
• Each of the first and the second image subframes has a subframe pixel resolution that is smaller than the sensor pixel resolution and a region in which the first and second image subframes overlap with each other.
• the subframe pixel resolution is predetermined by a processing capacity of the ISP 338. Thereafter, the first and the second image subframes are consecutively processed by the
• the image data is transferred back over the data interface 312, 332 to the image pre-processing apparatus 310 where it is streamed over the imager data interface 314, 336 to the front end of ISP 338.
• the image subframe is stored in the still image post-buffer 340 and reassembled into a complete image with a high image quality.
• the image processor 342 performs post-ISP processing on the image subframes in the still- image post-buffer.
• a finished image frame in a graphics image format for example, in the YCbCr tagged image file format (TIFF) or in the joint photographic experts group (JPEG) format may be stored to storage medium 150. It is also possible to compress the finished image frame by a JPEG encoder (not shown) before storing it to the storage medium 150.
• the processing of image subframes is asynchronous with the capture of image frames, is somewhat slower than the capture of the image frames and may continue as a background process after the image sensor 1 10 returns to live view mode.
• the still image pre-buffer 334 empties relatively slowly compared to the fill rate during a burst.
• captures will be blocked, even if the user holds the shutter button of the camera.
• additional burst frames can be captured.
• FIG. 4A shows an illustration of an image frame 410 generated by image sensor 110.
• FIG. 4A shows a frame which has a number of pixel columns (representing the pixel width) of the image frame that is larger than the number of pixel rows (representation the pixel height) of the image frame, any other ratio between width and height is possible.
• the image frame 410 is vertically divided, by way of example, at the dotted line shown in FIG. 4A into first and second image subframes.
• any other division of the image frame 410 is also possible.
• the image frame 410 may be horizontally divided into three subframes, the image frame 410 may also be further divided vertically, or otherwise, as long as the resulting subframes have an image subframe resolution that can be processed by ISP 338.
• FIG. 4B shows an illustration of a first image subframe 420 generated from the image frame 410 by image processor 342
• FIG. 4C shows an illustration of a second image subframe 440 generated from the image frame 410 by image processor 342.
• each of the first and second image subframes have a region 430, 450 in which the first and second subframes 420 and 440 overlap with each other.
• FIG. 5 (with reference to FIGS. 3 and 4A to 4C), in which a flow chart is shown illustrating method 500 including operations performed by the camera system 100 to process an image frame 410 according to an exemplary embodiment of the invention.
• Method 500 begins at step 510 where an image frame 410 is received from image sensor 110, which is operated in still image mode.
• the image frame 410 is divided into first and second image subframes 420 and 440, and at 530, the first and second image subframes 420 and 440 are consecutively processed by ISP 338.
• FIG. 6 (with continued reference to FIGS. 3 and 4A to 4C).
• FIG. 6 (with continued reference to FIGS. 3 and 4A to 4C).
• FIG. 6 is a flow chart showing a method 600 for operating the camera system 100 to generate a still image frame according to an exemplary embodiment of the invention.
• the method begins at step 605 in which defects in the image frame 410 are corrected by pixel processor 324.
• the method continues to 610 at which the corrected image frame is transmitted to still image pre-puffer 334 shown in FIG. 3 by TX-DMA controller 318 of image pre-processing apparatus 310.
• the image frame 410 is stored in still image pre-buffer 334 and at 620, first image subframe 420 is received by RX-DMA controller 316 of image pre-processing apparatus 310 via data interface 312, 332.
• the first image subframe 420 is generated by image processor 342 in still image pre-buffer before being transmitted to the image pre-processing apparatus.
• the method continues to 625 at which the first image subframe 420 is transmitted via imager data interface 314, 336 by RX-DMA controller 316 to ISP 338 where it is processed at 630 by the ISP 338.
• a second image subframe 440 is received from still image pre-buffer 334 by RX-DMA controller 316 of image pre-processing apparatus 310, and at 640, the second image subframe 440 is transmitted to the ISP 338 via imager data interface 314, 336 where it is processed by ISP 338 at 645.
• the method concludes with 650 at which the processed first and second image subframes 420 and 440, that are stored after being processed by ISP 338 in still image post-buffer 340, are reassembled in Still Image post-buffer 340 to a still image frame.
• the reassembling or subframe merging in the Still Image post-buffer 340 is configurable and overlapping regions are determined by the image quality parameters and spatial components determined in the ISP 338.
• the processing engines of the ISP 338 are configured in such a way that the spatial components in pre-processing and post-processing of the image may minimize the image quality which may affect the subframe processing and the final still image frame.
• Noise filtering and other fixed pattern noise removal are performed by the Image Processor 342 on the entire image.
• the pre-processing block is tuned in such a way that minimal sensor specifics like ADC, pedestal corrections, black offset removal processing, etc., is performed outside the ISP 338 to avoid any spatial artefacts which depend on the full frame capture.
• FIG. 7 shows a schematic illustration of an image pre-processing apparatus 720 of a camera system 700.
• the image pre-processing apparatus 720 includes an image data receiver 210 and an imager data interface 240 for still image processing.
• the configuration of the image data receiver 210 and of the data interface 240 has already been discussed with regard to FIG. 2.
• image pre-processing apparatus 720 does not require a DMA controller and a data interface to store image frame 410 to the buffer memory 250, to receive first and second image subframes 420 and 440 from the buffer memory 250 and to consecutively transmit first and second subframes to the ISP 260.
• image frame 410 generated by image sensor 110 is transmitted from image sensor 110 through image pre-processing apparatus 720 via image data receiver 210 and imager data interface 240 to pre-processor 730.
• Pre- processor 730 stores image frame 410 in buffer memory 250, receives first and second image subframes 420 and 430 from the buffer memory 250 and consecutively transmits the first and second subframes to the ISP 260.
• FIG. 8 shows a schematic illustration of a camera system 800 for processing digital images according to a third exemplary embodiment of the invention.
• image sensor 110 is directly connected to pre-processor 730 without an intermediate image pre-processing apparatus.
• image processor 110 directly transmits image frame 410 to pre-processor 730 which stores image frame 410 in buffer memory 250, receives first and second image subframes 420 and 430 from the buffer memory 250 and consecutively transmits the first and second subframes to the ISP 260.
• FIG. 9 shows a detailed schematic illustration of camera system 700 for processing digital images according to the second exemplary embodiment of the invention.
• the camera system includes an image pre-processing apparatus 910 to which image sensor 110 is connected, and an image processing apparatus 930.
• the image processing apparatus 930 is in communication with image pre-processing apparatus 910, display 140, and storage medium 150.
• image pre-processing apparatus 910 to which image sensor 110 is connected
• image processing apparatus 930 is in communication with image pre-processing apparatus 910, display 140, and storage medium 150.
• the description of elements that are the same as in the FIG 3 is omitted.
• the image processing apparatus 930 in the second exemplary embodiment of the camera system shown in FIG. 9 has a SoC architecture and integrates all components necessary to process an image frame received from an image sensor to generate digital images that can be displayed, printed or stored.
• image processing apparatus 930 includes pre-processor 932.
• Pre- processor 932 performs operations to allow image frame 410 received by image data receiver 336 to be stored in still-image pre-buffer 334.
• pre-processor 932 is configured to receive first and second image subframes 420 and 430 generated by image processor 342 from still-image pre-buffer 334 and to consecutively transmit the first and second image subframes 420 and 430 to ISP 338.
• pre-processor 932 provides a data path for the first and second image subframes 420 and 430 from the still image pre-buffer 334 back into the front end of ISP 1 without modifying the data.
• ISPs such as ISP 338 are typically complex structures and permit a plurality of operating parameters to be set. Therefore, the data path is needed to permit a test image to be processed by the ISP a plurality of times with different parameters. This allows the same image to be replayed multiple times until the correct parameters are set. Thereby, the ISP and the configuration of the ISP can be tested. Such a test within the SoC would not be possible if the configuration of the image processing apparatus would only allow the ISP to receive live data from an image sensor.
• the image processing apparatus 310 shown in FIG. 3 does not include the data path between still image pre-buffer 334 and ISP 338. Therefore, the data path in the exemplary embodiment shown in FIG. 3 is created by pre-processing apparatus 310 via data interfaces 332 and 312, receive DMA controller 316, FIFO 320, image data transmitter 314, and image data receiver 336.
• FIG. 10 shows a detailed schematic illustration of camera system 800 for processing digital images according to the third exemplary embodiment of the invention.
• the image processing apparatus is entirely omitted and the image sensor 110 is directly connected to image data receiver 336 of the image processing apparatus 1030.
• the image sensor 110 outputs image frame 410 in accordance with the communication standards of the image data receiver 336 and the corrections of the image data performed by the pixel processor 324 in the exemplary embodiments shown in FIGS. 3 and 9 are either performed by ISP 338 or are dispensed with.
• image processing apparatus 1030 does not include a second data path for live view via a separate ISP, such as ISP 350 in FIGS. 3 and 9. Therefore, in the exemplary embodiment shown in FIG. 10, view data is generated from still image post-buffer 340 and transmitted from still image post-buffer 340 to display controller 352.
• the pre-processor 932 in FIGS. 9 and 10 can also be omitted and direct data paths (not shown) can be established between image data receiver 336 and still image pre-buffer 334, and between still image pre buffer 334 and ISP 338.
• FIG. 11 shows a detailed schematic illustration of a camera system for processing digital images according to a fourth exemplary embodiment of the invention.
• image processing apparatus 1130 includes a second data path for live view via ISP 350. That is, the image processing apparatus 1130 includes data paths from the imager data receiver 336 that allow one image sensor input to be routed to either of two or more ISPs.
• the pre-image processor 932 may not be capable of downsizing the image frames in live view, or, as discussed above, the pre-processor 932 may be omitted entirely.
• image sensors allow skipping or binning in both horizontal and vertical dimensions to reduce output resolution which allows directly connecting the image sensor 110 to the image processing apparatus 1130 and to perform live view via ISP 350, for example.
• image sensors can also be connected to the image processing apparatus 1030 or to any other image processing apparatus of the other exemplary embodiments discussed above to provide additional capabilities to reduce output resolution.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• General Engineering & Computer Science (AREA)
• Studio Devices (AREA)
• Image Processing (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Related Parent Applications (1)

Related Child Applications (1)

Publications (1)

Family

ID=68764434

Family Applications (1)

Country Status (5)

Families Citing this family (5)

Citations (10)

Family Cites Families (15)

• 2018
  • 2018-06-12 US US16/006,637 patent/US10681266B2/en not_active Expired - Fee Related
• 2019
  • 2019-06-12 EP EP19818615.7A patent/EP3808071A4/en not_active Withdrawn
  • 2019-06-12 WO PCT/US2019/036687 patent/WO2019241331A1/en not_active Ceased
  • 2019-06-12 JP JP2020569907A patent/JP2021527273A/ja active Pending
  • 2019-06-12 CN CN201980040017.0A patent/CN112703728B/zh not_active Expired - Fee Related

Patent Citations (10)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events