WO2010064374A1 - Image processing device - Google Patents

Image processing device Download PDF

Info

Publication number
WO2010064374A1
WO2010064374A1 PCT/JP2009/006285 JP2009006285W WO2010064374A1 WO 2010064374 A1 WO2010064374 A1 WO 2010064374A1 JP 2009006285 W JP2009006285 W JP 2009006285W WO 2010064374 A1 WO2010064374 A1 WO 2010064374A1
Authority
WO
WIPO (PCT)
Prior art keywords
image processing
temporary storage
image
processing apparatus
image data
Prior art date
Application number
PCT/JP2009/006285
Other languages
French (fr)
Japanese (ja)
Inventor
古田暁広
高橋雄一郎
坪田一広
Original Assignee
パナソニック株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by パナソニック株式会社 filed Critical パナソニック株式会社
Publication of WO2010064374A1 publication Critical patent/WO2010064374A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/21—Intermediate information storage
    • H04N1/2104—Intermediate information storage for one or a few pictures
    • H04N1/2112—Intermediate information storage for one or a few pictures using still video cameras
    • H04N1/2137—Intermediate information storage for one or a few pictures using still video cameras with temporary storage before final recording, e.g. in a frame buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
    • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
    • H04N2201/3285—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device using picture signal storage, e.g. at transmitter
    • H04N2201/329—Storage of less than a complete document page or image frame
    • H04N2201/3292—Storage of less than a complete document page or image frame of one or two complete lines

Definitions

  • the present invention relates to an image processing apparatus that processes image data.
  • the image processing apparatus reduces the processing load of a host that controls the entire image processing system, such as processing with large processing load such as adjustment and quality improvement of video signals fetched from a camera module, compression and decompression of moving images, etc. And is generally configured using a dedicated processing engine.
  • FIG. 1 is a block diagram showing the configuration of a conventional image processing apparatus.
  • the image processing apparatus 10 includes an image processing unit 11 that performs fixed processing A, an image processing unit 12 that performs fixed processing B, an image processing unit 13 that performs fixed processing C, and SDRAM (Synchronous Dynamic Random Access Memory). And the like, and a bus 15 interconnecting these parts.
  • a camera module 16 incorporating a charge coupled device (CCD) camera or a complementary metal oxide semiconductor (CMOS) camera is connected to the image processing unit 11, and captured image data is input as a video signal.
  • the image processing unit 11 is a processing engine that performs fixed processing A (for example, RAW data processing (black level correction, gain adjustment, white balance correction, gamma correction)).
  • the image processing unit 12 is a processing engine that performs fixed processing B (for example, YC (brightness and color difference) separation).
  • the image processing unit 13 is a processing engine that performs fixed processing C (for example, output processing).
  • the shared memory 14 is connected to the bus 15 and temporarily stores data to be transferred between a plurality of processing engines (image processing units 11 to 13).
  • the shared memory 14 is controlled by an SDRAM controller (not shown).
  • the SDRAM controller performs access control to the SDRAM by exchanging addresses, data, and control signals with each processing engine.
  • Patent Document 1 also discloses, in FIG. 2 thereof, a buffer memory which is controlled by a DMA control unit, is connected to a bus, and temporarily stores data transferred between a plurality of processing engines.
  • the object of the present invention is made in view of the above, and is to solve the bottleneck of the memory IO and to provide a high-performance and programmable image processing apparatus.
  • An image processing apparatus is an image processing apparatus for processing image data composed of a plurality of divided image data, and includes a plurality of image processing means for executing a specific fixing process on the divided image data; A plurality of temporary storage means for temporarily storing image data; a bus for selectively connecting the image processing means and the temporary storage means; and a switching control means for selectively controlling the connection of the buses. Take the configuration provided.
  • the image processing means performs input / output in units of divided image data (for example, one line) with a bus that can be selectively combined
  • unnecessary processing skips and processing order Can be used as a common platform applicable to a plurality of different types of camera products with different applications.
  • high-resolution image processing can be performed without using a plurality of external memories, so both high performance and low price can be achieved.
  • FIG. 6 is a diagram showing a bus switching control sequence of the image processing apparatus according to the first embodiment of the present invention.
  • FIG. 6 is a diagram showing processing timing of the image processing apparatus according to the first embodiment of the present invention.
  • Block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention The figure which shows the data flow of the bus of the image processing apparatus which concerns on Embodiment 2 of this invention.
  • Block diagram showing a configuration of an image processing apparatus according to Embodiment 3 of the present invention The figure which shows the structure of the temporary memory part of the image processing apparatus which concerns on Embodiment 4 of this invention. The figure which shows the structure of the temporary storage part of the image processing apparatus which concerns on Embodiment 5 of this invention.
  • Block diagram showing a configuration of an image processing apparatus according to Embodiment 6 of the present invention A diagram showing a configuration of a temporary storage unit of an image processing apparatus according to Embodiment 6 of the present invention.
  • FIG. 6 is a diagram showing processing timing of the image processing apparatus according to the first embodiment of the present invention.
  • FIG. 16 is a diagram showing processing timing of the image processing apparatus according to the sixth embodiment of the present invention.
  • Block diagram showing a configuration of an image processing apparatus according to Embodiment 7 of the present invention Block diagram showing a configuration of an image processing apparatus according to Embodiment 7 of the present invention
  • FIG. 2 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention.
  • the present embodiment is an example applied to an apparatus that performs signal processing of a digital camera.
  • the image processing apparatus 200 includes an input interface (I / F) 201, an output I / F 202, image processing units 211 to 214, a bus 220, and a temporary storage unit 231 to 233. Mainly composed.
  • the image processing apparatus 200 also performs predetermined signal processing based on a control signal input from the external main control unit 250.
  • the image processing apparatus 200 is an apparatus that performs signal processing of a digital camera, and receives an output signal (Bayer array RAW data) of an image sensor (camera) to generate and output a YUV signal.
  • the resolution of the image sensor is, for example, 2560 horizontal pixels ⁇ 1920 vertical pixels (about 4.9 M pixels).
  • the input I / F 201 captures RAW data into the image processing apparatus 200 using an input clock and horizontal and vertical synchronization signals associated therewith.
  • the output I / F 202 adds an output clock and a horizontal / vertical synchronization signal to the YUV signal and outputs the same to the outside of the image processing apparatus 200.
  • the image processing unit 211 performs RAW data processing (black level correction, gain adjustment, white balance correction, gamma correction).
  • the image processing unit 212 is a two-dimensional digital filter with variable coefficients.
  • the image processing unit 212 incorporates a data buffer for four horizontal lines, and executes a filter operation of up to 100 taps (horizontal 20 ⁇ vertical 5).
  • the image processing unit 213 performs YC (brightness and color difference) separation.
  • the image processing unit 213 has a built-in data buffer for four horizontal lines, and generates luminance / color difference data (YUV signal) while performing interpolation processing with the data in this buffer.
  • the image processing unit 214 performs graphic superposition (processing of superposing a rectangle or a rectangular frame on an image). By superimposing a rectangle or a rectangular frame on an image, the photographer can be alerted and privacy of a subject to be photographed can be protected.
  • the bus 220 selectively couples the image processing units 211 to 214 with the temporary storage units 231 to 233.
  • the bus 220 is composed of 2-input 1-output selectors 221 to 224 (selectors S1 to S4) and a register 225 for switching and controlling the selectors 221 to 224.
  • the selectors 221 to 224 switch the 2-input connection bus to any one.
  • the register 225 outputs control signals for switching the selectors 221 to 224 for each application to the selectors 221 to 224 according to an instruction from the main control unit 250 outside the image processing apparatus 200.
  • the temporary storage units 231 to 233 are 2-port SRAMs capable of storing data (equivalent to 5120 pixels) for two horizontal lines.
  • the 2-port SRAM has, for example, 16 bits x 5120 words in 1 read 1 write.
  • the image processing units 211 to 214 are connected to the signal lines for the read address (not shown), the write address (not shown), the read data, and the write data through the bus 220.
  • FIG. 3 is a diagram showing processing contents and a selector control method of the bus 220 for each of three types of applications in which the present apparatus is used.
  • Information obtained by coding the content of this figure is stored in the internal memory of the main control unit 250 outside the image processing apparatus 200.
  • the information stored in the main control unit 250 includes the image processing unit 212 (for each of “1. normal use”, “2. monitoring under dark environment”, and “3. medical use”). 2) filtering (filter coefficients) of the two-dimensional digital filter and selection directions of the selectors 221 to 224 of the bus 220.
  • the register 225 of FIG. 2 outputs a control signal (for example, 1-bit information) to the selectors 221 to 224 (selectors S1 to S4) according to the value set from the main control unit 250, thereby selecting the selectors 221 to 224 (selectors). Switch control of S1 to S4).
  • the selectors 221 to 224 select one of the connection buses in this figure (“a” or “b”) of the two inputs “a” and “b” (see FIG. 5). In FIG. 2, the upper inputs of the selectors 221 to 224 are “a” and the lower inputs are “b”.
  • processing is performed in the order of RAW data processing in the image processing unit 211 ⁇ YC separation in the image processing unit 213 ⁇ filter processing in the image processing unit 212 ⁇ graphic superposition in the image processing unit 214.
  • the filter processing of the image processing unit 212 (two-dimensional digital filter) is capillary blood vessel enhancement (red high frequency component enhancement).
  • the selector 221 (selector S1) is “b”
  • the selector 222 (selector S2) is “a”
  • the selector 223 (selector S3) is “a”
  • the selector 224 selectives “b” respectively.
  • the image processing units 211 to 214 all input or output data in units of one line in the image because of the nature of the processing. For example, when the image processing unit 212 (two-dimensional digital filter) receives image data for one new line from the bus 220, the image data for four lines held in the built-in data buffer and the image data for one new line A maximum of 100 taps filter operation is performed on the data of the image data of 5 lines in total to output the image data of 1 line, and the image data of the oldest 1 line of the built-in data buffer is new Replace with one line of image data.
  • the image processing unit 212 processes the entire one screen by repeating the processing for each line.
  • FIG. 4 is a diagram showing processing timings of the image processing apparatus 200 in “1. normal use” shown in FIG.
  • the timing of image input the processing latency of the image processing unit 211 (RAW data processing), the writing operation to the temporary storage unit 231 by the image processing unit 211, and the temporary storage unit 231 by the image processing unit 213 (YC separation) Reading operation, processing latency of the image processing unit 213, writing operation to the temporary storage unit 233 by the image processing unit 213, reading operation from the temporary storage unit 233 by the image processing unit 214 (graphic superposition), processing latency of the image processing unit 214, And timing of image output.
  • the vertical axes of the temporary storage units 231 and 233 indicate addresses.
  • Raw data from the input I / F 201 is input in a 16 ⁇ s period in a 40 ⁇ s cycle per line, and the data rate is 160 MHz / pixel.
  • the image processing unit 211 (RAW data processing), the image processing unit 213 (YC separation), and the image processing unit 214 (graphics superposition) operate in synchronization in 40 ⁇ s by the above horizontal synchronization signal and process 10 ⁇ s, 24 ⁇ s, and 6 ⁇ s, respectively. Has a latency.
  • Each of the image processing units 211, 213, and 214 generates a read / write address for the temporary storage units 231 and 233, and the write area and the read area are alternately switched in a cycle of 40 ⁇ s.
  • the image processing unit 211 performs RAW data processing on the input image data with a processing latency of 10 ⁇ s, and sequentially writes the processed data in the temporary storage unit 231. Then, 40 ⁇ s after the start of processing by the image processing unit 211, the image processing unit 213 sequentially reads data from the temporary storage unit 231, performs YC separation with a processing latency of 24 ⁇ s, and sequentially completes processed data in the temporary storage unit 233. Write. Then, 40 ⁇ s after the start of processing by the image processing unit 213, the image processing unit 214 sequentially reads data from the temporary storage unit 233, performs graphic superposition with a processing latency of 6 ⁇ s, and sequentially outputs processed data.
  • the image processing units 211 to 214 operate in parallel as pipeline processing in units of lines.
  • the image processing apparatus 200 is used for the three types of applications shown in FIG. 3, and switches the execution existence and the order of the image processing units 211 to 214 for each application.
  • the processing content of the image processing unit 212 differs depending on the application. For example, when the image processing apparatus 200 is used for “2. Monitoring in a dark environment”, the amount of light input to the image sensor decreases and noise increases, so the image processing unit 212 processes high frequency components after RAW data processing. Remove and improve the image quality to make it easy for the observer to see.
  • FIG. 5 is a diagram showing a data flow of the bus 220 of the image processing apparatus 200. Thick solid lines in FIG. 5 indicate buses through which data flows by selection of the selectors 221 to 224 (selectors S1 to S4), and broken lines in the same figure indicate buses through which data does not flow due to non-selection of the selectors 221 to 224.
  • FIG. 5A is a data flow of the processing order (image processing unit 211 (RAW data processing) ⁇ image processing unit 213 (YC separation) ⁇ image processing unit 214 (graphics superposition)) in “1. normal use”.
  • FIG. 5B shows the processing order in “2. Monitoring under dark environment” (image processing unit 211 (RAW data processing) ⁇ image processing unit 212 (two-dimensional digital filter) ⁇ image processing unit 213 (YC separation) ⁇ image processing It is a data flow of part 214 (figure superposition).
  • FIG. 5C shows the processing order in “3. medical use” (image processing unit 211 (RAW data processing) ⁇ image processing unit 213 (YC separation) ⁇ image processing unit 212 (two-dimensional digital filter) ⁇ image processing unit 214 (figure Data flow).
  • image processing unit 211 RAW data processing
  • image processing unit 213 YC separation
  • image processing unit 212 two-dimensional digital filter
  • image processing unit 214 figure Data flow
  • the image processing apparatus 200 sets the four selectors 221 to 224 (selectors S1 to S4) included in the bus 220 as shown in FIG.
  • the temporary storage units 231 to 233 are selectively coupled. As a result, the execution existence and the order of the image processing units 211 to 214 are changed.
  • the memory in units of lines is used instead of the shared memory, the memory IO bottleneck can be eliminated and the cost can be suppressed with respect to the conventional device.
  • the conventional device uses a shared memory such as an SDRAM. Since the throughput of currently available DDR SDRAM is about 1.3 [GByte / s] at the maximum, the memory IO becomes a bottleneck in one SDRAM and the device can not be configured. Two SDRAMs are required to configure the conventional device. As described above, although the conventional device is unnecessary as a storage capacity, it is necessary to prepare a plurality of external memories to secure the throughput, which is expensive.
  • three small-capacity SRAMs of 640 [MByte / s] may be prepared, and therefore, they can be integrated on one semiconductor device (LSI), which is inexpensive
  • LSI semiconductor device
  • the register 225 switches and controls the selectors 221 to 224 according to the external control, thereby combining the image processing units 211 to 214 with the temporary storage units 231 to 233.
  • the form is changed to specify the execution order of the image processing units 211 to 214.
  • unnecessary processing can be skipped and the processing order can be changed, and a memory IO bottleneck can be eliminated, and a high performance and programmable image processing apparatus can be realized.
  • platforms for various camera products with different applications can be made common.
  • clock supply or power supply to the image processing units 211 to 214 skipped as unnecessary processing and the unused temporary storage units 231 to 233 can be stopped. In this way, power consumption can be reduced, and in battery-powered devices, continuous operation time can be extended.
  • the second embodiment is an example in which an external memory is connected to the image processing apparatus described in the first embodiment.
  • FIG. 6 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention.
  • the same components as in FIG. 2 will be assigned the same reference numerals and overlapping description will be omitted.
  • the image processing apparatus 300 includes an image processing apparatus main body 310 and an external memory 320.
  • the image processing apparatus main body 310 includes an input I / F 201, an output I / F 202, and an image processing unit 211.
  • a DMA (Direct Memory Access) controller 311 To 214A, a DMA (Direct Memory Access) controller 311, a bus 220A, and temporary storage units 231 to 234.
  • DMA Direct Memory Access
  • the image processing apparatus 300 is an apparatus that performs signal processing of a digital camera as in the image processing apparatus 200 of FIG. 2 and generates an YUV signal using an output signal (Bayer array RAW data) of an image sensor (camera) as an input. And output.
  • the image processing unit 214A reads the image data of one line at the same position from the temporary storage unit 233 and the temporary storage unit 234, in addition to the same function as the image processing unit 214 in FIG. Then, the presence or absence of a moving object (whether it is a still image or not) is determined.
  • the bus 220A selectively couples the image processing units 211 to 214A, the DMA controller 311, and the temporary storage units 231 to 234.
  • the bus 220A has a configuration in which a bus for connecting the DMA controller 311 and a selector are added to the bus 220 in FIG. 2 and is basically a configuration in which the bus 220 in FIG. 2 is expanded.
  • the temporary storage unit 234 is a two-port SRAM that can store data (corresponding to 5120 pixels) for two horizontal lines, similarly to the temporary storage units 231 to 233.
  • the 2-port SRAM has, for example, 16 bits x 5120 words in 1 read 1 write.
  • the DMA controller 311 performs data transfer between the bus 220A and the external memory 320 in units of one line.
  • the DMA controller 311 reads image data of one line from the temporary storage unit 233 and writes the image data to the external memory 320. Further, the DMA controller 311 reads the image data for one line at the same position of the old frame from the external memory 320 and writes the image data in the temporary storage unit 234.
  • the external memory 320 is a memory (for example, an SDRAM) having a large capacity as compared with the temporary storage units 231 to 234.
  • the external memory 320 is provided to hold past image data in frame units.
  • the external memory 320 stores image data of two old and new frames.
  • FIG. 7 is a diagram showing a data flow of the bus 220A of the image processing apparatus 300.
  • the image data is input to the image processing unit 211 and then the image is stored in the temporary storage unit 233.
  • the operation until the data is written is the same as that of the image processing apparatus 200 (FIG. 5) in the first embodiment, so the description will be omitted.
  • the DMA controller 311 When the image data is written to the temporary storage unit 233, the DMA controller 311 reads the image data of one line from the temporary storage unit 233 and writes the image data to the external memory 320. At the same time, the image data for one line is input to the image processing unit 214A.
  • the DMA controller 311 reads the image data of one line of the old frame from the external memory 320 and writes the image data in the temporary storage unit 234.
  • the image processing unit 214A reads the image data of one line of the old frame from the temporary storage unit 234.
  • the DMA control is performed such that the image data of one line of the old frame and the image data of one line of the above (new frame) are at the same position on the original frame.
  • the image processing unit 214A determines the presence or absence of a moving object from the difference between new and old two image data of one line at the same position.
  • the image processing apparatus 300 holds the external image 320 holding the past image data in frame units, and the DMA controller performs data transfer between the bus 220A and the external memory 320.
  • the DMA controller performs data transfer between the bus 220A and the external memory 320.
  • the external memory 320 having a large capacity is used as an external component, and mounting of the external memory can be omitted when the signal processing for referring to the past frame is unnecessary. This makes it possible to construct a platform that can be adapted to both high performance and low cost products.
  • the third embodiment is an example of connecting an external device to the image processing apparatus described in the second embodiment.
  • FIG. 8 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 3 of the present invention.
  • the same components as in FIG. 6 will be assigned the same reference numerals and overlapping descriptions will be omitted.
  • the image processing apparatus 300 comprises an image processing apparatus main body 310, an external memory 320, a bus 330, and an external device 340.
  • the image processing apparatus main body 310 has an input I / F 201 and an output.
  • An I / F 202, image processing units 211 to 214A, a DMA controller 311, a bus 220A, and temporary storage units 231 to 234 are provided.
  • the DMA controller 311 writes image data to the external memory 320 and reads image data from the external memory 320 via the bus 330.
  • the external device 340 is, for example, a general purpose processor, an FPGA (Field Programmable Gate Array).
  • the external device 340 reads image data from the external memory 320 and performs predetermined processing, for example, still image object recognition such as face recognition. Then, the external device 340 outputs a control signal indicating the result of the process to the main control unit 250. For example, when the external device 340 performs face recognition, when the face of a person is recognized from image data, a control signal indicating that is output to the main control unit 250.
  • the main control unit 250 controls the type of signal processing of the image processing apparatus 300 at the timing when the control signal is input from the external device 340. For example, when the image processing apparatus 300 is performing “1. normal use”, when the main control unit 250 receives, from the external device 340, a control signal indicating that a human face has been recognized, the image processing apparatus Instructs 300 to increase the resolution for processing.
  • the fixed processing by the image processing units 211 to 214A and the variable processing by the external device 340 can be combined freely in the execution order, and a signal processing platform with high flexibility is constructed. can do.
  • face recognition can be changed to vehicle recognition only by changing the program of the external device 340.
  • the external device 340 which is a general purpose processor, the bus 330, the image processing units 211 to 214A, the DMA controller 311, the bus 220A, and the temporary storage units 231 to 234 can be integrated on one semiconductor device (LSI). According to this configuration, the number of parts and the mounting area of the device can be reduced, and the downsizing and cost reduction of the device can be realized.
  • LSI semiconductor device
  • Embodiment 4 is a configuration example of a temporary storage unit.
  • FIG. 9 is a diagram showing the configuration of a temporary storage unit of an image processing apparatus according to a fourth embodiment of the present invention.
  • the same components as in FIG. 2 are assigned the same reference numerals.
  • FIG. 9 representatively shows the temporary storage unit 231 among the temporary storage units 231 to 233 in FIG.
  • the temporary storage unit 231 includes a FIFO (First-In First-Out) 251 which can store divided image data alone, and a control circuit 252 which controls the FIFO 251.
  • FIFO First-In First-Out
  • the image processing units 211 to 214 need to perform processing by recognizing the positional relationship of the divided images to be processed. For example, in camera signal processing, since the color arrangement of the image sensor is generally different between the even lines and the odd lines, it is necessary to perform processing by recognizing even and odd of the line being processed.
  • even and odd of the line being processed are recognized.
  • even-numbered lines are stored at addresses 0 to 2559
  • odd-numbered lines are stored at addresses 2560 to 5119.
  • the bus 220 needs a signal line for the write address and a signal line for the read address.
  • the control circuit 252 when storing the divided image data in the FIFO 251 of the temporary storage unit 231, the control circuit 252 adds information indicating the positional relationship of the divided image in the image. For example, when the divided image is one line, a line number is added to the stored line.
  • control circuit 252 stores the divided image 1 position information in the FIFO 251 after storing the divided image 1 data, and then stores the divided image 2 position information and then stores the divided image 2 data.
  • the FIFO 251 sequentially outputs the data input first.
  • the image processing unit 212 can recognize the positional relationship of the corresponding divided image only by reading out the divided image data sequentially from the FIFO 251 of the temporary storage unit 231.
  • the image processing units 211 to 214 do not need to generate addresses for the temporary storage units 231 to 233, so a large number of signal lines (address lines) are deleted from the bus 220. can do.
  • each circuit processing engine
  • each circuit such as an image processing unit can be arranged close to each other, thereby shortening the signal transmission time. Operation speed can be increased.
  • the information indicating the positional relationship of the divided images in the present embodiment may be any as long as it indicates the top line of the image (1 bit flag information), and the image processing units 211 to 214 divide the top line of the image. For example, the positional relationship of the divided images to be processed can be calculated.
  • the fifth embodiment is another configuration example of the temporary storage unit.
  • FIG. 10 is a diagram showing the configuration of a temporary storage unit of the image processing apparatus according to the fifth embodiment of the present invention. The same components as in FIG. 2 are assigned the same reference numerals.
  • FIG. 10 is a representative of the temporary storage unit 231 among the temporary storage units 231 to 233 in FIG.
  • the temporary storage unit 231 includes a control circuit 261, two or more SRAMs 263 and 265, and registers 262 and 264 associated with the SRAMs 263 and 265.
  • the control circuit 261 controls the division image data input from the bus 220 to be alternately stored in the SRAM 263 or the SRAM 265.
  • the control circuit 261 rewrites the value of the register 262 when starting storage of the divided image data in the SRAM 263, and rewrites the value of the register 264 when starting storage of the divided image data in the SRAM 265. Further, when outputting the divided image data from the SRAM 263 or the SRAM 265 of the temporary storage unit 231, the control circuit 261 adds the values of the registers 262 and 264 (information indicating the positional relationship of the corresponding divided image in the image).
  • the registers 262 and 264 hold information indicating the positional relationship of the corresponding divided image.
  • the SRAMs 263 and 265 are memories capable of storing divided image data alone.
  • the divided image data is alternately written in the two SRAMs 263 and 265, and has a 2-bank configuration in which one is read from the other during writing.
  • the image processing units 211 to 214 need to perform processing by recognizing the positional relationship between the divided images to be processed. Usually, this positional relationship is recognized by storing predetermined data at a predetermined address of the temporary storage unit.
  • the control circuit 261 when the control circuit 261 stores the divided image data in the SRAM 263 or the SRAM 265 of the temporary storage unit 231, the register (262 or 264) associated with the divided image data Stores information indicating the positional relationship. For example, when the divided image is one line, the line number for the stored line is stored.
  • control circuit 261 when outputting the divided image data from the SRAM 263 or the SRAM 265 of the temporary storage unit 231, the control circuit 261 adds the value of the register 262 or 264 as information indicating the positional relationship of the corresponding divided image in the image.
  • the image processing unit 212 can recognize the positional relationship of the corresponding divided image with respect to the input divided image data.
  • the image processing units 211 to 214 do not need to generate addresses for the temporary storage units 231 to 233, so a large number of signal lines (address lines) are deleted from the bus 220. can do.
  • the circuits processing engines
  • the circuits can be arranged close to each other by deleting a large number of signal lines, thereby shortening the signal transmission time and operating speed. You can raise it.
  • the values of the registers 262 and 264 may be those indicating the top line of the image (1 bit flag information), and the image processing units 211 to 214 If the top line of the image is known, the positional relationship of the divided image to be processed can be calculated.
  • the configuration of the temporary storage unit it is optional to adopt either the configuration of the fourth embodiment or the configuration of the fifth embodiment.
  • the temporary storage unit can be configured by two 1-port SRAMs, the circuit scale is reduced compared to the configuration of the fourth embodiment using FIFO (usually configured using 2-port SRAM). can do.
  • FIG. 11 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 6 of the present invention.
  • the same components as in FIG. 2 will be assigned the same reference numerals.
  • the image processing apparatus 400 includes an input I / F 201, an output I / F 202, image processing units 211 to 214B, a bus 220B, and temporary storage units 231B to 233B.
  • the image processing units 212B to 214B receive, from the temporary storage units 231B to 233B, a control signal indicating that storage of divided image data is completed. The operation is started at the timing when the signal is input.
  • the bus 220B selectively couples the image processing units 211 to 214B and the temporary storage units 231B to 233B.
  • the bus 220B has a configuration in which a signal line for supplying the control signals from the temporary storage units 231B to 233B to the image processing units 212B to 214B is added to the bus 220 in FIG. Is an expanded configuration of the bus 220 of FIG.
  • FIG. 12 is a diagram showing the configuration of a temporary storage unit of the image processing apparatus 400. As shown in FIG. In FIG. 12, the same components as in FIG. 10 and FIG. Among the temporary storage units 231B to 233B in FIG. 11, the temporary storage unit 231B is representatively shown.
  • the temporary storage unit 231B includes a control circuit 261B, two or more SRAMs 263 and 265, and registers 262 and 264 associated with the SRAMs 263 and 265.
  • control circuit 261B when storing the divided image data in the SRAMs 263 and 265, the control circuit 261B writes, in the registers 262 and 264, information indicating the positional relationship of the corresponding divided image in the image. Furthermore, the control circuit 261B generates a control signal indicating that storage of the divided image data in the SRAMs 263 and 265 is completed, and outputs the control signal to the image processing units 212B to 214B. Further, the control circuit 261 B performs bank switching simultaneously with the output of the control signal, and inputs new divided image data.
  • the present embodiment is characterized in the bank switching timing of the temporary storage units 231B to 233B and the operation start timing of each of the image processing units 212B to 214B.
  • FIG. 13 is a diagram showing processing timings of the image processing apparatus according to the first embodiment for comparison with the present embodiment.
  • FIG. 14 is a diagram showing processing timings of the image processing apparatus 400 according to the present embodiment.
  • FIGS. 13 and 14 each show the processing timing in “1. normal use” shown in FIG.
  • the data of the image processing unit 211 (RAW data processing) ⁇ temporary storage unit 231B ⁇ image processing unit 213B (YC separation) ⁇ temporary storage unit 233B ⁇ image processing unit 214B (graphics overlap)
  • the connection of the bus 220 is made to flow in order.
  • Raw data from the input I / F 201 is input in a 16 ⁇ s period, and the data rate is 160 MHz / pixel.
  • the image processing unit 211 (RAW data processing), the image processing unit 213B (YC separation), and the image processing unit 214B (graphics superposition) have processing latencies of 10 ⁇ s, 24 ⁇ s, and 6 ⁇ s, respectively.
  • the image processing apparatus when bank switching is simultaneously performed, the switching timing is limited by the processing time of the image processing unit 213B that requires the longest processing time. Therefore, the image processing apparatus according to the first embodiment can not improve the processing performance.
  • the image processing apparatus of the first embodiment requires a 40 ⁇ s period to output one line of image data, and Only 40% of the performance is out.
  • the image processing units 212B to 214B operate at timings at which control signals indicating that storage of divided image data is completed are input from the temporary storage units 231B to 233B. Therefore, Bank switching can be performed individually for each of the temporary storage units 231B to 233B, and the original performance can be obtained.
  • the image processing unit 211 processes the RAW data with a processing latency of 10 ⁇ s on the input divided image data of 16 ⁇ s per line, and sequentially writes the processed data in the temporary storage unit 231B. Further, the image processing unit 211 processes the segmented image data of the first line, and subsequently processes the segmented image data of the second line.
  • the image processing unit 213B sequentially reads data from the temporary storage unit 231B, performs YC separation with a processing latency of 24 ⁇ s, and sequentially stores processed data in the temporary storage unit 233B. Write. Further, the image processing unit 213B processes the divided image data of the first line, and subsequently processes the divided image data of the second line.
  • the image processing unit 214B sequentially reads data from the temporary storage unit 233B, performs graphic superposition with a processing latency of 6 ⁇ s, and sequentially outputs processed data.
  • the image processing unit 214B processes the divided image data of the first line, and subsequently processes the divided image data of the second line.
  • the image processing apparatus 400 can output image data of one line in a period of 16 ⁇ s with respect to RAW data input in a period of 16 ⁇ s per line.
  • the processing performance of the entire apparatus is improved as compared with the case where bank switching is performed simultaneously even if the processing time of each of the image processing units 211 to 214 B is uneven. Can.
  • FIG. 15 is a diagram showing a configuration of a temporary storage unit of an image processing apparatus according to Embodiment 7 of the present invention.
  • the same components as in FIG. 12 will be assigned the same reference numerals.
  • the temporary storage unit 231C having the same configuration as the temporary storage unit 231B is connected to the bus.
  • Temporary storage units 231B and 231C are configured of, for example, an SRAM.
  • one temporary storage unit 231D having a large capacity can be configured using a plurality of temporary storage units 231B and 231C. This can be realized by using an SRAM for the temporary storage unit 231D.
  • the temporary storage unit 231D for horizontal 2560 pixel 1 line can be configured by using two temporary storage units 231B and 231C that store a divided image of horizontal 1280 pixel 1 line. In this case, the register indicated by the broken line in FIG. 15B can be reduced.
  • FIGS. 16 and 17 are block diagrams showing the configuration of an image processing apparatus according to a seventh embodiment of the present invention.
  • the same components as in FIG. 2 will be assigned the same reference numerals and overlapping explanations will be omitted.
  • FIGS. 16 and 17 are block diagrams showing the configuration of an image processing apparatus according to a seventh embodiment of the present invention that performs stereo camera input processing.
  • the same components as in FIG. 2 will be assigned the same reference numerals and overlapping explanations will be omitted.
  • the image processing apparatus 500 includes input I / Fs 201A and 201B, output I / F 202, image processing units 511A, 511B, 512A, 512B, 213 and 214, a bus 220C, and a temporary storage unit. 531A, 531B, 532A, 532B, and 234 are comprised.
  • the image processing units 511A, 511B, 512A, and 512B are provided with two image processing units 211 and 212 of FIG. 2 for 2ch correspondence.
  • two temporary storage units 231 and 232 shown in FIG. 2 are provided for the temporary storage units 531A, 531B, 532A, and 532B.
  • the bus 220C corresponds to the above, and has the same configuration as the bus 220 of FIG.
  • the temporary storage units 531A and 532A in FIG. 16 correspond to the temporary storage unit 231B in FIG. 15A
  • the temporary storage units 531B and 532B correspond to the temporary storage unit 231C in FIG.
  • the image processing units 511A and 511B perform RAW data processing, and the image processing units 512A and 512B2 perform YC separation. Further, the image processing unit 213 performs distance measurement, and the image processing unit 214 performs output screen generation processing.
  • the image processing apparatus 500 can process a 2-channel stereo camera input (horizontal resolution: 1280 pixels), add distance information of an object to a screen, and output it.
  • FIG. 17 shows an example of the same image processing apparatus 500 as that of FIG. 16 and processing a 1-ch camera input whose horizontal resolution is doubled (2560 pixels).
  • a plurality of temporary storage units 531A and 531B are used to configure one temporary storage unit 531C having a large capacity.
  • one temporary storage unit 532C having a large capacity is configured by using a plurality of temporary storage units 532A and 532B.
  • Temporary storage units 531C and 532C in FIG. 17 correspond to the temporary storage unit 231D in FIG.
  • a high resolution processing system of one channel and a low resolution processing system of a plurality of channels can be constructed on the same platform. This makes it possible to commonize platforms for various camera products (eg, multiview camera and high resolution camera).
  • Eighth Embodiment 18 and 19 are block diagrams showing the configuration of an image processing apparatus according to an eighth embodiment of the present invention.
  • 18 and FIG. 19 the same components as in FIG. 2 and FIG. 16 will be assigned the same reference numerals and overlapping explanations will be omitted.
  • the image processing apparatus 600 includes an input I / F 201, an output I / F 202, an image processing unit 211 to 214, a bus 220D, and temporary storage units 631A, 631B, 632A, 632B, 633A, 633B. , And a power control unit 610.
  • the image processing apparatus 600 has, for example, two or more operation modes such as a low power mode and a normal mode.
  • the image processing apparatus 600 reduces power consumption by reducing the processing resolution of the image or partially using the image processing units 211 to 214.
  • the image processing apparatus 600 adopts an aspect of increasing the processing resolution and the image processing units 211 to 214 to be used.
  • the specific factors are related to intelligent monitoring, such as detection of registrations and detection of suspicious objects.
  • FIG. 18 is an example of the low power mode.
  • the image processing apparatus 600 reduces the horizontal resolution to 1280 pixels and does not perform color processing in YC separation.
  • the image processing unit 211 performs RAW data processing
  • the image processing unit 212 performs only luminance processing.
  • the image processing unit 213 performs color processing (skip), and the image processing unit 214 performs output screen generation and abnormality detection.
  • the power control unit 610 turns off the image processing unit 213, the temporary storage unit 631B, the temporary storage unit 632B, and the temporary storage unit 633B.
  • FIG. 19 shows an example of the normal mode in the same image processing apparatus 600 as FIG.
  • the detection signal is sent to the power control unit 610, and the image processing apparatus 600 shifts to the normal mode.
  • the image processing apparatus 600 performs color processing in YC separation while increasing the horizontal resolution to 2560 pixels.
  • the power control unit 610 turns on the image processing unit 213, the temporary storage unit 631B, the temporary storage unit 632B, and the temporary storage unit 633B.
  • the present embodiment it is possible to suppress the power consumption at the normal time while processing the necessary image with high performance. It is very effective as a power control for regular (24 hours) operation devices such as surveillance cameras.
  • the image processing apparatus is not limited to camera signal processing.
  • the processing unit of the image (the size of the divided image) is not limited to “one line”, and may be a plurality of lines, or may be one tile in a tile configuration in which the image is divided in a grid shape.
  • the configuration of the bus is not limited to the configuration with four selectors.
  • the execution order of the image processing units 211 to 214 is 1-2-3-4, 1-3-2-4, and 1-3-4. There is no limitation to such a configuration.
  • circuits processing engines
  • shared memory that constitute the image processing apparatus, the number thereof, the connection method, and the like may be used.
  • the image processing apparatus is suitable for use in the camera field and in programmable image processing systems such as intelligent surveillance.
  • Image processing apparatus 200, 300, 400, 500, 600 Image processing apparatus 201 Input I / F 202 Output I / F 211 to 214 image processing unit 220 bus 221 to 224 selector 225 register 231 to 233 temporary storage unit 311 DMA controller 320 external memory 330 bus 340 external device

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Storing Facsimile Image Data (AREA)
  • Studio Devices (AREA)

Abstract

Provided is a programmable image processing device which has a high performance and can solve the problem of the memory IO bottleneck.  The image processing device (200) includes: image processing units (211 to 214) which execute a particular process in a unit of a partition image (such as an image of one line in an image); temporary storage units (231 to 233) which temporarily store data in the unit of the partition image; selectors (221 to 224) which selectively connect the image processing units (211 to 214) to the temporary storage units (231 to 233); and a register (225) which controls switching between the selectors (221 to 224).  The image processing device (200) defines the execution order of the image processing units (211 to 214) by the register (225) which controls switching between the selectors (221 to 224) according to an external control so as to modify the connection state between the image processing units (211 to 214) and the temporary storage units (231 to 233).

Description

画像処理装置Image processing device
　本発明は、画像データを処理する画像処理装置に関する。 The present invention relates to an image processing apparatus that processes image data.
　画像処理装置は、カメラモジュールから取り込まれた映像信号の調整・品質向上や、動画像の圧縮・伸張等、処理負荷の大きい処理を、画像処理システム全体を制御するホストの処理負荷を軽減するために代行して行うもので、一般的には専用の処理エンジンを用いて構成される。 The image processing apparatus reduces the processing load of a host that controls the entire image processing system, such as processing with large processing load such as adjustment and quality improvement of video signals fetched from a camera module, compression and decompression of moving images, etc. And is generally configured using a dedicated processing engine.
　図１は、従来の画像処理装置の構成を示すブロック図である。図１に示すように、画像処理装置１０は、固定処理Ａを行う画像処理部１１、固定処理Ｂを行う画像処理部１２、固定処理Ｃを行う画像処理部１３、ＳＤＲＡＭ（Synchronous Dynamic Random Access Memory）などからなる共有メモリ１４、及びこれら各部を相互に接続するバス１５を備える。 FIG. 1 is a block diagram showing the configuration of a conventional image processing apparatus. As shown in FIG. 1, the image processing apparatus 10 includes an image processing unit 11 that performs fixed processing A, an image processing unit 12 that performs fixed processing B, an image processing unit 13 that performs fixed processing C, and SDRAM (Synchronous Dynamic Random Access Memory). And the like, and a bus 15 interconnecting these parts.
　画像処理部１１には、ＣＣＤ（Charge Coupled Device）カメラ又はＣＭＯＳ（Complementary Metal Oxide Semiconductor）カメラを内蔵するカメラモジュール１６が接続され、撮像された画像データが映像信号として入力される。画像処理部１１は、固定処理Ａ（例えばＲＡＷデータ処理（黒レベル補正、ゲイン調整、ホワイトバランス補正、ガンマ補正））を行う処理エンジンである。画像処理部１２は、固定処理Ｂ（例えばＹＣ（輝度・色差）分離）を行う処理エンジンである。画像処理部１３は、固定処理Ｃ（例えば出力処理）を行う処理エンジンである。 A camera module 16 incorporating a charge coupled device (CCD) camera or a complementary metal oxide semiconductor (CMOS) camera is connected to the image processing unit 11, and captured image data is input as a video signal. The image processing unit 11 is a processing engine that performs fixed processing A (for example, RAW data processing (black level correction, gain adjustment, white balance correction, gamma correction)). The image processing unit 12 is a processing engine that performs fixed processing B (for example, YC (brightness and color difference) separation). The image processing unit 13 is a processing engine that performs fixed processing C (for example, output processing).
　共有メモリ１４は、バス１５と接続して、複数の処理エンジン（画像処理部１１～１３）間を転送するデータを一時的に格納する。また、共有メモリ１４は、図示しないＳＤＲＡＭコントローラにより制御される。ＳＤＲＡＭコントローラは、各処理エンジンとの間でアドレス、データ、制御信号をやり取りすることで、ＳＤＲＡＭへのアクセス制御を行う。 The shared memory 14 is connected to the bus 15 and temporarily stores data to be transferred between a plurality of processing engines (image processing units 11 to 13). The shared memory 14 is controlled by an SDRAM controller (not shown). The SDRAM controller performs access control to the SDRAM by exchanging addresses, data, and control signals with each processing engine.
　特許文献１にも、その図２に、ＤＭＡ制御部により制御され、バスと接続して、複数の処理エンジン間を転送するデータを一時的に格納するバッファメモリが開示されている。 Patent Document 1 also discloses, in FIG. 2 thereof, a buffer memory which is controlled by a DMA control unit, is connected to a bus, and temporarily stores data transferred between a plurality of processing engines.
特開２００２－３４４７１０号公報Japanese Patent Application Publication No. 2002-344710
　しかしながら、上記従来の画像処理装置は、複数の処理エンジン間のデータ転送に共有メモリを用いているため、メモリＩＯ（In-Out）がボトルネックとなり、処理エンジンの性能が十分であっても、装置全体としては処理が破綻するおそれがある。図１の場合、共有メモリ１４とバス１５間がボトルネックとなる。 However, since the above-mentioned conventional image processing apparatus uses shared memory for data transfer between a plurality of processing engines, even if the memory IO (In-Out) becomes a bottleneck and the performance of the processing engine is sufficient, There is a possibility that the processing may be broken as the whole apparatus. In the case of FIG. 1, a bottleneck occurs between the shared memory 14 and the bus 15.
　例えば、カメラ分野においては、近年の半導体技術の進歩によってＣＣＤやＣＭＯＳカメラの画素数および動作速度が著しく向上しており、カメラモジュールの出力画像を処理する装置には、高い解像度・フレームレートを取り扱える処理性能が求められている。今後、高解像度・高フレームレート化等の高性能化がさらに進展すると、処理エンジンが取り扱うデータのスループットが増大し、メモリＩＯがボトルネックとなって処理が破綻するおそれがある。 For example, in the camera field, recent advances in semiconductor technology have significantly improved the pixel count and operating speed of CCDs and CMOS cameras, and devices that process the output image of the camera module can handle high resolutions and frame rates. Processing performance is required. In the future, as performance advances such as high resolution and high frame rate progress further, the throughput of data handled by the processing engine may increase, and the memory IO may become a bottleneck and processing may be broken.
　本発明の目的は、上記に鑑みてなされたものであり、メモリＩＯのボトルネックを解消し、高性能かつプログラマブルな画像処理装置を提供することである。 The object of the present invention is made in view of the above, and is to solve the bottleneck of the memory IO and to provide a high-performance and programmable image processing apparatus.
　本発明の画像処理装置は、複数の区分画像データからなる画像データを処理する画像処理装置であって、前記区分画像データに対して特定の固定処理を実行する複数の画像処理手段と、前記区分画像データを一時的に記憶する複数の一時記憶手段と、前記画像処理手段と前記一時記憶手段とを選択的に結合するバスと、前記バスの結合の形態を選択制御する切り替え制御手段と、を備える構成を採る。 An image processing apparatus according to the present invention is an image processing apparatus for processing image data composed of a plurality of divided image data, and includes a plurality of image processing means for executing a specific fixing process on the divided image data; A plurality of temporary storage means for temporarily storing image data; a bus for selectively connecting the image processing means and the temporary storage means; and a switching control means for selectively controlling the connection of the buses. Take the configuration provided.
　本発明によれば、画像処理手段が区分画像データ（例えば１ライン分）を単位として入出力を行う性質と、選択的に結合可能なバスとを組み合わせることにより、不要な処理のスキップや処理順序の入れ替えが可能となるため、用途の異なる複数種のカメラ製品に適用可能な共通プラットフォームとして利用することができる。また、高解像度の画像処理を複数の外付けメモリを用いることなく実行できるため、高性能と低価格を両立させることができる。 According to the present invention, by combining the property that the image processing means performs input / output in units of divided image data (for example, one line) with a bus that can be selectively combined, unnecessary processing skips and processing order Can be used as a common platform applicable to a plurality of different types of camera products with different applications. In addition, high-resolution image processing can be performed without using a plurality of external memories, so both high performance and low price can be achieved.
従来の画像処理装置の構成を示すブロック図Block diagram showing the configuration of a conventional image processing apparatus 本発明の実施の形態１に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention 本発明の実施の形態１に係る画像処理装置のバスの切り替え制御シーケンスを示す図FIG. 6 is a diagram showing a bus switching control sequence of the image processing apparatus according to the first embodiment of the present invention. 本発明の実施の形態１に係る画像処理装置の処理タイミングを示す図FIG. 6 is a diagram showing processing timing of the image processing apparatus according to the first embodiment of the present invention. 本発明の実施の形態１に係る画像処理装置のバスのデータフローを示す図The figure which shows the data flow of the bus of the image processing apparatus which concerns on Embodiment 1 of this invention. 本発明の実施の形態２に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention 本発明の実施の形態２に係る画像処理装置のバスのデータフローを示す図The figure which shows the data flow of the bus of the image processing apparatus which concerns on Embodiment 2 of this invention. 本発明の実施の形態３に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 3 of the present invention 本発明の実施の形態４に係る画像処理装置の一時記憶部の構成を示す図The figure which shows the structure of the temporary memory part of the image processing apparatus which concerns on Embodiment 4 of this invention. 本発明の実施の形態５に係る画像処理装置の一時記憶部の構成を示す図The figure which shows the structure of the temporary storage part of the image processing apparatus which concerns on Embodiment 5 of this invention. 本発明の実施の形態６に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 6 of the present invention 本発明の実施の形態６に係る画像処理装置の一時記憶部の構成を示す図A diagram showing a configuration of a temporary storage unit of an image processing apparatus according to Embodiment 6 of the present invention 本発明の実施の形態１に係る画像処理装置の処理タイミングを示す図FIG. 6 is a diagram showing processing timing of the image processing apparatus according to the first embodiment of the present invention. 本発明の実施の形態６に係る画像処理装置の処理タイミングを示す図FIG. 16 is a diagram showing processing timing of the image processing apparatus according to the sixth embodiment of the present invention. 本発明の実施の形態７に係る画像処理装置の一時記憶部の構成を示す図The figure which shows the structure of the temporary storage part of the image processing apparatus which concerns on Embodiment 7 of this invention. 本発明の実施の形態７に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 7 of the present invention 本発明の実施の形態７に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 7 of the present invention 本発明の実施の形態８に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 8 of the present invention 本発明の実施の形態８に係る画像処理装置の構成を示すブロック図Block diagram showing a configuration of an image processing apparatus according to Embodiment 8 of the present invention
　以下、本発明の実施の形態について図面を参照して詳細に説明する。 Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
　（実施の形態１）
　図２は、本発明の実施の形態１に係る画像処理装置の構成を示すブロック図である。本実施の形態は、ディジタルカメラの信号処理を行う装置に適用した例である。 Embodiment 1
FIG. 2 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention. The present embodiment is an example applied to an apparatus that performs signal processing of a digital camera.
　図２に示すように、画像処理装置２００は、入力インターフェース（Ｉ／Ｆ）２０１と、出力Ｉ／Ｆ２０２と、画像処理部２１１～２１４と、バス２２０と、一時記憶部２３１～２３３と、から主に構成される。また、画像処理装置２００は、外部の主制御部２５０から入力した制御信号に基づいて所定の信号処理を行う。 As shown in FIG. 2, the image processing apparatus 200 includes an input interface (I / F) 201, an output I / F 202, image processing units 211 to 214, a bus 220, and a temporary storage unit 231 to 233. Mainly composed. The image processing apparatus 200 also performs predetermined signal processing based on a control signal input from the external main control unit 250.
　画像処理装置２００は、ディジタルカメラの信号処理を行う装置であり、イメージセンサ（カメラ）の出力信号（ベイヤー配列ＲＡＷデータ）を入力として、ＹＵＶ信号を生成し、出力する。イメージセンサの解像度は、例えば、水平２５６０ピクセル×垂直１９２０ピクセル（約４．９Ｍピクセル）である。 The image processing apparatus 200 is an apparatus that performs signal processing of a digital camera, and receives an output signal (Bayer array RAW data) of an image sensor (camera) to generate and output a YUV signal. The resolution of the image sensor is, for example, 2560 horizontal pixels × 1920 vertical pixels (about 4.9 M pixels).
　入力Ｉ／Ｆ２０１は、ＲＡＷデータをこれに付随する入力クロック及び水平・垂直同期信号を用いて画像処理装置２００内部に取り込む。 The input I / F 201 captures RAW data into the image processing apparatus 200 using an input clock and horizontal and vertical synchronization signals associated therewith.
　出力Ｉ／Ｆ２０２は、ＹＵＶ信号に出力クロック及び水平・垂直同期信号を付加して画像処理装置２００外部に出力する。 The output I / F 202 adds an output clock and a horizontal / vertical synchronization signal to the YUV signal and outputs the same to the outside of the image processing apparatus 200.
　画像処理部２１１は、ＲＡＷデータ処理（黒レベル補正、ゲイン調整、ホワイトバランス補正、ガンマ補正）を行う。 The image processing unit 211 performs RAW data processing (black level correction, gain adjustment, white balance correction, gamma correction).
　画像処理部２１２は、係数可変の２次元ディジタルフィルタである。画像処理部２１２は、水平４ライン分のデータバッファを内蔵し、最大１００タップ（水平２０×垂直５）のフィルタ演算を実行する。 The image processing unit 212 is a two-dimensional digital filter with variable coefficients. The image processing unit 212 incorporates a data buffer for four horizontal lines, and executes a filter operation of up to 100 taps (horizontal 20 × vertical 5).
　画像処理部２１３は、ＹＣ（輝度・色差）分離を行う。画像処理部２１３は、水平４ライン分のデータバッファを内蔵し、このバッファ内のデータで補間処理を行いながら輝度・色差のデータ（ＹＵＶ信号）を生成する。 The image processing unit 213 performs YC (brightness and color difference) separation. The image processing unit 213 has a built-in data buffer for four horizontal lines, and generates luminance / color difference data (YUV signal) while performing interpolation processing with the data in this buffer.
　画像処理部２１４は、図形重畳（画像に矩形や矩形枠を重ね合わせる処理）を行う。画像に矩形や矩形枠を重畳させることにより、撮影者に対して注意を促すことや、撮影対象のプライバシーを保護することができる。 The image processing unit 214 performs graphic superposition (processing of superposing a rectangle or a rectangular frame on an image). By superimposing a rectangle or a rectangular frame on an image, the photographer can be alerted and privacy of a subject to be photographed can be protected.
　バス２２０は、画像処理部２１１～２１４と一時記憶部２３１～２３３とを選択的に結合する。バス２２０は、２入力１出力セレクタ２２１～２２４（セレクタＳ１～Ｓ４）と、セレクタ２２１～２２４を切り替え制御するレジスタ２２５とから構成される。 The bus 220 selectively couples the image processing units 211 to 214 with the temporary storage units 231 to 233. The bus 220 is composed of 2-input 1-output selectors 221 to 224 (selectors S1 to S4) and a register 225 for switching and controlling the selectors 221 to 224.
　セレクタ２２１～２２４は、２入力接続バスをいずれか１つに切り替える。レジスタ２２５は、画像処理装置２００の外部の主制御部２５０からの指示に従って、セレクタ２２１～２２４を用途毎に切り替えるための制御信号をセレクタ２２１～２２４に出力する。 The selectors 221 to 224 switch the 2-input connection bus to any one. The register 225 outputs control signals for switching the selectors 221 to 224 for each application to the selectors 221 to 224 according to an instruction from the main control unit 250 outside the image processing apparatus 200.
　一時記憶部２３１～２３３は、各々水平２ライン分のデータ（５１２０ピクセル相当）を格納することができる２ポートＳＲＡＭである。この２ポートＳＲＡＭは、例えば１リード・１ライトで、１６ビット×５１２０ワードを有する。 The temporary storage units 231 to 233 are 2-port SRAMs capable of storing data (equivalent to 5120 pixels) for two horizontal lines. The 2-port SRAM has, for example, 16 bits x 5120 words in 1 read 1 write.
　このように、バス２２０を介して画像処理部２１１～２１４と、リードアドレス（図示せず）、ライトアドレス（図示せず）、リードデータ、ライトデータの各信号線が結合される。 As described above, the image processing units 211 to 214 are connected to the signal lines for the read address (not shown), the write address (not shown), the read data, and the write data through the bus 220.
　図３は、本装置が用いられる３種類の用途毎に、処理内容およびバス２２０のセレクタ制御方法を示す図である。この図の内容をコード化した情報は、画像処理装置２００の外部の主制御部２５０の内部メモリに格納される。図３に示すように、主制御部２５０に格納される情報は、「１．通常用途」、「２．暗環境下での監視」、「３．医療用途」毎に、画像処理部２１２（２次元ディジタルフィルタ）のフィルタ処理（フィルタ係数）およびバス２２０のセレクタ２２１～２２４の選択方向である。 FIG. 3 is a diagram showing processing contents and a selector control method of the bus 220 for each of three types of applications in which the present apparatus is used. Information obtained by coding the content of this figure is stored in the internal memory of the main control unit 250 outside the image processing apparatus 200. As illustrated in FIG. 3, the information stored in the main control unit 250 includes the image processing unit 212 (for each of “1. normal use”, “2. monitoring under dark environment”, and “3. medical use”). 2) filtering (filter coefficients) of the two-dimensional digital filter and selection directions of the selectors 221 to 224 of the bus 220.
　図２のレジスタ２２５は、主制御部２５０から設定された値に従って制御信号（例えば、１ビットの情報）をセレクタ２２１～２２４（セレクタＳ１～Ｓ４）に出力することにより、セレクタ２２１～２２４（セレクタＳ１～Ｓ４）を切り替え制御する。セレクタ２２１～２２４は、２入力「ａ」，「ｂ」（図５参照）のうち、この図の接続バスのいずれか（「ａ」または「ｂ」）を選択する。図２では、セレクタ２２１～２２４の上側入力が「ａ」、下側入力が「ｂ」である。 The register 225 of FIG. 2 outputs a control signal (for example, 1-bit information) to the selectors 221 to 224 (selectors S1 to S4) according to the value set from the main control unit 250, thereby selecting the selectors 221 to 224 (selectors). Switch control of S1 to S4). The selectors 221 to 224 select one of the connection buses in this figure (“a” or “b”) of the two inputs “a” and “b” (see FIG. 5). In FIG. 2, the upper inputs of the selectors 221 to 224 are “a” and the lower inputs are “b”.
　「１．通常用途」では、画像処理部２１１におけるＲＡＷデータ処理→画像処理部２１３におけるＹＣ分離→画像処理部２１４における図形重畳、の順序で処理が行われる。また、画像処理部２１２（２次元ディジタルフィルタ）のフィルタ処理はスキップする。「１．通常用途」の場合、セレクタ２２３（セレクタＳ３）は「ａ」を選択し、セレクタ２２４（セレクタＳ４）は「ｂ」を選択する。なお、画像処理部２１２（２次元ディジタルフィルタ）はスキップするので、セレクタ２２１（セレクタＳ１）及びセレクタ２２２（セレクタＳ２）の選択方向は任意である。 In “1. normal use”, processing is performed in the order of RAW data processing in the image processing unit 211 → YC separation in the image processing unit 213 → graphic superposition in the image processing unit 214. Also, the filter processing of the image processing unit 212 (two-dimensional digital filter) is skipped. In the case of “1. normal use”, the selector 223 (selector S3) selects “a”, and the selector 224 (selector S4) selects “b”. Since the image processing unit 212 (two-dimensional digital filter) is skipped, the selection direction of the selector 221 (selector S1) and the selector 222 (selector S2) is arbitrary.
　「２．暗環境下での監視」では、画像処理部２１１におけるＲＡＷデータ処理→画像処理部２１２におけるフィルタ処理→画像処理部２１３におけるＹＣ分離→画像処理部２１４における図形重畳、の順序で処理が行われる。また、画像処理部２１２（２次元ディジタルフィルタ）のフィルタ処理は、ノイズ除去（高周波数成分除去）である。「２．暗環境下での監視」の場合、セレクタ２２１（セレクタＳ１）は「ａ」を、セレクタ２２２（セレクタＳ２）は「ｂ」を、セレクタ２２３（セレクタＳ３）は「ｂ」を、セレクタ２２４（セレクタＳ４）は「ａ」を、それぞれ選択する。 In “2. Monitoring under dark environment”, processing is performed in the order of RAW data processing in the image processing unit 211 → filter processing in the image processing unit 212 → YC separation in the image processing unit 213 → graphic superposition in the image processing unit 214. To be done. The filter processing of the image processing unit 212 (two-dimensional digital filter) is noise removal (high frequency component removal). In the case of "2. Monitoring under dark environment", the selector 221 (selector S1) selects "a", the selector 222 (selector S2) selects "b", and the selector 223 (selector S3) selects "b" 224 (selector S4) selects “a” respectively.
　「３．医療用途」では、画像処理部２１１におけるＲＡＷデータ処理→画像処理部２１３におけるＹＣ分離→画像処理部２１２におけるフィルタ処理→画像処理部２１４における図形重畳、の順序で処理が行われる。また、画像処理部２１２（２次元ディジタルフィルタ）のフィルタ処理は、毛細血管強調（赤色高周波成分強調）である。「３．医療用途」の場合、セレクタ２２１（セレクタＳ１）は「ｂ」を、セレクタ２２２（セレクタＳ２）は「ａ」を、セレクタ２２３（セレクタＳ３）は「ａ」を、セレクタ２２４（セレクタＳ４）は「ｂ」を、それぞれ選択する。 In “3. Medical use”, processing is performed in the order of RAW data processing in the image processing unit 211 → YC separation in the image processing unit 213 → filter processing in the image processing unit 212 → graphic superposition in the image processing unit 214. The filter processing of the image processing unit 212 (two-dimensional digital filter) is capillary blood vessel enhancement (red high frequency component enhancement). In the case of “3. medical use”, the selector 221 (selector S1) is “b”, the selector 222 (selector S2) is “a”, the selector 223 (selector S3) is “a”, the selector 224 (selector S4 ) Selects “b” respectively.
　以下、上述のように構成された画像処理装置２００の動作を説明する。 Hereinafter, the operation of the image processing apparatus 200 configured as described above will be described.
　〔ライン単位処理〕
　まず、ライン単位処理について説明する。 [Line unit processing]
First, line-by-line processing will be described.
　画像処理部２１１～２１４は、いずれもその処理の性質から、画像中の１ラインを単位としてデータの入力もしくは出力を行う。例えば、画像処理部２１２（２次元ディジタルフィルタ）は、バス２２０から新規の１ライン分の画像データを受けると、内蔵データバッファに保持された４ライン分の画像データと新規１ライン分の画像データとの合計５ライン分の画像データのデータに対し最大１００タップのフィルタ演算を実行して１ライン分の画像データの出力を行うとともに、内蔵データバッファの最古の１ライン分の画像データを新規１ライン分の画像データに置換する。画像処理部２１２は、この１ライン毎の処理を繰り返すことにより、１画面全体を処理する。 The image processing units 211 to 214 all input or output data in units of one line in the image because of the nature of the processing. For example, when the image processing unit 212 (two-dimensional digital filter) receives image data for one new line from the bus 220, the image data for four lines held in the built-in data buffer and the image data for one new line A maximum of 100 taps filter operation is performed on the data of the image data of 5 lines in total to output the image data of 1 line, and the image data of the oldest 1 line of the built-in data buffer is new Replace with one line of image data. The image processing unit 212 processes the entire one screen by repeating the processing for each line.
　〔動作タイミング〕
　次に、画像処理装置２００の動作タイミングについて説明する。図４は、図３に示した「１．通常用途」における、画像処理装置２００の処理タイミングを示す図である。 [Operation timing]
Next, the operation timing of the image processing apparatus 200 will be described. FIG. 4 is a diagram showing processing timings of the image processing apparatus 200 in “1. normal use” shown in FIG.
　詳細には、画像入力のタイミング、画像処理部２１１（ＲＡＷデータ処理）の処理レイテンシ、画像処理部２１１による一時記憶部２３１に対する書込み動作、画像処理部２１３（ＹＣ分離）による一時記憶部２３１からの読出し動作、画像処理部２１３の処理レイテンシ、画像処理部２１３による一時記憶部２３３に対する書込み動作、画像処理部２１４（図形重畳）による一時記憶部２３３からの読出し動作、画像処理部２１４の処理レイテンシ、及び画像出力のタイミングを示す。なお、図４中、一時記憶部２３１，２３３の縦軸はアドレスを示す。 Specifically, the timing of image input, the processing latency of the image processing unit 211 (RAW data processing), the writing operation to the temporary storage unit 231 by the image processing unit 211, and the temporary storage unit 231 by the image processing unit 213 (YC separation) Reading operation, processing latency of the image processing unit 213, writing operation to the temporary storage unit 233 by the image processing unit 213, reading operation from the temporary storage unit 233 by the image processing unit 214 (graphic superposition), processing latency of the image processing unit 214, And timing of image output. In FIG. 4, the vertical axes of the temporary storage units 231 and 233 indicate addresses.
　図４の例では、データが、画像処理部２１１（ＲＡＷデータ処理）→一時記憶部２３１→画像処理部２１３（ＹＣ分離）→一時記憶部２３３→画像処理部２１４（図形重畳）の順序で流れるように、バス２２０の接続が行われる。 In the example of FIG. 4, data flows in the order of image processing unit 211 (RAW data processing) → temporary storage unit 231 → image processing unit 213 (YC separation) → temporary storage unit 233 → image processing unit 214 (graphics overlap) Thus, the connection of the bus 220 is made.
　入力Ｉ／Ｆ２０１からのＲＡＷデータは、１ラインあたり４０μｓ周期中の１６μｓ期間に入力され、データレートは１６０ＭＨｚ／ピクセルである。 Raw data from the input I / F 201 is input in a 16 μs period in a 40 μs cycle per line, and the data rate is 160 MHz / pixel.
　画像処理部２１１（ＲＡＷデータ処理）、画像処理部２１３（ＹＣ分離）及び画像処理部２１４（図形重畳）は、上記水平同期信号により４０μｓで同期して動作し、それぞれ１０μｓ、２４μｓ、６μｓの処理レイテンシを持つ。 The image processing unit 211 (RAW data processing), the image processing unit 213 (YC separation), and the image processing unit 214 (graphics superposition) operate in synchronization in 40 μs by the above horizontal synchronization signal and process 10 μs, 24 μs, and 6 μs, respectively. Has a latency.
　各画像処理部２１１，２１３，２１４は、一時記憶部２３１，２３３に対してそれぞれリード・ライトのアドレスを生成し、４０μｓ周期でライト領域とリード領域が交互に入れ替わる構成となっている。 Each of the image processing units 211, 213, and 214 generates a read / write address for the temporary storage units 231 and 233, and the write area and the read area are alternately switched in a cycle of 40 μs.
　したがって、図４の例では、画像処理部２１１が、入力した画像データに対して１０μｓの処理レイテンシでＲＡＷデータ処理し、処理完了したデータを順次一時記憶部２３１に書込む。そして、画像処理部２１１の処理開始から４０μｓ後に、画像処理部２１３が、一時記憶部２３１からデータを順次読出し、２４μｓの処理レイテンシでＹＣ分離を行い、処理完了したデータを順次一時記憶部２３３に書込む。そして、画像処理部２１３の処理開始から４０μｓ後に、画像処理部２１４が、一時記憶部２３３からデータを順次読出し、６μｓの処理レイテンシで図形重畳を行い、処理完了したデータを順次出力する。 Therefore, in the example of FIG. 4, the image processing unit 211 performs RAW data processing on the input image data with a processing latency of 10 μs, and sequentially writes the processed data in the temporary storage unit 231. Then, 40 μs after the start of processing by the image processing unit 211, the image processing unit 213 sequentially reads data from the temporary storage unit 231, performs YC separation with a processing latency of 24 μs, and sequentially completes processed data in the temporary storage unit 233. Write. Then, 40 μs after the start of processing by the image processing unit 213, the image processing unit 214 sequentially reads data from the temporary storage unit 233, performs graphic superposition with a processing latency of 6 μs, and sequentially outputs processed data.
　このように、画像処理装置２００は、画像処理部２１１～２１４が、ライン単位のパイプライン処理として並列動作する構成となる。 As described above, in the image processing apparatus 200, the image processing units 211 to 214 operate in parallel as pipeline processing in units of lines.
　〔バス切替方式〕
　次に、バス切替方式について説明する。 [Bus switching method]
Next, the bus switching method will be described.
　画像処理装置２００は、図３に示す３種類の用途に用いられ、用途毎に画像処理部２１１～２１４の実行有無や順序を切り替える。 The image processing apparatus 200 is used for the three types of applications shown in FIG. 3, and switches the execution existence and the order of the image processing units 211 to 214 for each application.
　ここで、画像処理部２１２（２次元ディジタルフィルタ）の処理内容は、用途毎に異なる。例えば、画像処理装置２００を「２．暗環境下での監視」に用いる場合、イメージセンサへの入力光量が減少しノイズが増大するため、画像処理部２１２は、ＲＡＷデータ処理後の高周波成分を除去し、監視者が見やすいよう画質改善を行う。 Here, the processing content of the image processing unit 212 (two-dimensional digital filter) differs depending on the application. For example, when the image processing apparatus 200 is used for “2. Monitoring in a dark environment”, the amount of light input to the image sensor decreases and noise increases, so the image processing unit 212 processes high frequency components after RAW data processing. Remove and improve the image quality to make it easy for the observer to see.
　また、画像処理装置２００を「３．医療用途」に用いる場合、医療者が毛細血管を視認しやすいよう赤色高周波成分を強調する必要があるため、画像処理部２１２は、ＹＣ分離後にこの赤色高周波成分強調処理を行う。 When the image processing apparatus 200 is used for “3. medical use”, it is necessary to emphasize the red high-frequency component so that the medical practitioner can easily view the capillary blood vessel. Perform component emphasis processing.
　これらの特殊処理は、主制御部２５０が画像処理部２１２（２次元ディジタルフィルタ）のフィルタ係数を変更することにより実現される。 These special processes are realized by the main control unit 250 changing the filter coefficients of the image processing unit 212 (two-dimensional digital filter).
　図５は、画像処理装置２００のバス２２０のデータフローを示す図である。図５中の太実線は、セレクタ２２１～２２４（セレクタＳ１～Ｓ４）の選択によってデータが流れるバスを示し、同図破線は、セレクタ２２１～２２４の非選択によってデータが流れないバスを示す。 FIG. 5 is a diagram showing a data flow of the bus 220 of the image processing apparatus 200. Thick solid lines in FIG. 5 indicate buses through which data flows by selection of the selectors 221 to 224 (selectors S1 to S4), and broken lines in the same figure indicate buses through which data does not flow due to non-selection of the selectors 221 to 224.
　図５Ａは、「１．通常用途」における処理順序（画像処理部２１１（ＲＡＷデータ処理）→画像処理部２１３（ＹＣ分離）→画像処理部２１４（図形重畳））のデータフローである。 FIG. 5A is a data flow of the processing order (image processing unit 211 (RAW data processing) → image processing unit 213 (YC separation) → image processing unit 214 (graphics superposition)) in “1. normal use”.
　図５Ｂは、「２．暗環境下での監視」における処理順序（画像処理部２１１（ＲＡＷデータ処理）→画像処理部２１２（２次元ディジタルフィルタ）→画像処理部２１３（ＹＣ分離）→画像処理部２１４（図形重畳））のデータフローである。 FIG. 5B shows the processing order in “2. Monitoring under dark environment” (image processing unit 211 (RAW data processing) → image processing unit 212 (two-dimensional digital filter) → image processing unit 213 (YC separation) → image processing It is a data flow of part 214 (figure superposition).
　図５Ｃは、「３．医療用途」における処理順序（画像処理部２１１（ＲＡＷデータ処理）→画像処理部２１３（ＹＣ分離）→画像処理部２１２（２次元ディジタルフィルタ）→画像処理部２１４（図形重畳））のデータフローである。 FIG. 5C shows the processing order in “3. medical use” (image processing unit 211 (RAW data processing) → image processing unit 213 (YC separation) → image processing unit 212 (two-dimensional digital filter) → image processing unit 214 (figure Data flow).
　図５に示すように、画像処理装置２００は、バス２２０に含まれる４つのセレクタ２２１～２２４（セレクタＳ１～Ｓ４）を、図３に示すように設定することにより、画像処理部２１１～２１４と一時記憶部２３１～２３３とを選択的に結合する。これにより、画像処理部２１１～２１４の実行有無や順序が変更される。 As shown in FIG. 5, the image processing apparatus 200 sets the four selectors 221 to 224 (selectors S1 to S4) included in the bus 220 as shown in FIG. The temporary storage units 231 to 233 are selectively coupled. As a result, the execution existence and the order of the image processing units 211 to 214 are changed.
　また、本実施の形態によれば、共有メモリを用いず、ライン単位のメモリを用いているため、メモリＩＯボトルネックが解消されると共に、従来の装置に対してコストを抑えることができる。 Further, according to the present embodiment, since the memory in units of lines is used instead of the shared memory, the memory IO bottleneck can be eliminated and the cost can be suppressed with respect to the conventional device.
　例えば、図３の「１．通常用途」に相当する処理を従来の装置で行う場合、共有メモリに求められる最大スループットは以下の通りである。
　　データレート：Ｒ＝１６０ [ＭＨｚ]
　　入出力系統数：Ｎ＝６（リード・ライト各３系統）
　　１ピクセル当たりデータ量：Ｂ＝２[Ｂｙｔｅ]
　　スループット：Ｓ＝Ｒ×Ｎ×Ｂ＝１９２０[ＭＢｙｔｅ／ｓ] For example, when the processing corresponding to “1. normal use” in FIG. 3 is performed by the conventional device, the maximum throughput required for the shared memory is as follows.
Data rate: R = 160 [MHz]
Number of input / output systems: N = 6 (3 systems for read and write)
Amount of data per pixel: B = 2 [Byte]
Throughput: S = R × N × B = 1920 [MByte / s]
　従来の装置は、ＳＤＲＡＭなどからなる共有メモリを用いている。現時点で入手し得るＤＤＲ ＳＤＲＡＭのスループットは最大１．３[ＧＢｙｔｅ／ｓ]程度であるため、１個のＳＤＲＡＭではメモリＩＯがボトルネックとなり、装置を構成できない。従来の装置を構成するには２個のＳＤＲＡＭが必要となる。このように、従来の装置は、記憶容量としては不必要であるにもかかわらず、スループットを確保するために複数個の外部メモリを用意しなければならず、高価なものとなる。 The conventional device uses a shared memory such as an SDRAM. Since the throughput of currently available DDR SDRAM is about 1.3 [GByte / s] at the maximum, the memory IO becomes a bottleneck in one SDRAM and the device can not be configured. Two SDRAMs are required to configure the conventional device. As described above, although the conventional device is unnecessary as a storage capacity, it is necessary to prepare a plurality of external memories to secure the throughput, which is expensive.
　これに対して、本実施の形態の装置は、６４０[ＭＢｙｔｅ／ｓ]の小容量ＳＲＡＭを３個用意すれば良いので、１つの半導体デバイス（ＬＳＩ）上に集積することが可能となり、安価に装置を構成することができる。 On the other hand, in the device according to the present embodiment, three small-capacity SRAMs of 640 [MByte / s] may be prepared, and therefore, they can be integrated on one semiconductor device (LSI), which is inexpensive The device can be configured.
　以上詳細に説明したように、本実施の形態によれば、外部制御に従って、レジスタ２２５がセレクタ２２１～２２４を切り替え制御することにより、画像処理部２１１～２１４と一時記憶部２３１～２３３との結合形態を変化させて画像処理部２１１～２１４の実行順序を規定する。これにより、不要な処理のスキップや処理順序の入れ替えが可能となり、メモリＩＯボトルネックを解消し、高性能かつプログラマブルな画像処理装置を実現することができる。例えば、用途の異なる種々のカメラ製品に対するプラットフォームを共通化することができる。 As described above in detail, according to the present embodiment, the register 225 switches and controls the selectors 221 to 224 according to the external control, thereby combining the image processing units 211 to 214 with the temporary storage units 231 to 233. The form is changed to specify the execution order of the image processing units 211 to 214. As a result, unnecessary processing can be skipped and the processing order can be changed, and a memory IO bottleneck can be eliminated, and a high performance and programmable image processing apparatus can be realized. For example, platforms for various camera products with different applications can be made common.
　また、本実施の形態によれば、不要処理としてスキップさせた画像処理部２１１～２１４、及び未使用の一時記憶部２３１～２３３に対するクロック供給、もしくは電源供給を停止させることもできる。このようにすれば、消費電力を削減することができ、バッテリ駆動の機器では、連続動作時間の長時間化を図ることができる。 Further, according to the present embodiment, clock supply or power supply to the image processing units 211 to 214 skipped as unnecessary processing and the unused temporary storage units 231 to 233 can be stopped. In this way, power consumption can be reduced, and in battery-powered devices, continuous operation time can be extended.
　（実施の形態２）
　実施の形態２は、実施の形態１で説明した画像処理装置に外部メモリを接続する例である。 Second Embodiment
The second embodiment is an example in which an external memory is connected to the image processing apparatus described in the first embodiment.
　図６は、本発明の実施の形態２に係る画像処理装置の構成を示すブロック図である。本実施の形態の説明に当たり、図２と同一の構成部分には同一符号を付して重複箇所の説明を省略する。 FIG. 6 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention. In the description of the present embodiment, the same components as in FIG. 2 will be assigned the same reference numerals and overlapping description will be omitted.
　図６に示すように、画像処理装置３００は、画像処理装置本体３１０と、外部メモリ３２０とからなり、画像処理装置本体３１０は、入力Ｉ／Ｆ２０１と、出力Ｉ／Ｆ２０２と、画像処理部２１１～２１４Ａと、ＤＭＡ（Direct Memory Access）コントローラ３１１と、バス２２０Ａと、一時記憶部２３１～２３４と、を備える。 As shown in FIG. 6, the image processing apparatus 300 includes an image processing apparatus main body 310 and an external memory 320. The image processing apparatus main body 310 includes an input I / F 201, an output I / F 202, and an image processing unit 211. To 214A, a DMA (Direct Memory Access) controller 311, a bus 220A, and temporary storage units 231 to 234.
　画像処理装置３００は、図２の画像処理装置２００と同様に、ディジタルカメラの信号処理を行う装置であり、イメージセンサ（カメラ）の出力信号（ベイヤー配列ＲＡＷデータ）を入力として、ＹＵＶ信号を生成し、出力する。 The image processing apparatus 300 is an apparatus that performs signal processing of a digital camera as in the image processing apparatus 200 of FIG. 2 and generates an YUV signal using an output signal (Bayer array RAW data) of an image sensor (camera) as an input. And output.
　画像処理部２１４Ａは、図２の画像処理部２１４と同様の機能に加え、一時記憶部２３３及び一時記憶部２３４から、それぞれ同位置の１ライン分の画像データを読み出し、２つの画像データの差分から動体の有無（静止画か否か）を判別する。 The image processing unit 214A reads the image data of one line at the same position from the temporary storage unit 233 and the temporary storage unit 234, in addition to the same function as the image processing unit 214 in FIG. Then, the presence or absence of a moving object (whether it is a still image or not) is determined.
　バス２２０Ａは、画像処理部２１１～２１４Ａ、ＤＭＡコントローラ３１１と一時記憶部２３１～２３４とを選択的に結合する。バス２２０Ａは、図２のバス２２０に、ＤＭＡコントローラ３１１接続用のバス及びセレクタが付加された構成であり、基本的には図２のバス２２０を拡張した構成である。 The bus 220A selectively couples the image processing units 211 to 214A, the DMA controller 311, and the temporary storage units 231 to 234. The bus 220A has a configuration in which a bus for connecting the DMA controller 311 and a selector are added to the bus 220 in FIG. 2 and is basically a configuration in which the bus 220 in FIG. 2 is expanded.
　一時記憶部２３４は、一時記憶部２３１～２３３と同様に、各々水平２ライン分のデータ（５１２０ピクセル相当）が格納可能な２ポートＳＲＡＭである。この２ポートＳＲＡＭは、例えば１リード・１ライトで、１６ビット×５１２０ワードを有する。 The temporary storage unit 234 is a two-port SRAM that can store data (corresponding to 5120 pixels) for two horizontal lines, similarly to the temporary storage units 231 to 233. The 2-port SRAM has, for example, 16 bits x 5120 words in 1 read 1 write.
　ＤＭＡコントローラ３１１は、１ライン単位でバス２２０Ａと外部メモリ３２０との間のデータ転送を行う。ＤＭＡコントローラ３１１は、一時記憶部２３３から１ライン分の画像データを読み出して外部メモリ３２０に書き込む。また、ＤＭＡコントローラ３１１は、外部メモリ３２０から旧フレームの同位置の１ライン分の画像データを読み出して一時記憶部２３４に書き込む。 The DMA controller 311 performs data transfer between the bus 220A and the external memory 320 in units of one line. The DMA controller 311 reads image data of one line from the temporary storage unit 233 and writes the image data to the external memory 320. Further, the DMA controller 311 reads the image data for one line at the same position of the old frame from the external memory 320 and writes the image data in the temporary storage unit 234.
　外部メモリ３２０は、一時記憶部２３１～２３４に比較して大容量のメモリ（例えばＳＤＲＡＭ）である。外部メモリ３２０は、過去の画像データをフレーム単位で保持するために設けられる。ここでは、外部メモリ３２０は、新旧２フレーム分の画像データを蓄積する。 The external memory 320 is a memory (for example, an SDRAM) having a large capacity as compared with the temporary storage units 231 to 234. The external memory 320 is provided to hold past image data in frame units. Here, the external memory 320 stores image data of two old and new frames.
　以下、上述のように構成された画像処理装置３００の動作を、図７を用いて説明する。図７は、画像処理装置３００のバス２２０Ａのデータフローを示す図である。なお、「１．通常用途」、「２．暗環境下での監視」及び「３．医療用途」のいずれにおいても、画像処理部２１１に画像データが入力されてから、一時記憶部２３３に画像データが書き込まれるまでの動作は、上記実施の形態１における画像処理装置２００（図５）と同様であるので、説明を省略する。 Hereinafter, the operation of the image processing apparatus 300 configured as described above will be described with reference to FIG. FIG. 7 is a diagram showing a data flow of the bus 220A of the image processing apparatus 300. In any of “1. normal use”, “2. monitoring in dark environment” and “3. medical use”, the image data is input to the image processing unit 211 and then the image is stored in the temporary storage unit 233. The operation until the data is written is the same as that of the image processing apparatus 200 (FIG. 5) in the first embodiment, so the description will be omitted.
　一時記憶部２３３に画像データが書き込まれると、ＤＭＡコントローラ３１１は、一時記憶部２３３から１ライン分の画像データを読み出して外部メモリ３２０に書き込む。また同時に、この１ライン分の画像データが画像処理部２１４Ａに入力される。 When the image data is written to the temporary storage unit 233, the DMA controller 311 reads the image data of one line from the temporary storage unit 233 and writes the image data to the external memory 320. At the same time, the image data for one line is input to the image processing unit 214A.
　次に、ＤＭＡコントローラ３１１は、外部メモリ３２０から旧フレームの１ライン分の画像データを読み出して一時記憶部２３４に書き込む。 Next, the DMA controller 311 reads the image data of one line of the old frame from the external memory 320 and writes the image data in the temporary storage unit 234.
　次に、画像処理部２１４Ａは、一時記憶部２３４から旧フレームの１ライン分の画像データを読み出す。この旧フレームの１ライン分の画像データ、および上記（新フレームの）１ライン分の画像データは、元のフレーム上で同位置となるようにＤＭＡ制御が行われる。画像処理部２１４Ａは、同位置の１ライン分の新旧２つの画像データの差分から動体の有無を判別する。 Next, the image processing unit 214A reads the image data of one line of the old frame from the temporary storage unit 234. The DMA control is performed such that the image data of one line of the old frame and the image data of one line of the above (new frame) are at the same position on the original frame. The image processing unit 214A determines the presence or absence of a moving object from the difference between new and old two image data of one line at the same position.
　このように、本実施の形態によれば、画像処理装置３００が、過去の画像データをフレーム単位で保持する外部メモリ３２０、及び、バス２２０Ａと外部メモリ３２０との間のデータ転送を行うＤＭＡコントローラ３１１を有することにより、実施の形態１の機能に加えて、動体の有無の判別等、過去のフレームを使用する画像処理を行うことができる。 As described above, according to the present embodiment, the image processing apparatus 300 holds the external image 320 holding the past image data in frame units, and the DMA controller performs data transfer between the bus 220A and the external memory 320. By including 311, in addition to the functions of the first embodiment, it is possible to perform image processing using a past frame, such as determination of the presence or absence of a moving object.
　また、本実施の形態によれば、大容量の外部メモリ３２０を外付け部品としており、過去のフレームを参照する信号処理が不要な場合、外部メモリの実装を省略することができる。これにより、高性能品・廉価品の双方に適応できるプラットフォームを構築することができる。 Further, according to the present embodiment, the external memory 320 having a large capacity is used as an external component, and mounting of the external memory can be omitted when the signal processing for referring to the past frame is unnecessary. This makes it possible to construct a platform that can be adapted to both high performance and low cost products.
　（実施の形態３）
　実施の形態３は、実施の形態２で説明した画像処理装置に外部デバイスを接続する例である。 Third Embodiment
The third embodiment is an example of connecting an external device to the image processing apparatus described in the second embodiment.
　図８は、本発明の実施の形態３に係る画像処理装置の構成を示すブロック図である。本実施の形態の説明に当たり、図６と同一の構成部分には同一符号を付して重複箇所の説明を省略する。 FIG. 8 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 3 of the present invention. In the description of the present embodiment, the same components as in FIG. 6 will be assigned the same reference numerals and overlapping descriptions will be omitted.
　図８に示すように、画像処理装置３００は、画像処理装置本体３１０と、外部メモリ３２０と、バス３３０と、外部デバイス３４０とからなり、画像処理装置本体３１０は、入力Ｉ／Ｆ２０１と、出力Ｉ／Ｆ２０２と、画像処理部２１１～２１４Ａと、ＤＭＡコントローラ３１１と、バス２２０Ａと、一時記憶部２３１～２３４と、を備える。 As shown in FIG. 8, the image processing apparatus 300 comprises an image processing apparatus main body 310, an external memory 320, a bus 330, and an external device 340. The image processing apparatus main body 310 has an input I / F 201 and an output. An I / F 202, image processing units 211 to 214A, a DMA controller 311, a bus 220A, and temporary storage units 231 to 234 are provided.
　ＤＭＡコントローラ３１１は、バス３３０を介して、外部メモリ３２０への画像データの書き込み、外部メモリ３２０からの画像データの読み出し、を行う。 The DMA controller 311 writes image data to the external memory 320 and reads image data from the external memory 320 via the bus 330.
　外部デバイス３４０は、例えば汎用プロセッサ、ＦＰＧＡ（Field Programmable Gate Array）である。外部デバイス３４０は、外部メモリ３２０から画像データを読み出し、所定の処理、例えば顔認識のような静止画オブジェクト認識を行う。そして、外部デバイス３４０は、処理の結果を示す制御信号を主制御部２５０に出力する。例えば、外部デバイス３４０が顔認識を行う場合、画像データから人物の顔を認識したときに、その旨を示す制御信号を主制御部２５０に出力する。 The external device 340 is, for example, a general purpose processor, an FPGA (Field Programmable Gate Array). The external device 340 reads image data from the external memory 320 and performs predetermined processing, for example, still image object recognition such as face recognition. Then, the external device 340 outputs a control signal indicating the result of the process to the main control unit 250. For example, when the external device 340 performs face recognition, when the face of a person is recognized from image data, a control signal indicating that is output to the main control unit 250.
　主制御部２５０は、外部デバイス３４０から制御信号を入力したタイミングで、画像処理装置３００の信号処理の種別を制御する。例えば、画像処理装置３００が「１．通常用途」を行っている場合に、主制御部２５０は、外部デバイス３４０から人物の顔を認識した旨を示す制御信号を入力したときに、画像処理装置３００に対して、解像度を上げて処理を行うように指示する。 The main control unit 250 controls the type of signal processing of the image processing apparatus 300 at the timing when the control signal is input from the external device 340. For example, when the image processing apparatus 300 is performing “1. normal use”, when the main control unit 250 receives, from the external device 340, a control signal indicating that a human face has been recognized, the image processing apparatus Instructs 300 to increase the resolution for processing.
　このように、本実施の形態によれば、画像処理部２１１～２１４Ａによる固定処理と、外部デバイス３４０による可変処理とを実行順序自在に組合せることができ、フレキシビリティの高い信号処理プラットフォームを構築することができる。例えば、外部デバイス３４０のプログラムを変更するだけで、顔認識を車輛認識に変更することができる。 As described above, according to the present embodiment, the fixed processing by the image processing units 211 to 214A and the variable processing by the external device 340 can be combined freely in the execution order, and a signal processing platform with high flexibility is constructed. can do. For example, face recognition can be changed to vehicle recognition only by changing the program of the external device 340.
　また、汎用プロセッサである外部デバイス３４０、バス３３０、画像処理部２１１～２１４Ａ、ＤＭＡコントローラ３１１、バス２２０Ａ、及び一時記憶部２３１～２３４を１つの半導体デバイス（ＬＳＩ）上に集積することもできる。このように構成すれば、機器の部品点数及び実装面積を削減することができ、機器の小型化、及びコストダウンを実現することができる。 The external device 340 which is a general purpose processor, the bus 330, the image processing units 211 to 214A, the DMA controller 311, the bus 220A, and the temporary storage units 231 to 234 can be integrated on one semiconductor device (LSI). According to this configuration, the number of parts and the mounting area of the device can be reduced, and the downsizing and cost reduction of the device can be realized.
　（実施の形態４）
　実施の形態４は、一時記憶部の構成例である。 Embodiment 4
The fourth embodiment is a configuration example of a temporary storage unit.
　図９は、本発明の実施の形態４に係る画像処理装置の一時記憶部の構成を示す図である。図２と同一構成部分には同一符号を付す。図９は、図２の一時記憶部２３１～２３３のうち、一時記憶部２３１を代表して示す。 FIG. 9 is a diagram showing the configuration of a temporary storage unit of an image processing apparatus according to a fourth embodiment of the present invention. The same components as in FIG. 2 are assigned the same reference numerals. FIG. 9 representatively shows the temporary storage unit 231 among the temporary storage units 231 to 233 in FIG.
　図９に示すように、一時記憶部２３１は、単体で区分画像データを格納することができるＦＩＦＯ（First-In First-Out）２５１と、ＦＩＦＯ２５１を制御する制御回路２５２とから構成される。 As shown in FIG. 9, the temporary storage unit 231 includes a FIFO (First-In First-Out) 251 which can store divided image data alone, and a control circuit 252 which controls the FIFO 251.
　ここで、画像処理部２１１～２１４（図２参照）は、処理対象の区分画像の位置関係を認識して処理を行う必要がある。例えば、カメラ信号処理では、一般に偶数ラインと奇数ラインでイメージセンサの色配列が異なるため、処理中のラインの偶奇を認識して処理を行う必要がある。 Here, the image processing units 211 to 214 (see FIG. 2) need to perform processing by recognizing the positional relationship of the divided images to be processed. For example, in camera signal processing, since the color arrangement of the image sensor is generally different between the even lines and the odd lines, it is necessary to perform processing by recognizing even and odd of the line being processed.
　通常は、一時記憶部の所定のアドレスに所定のデータを格納することで、処理中のラインの偶奇を認識する。例えば、実施の形態１の一時記憶部２３１であれば、アドレス０番地～２５５９番地に偶数ライン、アドレス２５６０番地～５１１９番地に奇数ラインを格納する。これを行うためには、バス２２０にはライトアドレスの信号線およびリードアドレスの信号線がそれぞれ必要となる。 Usually, by storing predetermined data at a predetermined address of the temporary storage unit, even and odd of the line being processed are recognized. For example, in the temporary storage unit 231 according to the first embodiment, even-numbered lines are stored at addresses 0 to 2559, and odd-numbered lines are stored at addresses 2560 to 5119. In order to do this, the bus 220 needs a signal line for the write address and a signal line for the read address.
　これに対し、本実施の形態では、制御回路２５２が、一時記憶部２３１のＦＩＦＯ２５１に区分画像データを格納する際に、画像における当該区分画像の位置関係を示す情報を付加する。例えば、区分画像が１ラインの場合、格納されるラインに対するライン番号を付加する。 On the other hand, in the present embodiment, when storing the divided image data in the FIFO 251 of the temporary storage unit 231, the control circuit 252 adds information indicating the positional relationship of the divided image in the image. For example, when the divided image is one line, a line number is added to the stored line.
　図９の例では、制御回路２５２が、ＦＩＦＯ２５１に、区分画像１位置情報を格納した後に区分画像１データを格納し、続いて区分画像２位置情報を格納した後に区分画像２データを格納する。所定のデータが入力されると、ＦＩＦＯ２５１は最初に入力されたデータから順に出力する。 In the example of FIG. 9, the control circuit 252 stores the divided image 1 position information in the FIFO 251 after storing the divided image 1 data, and then stores the divided image 2 position information and then stores the divided image 2 data. When predetermined data is input, the FIFO 251 sequentially outputs the data input first.
　これにより、画像処理部２１２は、一時記憶部２３１のＦＩＦＯ２５１から順次区分画像データを読み出すだけで、該当区分画像の位置関係を認識することができる。 Thus, the image processing unit 212 can recognize the positional relationship of the corresponding divided image only by reading out the divided image data sequentially from the FIFO 251 of the temporary storage unit 231.
　このように、本実施の形態によれば、画像処理部２１１～２１４側で、一時記憶部２３１～２３３に対するアドレスを生成する必要がなくなるので、バス２２０から大量の信号線（アドレス線）を削除することができる。また、ＬＳＩに集積する場合には、大量の信号線を削除することにより、画像処理部等の各回路（処理エンジン）を互いに近くに寄せて配置することができ、信号の伝送時間を短縮して、動作速度を上げることができる。 As described above, according to the present embodiment, the image processing units 211 to 214 do not need to generate addresses for the temporary storage units 231 to 233, so a large number of signal lines (address lines) are deleted from the bus 220. can do. In addition, in the case of integration in an LSI, by deleting a large number of signal lines, each circuit (processing engine) such as an image processing unit can be arranged close to each other, thereby shortening the signal transmission time. Operation speed can be increased.
　なお、本実施の形態における区分画像の位置関係を示す情報は、画像のトップラインを示すもの（１ビットのフラグ情報）であれば良く、画像処理部２１１～２１４は、画像のトップラインが分かれば、処理対象の区分画像の位置関係を計算することができる。 The information indicating the positional relationship of the divided images in the present embodiment may be any as long as it indicates the top line of the image (1 bit flag information), and the image processing units 211 to 214 divide the top line of the image. For example, the positional relationship of the divided images to be processed can be calculated.
　（実施の形態５）
　実施の形態５は、一時記憶部の他の構成例である。 Fifth Embodiment
The fifth embodiment is another configuration example of the temporary storage unit.
　図１０は、本発明の実施の形態５に係る画像処理装置の一時記憶部の構成を示す図である。図２と同一構成部分には同一符号を付す。図１０は、図２の一時記憶部２３１～２３３のうち、一時記憶部２３１を代表して示す。 FIG. 10 is a diagram showing the configuration of a temporary storage unit of the image processing apparatus according to the fifth embodiment of the present invention. The same components as in FIG. 2 are assigned the same reference numerals. FIG. 10 is a representative of the temporary storage unit 231 among the temporary storage units 231 to 233 in FIG.
　図１０に示すように、一時記憶部２３１は、制御回路２６１と、２以上のＳＲＡＭ２６３，２６５と、ＳＲＡＭ２６３，２６５に関連づけられたレジスタ２６２，２６４とから構成される。 As shown in FIG. 10, the temporary storage unit 231 includes a control circuit 261, two or more SRAMs 263 and 265, and registers 262 and 264 associated with the SRAMs 263 and 265.
　制御回路２６１は、バス２２０から入力した区分画像データを、ＳＲＡＭ２６３またはＳＲＡＭ２６５に交互に格納するように制御する。また、制御回路２６１は、ＳＲＡＭ２６３に対して区分画像データの格納を開始する際にレジスタ２６２の値を書き換え、ＳＲＡＭ２６５に対して区分画像データの格納を開始する際にレジスタ２６４の値を書き換える。また、制御回路２６１は、一時記憶部２３１のＳＲＡＭ２６３またはＳＲＡＭ２６５から区分画像データを出力する際に、レジスタ２６２，２６４の値（画像における該当区分画像の位置関係を示す情報）を付加する。 The control circuit 261 controls the division image data input from the bus 220 to be alternately stored in the SRAM 263 or the SRAM 265. The control circuit 261 rewrites the value of the register 262 when starting storage of the divided image data in the SRAM 263, and rewrites the value of the register 264 when starting storage of the divided image data in the SRAM 265. Further, when outputting the divided image data from the SRAM 263 or the SRAM 265 of the temporary storage unit 231, the control circuit 261 adds the values of the registers 262 and 264 (information indicating the positional relationship of the corresponding divided image in the image).
　レジスタ２６２，２６４は、該当区分画像の位置関係を示す情報を保持する。ＳＲＡＭ２６３，２６５は、単体で区分画像データを格納することができるメモリである。 The registers 262 and 264 hold information indicating the positional relationship of the corresponding divided image. The SRAMs 263 and 265 are memories capable of storing divided image data alone.
　以上の構成において、区分画像データは、２つのＳＲＡＭ２６３，２６５に交互に書き込まれ、一方の書き込み中に他方から読み出す２－Ｂａｎｋの構成を採っている。 In the above configuration, the divided image data is alternately written in the two SRAMs 263 and 265, and has a 2-bank configuration in which one is read from the other during writing.
　上記実施の形態４で説明したように、画像処理部２１１～２１４は、処理対象の区分画像の位置関係を認識して処理を行う必要がある。通常は、一時記憶部の所定のアドレスに所定のデータを格納することで、この位置関係を認識する。 As described in the fourth embodiment, the image processing units 211 to 214 need to perform processing by recognizing the positional relationship between the divided images to be processed. Usually, this positional relationship is recognized by storing predetermined data at a predetermined address of the temporary storage unit.
　これに対し、本実施の形態では、制御回路２６１が、一時記憶部２３１のＳＲＡＭ２６３またはＳＲＡＭ２６５に区分画像データを格納する際に、関連づけられたレジスタ（２６２または２６４）に、画像における該当区分画像の位置関係を示す情報を格納する。例えば、区分画像が１ラインの場合、格納されるラインに対するライン番号を格納する。 On the other hand, in the present embodiment, when the control circuit 261 stores the divided image data in the SRAM 263 or the SRAM 265 of the temporary storage unit 231, the register (262 or 264) associated with the divided image data Stores information indicating the positional relationship. For example, when the divided image is one line, the line number for the stored line is stored.
　さらに、制御回路２６１は、一時記憶部２３１のＳＲＡＭ２６３またはＳＲＡＭ２６５から区分画像データを出力する際に、レジスタ２６２または２６４の値を、画像における該当区分画像の位置関係を示す情報として付加する。 Furthermore, when outputting the divided image data from the SRAM 263 or the SRAM 265 of the temporary storage unit 231, the control circuit 261 adds the value of the register 262 or 264 as information indicating the positional relationship of the corresponding divided image in the image.
　これにより、画像処理部２１２は、入力した区分画像データについて、該当区分画像の位置関係を認識することができる。 Thereby, the image processing unit 212 can recognize the positional relationship of the corresponding divided image with respect to the input divided image data.
　このように、本実施の形態によれば、画像処理部２１１～２１４側で、一時記憶部２３１～２３３に対するアドレスを生成する必要がなくなるので、バス２２０から大量の信号線（アドレス線）を削除することができる。また、ＬＳＩに集積する場合には、大量の信号線を削除することにより、各回路（処理エンジン）を互いに近くに寄せて配置することができ、信号の伝送時間を短縮して、動作速度を上げることができる。 As described above, according to the present embodiment, the image processing units 211 to 214 do not need to generate addresses for the temporary storage units 231 to 233, so a large number of signal lines (address lines) are deleted from the bus 220. can do. In addition, in the case of integration in an LSI, the circuits (processing engines) can be arranged close to each other by deleting a large number of signal lines, thereby shortening the signal transmission time and operating speed. You can raise it.
　なお、本実施の形態におけるレジスタ２６２，２６４の値（区分画像の位置関係を示す情報）は、画像のトップラインを示すもの（１ビットのフラグ情報）であれば良く、画像処理部２１１～２１４は、画像のトップラインが分かれば、処理対象の区分画像の位置関係を計算することができる。 Note that the values of the registers 262 and 264 (information indicating the positional relationship of the divided images) in the present embodiment may be those indicating the top line of the image (1 bit flag information), and the image processing units 211 to 214 If the top line of the image is known, the positional relationship of the divided image to be processed can be calculated.
　なお、一時記憶部の構成として、実施の形態４と実施の形態５のどちらの構成を採るかは任意である。実施の形態５では、一時記憶部を１ポートＳＲＡＭ×２個で構成できるため、ＦＩＦＯ（通常２ポートＳＲＡＭを用いて構成される）を用いる実施の形態４の構成に比べて、回路規模を削減することができる。 As the configuration of the temporary storage unit, it is optional to adopt either the configuration of the fourth embodiment or the configuration of the fifth embodiment. In the fifth embodiment, since the temporary storage unit can be configured by two 1-port SRAMs, the circuit scale is reduced compared to the configuration of the fourth embodiment using FIFO (usually configured using 2-port SRAM). can do.
　（実施の形態６）
　図１１は、本発明の実施の形態６に係る画像処理装置の構成を示すブロック図である。なお、図１１において、図２と同一構成部分には同一符号を付す。 Sixth Embodiment
FIG. 11 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 6 of the present invention. In FIG. 11, the same components as in FIG. 2 will be assigned the same reference numerals.
　図１１に示すように、画像処理装置４００は、入力Ｉ／Ｆ２０１と、出力Ｉ／Ｆ２０２と、画像処理部２１１～２１４Ｂと、バス２２０Ｂと、一時記憶部２３１Ｂ～２３３Ｂと、を備える。 As shown in FIG. 11, the image processing apparatus 400 includes an input I / F 201, an output I / F 202, image processing units 211 to 214B, a bus 220B, and temporary storage units 231B to 233B.
　画像処理部２１２Ｂ～２１４Ｂは、図２の画像処理部２１２～２１４と同様の機能に加え、一時記憶部２３１Ｂ～２３３Ｂから区分画像データの格納が終了したことを示す制御信号が入力され、この制御信号が入力されたタイミングで動作を開始する。 In addition to the same functions as the image processing units 212 to 214 in FIG. 2, the image processing units 212B to 214B receive, from the temporary storage units 231B to 233B, a control signal indicating that storage of divided image data is completed. The operation is started at the timing when the signal is input.
　バス２２０Ｂは、画像処理部２１１～２１４Ｂと一時記憶部２３１Ｂ～２３３Ｂとを選択的に結合する。バス２２０Ｂは、図２のバス２２０に、一時記憶部２３１Ｂ～２３３Ｂからの上記制御信号を画像処理部２１２Ｂ～２１４Ｂへ供給するための信号線が付加された構成であり、基本的には図２のバス２２０を拡張した構成である。 The bus 220B selectively couples the image processing units 211 to 214B and the temporary storage units 231B to 233B. The bus 220B has a configuration in which a signal line for supplying the control signals from the temporary storage units 231B to 233B to the image processing units 212B to 214B is added to the bus 220 in FIG. Is an expanded configuration of the bus 220 of FIG.
　図１２は、上記画像処理装置４００の一時記憶部の構成を示す図である。図１２において、図１０及び図１１と同一構成部分には同一符号を付す。図１１の一時記憶部２３１Ｂ～２３３Ｂのうち、一時記憶部２３１Ｂを代表して示す。 FIG. 12 is a diagram showing the configuration of a temporary storage unit of the image processing apparatus 400. As shown in FIG. In FIG. 12, the same components as in FIG. 10 and FIG. Among the temporary storage units 231B to 233B in FIG. 11, the temporary storage unit 231B is representatively shown.
　図１２に示すように、一時記憶部２３１Ｂは、制御回路２６１Ｂと、２以上のＳＲＡＭ２６３，２６５と、ＳＲＡＭ２６３，２６５に関連づけられたレジスタ２６２，２６４とから構成される。 As shown in FIG. 12, the temporary storage unit 231B includes a control circuit 261B, two or more SRAMs 263 and 265, and registers 262 and 264 associated with the SRAMs 263 and 265.
　制御回路２６１Ｂは、図１０の制御回路２６１と同様に、ＳＲＡＭ２６３，２６５に区分画像データを格納する際に、画像における該当区分画像の位置関係を示す情報をレジスタ２６２，２６４に書き込む。さらに、制御回路２６１Ｂは、ＳＲＡＭ２６３，２６５への区分画像データの格納が終了したことを示す制御信号を生成し、画像処理部２１２Ｂ～２１４Ｂに制御信号を出力する。また、制御回路２６１Ｂは、制御信号の出力と同時にＢａｎｋ切り替えを行い、新たな区分画像データを入力する。 Similar to the control circuit 261 of FIG. 10, when storing the divided image data in the SRAMs 263 and 265, the control circuit 261B writes, in the registers 262 and 264, information indicating the positional relationship of the corresponding divided image in the image. Furthermore, the control circuit 261B generates a control signal indicating that storage of the divided image data in the SRAMs 263 and 265 is completed, and outputs the control signal to the image processing units 212B to 214B. Further, the control circuit 261 B performs bank switching simultaneously with the output of the control signal, and inputs new divided image data.
　本実施の形態は、一時記憶部２３１Ｂ～２３３ＢのＢａｎｋ切り替えのタイミング、及び、各画像処理部２１２Ｂ～２１４Ｂの動作開始タイミングに特徴がある。 The present embodiment is characterized in the bank switching timing of the temporary storage units 231B to 233B and the operation start timing of each of the image processing units 212B to 214B.
　図１３は、本実施の形態と比較するため、実施の形態１の画像処理装置の処理タイミングを示す図である。図１４は、本実施の形態の画像処理装置４００の処理タイミングを示す図である。なお、図１３及び図１４は、いずれも、図３に示した「１．通常用途」における処理タイミングを示す。 FIG. 13 is a diagram showing processing timings of the image processing apparatus according to the first embodiment for comparison with the present embodiment. FIG. 14 is a diagram showing processing timings of the image processing apparatus 400 according to the present embodiment. FIGS. 13 and 14 each show the processing timing in “1. normal use” shown in FIG.
　図１３及び図１４の例では、データが、画像処理部２１１（ＲＡＷデータ処理）→一時記憶部２３１Ｂ→画像処理部２１３Ｂ（ＹＣ分離）→一時記憶部２３３Ｂ→画像処理部２１４Ｂ（図形重畳）の順序で流れるように、バス２２０の接続が行われる。 In the example of FIG. 13 and FIG. 14, the data of the image processing unit 211 (RAW data processing) → temporary storage unit 231B → image processing unit 213B (YC separation) → temporary storage unit 233B → image processing unit 214B (graphics overlap) The connection of the bus 220 is made to flow in order.
　入力Ｉ／Ｆ２０１からのＲＡＷデータは、１６μｓ期間に入力され、データレートは１６０ＭＨｚ／ピクセルである。 Raw data from the input I / F 201 is input in a 16 μs period, and the data rate is 160 MHz / pixel.
　画像処理部２１１（ＲＡＷデータ処理）、画像処理部２１３Ｂ（ＹＣ分離）及び画像処理部２１４Ｂ（図形重畳）は、それぞれ１０μｓ、２４μｓ、６μｓの処理レイテンシを持つ。 The image processing unit 211 (RAW data processing), the image processing unit 213B (YC separation), and the image processing unit 214B (graphics superposition) have processing latencies of 10 μs, 24 μs, and 6 μs, respectively.
　実施の形態１の画像処理装置において、Ｂａｎｋ切り替えを一斉に行う場合、切り替えタイミングは、最も処理の時間のかかる画像処理部２１３Ｂの処理時間に律速される。このため、実施の形態１の画像処理装置は、処理性能を上げることができない。 In the image processing apparatus according to the first embodiment, when bank switching is simultaneously performed, the switching timing is limited by the processing time of the image processing unit 213B that requires the longest processing time. Therefore, the image processing apparatus according to the first embodiment can not improve the processing performance.
　図１３の例では、ＲＡＷデータが１ライン当たり１６μｓ期間に入力されるにもかかわらず、実施の形態１の画像処理装置は、１ラインの画像データを出力するために４０μｓ期間を必要とし、本来性能の４０％しか出ていない。 In the example of FIG. 13, although the RAW data is input in a period of 16 μs per line, the image processing apparatus of the first embodiment requires a 40 μs period to output one line of image data, and Only 40% of the performance is out.
　これに対し、本実施の形態の画像処理装置４００では、各画像処理部２１２Ｂ～２１４Ｂが、一時記憶部２３１Ｂ～２３３Ｂから区分画像データの格納が終了したことを示す制御信号を入力したタイミングで動作することができるので、一時記憶部２３１Ｂ～２３３Ｂ毎に個別にＢａｎｋ切り替えを行うことができ、本来の性能を引き出すことができる。 On the other hand, in the image processing apparatus 400 according to the present embodiment, the image processing units 212B to 214B operate at timings at which control signals indicating that storage of divided image data is completed are input from the temporary storage units 231B to 233B. Therefore, Bank switching can be performed individually for each of the temporary storage units 231B to 233B, and the original performance can be obtained.
　図１４の例では、画像処理部２１１は、入力した１ライン当たり１６μｓ期間の区分画像データに対して１０μｓの処理レイテンシでＲＡＷデータ処理し、処理完了したデータを順次一時記憶部２３１Ｂに書込む。また、画像処理部２１１は、１番目のラインの区分画像データを処理した後、続けて２番目のラインの区分画像データを処理する。 In the example of FIG. 14, the image processing unit 211 processes the RAW data with a processing latency of 10 μs on the input divided image data of 16 μs per line, and sequentially writes the processed data in the temporary storage unit 231B. Further, the image processing unit 211 processes the segmented image data of the first line, and subsequently processes the segmented image data of the second line.
　そして、画像処理部２１１の処理開始から２６μｓ後に、画像処理部２１３Ｂは、一時記憶部２３１Ｂからデータを順次読出し、２４μｓの処理レイテンシでＹＣ分離を行い、処理完了したデータを順次一時記憶部２３３Ｂに書込む。また、画像処理部２１３Ｂは、１番目のラインの区分画像データを処理した後、続けて２番目のラインの区分画像データを処理する。 Then, 26 μs after the start of processing by the image processing unit 211, the image processing unit 213B sequentially reads data from the temporary storage unit 231B, performs YC separation with a processing latency of 24 μs, and sequentially stores processed data in the temporary storage unit 233B. Write. Further, the image processing unit 213B processes the divided image data of the first line, and subsequently processes the divided image data of the second line.
　そして、画像処理部２１３Ｂの処理開始から４０μｓ後に、画像処理部２１４Ｂは、一時記憶部２３３Ｂからデータを順次読出し、６μｓの処理レイテンシで図形重畳を行い、処理完了したデータを順次出力する。また、画像処理部２１４Ｂは、１番目のラインの区分画像データを処理した後、続けて２番目のラインの区分画像データを処理する。 Then, 40 μs after the start of processing by the image processing unit 213B, the image processing unit 214B sequentially reads data from the temporary storage unit 233B, performs graphic superposition with a processing latency of 6 μs, and sequentially outputs processed data. In addition, the image processing unit 214B processes the divided image data of the first line, and subsequently processes the divided image data of the second line.
　この結果、図１４の例では、１ライン当たり１６μｓ期間に入力されるＲＡＷデータに対し、本実施の形態の画像処理装置４００は、１ラインの画像データを１６μｓ期間で出力することができる。 As a result, in the example of FIG. 14, the image processing apparatus 400 according to the present embodiment can output image data of one line in a period of 16 μs with respect to RAW data input in a period of 16 μs per line.
　このように、本実施の形態によれば、各画像処理部２１１～２１４Ｂの処理時間が不均一であっても、Ｂａｎｋ切り替えを一斉に行う場合と比較して装置全体の処理性能を向上させることができる。 As described above, according to the present embodiment, the processing performance of the entire apparatus is improved as compared with the case where bank switching is performed simultaneously even if the processing time of each of the image processing units 211 to 214 B is uneven. Can.
　（実施の形態７）
　図１５は、本発明の実施の形態７に係る画像処理装置の一時記憶部の構成を示す図である。図１５において、図１２と同一構成部分には同一符号を付す。 Seventh Embodiment
FIG. 15 is a diagram showing a configuration of a temporary storage unit of an image processing apparatus according to Embodiment 7 of the present invention. In FIG. 15, the same components as in FIG. 12 will be assigned the same reference numerals.
　図１５（ａ）では、一時記憶部２３１Ｂと同様の構成の一時記憶部２３１Ｃを、バスに対して接続している。一時記憶部２３１Ｂ，２３１Ｃは、例えばＳＲＡＭにより構成される。 In FIG. 15A, the temporary storage unit 231C having the same configuration as the temporary storage unit 231B is connected to the bus. Temporary storage units 231B and 231C are configured of, for example, an SRAM.
　図１５（ｂ）に示すように、複数の一時記憶部２３１Ｂ，２３１Ｃを用いて、容量の大きな１つの一時記憶部２３１Ｄを構成できる。一時記憶部２３１ＤにＳＲＡＭを用いることにより実現可能である。 As shown in FIG. 15B, one temporary storage unit 231D having a large capacity can be configured using a plurality of temporary storage units 231B and 231C. This can be realized by using an SRAM for the temporary storage unit 231D.
　例えば、水平１２８０ピクセル１ラインの区分画像を格納する一時記憶部２３１Ｂ，２３１Ｃを２個用いて、水平２５６０ピクセル１ラインに対する一時記憶部２３１Ｄを構成することができる。この場合、図１５（ｂ）において破線で示すレジスタを削減することができる。 For example, the temporary storage unit 231D for horizontal 2560 pixel 1 line can be configured by using two temporary storage units 231B and 231C that store a divided image of horizontal 1280 pixel 1 line. In this case, the register indicated by the broken line in FIG. 15B can be reduced.
　図１６及び図１７は、本発明の実施の形態７に係る画像処理装置の構成を示すブロック図である。なお、図１６及び図１７において、図２と同一構成部分には同一符号を付して重複箇所の説明を省略する。 FIGS. 16 and 17 are block diagrams showing the configuration of an image processing apparatus according to a seventh embodiment of the present invention. In FIGS. 16 and 17, the same components as in FIG. 2 will be assigned the same reference numerals and overlapping explanations will be omitted.
　図１６及び図１７は、ステレオカメラ入力処理を行う、本発明の実施の形態７に係る画像処理装置の構成を示すブロック図である。なお、図１６及び図１７において、図２と同一構成部分には同一符号を付して重複箇所の説明を省略する。 FIGS. 16 and 17 are block diagrams showing the configuration of an image processing apparatus according to a seventh embodiment of the present invention that performs stereo camera input processing. In FIGS. 16 and 17, the same components as in FIG. 2 will be assigned the same reference numerals and overlapping explanations will be omitted.
　図１６及び図１７に示すように、画像処理装置５００は、入力Ｉ／Ｆ２０１Ａ，２０１Ｂ、出力Ｉ／Ｆ２０２、画像処理部５１１Ａ，５１１Ｂ，５１２Ａ，５１２Ｂ，２１３，２１４、バス２２０Ｃ、及び一時記憶部５３１Ａ，５３１Ｂ，５３２Ａ，５３２Ｂ，２３４を備えて構成される。 As illustrated in FIGS. 16 and 17, the image processing apparatus 500 includes input I / Fs 201A and 201B, output I / F 202, image processing units 511A, 511B, 512A, 512B, 213 and 214, a bus 220C, and a temporary storage unit. 531A, 531B, 532A, 532B, and 234 are comprised.
　画像処理部５１１Ａ，５１１Ｂ，５１２Ａ，５１２Ｂは、２ｃｈ対応用に、図２の画像処理部２１１，２１２を２つ設けたものである。また、一時記憶部５３１Ａ，５３１Ｂ，５３２Ａ，５３２Ｂについても同様に、図２の一時記憶部２３１，２３２を２つ設けたものである。バス２２０Ｃは、上記に対応したもので図２のバス２２０と同様の構成を有する。図１６の一時記憶部５３１Ａ，５３２Ａは図１５（ａ）の一時記憶部２３１Ｂに対応し、一時記憶部５３１Ｂ，５３２Ｂは図１５（ａ）の一時記憶部２３１Ｃに対応する。 The image processing units 511A, 511B, 512A, and 512B are provided with two image processing units 211 and 212 of FIG. 2 for 2ch correspondence. Similarly, two temporary storage units 231 and 232 shown in FIG. 2 are provided for the temporary storage units 531A, 531B, 532A, and 532B. The bus 220C corresponds to the above, and has the same configuration as the bus 220 of FIG. The temporary storage units 531A and 532A in FIG. 16 correspond to the temporary storage unit 231B in FIG. 15A, and the temporary storage units 531B and 532B correspond to the temporary storage unit 231C in FIG.
　画像処理部５１１Ａ，５１１ＢはＲＡＷデータ処理を行い、画像処理部５１２Ａ，５１２Ｂ２はＹＣ分離を行う。また、画像処理部２１３は距離計測を行い、画像処理部２１４は出力画面生成処理を行う。 The image processing units 511A and 511B perform RAW data processing, and the image processing units 512A and 512B2 perform YC separation. Further, the image processing unit 213 performs distance measurement, and the image processing unit 214 performs output screen generation processing.
　画像処理装置５００は、２ｃｈのステレオカメラ入力（水平解像度：１２８０ピクセル）を処理し、画面に物体の距離情報を付加して出力することができる。 The image processing apparatus 500 can process a 2-channel stereo camera input (horizontal resolution: 1280 pixels), add distance information of an object to a screen, and output it.
　図１７は、図１６と同一の画像処理装置５００であり、水平解像度が２倍（２５６０ピクセル）の１ｃｈカメラ入力を処理する例を示す。この場合、複数の一時記憶部５３１Ａ，５３１Ｂを用いて、容量の大きな１つの一時記憶部５３１Ｃを構成する。同様に、複数の一時記憶部５３２Ａ，５３２Ｂを用いて、容量の大きな１つの一時記憶部５３２Ｃを構成する。図１７の一時記憶部５３１Ｃ，５３２Ｃは図１５（ｂ）の一時記憶部２３１Ｄに対応する。 FIG. 17 shows an example of the same image processing apparatus 500 as that of FIG. 16 and processing a 1-ch camera input whose horizontal resolution is doubled (2560 pixels). In this case, a plurality of temporary storage units 531A and 531B are used to configure one temporary storage unit 531C having a large capacity. Similarly, one temporary storage unit 532C having a large capacity is configured by using a plurality of temporary storage units 532A and 532B. Temporary storage units 531C and 532C in FIG. 17 correspond to the temporary storage unit 231D in FIG.
　このように、本実施の形態によれば、例えば１チャネルの高解像度処理系と、複数チャネルの低解像度処理系とを、同一のプラットフォームで構築することができる。これにより、種々のカメラ製品（例えばマルチビューカメラと高解像度カメラ）に対するプラットフォームを共通化することができる。 Thus, according to the present embodiment, for example, a high resolution processing system of one channel and a low resolution processing system of a plurality of channels can be constructed on the same platform. This makes it possible to commonize platforms for various camera products (eg, multiview camera and high resolution camera).
　（実施の形態８）
　図１８及び図１９は、本発明の実施の形態８に係る画像処理装置の構成を示すブロック図である。なお、図１８及び図１９において、図２及び図１６と同一構成部分には同一符号を付して重複箇所の説明を省略する。 Eighth Embodiment
18 and 19 are block diagrams showing the configuration of an image processing apparatus according to an eighth embodiment of the present invention. 18 and FIG. 19, the same components as in FIG. 2 and FIG. 16 will be assigned the same reference numerals and overlapping explanations will be omitted.
　図１８及び図１９に示すように、画像処理装置６００は、入力Ｉ／Ｆ２０１、出力Ｉ／Ｆ２０２、画像処理部２１１～２１４、バス２２０Ｄ、一時記憶部６３１Ａ，６３１Ｂ，６３２Ａ，６３２Ｂ，６３３Ａ，６３３Ｂ、及び電力制御部６１０を備えて構成される。 As shown in FIGS. 18 and 19, the image processing apparatus 600 includes an input I / F 201, an output I / F 202, an image processing unit 211 to 214, a bus 220D, and temporary storage units 631A, 631B, 632A, 632B, 633A, 633B. , And a power control unit 610.
　画像処理装置６００は、例えば、ローパワーモードと通常モードなど２以上の動作モードを持つ。 The image processing apparatus 600 has, for example, two or more operation modes such as a low power mode and a normal mode.
　画像処理装置６００は、ローパワーモードでは、画像の処理解像度を落とす、あるいは、画像処理部２１１～２１４を一部使用しない等により、消費電力を抑える。 In the low power mode, the image processing apparatus 600 reduces power consumption by reducing the processing resolution of the image or partially using the image processing units 211 to 214.
　また、特定の要因によりローパワーモードから通常モードに移行した場合、画像処理装置６００は、処理解像度や使用する画像処理部２１１～２１４を増やす態様を採る。特定の要因とは、登録物の検出、不審物の検出など、インテリジェント監視関連がある。 Further, in the case of transition from the low power mode to the normal mode due to a specific factor, the image processing apparatus 600 adopts an aspect of increasing the processing resolution and the image processing units 211 to 214 to be used. The specific factors are related to intelligent monitoring, such as detection of registrations and detection of suspicious objects.
　図１８は、ローパワーモードの例である。ローパワーモードでは、画像処理装置６００は、例えば、水平解像度を１２８０ピクセルに落とすと共に、ＹＣ分離における色処理を行わない。この場合、画像処理部２１１はＲＡＷデータ処理を行い、画像処理部２１２は輝度処理のみを行う。また、画像処理部２１３は色処理（スキップ）を行い、画像処理部２１４は出力画面生成と異常検出を行う。また、電力制御部６１０は、画像処理部２１３、一時記憶部６３１Ｂ、一時記憶部６３２Ｂ、一時記憶部６３３Ｂをオフにする。 FIG. 18 is an example of the low power mode. In the low power mode, the image processing apparatus 600, for example, reduces the horizontal resolution to 1280 pixels and does not perform color processing in YC separation. In this case, the image processing unit 211 performs RAW data processing, and the image processing unit 212 performs only luminance processing. The image processing unit 213 performs color processing (skip), and the image processing unit 214 performs output screen generation and abnormality detection. In addition, the power control unit 610 turns off the image processing unit 213, the temporary storage unit 631B, the temporary storage unit 632B, and the temporary storage unit 633B.
　図１９は、図１８と同一の画像処理装置６００における通常モードの例である。ローパワーモード時に、画像処理部２１４が不審物を検出すると、電力制御部６１０にその検出信号を送り、画像処理装置６００は、通常モードに移行する。通常モードでは、画像処理装置６００は、水平解像度を２５６０ピクセルに増すとともに、ＹＣ分離における色処理を行う。また、電力制御部６１０は、画像処理部２１３、一時記憶部６３１Ｂ、一時記憶部６３２Ｂ、一時記憶部６３３Ｂをオンにする。 FIG. 19 shows an example of the normal mode in the same image processing apparatus 600 as FIG. In the low power mode, when the image processing unit 214 detects a suspicious object, the detection signal is sent to the power control unit 610, and the image processing apparatus 600 shifts to the normal mode. In the normal mode, the image processing apparatus 600 performs color processing in YC separation while increasing the horizontal resolution to 2560 pixels. Also, the power control unit 610 turns on the image processing unit 213, the temporary storage unit 631B, the temporary storage unit 632B, and the temporary storage unit 633B.
　このように、本実施の形態によれば、必要な画像は高性能で処理しつつ、通常時の消費電力は抑制することができる。監視カメラなど常時（２４時間）稼動機器に対する電力抑制として非常に有効である。 As described above, according to the present embodiment, it is possible to suppress the power consumption at the normal time while processing the necessary image with high performance. It is very effective as a power control for regular (24 hours) operation devices such as surveillance cameras.
　以上の説明は本発明の好適な実施の形態の例証であり、本発明の範囲はこれに限定されることはない。 The above description is an illustration of a preferred embodiment of the present invention, and the scope of the present invention is not limited to this.
　例えば、画像処理装置は、カメラ信号処理に限定されない。また、画像の処理単位（区分画像のサイズ）は「１ライン」に限定されず、複数ラインでもよく、画像を格子状に分割したタイル構成における１タイルでもよい。さらに、バスの構成は、セレクタ４個で構成などに限定されない。 For example, the image processing apparatus is not limited to camera signal processing. Further, the processing unit of the image (the size of the divided image) is not limited to “one line”, and may be a plurality of lines, or may be one tile in a tile configuration in which the image is divided in a grid shape. Furthermore, the configuration of the bus is not limited to the configuration with four selectors.
　さらに、例えば図３に示すように、画像処理部２１１～２１４の実行順序として、１－２－３－４、１－３－２－４、１－３－４としているが、これらは一例であり、このような構成に限定されるものではない。 Further, for example, as shown in FIG. 3, the execution order of the image processing units 211 to 214 is 1-2-3-4, 1-3-2-4, and 1-3-4. There is no limitation to such a configuration.
　また、上記各実施の形態では画像処理装置という名称を用いたが、これは説明の便宜上であり、他の名称を用いてもよい。 Further, although the name of the image processing apparatus is used in each of the above embodiments, this is for convenience of description, and other names may be used.
　さらに、上記画像処理装置を構成する各回路（処理エンジン）及び共有メモリの種類、その数及び接続方法などはどのようなものでもよい。 Furthermore, the types of circuits (processing engines) and shared memory that constitute the image processing apparatus, the number thereof, the connection method, and the like may be used.
　２００８年１２月５日出願の特願２００８－３１０７９９の日本出願に含まれる明細書、図面および要約書の開示内容は、すべて本願に援用される。 The disclosures of the specification, drawings, and abstract included in the Japanese application of Japanese Patent Application No. 2008-310799 filed on Dec. 5, 2008 are all incorporated herein by reference.
　本発明に係る画像処理装置は、カメラ分野及び、インテリジェント監視等のプログラマブルな画像処理システムに用いるに好適である。 The image processing apparatus according to the present invention is suitable for use in the camera field and in programmable image processing systems such as intelligent surveillance.
　２００，３００，４００，５００，６００　画像処理装置
　２０１　入力Ｉ／Ｆ
　２０２　出力Ｉ／Ｆ
　２１１～２１４　画像処理部
　２２０　バス
　２２１～２２４　セレクタ
　２２５　レジスタ
　２３１～２３３　一時記憶部
　３１１　ＤＭＡコントローラ
　３２０　外部メモリ
　３３０　バス
　３４０　外部デバイス
                                                                                 200, 300, 400, 500, 600 Image processing apparatus 201 Input I / F
202 Output I / F
211 to 214 image processing unit 220 bus 221 to 224 selector 225 register 231 to 233 temporary storage unit 311 DMA controller 320 external memory 330 bus 340 external device

Claims (10)

  1. 　複数の区分画像データからなる画像データを処理する画像処理装置であって、
    　前記区分画像データに対して互いに異なる処理を実行する複数の画像処理部と、
    　前記区分画像データを一時的に記憶する複数の一時記憶部と、
    　前記画像処理部と前記一時記憶部とを選択的に結合するバスと、
    　前記バスの結合の形態を選択制御する切り替え制御部と、
    　を備える画像処理装置。     An image processing apparatus for processing image data composed of a plurality of divided image data, comprising:
    A plurality of image processing units that execute different processes on the divided image data;
    A plurality of temporary storage units for temporarily storing the divided image data;
    A bus selectively coupling the image processing unit and the temporary storage unit;
    A switching control unit that selects and controls a mode of connection of the buses;
    An image processing apparatus comprising:
  2. 　少なくとも１フレーム以上の画像データを格納可能な外部メモリと、
    　前記バスを介して、前記外部メモリと前記一時記憶部との間で前記区分画像データの転送を行うＤＭＡコントローラと、
    　をさらに備える請求項１記載の画像処理装置。     An external memory capable of storing image data of at least one frame or more;
    A DMA controller for transferring the divided image data between the external memory and the temporary storage unit via the bus;
    The image processing apparatus according to claim 1, further comprising:
  3. 　前記外部メモリに接続し、前記外部メモリに格納された画像データを用いて特定処理を行う外部デバイスをさらに備え、
    　前記切り替え制御部は、前記外部デバイスの処理結果に基づいて前記バスの結合の形態を選択制御する、
    　請求項２記載の画像処理装置。     It further comprises an external device connected to the external memory and performing specific processing using the image data stored in the external memory,
    The switching control unit selects and controls the form of coupling of the bus based on the processing result of the external device.
    The image processing apparatus according to claim 2.
  4. 　前記外部デバイス、前記バス、前記画像処理部、前記ＤＭＡコントローラ及び前記一時記憶部を１つの半導体デバイス上に集積する、請求項３記載の画像処理装置。 The image processing apparatus according to claim 3, wherein the external device, the bus, the image processing unit, the DMA controller, and the temporary storage unit are integrated on one semiconductor device.
  5. 　前記一時記憶部は、単体で区分画像データを格納することができるＦＩＦＯ（First-In First-Out）と、前記ＦＩＦＯに区分画像データを格納する際に、画像における該当区分画像の位置関係を示す情報を付加する制御部と、
    　を備える請求項１記載の画像処理装置。     The temporary storage unit indicates a positional relationship between a FIFO (First-In First-Out) that can store divided image data alone and the divided image in the image when storing the divided image data in the FIFO. A control unit that adds information;
    The image processing apparatus according to claim 1, comprising:
  6. 　前記一時記憶部は、単体で区分画像データを格納することができる複数のＳＲＡＭと、画像における該当区分画像の位置関係を示す情報を保持する複数のレジスタと、前記ＳＲＡＭに区分画像データを格納する際に前記レジスタの値を書き換え、前記ＳＲＡＭから区分画像データを出力する際に前記レジスタの値を付加する制御部と、
    　を備える請求項１記載の画像処理装置。     The temporary storage unit stores divided image data in a plurality of SRAMs that can separately store divided image data, a plurality of registers that hold information indicating the positional relationship of the corresponding divided image in the image, and the SRAM. A control unit that rewrites the value of the register and adds the value of the register when the divided image data is output from the SRAM;
    The image processing apparatus according to claim 1, comprising:
  7. 　前記一時記憶部は、前記区分画像データの格納が終了したことを示す制御信号を生成し、
    　前記画像処理部は、前記制御信号を入力した時点で処理を開始する、
    　請求項６記載の画像処理装置。     The temporary storage unit generates a control signal indicating that storage of the divided image data is completed,
    The image processing unit starts processing when the control signal is input.
    The image processing apparatus according to claim 6.
  8. 　複数の前記一時記憶部を容量の大きな１つの一時記憶部として構成する、請求項５記載の画像処理装置。 The image processing apparatus according to claim 5, wherein the plurality of temporary storage units are configured as one temporary storage unit having a large capacity.
  9. 　前記画像処理部および前記一時記憶部のうち未使用のものに対するクロック供給若しくは電源供給を停止させる、請求項１記載の画像処理装置。 The image processing apparatus according to claim 1, wherein clock supply or power supply to an unused one of the image processing unit and the temporary storage unit is stopped.
  10. 　画像の処理解像度を下げる、あるいは、使用する前記画像処理部を削減することにより消費電力を抑える電力制御部をさらに具備する請求項９記載の画像処理装置。

                                                                                         10. The image processing apparatus according to claim 9, further comprising: a power control unit for reducing power consumption by reducing the processing resolution of the image or reducing the image processing unit to be used.

PCT/JP2009/006285 2008-12-05 2009-11-20 Image processing device WO2010064374A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2008-310799 2008-12-05
JP2008310799A JP2010134743A (en) 2008-12-05 2008-12-05 Image processor

Publications (1)

Publication Number Publication Date
WO2010064374A1 true WO2010064374A1 (en) 2010-06-10

Family

ID=42233032

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2009/006285 WO2010064374A1 (en) 2008-12-05 2009-11-20 Image processing device

Country Status (2)

Country Link
JP (1) JP2010134743A (en)
WO (1) WO2010064374A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012008715A (en) * 2010-06-23 2012-01-12 Fuji Xerox Co Ltd Data processing device
JP2013029962A (en) * 2011-07-28 2013-02-07 Nikon Corp Electronic apparatus and program
JP2015508528A (en) * 2011-12-28 2015-03-19 インテル・コーポレーション Pipelined image processing sequencer

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
MY165813A (en) * 2011-02-25 2018-04-27 Photonis Netherlands B V Acquiring and displaying images in real-time
JP5899918B2 (en) 2011-12-27 2016-04-06 株式会社リコー Image processing apparatus and image processing method
KR102305470B1 (en) 2015-02-13 2021-09-28 삼성전자주식회사 Image signal processing device performing image signal processing in parallel through plurality of image processing channels

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08171536A (en) * 1994-12-16 1996-07-02 Com Syst:Kk Data processor
JP2004021645A (en) * 2002-06-17 2004-01-22 Canon Inc Image processing device and its control method
JP2004240885A (en) * 2003-02-07 2004-08-26 Makoto Ogawa Image processor and image processing method
JP2006157977A (en) * 2006-03-13 2006-06-15 Ricoh Co Ltd Image processing apparatus

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008293267A (en) * 2007-05-24 2008-12-04 Canon Inc Image forming apparatus equipped with original conveyance device and original reading device and method for controlling image processor

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08171536A (en) * 1994-12-16 1996-07-02 Com Syst:Kk Data processor
JP2004021645A (en) * 2002-06-17 2004-01-22 Canon Inc Image processing device and its control method
JP2004240885A (en) * 2003-02-07 2004-08-26 Makoto Ogawa Image processor and image processing method
JP2006157977A (en) * 2006-03-13 2006-06-15 Ricoh Co Ltd Image processing apparatus

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012008715A (en) * 2010-06-23 2012-01-12 Fuji Xerox Co Ltd Data processing device
JP2013029962A (en) * 2011-07-28 2013-02-07 Nikon Corp Electronic apparatus and program
JP2015508528A (en) * 2011-12-28 2015-03-19 インテル・コーポレーション Pipelined image processing sequencer

Also Published As

Publication number Publication date
JP2010134743A (en) 2010-06-17

Similar Documents

Publication Publication Date Title
KR101034493B1 (en) Image transforming apparatus, dma apparatus for image transforming, and camera interface supporting image transforming
WO2010064374A1 (en) Image processing device
US20020159656A1 (en) Image processing apparatus, image processing method and portable imaging apparatus
KR100997619B1 (en) Techniques to facilitate use of small line buffers for processing of small or large images
JP6210743B2 (en) Data processing device and data transfer control device
TWI360347B (en) Image processor, imaging device, and image process
JP4189252B2 (en) Image processing apparatus and camera
JP2004304387A (en) Image processing apparatus
JPH11103407A (en) Ccd data pixel interpolating circuit and digital still camera provided with the same
JP6722278B2 (en) Image processing device
JP2007088806A (en) Image signal processor and image signal processing method
US20120203942A1 (en) Data processing apparatus
US10453166B2 (en) Image processing device and image processing method
JP2008172410A (en) Imaging apparatus, image processing apparatus, image processing method, program for image processing method, and recording medium recorded with program for image processing method
JP4487454B2 (en) Electronic camera and control IC for electronic camera
JP2011030268A (en) Image processing apparatus
US20120144150A1 (en) Data processing apparatus
JP3810685B2 (en) Resolution converter and digital camera
US20230388661A1 (en) Integrated circuit with multi-application image processing
JP5224492B2 (en) Image data transfer control device, image data transfer method, and camera having the image data transfer device
JP2009260788A (en) Imaging unit
US20070247644A1 (en) Image processing system
JP2000059800A (en) Image signal processing circuit
US9277145B2 (en) Imaging device dividing imaging region into first divided image data and second divided image data
JP2005159596A (en) Digital camera

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09830145

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 09830145

Country of ref document: EP

Kind code of ref document: A1