JP6245389B2 - Imaging apparatus and image processing apparatus - Google Patents

Description

  The present invention relates to an imaging apparatus and an image processing apparatus that image a subject using a plurality of imaging elements.

  2. Description of the Related Art Conventionally, there are known omnidirectional imaging apparatuses that shoot an omnidirectional image using a plurality of image sensors (CMOS sensors, etc.), perform synthesis processing on a plurality of image data obtained by the imaging, and generate a panoramic image. ing.

  However, in the conventional omnidirectional imaging device, for a plurality of image data captured using a plurality of imaging elements, black level correction, color interpolation, defective pixel correction, etc., respectively with a plurality of image processing circuits corresponding to the imaging elements, Necessary image processing is performed. For this reason, as the number of image pickup devices increases, a large amount of image processing hardware is required, which increases the cost. Further, since a plurality of image data are handled separately, there is a problem that data handling becomes complicated.

  For example, Patent Document 1 describes a multi-sensor panoramic network camera configured with a plurality of image sensors (imaging devices), a plurality of image processors (image processing circuits), an image post processor, a network interface, and the like. There are as many image processing circuits as the number of image sensors.

  The present invention provides an image pickup apparatus such as an omnidirectional image pickup apparatus that uses a plurality of image pickup devices, which increases costs due to an increase in image processing hardware associated with the number of image pickup devices, and handles data by handling a plurality of image data separately. This is to solve the complication of the problem and to improve the reliability.

  In order to solve the above-described problems and achieve the object, the present invention is an imaging apparatus having a plurality of imaging elements, each storing the hemispherical image data output from the plurality of imaging elements. A plurality of memories corresponding to the imaging elements, a reading means for reading out the hemispherical image data stored in the plurality of memories in a time division manner, and the read hemispherical image data corresponding to the plurality of imaging elements A single image processing unit that performs predetermined image processing, and an image generation unit that generates omnidirectional image data based on the plurality of hemispherical image data subjected to predetermined image processing by the single image processing unit And having.

  According to the imaging apparatus of the present invention, a plurality of image processing means corresponding to a plurality of imaging elements is unnecessary, and an increase in cost can be suppressed. Also, by using a single image processing means, it becomes possible to handle the image data of a plurality of image sensors as image data of a single image element, and the complexity of data handling is eliminated. Furthermore, the use of the write monitoring means can correctly send the image data of the same line of the plurality of image elements to the image processing means, so that the reliability is improved.

1 is an overall configuration diagram of an omnidirectional imaging apparatus according to an embodiment of an imaging apparatus of the present invention. It is a detailed block diagram of one Embodiment of the image processing unit in FIG. It is a figure explaining the mode of transfer of the image data of this embodiment. It is a figure explaining the mode of storage of the image data in the page memory of this embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the imaging apparatus is an omnidirectional imaging apparatus using two imaging elements. However, in general, there may be any number of two or more imaging elements, and it is not always necessary to be an omnidirectional imaging apparatus.

  FIG. 1 is an overall configuration diagram of an omnidirectional imaging apparatus according to the present embodiment. The omnidirectional imaging apparatus includes an imaging unit 10, an image processing unit 20, an imaging control unit 30, a CPU 40, a memory such as a ROM 50, an SRAM 60, and a DRAM 70, an operation unit 80, and an external I / F circuit 90. Contained.

  The imaging unit 10 includes two imaging elements 11 and 12. The image sensors 11 and 12 are each a fisheye lens having an angle of view of 180 ° or more for forming a hemispherical image, an image sensor such as a CMOS sensor that converts an optical image obtained by the fisheye lens into image data of an electrical signal, and outputs it. It has a timing generation circuit for generating a horizontal / vertical synchronization signal and a pixel clock of the image sensor, a register group in which various commands and parameters necessary for the operation of the image sensor are set. Since this type of image sensor is well known, the specific configuration of the image sensors 11 and 12 is omitted.

  The imaging elements 11 and 12 of the imaging unit 10 are each connected to the image processing unit 20 via a parallel I / F bus, while being separately connected to the imaging control unit 30 via a serial I / F bus (I2C bus or the like). It is connected. The image processing unit 20 and the imaging control unit 30 are connected to the CPU 40 via the bus 100. Furthermore, ROM 50, SRAM 60, DRAM 70, operation unit 80, external I / F 90, and the like are also connected to the bus 100.

  The image processing unit 20 takes in the image data output from the image sensors 11 and 12 through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. Omnidirectional image data is generated. The present invention particularly relates to the image processing unit 20. The configuration and operation of the image processing unit 20 will be described in detail later.

  The imaging control unit 30 generally sets commands and the like in the register groups of the imaging elements 11 and 12 using the I2C bus, with the imaging control unit 30 as a master device and the imaging elements 11 and 12 as slave devices. Necessary commands and the like are received from the CPU 40. The imaging control unit 30 also uses the I2C bus to capture the status data of the register groups of the imaging elements 11 and 12 and send them to the CPU 40.

  Further, the imaging control unit 30 instructs the imaging elements 11 and 12 to output image data at the timing when the shutter button of the operation unit 80 is pressed. Depending on the imaging device, there may be a preview display function using a display (not shown in FIG. 1) and a moving image support function. In this case, image data output from the image sensors 11 and 12 is continuously performed at a predetermined frame rate (frame / minute). Since the imaging control unit 30 is basically the same as the conventional one, further explanation is omitted.

  The CPU 40 controls the operation of the entire imaging apparatus and executes necessary processes. The ROM 50 stores various programs for the CPU 40. The SRAM 60 and the DRAM 70 are work memories, and store programs executed by the CPU 40, data being processed, and the like. In particular, the DRAM 70 stores image data being processed by the image processing unit 20 and processed omnidirectional image data.

  The operation unit 80 is a general term for various operation buttons, a power switch, a shutter button, and the like. The user inputs various shooting modes and shooting conditions by operating the operation buttons.

  The external I / F 90 is a general term for an interface circuit (USB I / F, etc.) with an external memory (SD card, flash memory, etc.) and a personal computer. Further, the external I / F 90 may be a network interface regardless of wireless or wired. The omnidirectional image data stored in the DRAM 70 is stored in an external memory via the external I / F 90 or transferred to a personal computer via the external I / F 90 as a network I / F as necessary. The

  Hereinafter, the configuration and operation of the image processing unit 20 will be described in detail.

  FIG. 2 is a detailed configuration diagram of the image processing unit 20 according to the present embodiment. The image processing unit 20 includes a page memory 210 corresponding to the image sensor (1) 11, a page memory 220 corresponding to the image sensor (2) 12, a page memory read circuit 230, a write monitoring circuit 240, and a single image processing circuit. 250, an image composition circuit 260, a bus I / F circuit 270, and an internal bus 280 connecting the image processing circuit 250, the image composition circuit 260, and the bus I / F circuit 270.

  The page memories 210 and 220 sequentially store image data output in units of lines from the image sensors 11 and 12, respectively. Here, the page memories 210 and 220 are configured to have page memories for two pages, corresponding to the image sensors 11 and 12, respectively. That is, the page memories 210 and 220 can store up to two pages of image data output from the image sensors 11 and 12, respectively. The page memory 210 sequentially accumulates the image data output from the image sensor 11 in a toggle format such as page memory 1, page memory 2, page memory 1,. Similarly, the page memory 220 sequentially accumulates the image data output from the image sensor 12 in page units in a toggle format such as page memory 1, page memory 2, page memory 1,.

  Reading of the image data from the page memories 210 and 220 is controlled by the page memory reading circuit 230 independently of the writing operation. The page memory read circuit 230 has a read pointer indicating a page memory to be read next to the page memories 210 and 220. When the page memory read circuit 230 receives the page memory read start instruction signal from the write monitoring circuit 240, the page memory read circuit 230 reads the page memory image data indicated by the read pointer from the page memories 210 and 220 in a time-sharing manner in units of lines. When the reading of the page memory indicated by the read pointer is completed, the read pointer is updated so that the next page memory can be read. That is, the read pointer is updated in a toggle format of 1 → 2 → 1 → 2 → 1 →. In this manner, when the page memory read circuit 230 receives the page memory read start instruction signal from the write monitoring circuit 240, the page memory 1, page memory 2, page memory 1, page memory 2, respectively. The image data is read out in a time-sharing manner in units of lines in order of the page memory 1,. The write monitoring circuit 240 will be described later.

  The image processing circuit 250 inputs the image data read in a time-sharing manner in line units from the page memories of the page memories 210 and 220 by the page memory reading circuit 230, and sequentially performs predetermined image processing in real time. Go. The image processing circuit 250 is also supplied with a synchronization signal or the like from the page memory reading circuit 230. Here, the image processing in the image processing circuit 250 includes black level correction, color correction, defective pixel correction, white balance correction, and the like. The image data of the image sensors 11 and 12 subjected to image processing by the image processing circuit 250 is transferred to the DRAM 70 via the bus I / F circuit 270. The image data of the image sensors 11 and 12 transferred to the DRAM 70 is separated for each of the image sensors 11 and 12 and written in the storage areas corresponding to the image sensors 11 and 12 in the DRAM 30.

  Note that some image processing in the image processing circuit 250 cannot process image data of the imaging elements 11 and 12 collectively, such as lens correction. What is necessary is just to process this as follows. First, processed image data for one screen of each of the image sensors 11 and 12 is stored in the DRAM 70. Thereafter, the CPU 40 reads out the image data of each of the image pickup devices 11 and 12 and sends the image data to the image processing circuit 250. The image processing circuit 250 further performs predetermined image processing, and writes the image data in the DRAM 70 again.

  The image composition circuit 260 takes in the image data of the image sensors 11 and 12 from the DRAM 70 via the bus I / F circuit 270 and performs composition processing. In the DRAM 70, two hemispherical image data obtained by photographing with the image pickup devices 11 and 12 are stored after being subjected to predetermined image processing. Since the two hemispherical image data are image data having an angle of view of 180 degrees or more as described above, they have overlapping regions. The image synthesizing circuit 260 synthesizes the two hemispherical image data based on the overlapping areas of each other to generate omnidirectional image data.

  The generated omnidirectional image data is stored again in the DRAM 70 via the bus I / F circuit 270. Thereafter, the omnidirectional image data is stored in the external memory from the DRAM 70 or transferred to a personal computer or the like through the external I / F circuit 90 as necessary.

  Next, the write monitoring circuit 240 will be described. The write monitoring circuit 240 is a circuit that monitors the writing of image data from the image sensors 11 and 12 to the page memories 210 and 220. The image sensors 11 and 12 output horizontal / vertical synchronization signals, pixel clocks, and the like simultaneously with the image data. The write monitoring circuit 240 monitors the synchronization signal output from the image sensors 11 and 12, and the page memory read circuit 230 is at a timing when the image data for one page of the image sensors 11 and 12 is aligned in the page memories 210 and 220. In response to this, a page memory read start instruction signal is transmitted.

  In this embodiment, the page memories 210 and 220 have a page memory for two pages for each of the image sensors 11 and 12. In this case, the synchronization deviation of the image data of the image sensors 11 and 12 is allowable up to one page. The write monitoring circuit 240 sends a page memory read start instruction signal to the page memory read circuit 230 at a timing when the image data for one page output from the image pickup devices 11 and 12 is arranged in the page memories 210 and 220, respectively. .

  When the page memory read circuit 230 receives the page memory read start instruction signal from the write monitoring circuit 240, the page memory read circuit 230 starts reading image data from the page memories 210 and 220. That is, in the case of the present embodiment, the page memory reading circuit 230 selects a page memory in which image data for one page in the page memories 210 and 220 is already stored in a predetermined order in a toggle format, and the image data Can be read in a time-sharing manner in line units. As a result, even if there is a synchronization shift in the image data output from the image sensors 11 and 12, the image data from the top of the page of the image sensors 11 and 12 can be correctly sent to the subsequent image processing circuit 250. .

  In the present embodiment, the page memories 210 and 220 each have a page memory for two pages. However, the number of page memories may be determined according to the characteristics of the image sensor (CMOS sensor). If each image sensor (CMOS sensor) has the same type and the same characteristics with little variation, a page memory configuration for one page may be used.

  In the page memories 210 and 220, the write operation and the read operation are performed independently. However, the write and read clocks have different frequencies, and the read clock frequency is n times (n ≧ 2) or more of the write clock frequency. To do. As a result, write / read operations of the page memories 210 and 220 and real-time processing of the image processing circuit 250 in units of lines are realized.

  FIG. 3 is a diagram showing how image data is transferred according to this embodiment, and FIG. 4 is a diagram showing how image data is stored in the page memories 210 and 220. Note that the signals in FIG. 3 are written with the horizontal axis as the time axis.

  In FIG. 3, the upper row is a signal output from the image sensor 11, Vsync_A is a vertical synchronization signal (output only once at the beginning of one page of a two-dimensional image), Hsync_A is a horizontal synchronization signal (output at the beginning of a line unit), DE_A is a data enable signal (indicating data validity), A (1), A (2), A (3),... Are image data of each line. The middle stage is a signal output from the image sensor 12, Vsync_B is a vertical synchronization signal, Hsync_B is a horizontal synchronization signal, DE_B is a data enable signal, B (1), B (2), B (3),. Image data. Note that a pixel clock is also output from the image sensors 11 and 12, but is omitted in FIG.

  As shown in the upper and middle stages of FIG. 3, it is assumed here that the image data output from the image sensors 11 and 12 is shifted by an arbitrary line. Specifically, it is assumed that the output image data of the image sensor 12 is delayed by several lines with respect to the output image data of the image sensor 11.

  The image data output from the image sensors 11 and 12 are sequentially stored in line units in the page memories 210 and 220, respectively. FIG. 4 shows this state. On the other hand, the writing monitoring circuit 240 monitors the writing state of the image data output from the image sensors 11 and 12 to the page memories 210 and 220, and the image data output from the image sensors 11 and 12 is for one page, respectively. A page memory read start instruction signal is sent to the page memory read circuit 230 at the timing when the page memories 210 and 220 are aligned.

  In the embodiment, image data A (1), A (2), A (3)... At the top of the page of the image sensor 11 are sequentially stored in the page memory 1 of the page memory 210, and delayed by several lines from this. Image data B (1), B (2),... At the top of the page of the image sensor 12 are sequentially stored in the page memory 1 of the page memory 220. Thus, at the timing when all the image data for one page output from the image sensors 11 and 12 is stored, a page memory read start instruction signal is sent from the write monitoring circuit 240 to the page memory read circuit 230. . In the example, the page memory read start instruction signal is output at the timing when the image data B (n) of the image sensor 12 is stored in the page memory 1 of the page memory 220.

  The page memory reading circuit 230 receives the page memory read start instruction signal from the write monitoring circuit 240, and starts time-division reading of image data from the page memories 210 and 220. That is, the page memory reading circuit 230 first reads the image data A (1) of the first line of the page memory 1 in the page memory 210 and sends it to the image processing circuit 250, and then continues to 1 of the page memory 1 in the page memory 220. The image data B (1) of the line is read and sent to the image processing circuit 250. Similarly, the page memory reading circuit 230 receives image data A (2) and B (2), A (3) and B (3),... From each page memory 1 of the page memories 210 and 220. The data are sequentially read and sent to the image processing circuit 250. The page memory reading circuit 230 also sends a synchronization signal and the like to the image processing circuit 250.

  The image processing circuit 250 has image data A (1) and B (1), A (2) and B (2), A (3) and B (3) sent from the page memory reading circuit 230,.・ Sequentially, predetermined image processing is performed in real time and output. This is shown in the lower part of FIG. Here, Vsync_O is the vertical synchronization signal of the image processing circuit 250, Hsync_O is the same synchronization signal (output at the head of the line unit), and DE_O is the same data enable signal. O (1) means output image data that has been subjected to image processing of A (1) and B (1). Similarly, O (2) means output image data subjected to image processing of A (2) and B (2).

  As described above, in the present embodiment, image data output from the image sensors 11 and 12 is accumulated in the page memories 210 and 220 using the page memories 210 and 220 each including page memories for two pages, and the page memory The read circuit 230 reads the image data of the image sensors 11 and 12 from the page memories 210 and 220 in a time-sharing manner and sends the image data to the single image processing circuit 250. The image processing circuit 250 converts the image data of the image sensors 11 and 12 into the image data. On the other hand, predetermined image processing is performed collectively. Thereby, an image processing circuit corresponding to each image sensor is not required, the number of hardware of the image processing circuit can be reduced, and an increase in cost can be suppressed.

  Further, in the present embodiment, the page monitoring readout circuit 240 causes the page memory read circuit 230 to page the image data for one page of the image sensors 11 and 12 in a page memory with the page memories 210 and 220. A memory read start instruction signal is transmitted. Thereby, the image data of the same page of the image sensors 11 and 12 can be correctly sent to the subsequent image processing circuit 250.

  As mentioned above, although one Embodiment of this invention was described, the imaging device of this invention is not limited to the structure of illustration. As described above, there may be three or more image sensors. Also, the page memory need not be two pages.

DESCRIPTION OF SYMBOLS 11, 12 Image pick-up element 20 Image processing unit 30 Imaging control unit 40 CPU
70 DRM
210, 220 Page memory 230 Page memory read circuit 240 Write monitor circuit 250 Image processing circuit 260 Image composition circuit

JP 2006-033810 A

Claims (6)

An imaging apparatus having a plurality of imaging elements,
A plurality of memories corresponding to the plurality of image sensors, each storing hemispherical image data output from the plurality of image sensors;
Reading means for reading out the hemispherical image data stored in the plurality of memories in a time-sharing manner;
A single image processing means for performing predetermined image processing on the hemispherical image data corresponding to the read image sensors;
Image generating means for generating omnidirectional image data based on a plurality of the hemispherical image data subjected to predetermined image processing by the single image processing means;
An imaging device.   The imaging apparatus according to claim 1, further comprising a writing monitoring unit that monitors writing states in the plurality of memories and controls reading start of the reading unit.   The imaging apparatus according to claim 2, wherein the memory is a page memory. Each of the plurality of page memories has a page memory for at least one page;
The write monitoring unit instructs the page memory reading unit to start page memory reading at a timing when one page of the hemispherical image data of the plurality of image sensors is aligned in each page memory. The imaging device described.   The imaging apparatus according to claim 1, wherein the reading unit reads the hemispherical image data in units of lines. Reading means for reading out the hemispherical image data in a time-sharing manner from a plurality of memories each storing hemispherical image data output from a plurality of imaging elements included in the imaging device;
A single image processing means for performing predetermined image processing on the hemispherical image data corresponding to the plurality of read image sensors;
Image generating means for generating omnidirectional image data based on a plurality of the hemispherical image data subjected to predetermined image processing by the single image processing means;
An image processing apparatus.

Images