JP2018137592A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, an imaging method, and a program capable of guaranteeing the quality of a joint while suppressing increase in costs.SOLUTION: The imaging apparatus according to this invention, which generates one image by joining images captured by respective image pick-up devices corresponding to plural optical systems, comprises a correction control part. The correction control part controls the image pick-up devices so as to correct deviation of the exposure timing of the respective image pick-up devices.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program.

  For example, there is known an imaging apparatus that generates one image by connecting images captured by each of a plurality of imaging elements (image sensors), as represented by an omnidirectional image. In such an imaging apparatus, a technique is known in which the exposure timing is adjusted by aligning the scanning direction in order to improve the quality of the joint (for example, see Patent Document 1).

  However, for example, in order to physically align the scanning direction, it may be possible to prepare a plurality of different image pickup device substrates (substrates on which the image pickup devices are mounted). However, in this case, there is a problem that costs increase. When using one type of substrate (one substrate on which a plurality of image sensors are mounted) in order to reduce costs, the reading direction from the image sensor (direction for reading an image in line units) is operated in the reverse direction, Although it is possible to align the scanning direction, in this case, there is a possibility that the readout timing between the image sensors is shifted by several lines. That is, in the related art, there is a problem that the quality of the joint cannot be secured while suppressing an increase in cost.

  This invention is made | formed in view of the above, and it aims at ensuring the quality of a joint, suppressing the increase in cost.

  In order to solve the above-described problems and achieve the object, the present invention provides an image pickup apparatus that generates a single image by connecting images picked up by a plurality of image pickup elements corresponding to a plurality of optical systems. The imaging apparatus includes a correction control unit that controls the imaging element so as to correct a deviation in exposure timing of each imaging element.

  According to the present invention, it is possible to ensure the quality of joints while suppressing an increase in cost.

FIG. 1 is a diagram illustrating an example of a hardware configuration of the imaging apparatus according to the embodiment. FIG. 2 is a diagram illustrating an example of a detailed configuration of the imaging unit according to the embodiment. FIG. 3 is a diagram for explaining the arrangement of the image sensor and the scanning direction. FIG. 4 is a diagram for explaining the influence of an image due to a shift in exposure timing. FIG. 5 is a diagram for simply explaining how to connect two captured images to one omnidirectional image. FIG. 6 is a diagram for explaining a shift in exposure timing. FIG. 7 is a diagram illustrating an example of functions of the imaging apparatus according to the embodiment. FIG. 8 is a diagram for explaining a method for correcting a deviation in exposure timing. FIG. 9 is a diagram illustrating an example in which the input of the horizontal synchronization signal is delayed by the amount of deviation.

  Hereinafter, embodiments of an imaging apparatus, an imaging method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. In the following embodiment, an imaging device that generates an omnidirectional image obtained by connecting two hemispherical images will be described as an example, but the present invention is not limited to this. In short, the imaging device may be an imaging device that generates one image by connecting images captured by each of a plurality of imaging elements corresponding to a plurality of optical systems. Note that the omnidirectional image is an image captured over the entire visual field at a horizontal angle of 360 degrees and a vertical angle of 180 degrees.

  FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus 1 (an imaging apparatus 1 that generates an omnidirectional image) according to the present embodiment. Hereinafter, the optical systems 10b and 10c and the imaging elements 13 and 14 constituting the fisheye lens will be collectively referred to as the imaging unit 10. The optical system 10 b corresponds to the image sensor 13, and the optical system 10 c corresponds to the image sensor 14. In FIG. 1, image sensors 13 and 14 convert an optical image obtained by the optical systems 10 b and 10 c into image data of an electrical signal and output the image sensor, such as a CMOS sensor or a CCD sensor, and horizontal / vertical synchronization of the image sensor. It includes a timing generation circuit that generates a signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set. It is necessary to synchronize the synchronization signals of these two image sensors 13 and 14 so that images can be captured at the same timing. A method for this will be described later.

  Each of the imaging elements 13 and 14 of the imaging unit 10 is connected to the image processing unit 20 via a parallel I / F bus. In addition, each of the imaging elements 13 and 14 of the imaging unit 10 is separately connected to the imaging control unit 30 via a serial I / F bus (I2C bus or the like). The image processing unit 20 and the imaging control unit 30 are connected to the CPU 40 via the bus 100. Further, ROM 50, SRAM 60, DRAM 70, operation unit 80, external I / F circuit 90, and the like are connected to bus 100. The image processing unit 20 takes in the image data output from the image sensors 13 and 14 through the parallel I / F bus, performs predetermined processing on each image data, and then combines these image data. Generate an omnidirectional image.

  In general, the imaging control unit 30 sets a command or the like in a register group of the imaging elements 13 and 14 using the I2C bus with the imaging control unit 30 itself as a master device and the imaging elements 13 and 14 as slave devices. Necessary commands and the like are received from the CPU 40. The imaging control unit 30 also uses the I2C bus to capture the status data of the register groups of the imaging elements 13 and 14 and send them to the CPU 40. Further, the imaging control unit 30 instructs the imaging elements 13 and 14 to output image data at the timing when the shutter button of the operation unit 80 is pressed. In the present embodiment, the CPU 40 and the imaging control unit 30 cooperate to control the imaging element 13 or 14 so as to correct a shift in exposure timing (timing for reading an image) of the imaging elements 13 and 14. (Correction control unit) is realized. This function is realized by the CPU 40 executing a predetermined program to operate the imaging control unit.

  Depending on the type of the imaging device 1 that generates an omnidirectional image, it may have a preview display function by a display or a function corresponding to a moving image display. In this case, image data output from the image sensors 13 and 14 is continuously performed at a predetermined frame rate (frame / min). The CPU 40 controls the overall operation of the imaging apparatus 1 and executes necessary processes. The ROM 50 stores various programs for the CPU 40. The SRAM 60 and the DRAM 70 are work memories, and store programs executed by the CPU 40, data being processed, and the like. Here, the DRAM 70 is also used to store image data being processed by the image processing unit 20 and processed spherical image data.

  The operation unit 80 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user inputs various shooting modes and shooting conditions by operating the operation buttons. The external I / F circuit 90 is a general term for an interface circuit (USB I / F, etc.) with an external memory (SD card, flash memory, etc.) and a personal computer. Further, the external I / F circuit 90 may be a network interface regardless of wireless or wired. The spherical image stored in the DRAM 70 is stored in an external memory via the external I / F circuit 90, and a personal computer via the external I / F circuit 90 serving as a network I / F if necessary. Or transferred to a smartphone or the like.

  FIG. 2 is a diagram illustrating an example of a detailed configuration of the imaging unit 10. As shown in FIG. 2, the imaging unit 10 of the present embodiment includes an optical system 10b, a cover glass CGb, an imaging element 13, an optical system 10c, a cover glass CGc, and an imaging element 14. . The optical systems 10b and 10c employ an equidistant projection method in order to form a fisheye lens. The optical system 10b includes a front lens group (first lens group), an aperture 4b, and a rear lens group (second lens group). The optical system 10c includes a front lens group (first lens group), a stop 4c, and a rear lens group (second lens group).

  The lens group that is the front group of the optical system 10b is disposed on the object side of the optical system 10b. This lens group takes in light rays having a wide field angle exceeding 180 degrees with negative refractive power as a whole. This lens group is configured by arranging a lens L1b, a lens L2b, and a right-angle prism PSb in series in this order from the object side to the image side. The right-angle prism PSb has a mirror surface MSb in which a reflective film is formed on the inclined surface by an aluminum coating. The right-angle prism PSb causes the mirror surface MSb to internally reflect the light beam from the front lens group toward the rear lens group. That is, the optical axis OA2b of the optical system 10b is bent by 90 degrees on the mirror surface MSb of the right-angle prism PSb, and goes toward the image sensor 13 through the rear lens group.

  The lens group that is the rear group of the optical system 10b is disposed on the image side of the optical system 10b. This lens group mainly corrects the aberration of the captured image with a positive refractive power as a whole. This lens group is composed of a lens L3b, a lens L4b, a lens L5b, a lens L6b, and a lens L7b arranged in series in this order from the object side to the image side.

  The lens group that is the front group of the optical system 10c is disposed on the object side of the optical system 10c. This lens group takes in light rays having a wide field angle exceeding 180 degrees with negative refractive power as a whole. This lens group is configured by arranging a lens L1c, a lens L2c, and a right-angle prism PSc in series in this order from the object side to the image side.

  The right-angle prism PSc has a mirror surface MSc in which a reflective film is formed on the inclined surface by an aluminum coating. The right-angle prism PSc causes the mirror surface MSc to internally reflect light rays from the front lens group toward the rear lens group. That is, the optical axis OA2c of the optical system 10c is bent by 90 degrees on the mirror surface MSc of the right-angle prism PSc, and goes to the image sensor 14 through the lens group as the rear group.

  The lens group that is the rear group of the optical system 10c is disposed on the image side of the optical system 10c. This lens group mainly corrects the aberration of the captured image with a positive refractive power as a whole. This lens group is composed of a lens L3c, a lens L4c, a lens L5c, a lens L6c, and a lens L7c arranged in series in this order from the object side to the image side.

  Further, the outer surface of the right-angle prism PSb of the optical system 10b on which the mirror surface MSb is formed and the outer surface of the right-angle prism PSc of the optical system 10c on which the mirror surface MSc is formed are bonded and fixed. That is, since it has a configuration in which two imaging systems are combined, the entire imaging unit 10 can be reduced in size and a hand-held system can be realized. Further, since the maximum angles of view of the optical system 10b and the optical system 10c are each 190 degrees or more, they are combined as shown in FIG. The unit 10 can be configured.

  FIG. 3 is a diagram for explaining the arrangement of the image sensor and the scanning direction. FIG. 3A is an image diagram illustrating the configuration of FIG. 2 in a simplified manner. The imaging device 1 that generates an omnidirectional image can reduce a region where imaging is impossible (an imaging impossible region 6) when the distance between the optical systems 10b and 10c is narrower. In general, an image pickup device represented by a CMOS image sensor used in a camera has a 3: 2 or 4: 3 horizontally long shape. In order to narrow the interval between the optical systems 10b and 10c, it is advantageous to place the image sensors 13 and 14 vertically as shown in FIG. In general, since the column ADC of the CMOS image sensor is often provided on the long side, the scanning direction is horizontal as indicated by 131a and 131b in FIG. When the scanning directions 131a and 131b of the image sensors 13 and 14 are aligned as shown in FIG. 3B, the light passes through the lens optical system and the mirror of the prism, so the scanning directions 132a and 132b in space are The direction is as shown in FIG. Although details will be described later, as shown in FIG. 3B, the scanning direction in the space is aligned, and the reading of images (line unit images) from each of the two image sensors 13 and 14 is synchronized. For example, since the exposure timings of the contact portions connecting the images picked up by the two image pickup devices 13 and 14 coincide with each other, the quality of the joint for the moving object can be improved. If a mechanical shutter is mounted in the optical system, the influence of such exposure timing is almost eliminated in a still image, but there is a case where there is no mechanical shutter and a rolling shutter in an image sensor is used. A rolling shutter is used for moving image shooting regardless of the presence or absence of a mechanical shutter.

  On the other hand, with respect to the substrate on which the image pickup devices 13 and 14 are mounted, when one type of substrate is used, due to restrictions on connection from the sensor (image pickup devices 13 and 14) to the main substrate, FIG. Thus, it is necessary to rotate them in the horizontal direction so that they face each other. Then, as shown in FIG. 3C, the scanning directions 132a and 132b 'in space are not aligned. Therefore, for example, the mounting directions of the image sensors 13 and 14 can be reversed to align the scanning direction, but in this case, it is necessary to create two types of substrates, which is disadvantageous in terms of cost and quality. On the other hand, the imaging elements 13 and 14 have a function of inverting the readout direction. However, when this function is used, there is a possibility that the readout timing between the image sensors is shifted by several lines.

  Next, the influence of the image due to the shift of the exposure timing will be described. In the present embodiment, each of the image sensors 13 and 14 uses a rolling shutter system in which exposure and image reading are performed in line units (row units). FIG. 4A shows a subject to be imaged. In FIG. 4A, it is assumed that the subject 8 and the subject 9 exist so as to straddle the joint 7 of the omnidirectional image, the subject 8 is moving in the direction of the arrow, and the subject 9 is stationary. .

  FIG. 4B is a diagram illustrating an image obtained by capturing FIG. 4A when the exposure timing is correct. It is assumed that the scan 130 of the image sensor 13 and the scan 140 of the image sensor 14 are coincident in both direction and timing. The subjects 10 and 11 imaged at that time can be connected without any problem. For convenience of explanation, the joint 7 and the scanning direction are shown, but they are not shown in the actual image.

  FIG. 4C is a diagram illustrating an image obtained by capturing FIG. 4A when the exposure timing is shifted. Although the scanning 130 of the image sensor 13 and the scan 140 ′ of the image sensor 14 are in the same direction, the timing is shifted. Since the imaged subject 11 is stationary at that time, it can be connected without any problem regardless of the exposure timing. However, since the moving subject 10 causes a time difference, it shifts as shown in the figure, and is accurate. I can't connect.

  Next, how to connect two captured images to one omnidirectional image will be briefly described with reference to FIG. In FIG. 5, the solid line on the photographed image 141 and the solid line on the photographed image 150 indicate the image circle of the fisheye lens, and each of the optical systems 10b and 10c shown in FIG. Actually, since the omnidirectional image 160 is generated by combining the images of 180 degrees, the region to be used is indicated by a dotted line in FIG.

  In the example of FIG. 5, it is assumed that the captured image 141 is arranged in the center, and the captured image 150 is connected to the left and right to form the omnidirectional image 160. At that time, it is assumed that the omnidirectional image 160 in which the points Pa1 to Pa4 and Pb1 to Pb4 on the joint are connected to Pa1 and Pb2, Pa2 and Pb1, Pa3 and Pb4, and Pa4 and Pb3 at the time of joining is generated. Continue to explain.

  Next, a shift in exposure timing (timing for reading an image in line units) will be described with reference to FIG. FIG. 6A shows a diagram in which the two image pickup devices 13 and 14 are synchronized and an image in an ideal state with no timing shift is imagined (assuming the image described in FIG. 5). In the following description, when the imaging devices 13 and 14 are not distinguished from each other, they are simply referred to as “imaging devices”. As described above, here, it is assumed that the image pickup element adopts a rolling shutter system and reads out line by line from the top to the bottom of the arrow direction (exposes an image by reading line by line).

  The time for reading the points Pa1 and Pa2 on the captured image 141 is ta1, and the time for reading the points Pa3 and Pa4 is ta2. Similarly, the time for reading the points Pb1 and Pb2 on the captured image 150 is tb1, and the time for reading the points Pb3 and Pb4 is tb2. Since (A) in FIG. 6 is an ideal state with no deviation, ta1 = tb1 and ta2 = tb2, and as described with reference to FIG. 5, the connection timings coincide with each other.

  FIG. 6B illustrates a case where the readout time of the captured image 150 has changed due to a synchronization error of the image sensor or a timing error due to inversion reading. In this case, since ta1 and tb1 are different, and ta2 and tb2 are also different, the exposure timing is shifted between connected points, and the state described with reference to FIG.

  FIG. 6C shows a photographed image when the image circle does not come to the center due to mechanical restrictions or when the image circle is shifted due to an assembly error. Also in this case, since ta1 and tb1 are different, and ta2 and tb2 are also different, the exposure timing is shifted between the connected points, and the state described with reference to FIG. Such unintended misalignment of the image circle can be corrected by software in the adjustment process, but the exposure timing shift is not corrected.

  Therefore, the imaging apparatus 1 according to the present embodiment has a function (correction control unit) that controls the imaging element so as to correct a shift in exposure timing of each imaging element. As described above, this function is realized by the cooperation of the CPU 40 and the imaging control unit 30. Further, as described above, it is assumed that each image sensor uses a rolling shutter system that performs exposure and image reading in units of lines, and the scanning direction of each image sensor (direction in which light is scanned for each line) is Assuming they are in the same direction.

  FIG. 7 is a diagram illustrating functions related to the present invention among the functions of the imaging apparatus 1 of the present embodiment. For convenience of explanation, only functions related to the present invention are illustrated here, but the functions of the imaging apparatus 1 are not limited thereto. As illustrated in FIG. 7, the imaging device 1 includes a deviation amount storage unit 201 and a correction control unit 202.

  The shift amount storage unit 201 stores in advance a shift amount between the exposure timing of the image sensor 13 and the exposure timing of the image sensor 14. This deviation amount can be grasped for each individual in the adjustment process.

  The correction control unit 202 controls the image sensor so as to correct the deviation amount stored in advance in the deviation amount storage unit 201. In the present embodiment, the correction control unit 202 crops the image sensor whose exposure timing is earlier (in this example, one of the image sensors 13 and 14) by the amount of deviation (the amount of deviation of the exposure timing). And read.

  FIG. 8A corresponds to the state of FIG. In FIG. 8B, the correction control unit 202 crops and reads out an image pickup element (an image pickup element with an earlier exposure timing) for picking up the picked-up image 150 by an exposure timing shift (line). It is shown that it is possible to make the readout time uniform (ta1 = tb1, ta2 = tb2) by exposing the image in units and reading out the image.

  Note that the correction method of the shift amount is not limited to the above, and various correction methods can be employed. For example, the correction control unit 202 may read a blank line or a dummy line by an amount corresponding to the shift amount with respect to an image sensor having an earlier exposure timing. FIG. 8C shows a blank line or a dummy line corresponding to the deviation of the exposure timing for the image sensor (the image sensor with the earlier exposure timing) for which the correction control unit 202 captures the captured image 150. It is shown that it is possible to make the reading times uniform (ta1 = tb1, ta2 = tb2).

  Further, for example, the correction control unit 202 may delay the input of the horizontal synchronization signal for reading out the image in units of lines for the image sensor having the earlier exposure timing by the amount of deviation. For example, as shown in FIG. 9, a horizontal synchronization signal 180 for reading the image of the captured image 150 in units of lines (the one with the earlier exposure timing) is compared to the horizontal synchronization signal 170 for reading the image of the captured image 141 in units of lines. By delaying the horizontal synchronization signal 180) input to the image pickup device by the correction time 190 (time ta1-tb1 corresponding to the number of correction lines), it is possible to make the readout time uniform (ta1 = tb1, ta2 = tb2). become.

  As described above, in the present embodiment, the captured image captured by each image sensor is controlled by controlling the image sensor so as to correct the deviation of the exposure timing of each image sensor in which the respective scanning directions are aligned. When connecting and generating one image, it is possible to ensure the quality of a joint between captured images while suppressing an increase in cost (since it is not necessary to provide a plurality of imaging substrates).

  Although the embodiments according to the present invention have been described above, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment.

  The program executed by the imaging apparatus 1 of the above-described embodiment is an installable or executable file, which is a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), It may be configured to be recorded on a computer-readable recording medium such as a USB (Universal Serial Bus), or provided or distributed via a network such as the Internet. Various programs may be provided by being incorporated in advance in a ROM or the like.

DESCRIPTION OF SYMBOLS 1 Imaging device 10 Imaging unit 10b Optical system 10c Optical system 13 Imaging element 14 Imaging element 20 Image processing unit 30 Imaging control unit 40 CPU
50 ROM
60 SRAM
70 DRAM
80 Operation unit 90 External I / F circuit 100 Bus 201 Deviation amount storage unit 202 Correction control unit

Japanese Patent Laying-Open No. 2015-61125

Claims (8)

In an imaging apparatus that generates one image by joining images captured by each of a plurality of imaging elements corresponding to a plurality of optical systems,
A correction control unit for controlling the image sensor so as to correct a shift in exposure timing of each image sensor;
Imaging device. Each of the imaging elements uses a rolling shutter system that performs exposure and readout of an image in units of lines.
The imaging device according to claim 1. The correction control unit
For the image sensor with the earlier exposure timing, the amount of deviation of the exposure timing is cropped and read.
The imaging device according to claim 2. The correction control unit
For the image sensor with the earlier exposure timing, a blank line is read by the amount of deviation of the exposure timing.
The imaging device according to claim 2. The correction control unit
For the image sensor with the earlier exposure timing, a dummy line is read out by the amount of deviation of the exposure timing.
The imaging device according to claim 2. The correction control unit
For the image sensor with the earlier exposure timing, the input of a horizontal synchronization signal for reading an image in line units is delayed by the amount of deviation of the exposure timing.
The imaging device according to claim 2. An image pickup method by an image pickup apparatus that generates a single image by joining images picked up by a plurality of image pickup elements corresponding to a plurality of optical systems,
A correction control step for controlling the image sensor so as to correct a shift in exposure timing of each image sensor;
Imaging method. To an imaging device that generates one image by joining images captured by each of a plurality of imaging elements corresponding to a plurality of optical systems,
A program for executing a correction control step for controlling the image sensor so as to correct a deviation in exposure timing of each image sensor.

Images