JP6035789B2 - Image composition apparatus and program - Google Patents

Description

  The present invention relates to an image synthesizing apparatus and a program for generating an omnidirectional image by connecting a plurality of captured images.

  A method of connecting images captured by a plurality of cameras to generate a wide-angle image, a high-definition image, and an omnidirectional image overlooking 360 degrees is already known. For example, Patent Document 1 discloses an invention for generating a three-dimensional image of an object by irradiating an infrared pattern.

  The invention according to Patent Document 1 can generate a three-dimensional image of a subject by measuring the distance to the subject using an infrared pattern (invisible light). However, as in the invention described in Patent Document 1, until now, images are joined with parameters fixed based on the camera position, imaging direction, and optical characteristics, so depending on the distance to the subject, the images are not joined correctly. There is a problem that it becomes double.

  There is also a method for recognizing images and adaptively connecting them, but image recognition requires a large amount of processing and may not be recognized correctly depending on the type of image.

  Therefore, the present invention provides an image composition device that correctly joins captured images so that the captured images are not duplicated regardless of the distance to the subject and without depending on image recognition. Objective.

The present invention has been made in view of the above problems, and several have a optical system with X degrees or more angle, a plurality of imaging that are arranged corresponding to the optical system to image the omnidirectional The reference light in a plurality of directions arranged between the optical system and the recording unit for recording a plurality of picked-up images picked up by the plurality of image pickup devices and two optical systems and shifted by X / 2 degrees from the optical axis of the optical system A reference light irradiating unit that irradiates and an image synthesizing device that synthesizes a plurality of captured images picked up by a plurality of image sensors and generates an omnidirectional image. A plurality of picked-up images are synthesized so that the reference lights irradiated in a plurality of directions shifted by X / 2 degrees from the optical axis of the system overlap.

  According to the present invention, since the reference light is irradiated to the region where the angle of view by the plurality of image sensors overlaps, and the plurality of captured images are synthesized so that the captured reference light overlaps, the image is not duplicated. Thus, it is possible to correctly connect the captured images regardless of the distance to the subject.

1 is a block diagram illustrating an overall configuration of an image composition device according to an embodiment of the present invention. It is the schematic which shows a mode that the omnidirectional image is produced | generated from the image sensor by the fisheye lens which opposes the image synthesizing | combining apparatus of 1st Embodiment of this invention. It is explanatory drawing which shows the example of the difference of the picked-up image by the distance of a to-be-photographed object at the time of using the image sensor by the fisheye lens which opposes for the image composition apparatus of 1st Embodiment of this invention. It is the schematic which shows a mode that a wide angle image is produced | generated from the two image sensors which faced the same direction which concerns on the image synthesizing | combining apparatus of 2nd Embodiment of this invention. It is explanatory drawing which shows the example of the difference of the picked-up image by the distance of a to-be-photographed object at the time of using the two image sensors which face the same direction based on the image synthesizing | combining apparatus of 2nd Embodiment of this invention. It is explanatory drawing which shows the example of the preservation | save area | region as an overlapping area | region of the captured image by the image sensor of 1st Embodiment and 2nd Embodiment of this invention.

  An embodiment of the present invention will be described. The present invention has the following characteristics with respect to an omnidirectional image in which a plurality of captured images are connected to look around 360 degrees and an image composition device that generates an image. In short, the reference light is irradiated to the overlapping area of a plurality of sensors, and the captured images are connected (synthesized) in accordance with the position of the reference light. Features of the image composition apparatus according to the embodiment of the present invention will be described below with reference to the drawings.

  The overall configuration of the image composition apparatus 1 of the present embodiment will be described with reference to the block diagram of FIG. An image composition device 1 according to an embodiment of the present invention includes an image sensor 11a (imaging device), an image sensor 11b (imaging device), a reference light irradiation unit 12, a nonvolatile memory 13 (recording unit), and an overall control unit 14. .

  The image sensor 11a and the image sensor 11b are image sensors provided in correspondence with an optical system such as a fisheye lens described later. This image sensor is an apparatus for capturing an image of a subject and capturing the captured image in the main body of the image composition apparatus 1. An image sensor typified by a CCD or CMOS is used for the image sensor. In the present embodiment, a configuration using two image sensors is shown, but the configuration is not limited to this, and a configuration using two or more image sensors may be used. In addition, the imaging element of the present embodiment is an element having sensitivity to invisible light described later.

  The reference light irradiation unit 12 (reference light irradiation unit) is an apparatus that irradiates reference light to an overlapping portion of a plurality of captured images captured by the image sensor 11a and the like. The reference light refers to light serving as a reference for combining a plurality of captured images by superimposing the light irradiated on the overlapped portion of the captured images. By synthesizing a plurality of captured images using the reference light, it is possible to prevent the generation of a double image due to the overlapping of the overlapping portions of the plurality of captured images.

  Moreover, it is good also as a structure which uses invisible lights, such as infrared rays, for reference light. This is because when visible light is used as the reference light, the reference light can be confirmed with the naked eye at the time of imaging, which is troublesome. That is, if the reference light is invisible light, it is possible to prevent the troublesome reference light from being seen with the naked eye during photographing. The reference light irradiation unit 12 uses, for example, an illuminator such as a light emitting element, but is not limited thereto, and any device that can irradiate reference light may be used.

  Furthermore, in this embodiment, the structure which irradiates a reference light intermittently to the overlapping part of a some captured image may be taken. The captured image may be affected by continuously irradiating the captured image with the reference light. Therefore, it is possible to reduce the influence on the captured image by intermittently irradiating the reference light.

  The non-volatile memory 13 is a memory that records captured images. In this embodiment, for example, a non-volatile storage device such as a hard disk, a flash memory, or an SD memory is used.

  The overall control unit 14 is a module for controlling the entire image composition apparatus 1 including a CPU / ROM / RAM and the like. The overall control unit 14 performs various processes such as synthesis of captured images, which will be described later, input / output processing of captured images and synthesized images to / from the nonvolatile memory 13, reference light irradiation command processing to the reference light irradiation unit 12, and the like.

[First Embodiment]
Next, the synthesized process of the captured image of 1st Embodiment of this invention is demonstrated using FIG. In this embodiment, the image sensor 11a corresponding to the fisheye lens 110a having an angle of view of 180 degrees or more and the image sensor 11b corresponding to the fisheye lens 110b having an angle of view of 180 degrees or more are arranged to face each other by 180 degrees. Here, two lenses are used, and the angle of view is 180 degrees or more in order to cause overlapping of images. However, the angle of view is adjusted as the number of image sensors increases. For example, if there are four image sensors, four lenses having an angle of view of 90 degrees or more are used.

  An angle x formed by oblique lines (a, b) extending from both ends of the fisheye lens 110a and an angle y formed by oblique lines (c, d) extended from both ends of the fisheye lens 110b indicate the angle of view of each lens. In addition, the hatched portion indicated by the parallel oblique lines in the figure is an area (hereinafter, referred to as an overlapping area C) reflected in both image sensors. In the overlapping area C, a short-distance subject A and a long-distance subject B are indicated by black dots.

  With the configuration as shown in FIG. 2, a 360-degree omnidirectional image can be generated by connecting (combining) the captured images of the overlapping region C based on the images. However, if the parameters are simply connected based on the camera position, imaging direction, and optical characteristics, the subject may be doubled depending on the distance to the subject.

  Therefore, in this embodiment, the reference light irradiation unit 12 is arranged so as to irradiate the reference light 121 in a direction (reference light optical axis f) shifted by 90 degrees from the optical axis e of the fisheye lens (optical system). That is, the reference light irradiation unit is provided in accordance with the position where the overlapping region is generated as viewed from the optical axis of the optical system. In FIG. 2, the reference light optical axis f is extended so as to emit the reference light 121 only in the upper part of the figure, but actually, the reference light 121 is irradiated in all directions orthogonal to the optical system optical axis e. Depending on how the reference light 121 is reflected in the overlapping area C, the distance to the subject can be measured.

  FIG. 3 shows captured images (D, E) acquired by the image sensors (11a, 11b) of the present embodiment described above and reference light (A1, B1) by the reference light irradiation unit 12. FIG. 3A shows a state in which the reference light A1 irradiated to the subject A at a short distance is projected. The reference light A1 is projected on a position far from the center of the captured image (D, E) on the overlapping region C.

  On the other hand, FIG. 5B shows a state in which the reference light B1 irradiated to the subject B at a long distance is projected. The reference light B1 is projected on a position close to the center of the captured image (D, E) on the overlapping region C. That is, when the subject is near, the reference light is projected at a position far from the center of the image as shown in the figure, and when the subject is far away, the reference light is projected at a position near the center of the image.

  By deforming and synthesizing two images so that the reference light (A1, B1) overlaps, the image does not become double. For example, the following steps can be considered for the synthesis process. In the two captured images, reference light that is recognized as equidistant in the overlapping region is specified (step 1). As a specific method, it is possible to measure the distance from the center of the captured images D and E. The images are joined so that the specified reference lights overlap each other (step 2). In addition to such synthesis processing, synthesis processing can be applied as appropriate. In this embodiment, the reference light is a disc-shaped slit light. However, the reference light may be irradiated in such a form that the distance such as a lattice or a dot pattern can be recognized over the entire overlapping region C.

[Second Embodiment]
The composite processing of captured images according to the second embodiment of the present invention will be described with reference to FIG. In the present embodiment, the image sensor 21a corresponding to the wide-angle lens 210a and the image sensor 21b corresponding to the wide-angle lens 210b are arranged side by side in the same direction.

  Further, as shown in FIG. 4A, in this embodiment, the reference light irradiation unit 120 is arranged in the same direction as the image sensors (21a, 21b). Further, as shown in the figure, the reference light 221 is emitted toward the subject G from the reference light irradiation unit 120 arranged between the image sensors. Therefore, the optical axis of the reference light 221 is parallel to the optical axis f of the image sensor 21a and the optical axis i of the image sensor 21b. Here, as in the first embodiment, the reference light irradiation unit is provided at a position where the reference light can be irradiated toward the overlapping region.

  A hatched portion with parallel diagonal lines in the figure is an area (hereinafter, referred to as an overlapping area F) reflected in both the image sensor 21a and the image sensor 21b.

  The schematic view of the imaging target H including the subject G by the image sensor 21a, the imaging target I including the subject G by the image sensor 21b, and the irradiation state of the reference light 221 to the subject by the reference light irradiation unit 120 as viewed from the front. Shown in 4 (b).

  A difference between the captured image H including the reference light 221 acquired by the image sensor 21a and the captured image I including the reference light 221 acquired by the image sensor 21b will be described with reference to FIG. As shown in the figure, the position of the reference light 221 is shifted and projected in accordance with the distance to the subject G.

  That is, in the captured image H, the reference light 221 irradiated on the spherical surface of the subject G is displayed as a semicircular portion G1 with the arc on the left side. On the other hand, in the captured image I, the reference light 221 irradiated on the spherical surface of the subject is displayed as a semicircular portion G2 whose arc is on the right side. The semicircular portions G1 and G2 are closer to each other than the linear portions G3 and G4 of the reference light projected on the imaging target H and I as viewed from the image sensors 21a and 21b.

  Also in the present embodiment, as in the first embodiment, for example, the following steps can be considered for the synthesis process. In the two captured images, the reference light recognized as equidistant in the overlapping region is specified (step 1). The distance can be identified by measuring the distance of the reference light from the end of the overlapping region to determine whether the distance is equal. The images are joined so that the specified reference lights overlap each other (step 2). In addition to such synthesis processing, synthesis processing can be applied as appropriate. By deforming and synthesizing the two captured images so that the reference light overlaps, the image is not duplicated and a wider area can be photographed.

  FIG. 6A shows a case where the overlapping region C (parallel oblique line portion) is a storage region when the fisheye lens (110a, 110b) in the first embodiment of the present invention is used, and FIG. 6B shows the present invention. The case where the overlapping area F (parallel oblique line portion) when the wide-angle lens (210a, 210b) in the second embodiment is used as a storage area will be described.

  By saving only the overlap area as the processing data for image composition processing as shown in this figure (that is, by making the overlap area function as a part of the image sensor, other previously required image composition for image composition) Therefore, it is possible to reduce the storage data capacity stored in the recording unit such as the nonvolatile memory. The figure is an example, and changes depending on the focal length, the number of image sensors, and the arrangement method of the image sensors.

  In addition to the above, a configuration may be adopted in which only the coordinate information of the position irradiated with the reference light is stored in the recording unit as the processing data for the image composition processing. As a result, similarly to the above, the storage data capacity stored in the recording unit such as the nonvolatile memory can be reduced.

  The present invention may be characterized in that the reference light irradiation unit irradiates the reference light intermittently. Thereby, it is possible to reduce the influence on the captured image by the irradiation of the reference light.

  The present invention may be characterized in that the reference light is invisible light having sensitivity of the imaging device. By making the reference light invisible, it is possible to prevent the troublesome reference light from being seen with the naked eye during photographing.

  The present invention may be characterized in that only the overlapping portion of a plurality of captured images irradiated with the reference light is recorded in the recording unit. Thereby, it is possible to reduce the capacity of the captured image stored (recorded) in the recording unit.

  The present invention may be characterized in that only the coordinate information of the position irradiated with the reference light is recorded in the recording unit. Thereby, it is possible to reduce the capacity of the captured image stored (recorded) in the recording unit.

  The present invention may be characterized by causing a computer to execute processing for intermittently irradiating reference light with a reference light irradiator. Thereby, it is possible to reduce the influence on the captured image by the irradiation of the reference light.

  The present invention may be characterized in that a computer executes a recording process including a process of recording only an overlapping portion of a plurality of captured images irradiated with reference light. This makes it possible to reduce the capacity of the captured image to be saved (recorded).

  The present invention may be characterized in that a computer executes a recording process including a process of recording only coordinate information of a position irradiated with reference light. This makes it possible to reduce the capacity of the captured image to be saved (recorded).

  Each process such as the image composition process may be executed by a processor such as a CPU reading a process program stored in a predetermined storage area (ROM or the like). Each of the above-described embodiments is a preferred embodiment of the present invention, and various modifications can be made without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 Image composition apparatus 11a, 11b, 21a, 21b Image sensor 12, 120 Reference light irradiation unit 13 Non-volatile memory 14 Whole control part 110a, 110b Fisheye lens 210a, 210b Wide angle lens

Special table 2008-537190 gazette

Claims (8)

A plurality of imaging elements arranged corresponding to the optical system as possess multiple captures an omnidirectional optical system with X degrees than the angle of view,
A recording unit for recording a plurality of captured images captured by the plurality of imaging elements;
A reference light irradiating unit disposed between the two optical systems and irradiating the reference light in a plurality of directions shifted by X / 2 degrees from the optical axis of the optical system;
An image composition device comprising: an image composition unit that composes a plurality of captured images captured by the plurality of image sensors and generates an omnidirectional image;
The image synthesizing device, wherein the image synthesizing unit synthesizes the plurality of captured images so that reference lights irradiated in a plurality of directions shifted by X / 2 degrees from an optical axis of the optical system overlap. Two image sensors having two optical systems having an angle of view of 180 degrees or more and arranged opposite to each other so as to image all directions;
A recording unit that records two captured images captured by the two image sensors;
A reference light irradiating unit disposed between the two optical systems and irradiating the reference light in a plurality of directions shifted by 90 degrees from the optical axis of the optical system;
An image composition device comprising: an image composition unit that composes two captured images captured by the two image sensors and generates an omnidirectional image;
The image synthesizing device, wherein the image synthesizing unit synthesizes the two captured images so that the reference light emitted in a plurality of directions shifted by 90 degrees from the optical axis of the optical system overlaps. Four image sensors having four optical systems having an angle of view of 90 degrees or more, and two image sensors corresponding to the optical system so as to image all directions;
A recording unit that records four captured images captured by the four image sensors;
A reference light irradiating unit disposed between the two optical systems and irradiating the reference light in a plurality of directions shifted by 45 degrees from the optical axis of the optical system;
An image composition device including an image composition unit that composes four captured images captured by the four image sensors and generates an omnidirectional image,
The image synthesizing device, wherein the image synthesizing unit synthesizes the four captured images so that the reference lights irradiated in a plurality of directions shifted by 45 degrees from the optical axis of the optical system overlap.   The image synthesizer according to any one of claims 1 to 3, wherein the reference light irradiation unit irradiates the reference light intermittently.   5. The image composition apparatus according to claim 1, wherein the reference light is invisible light in which the imaging device has sensitivity. 6.   6. The image synthesizing apparatus according to claim 1, wherein only the overlapping portion of the two captured images is recorded in the recording unit.   6. The image synthesizing apparatus according to claim 1, wherein only the coordinate information of the position irradiated with the reference light is recorded in the recording unit. And multiple chromatic optical system with X degrees than the angle of view, a recording process for recording a plurality of images captured by the plurality of imaging elements arranged corresponding to the optical system to image the omnidirectional When,
The plurality of captured images are synthesized by the reference light irradiator disposed between the two optical systems so that the reference lights irradiated in a plurality of directions shifted by X / 2 degrees from the optical axis of the optical system overlap. A program for causing a computer to execute image composition processing for generating an omnidirectional image.

Images