JP6740996B2 - Data generation device, image processing device, method and program - Google Patents

Description

The present invention relates to files, and more particularly to an imaging system, an imaging device, a program for generating an image, and a file used in the system.

In recent years, a camera capable of capturing a still image in a hemispherical or omnidirectional field of view by one shot has been put into practical use. In addition, a size that allows you to shoot a still image of a spherical image by hand is also provided. In such a handheld camera, since the camera body is easily tilted, it is known that the camera body has a function of tilt-correcting and displaying a captured still image corresponding to shooting in a tilted state. There is.

Japanese Patent Application Laid-Open No. 2013-214947 (Patent Document 1) is known as a technique for correcting the inclination of a still image. Patent Document 1 discloses a still image capturing apparatus having two fish-eye lenses for the purpose of obtaining a still image with a correct vertical direction, in accordance with a tilt angle detected by an acceleration sensor for obtaining a tilt from the vertical direction provided inside. A configuration for calculating a conversion parameter for converting a fish-eye image into a regular image is disclosed.

Further, not only still images but also cameras capable of storing a wide range as moving images by using an ultra wide-angle lens have been developed.

However, although it is known that a camera that captures a moving image of the related art has a camera shake correction function, when the camera body is imaged in a tilted state, the vertical direction and the zenith direction of the moving image capturing apparatus are There was a problem due to the difference between and. That is, when viewing a moving image captured with the camera body tilted, if the field of view is changed while keeping the vertical direction aligned, a complicated rotation occurs, which causes a 3 There was a possibility of giving discomfort such as plane sickness. The conventional technique of Patent Document 1 is for a still image and is not a technique capable of creating a moving image with a correct vertical direction. Therefore, it has been desired to develop a technology for generating a spherical video in which the vertical direction is correct regardless of the tilted state of the camera.

The present invention has been made in view of the inadequacies in the above-mentioned conventional art, and the present invention is data for generating moving image data whose inclination is corrected according to the detected inclination of the photographing means with respect to the reference direction. The purpose is to provide a file to store.

In order to solve the above problems, the present invention provides a file having the following features. This file is a file that stores data for generating moving image data whose coordinate system is converted by a computer. The file includes image data which is image data in a moving image format, tilt angle data which is tilt data with respect to a reference direction when the image is taken by the image pickup means, and additional information unique to the image pickup means. Have.

With the above configuration, it is possible to generate moving image data whose inclination is corrected according to the detected inclination of the image capturing unit with respect to the reference direction.

Sectional drawing of the omnidirectional camera which comprises the omnidirectional video imaging system by this embodiment. The hardware block diagram of the celestial sphere animation imaging system by this embodiment. FIG. 3 is a main functional block diagram relating to a spherical moving image capturing/recording function realized on the spherical moving image capturing system according to the present embodiment. 6 is a flowchart showing an imaging process performed on the omnidirectional camera side that constitutes the omnidirectional video imaging system according to the present embodiment. The figure explaining the projection relationship in the omnidirectional camera which used the fisheye lens in this embodiment. It is the schematic explaining the inclination of the spherical camera in this embodiment. 6 is a flowchart showing a recording process performed on the image processing device side that constitutes the spherical moving image capturing system according to the present embodiment. FIG. 6 is a diagram illustrating a configuration in which, in the preferred embodiment, only images of a part of frames that make up a moving image are expanded on a RAM during correction. 5 is a flowchart showing a conversion process performed on the image processing device side that constitutes the spherical moving image capturing system according to the present embodiment. The figure explaining the case where it represents by the (A) plane and the case where it represents by the (B) spherical surface about the spherical image format in this embodiment, respectively. FIG. 3 is a diagram illustrating an image acquired by each image sensor in the omnidirectional camera and an image generated by the image processing device in the present embodiment. The figure explaining the conversion table of the spherical image in this embodiment. The flowchart explaining the operation|movement flow of the correction process with respect to the conversion table of the spherical image in this embodiment. The figure which demonstrates the vertical correction calculation of the spherical image in this embodiment using (A) camera coordinate system and (B) global coordinate system. The figure explaining the frame of the moving image before inclination correction. FIG. 6 is a diagram illustrating a frame of a moving image after tilt correction.

Hereinafter, the present embodiment will be described, but the present embodiment is not limited to the embodiment described below. In the following embodiments, as an example of an image capturing apparatus and an image capturing system, a spherical camera including an image capturing body including two fish-eye lenses in an optical system, a spherical camera, and a spherical camera separated from the spherical camera. An explanation will be given by using a celestial sphere moving image capturing system including the image processing device.

[overall structure]
Hereinafter, the overall configuration of the celestial sphere moving image capturing system according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a cross-sectional view of a celestial sphere camera 110 that constitutes a celestial sphere video imaging system 100 according to the present embodiment. The omnidirectional camera 110 shown in FIG. 1 includes an imaging body 12, a housing 14 that holds the imaging body 12 and components such as a controller and a battery, and a shutter button 18 provided on the housing 14. ..

The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B, and two imaging elements 22A and 22B such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. It Each of the image forming optical systems 20 is configured as, for example, a fisheye lens of 7 elements in 6 groups. In the embodiment shown in FIG. 1, the fisheye lens has a total angle of view larger than 180 degrees (=360 degrees/n; optical system number n=2), and preferably has a field angle of 185 degrees or more, More preferably, it has an angle of view of 190 degrees or more. A combination of the wide-angle imaging optical system 20 and the image sensor 22 one by one is referred to as a wide-angle imaging optical system.

The optical elements (lens, prism, filter, and aperture stop) of the two imaging optical systems 20A and 20B have a positional relationship with respect to the image pickup elements 22A and 22B. More specifically, the optical axes of the optical elements of the imaging optical systems 20A and 20B are positioned orthogonal to the center of the light receiving area of the corresponding image pickup element 22, and the light receiving area corresponds to the fisheye lens. The positioning is performed so as to be the image plane of.

In the embodiment shown in FIG. 1, the image forming optical systems 20A and 20B have the same specifications and are combined in opposite directions so that their optical axes coincide with each other. The image pickup devices 22A and 22B convert the received light distribution into image signals and sequentially output image frames to the image processing means on the controller. As will be described later in detail, the images respectively captured by the image pickup devices 22A and 22B are transferred to the image processing device 150 and subjected to a synthesis process, whereby an image with a solid angle of 4π steradian (hereinafter referred to as “all sphere image”). Is generated). The celestial sphere image is a photograph of all directions that can be viewed from the photographing point. Then, the celestial sphere moving image is constituted by the continuous frames of the celestial sphere image. Here, in the embodiment to be described, the explanation will be given on the assumption that the spherical image and the spherical moving image are generated, but a so-called panoramic image and panoramic moving image in which only the horizontal plane is photographed at 360 degrees may be used.

FIG. 2A shows the hardware configuration of the omnidirectional camera 110 that constitutes the omnidirectional moving image capturing system 100 according to the present embodiment. The spherical camera 110 is connected via a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a moving image compression block 118, and a DRAM (Dynamic Random Access Memory) interface 120. The DRAM 132 and the acceleration sensor 136 connected via the external sensor interface 124.

The CPU 112 controls the operation and overall operation of each unit of the omnidirectional camera 110. The ROM 114 stores a control program and various parameters written in a code readable by the CPU 112. The image processing block 116 is connected to the two image pickup devices 130A and 130B (the image pickup devices 22A and 22B in FIG. 1), and the image signals of the images picked up by the respective image pickup blocks 130A and 130B are input. The image processing block 116 is configured to include an ISP (Image Signal Processor) and the like, and performs shading correction, Bayer interpolation, white balance correction, gamma correction, and the like on the image signal input from the image sensor 130.

The moving picture compression block 118 is an MPEG-4 AVC/H. It is a codec block that performs video compression and decompression such as H.264. The DRAM 132 provides a storage area for temporarily storing data when performing various kinds of signal processing and image processing. The acceleration sensor 136 detects triaxial acceleration components, and the detected acceleration components are used to detect the vertical direction and perform zenith correction of the omnidirectional image.

The spherical camera 110 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, and a serial block 128. An external storage 134 is connected to the external storage interface 122. The external storage interface 122 controls reading and writing with respect to the external storage 134 such as a memory card inserted in the memory card slot. A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with an external device such as a personal computer connected via the USB connector 138. The serial block 128 controls serial communication with an external device such as a personal computer, and is connected to a wireless NIC (Network Interface Card) 140.

When the power is turned on by operating the power switch, the control program is loaded into the main memory. The CPU 112 controls the operation of each unit of the device according to the program read into the main memory, and temporarily stores the data necessary for the control in the memory. As a result, the functional units and processes of the spherical camera 110 described later are realized.

FIG. 2B shows the hardware configuration of the image processing apparatus 150 that constitutes the omnidirectional video image capturing system 100 according to the present embodiment. The image processing apparatus 150 shown in FIG. 2B includes a CPU 152, a RAM 154, an HDD (Hard Disk Drive) 156, an input device 158 such as a mouse and a keyboard, an external storage 160, a display 162, and a wireless NIC 164. , And a USB connector 166.

The CPU 152 controls the operation of each unit and the overall operation of the image processing apparatus 150. The RAM 154 provides a work area for the CPU 152. The HDD 156 stores a control program, such as an operating system, an application responsible for processing on the image processing apparatus 150 side according to the present embodiment, which is written in a code readable by the CPU 152.

The input device 158 is an input device such as a mouse, a keyboard, a touch pad, and a touch screen, and provides a user interface. The external storage 160 is a removable recording medium mounted in a memory card slot or the like, and records various data such as moving image format image data and still image data. The display 162 displays the spherical moving image reproduced in response to the user operation on the screen. The wireless NIC 164 establishes a wireless communication connection with an external device such as the spherical camera 110. The USB connector 166 makes a USB connection with an external device such as the spherical camera 110.

When the image processing apparatus 150 is powered on and turned on, the control program is read from the ROM or the HDD 156 and loaded into the RAM 154. The CPU 152 controls the operation of each part of the device according to the control program read in the RAM 154, and temporarily stores the data necessary for the control in the memory. As a result, each functional unit and processing of the image processing apparatus 150 described later are realized.

[Spherical video recording function]
Hereinafter, with reference to FIGS. 3 to 16, the omnidirectional moving image capturing/recording function of the omnidirectional moving image capturing system 100 according to the present embodiment will be described. FIG. 3 shows main functional blocks 200 related to the spherical moving image capturing/recording function realized on the spherical moving image capturing system 100 according to the present embodiment.

As shown in FIG. 3, the functional block 210 of the omnidirectional camera 110 includes an imaging unit 212, a moving image compression unit 214, a tilt detection unit 216, a tilt recording unit 218, and an output unit 220. .. On the other hand, the functional block 250 of the image processing device 150 includes a reading unit 252, an image restoration unit 254, a tilt acquisition unit 256, a tilt correction unit 258, and a moving image generation unit 264.

First, the spherical camera 110 side will be described. The image pickup unit 212 is configured to include the above-described two wide-angle image pickup optical systems and controls the two image pickup elements 130A and 130B to sequentially pick up consecutive frames. The image picked up by each of the image pickup devices 130 is a fish-eye image in which the hemisphere of the whole celestial sphere is included in the visual field, and constitutes a partial image of the whole celestial sphere image. Hereinafter, the image captured by each of the image sensors 130 may be referred to as a partial image.

The moving image compression unit 214 is configured to include the moving image compression block 118, and compresses the continuous frames captured by the image capturing unit 212 into image data in a predetermined moving image format. The moving image compression format is not particularly limited, but H.264. H.264/MPEG-4 AVC (Advanced Video Coding), H.264. Examples include various moving image compression formats such as H.265/HEVC (High Efficiency Video Coding), Motion JPEG (Joint Photographic Experts Group), and Motion JPEG2000.

The Motion JPEG system is a format for expressing a moving image as a continuous still image, and by adopting this method, a high quality moving image can be obtained. On the other hand, H.264. H.264/MPEG-4 AVC and H.264. Since the H.265/HEVC can perform compression in the time axis direction, the processing efficiency is high, and the requirement for delay in writing to the external storage can be relaxed. On the side of the omnidirectional camera 110 that is handheld, it is difficult to mount high-performance hardware due to the demand for downsizing and cost reduction. Therefore, in the preferred embodiment, compression in the time axis direction can be performed. H.264 capable of suppressing the bit rate. H.264/MPEG-4 AVC and H.264. 265/HEVC can be preferably adopted.

In the embodiment to be described, the image capturing unit 212 independently outputs two fish-eye images captured by the two image capturing elements 130A and 130B at the same timing, and the moving image compressing unit 214 outputs two independent images. Image data of two moving image formats are independently generated from the frame of the fisheye image. However, the expression format of the image data is not particularly limited. In another embodiment, the image capturing unit 212 outputs a single image configured by joining two fish-eye images captured by the two image capturing elements 130A and 130B, and the moving image compression unit 214 outputs the single image. It is possible to compress image data in a moving image format in which an image including two fisheye images is used as a frame.

The tilt detection unit 216 includes an acceleration sensor 136 and detects the tilt of the celestial sphere camera 110 with respect to a predetermined reference direction. Here, the predetermined reference direction is typically a vertical direction, and is a direction in which gravitational acceleration acts. The tilt detection unit 216 outputs signals of each acceleration component of the triaxial acceleration sensor 136 to the tilt recording unit 218. The inclination recording unit 218 samples the acceleration component signal input from the inclination detection unit 216 in synchronization with a frame of moving image format image data, and obtains an inclination angle with respect to a predetermined reference direction. Then, it is recorded as time-series data having a rate equivalent to the frame rate of the moving image format image data.

In addition, in the embodiment to be described, it is assumed that the tilt angles correspond to the frames in the image data of the moving image in a one-to-one relationship, and the frames and the tilt angles are stored in synchronization. However, the rate of the tilt angle to be stored does not necessarily have to be the same as the frame rate. If they do not match, the tilt angle corresponding to the frame may be obtained by resampling at the frame rate. ..

Each of the moving image format image data generated by the moving image compression unit 214 and the tilt angle data recorded by the tilt recording unit 218 are temporarily stored in the DRAM 132. The output unit 220 writes each of the moving image format image data and the tilt angle data held in the DRAM 132 to the external storage 134 as two moving image files 230 and a tilt angle file 240 at the timing when the write size becomes appropriate. .. When the shooting ends, the two moving image files 230 and the tilt angle file 240 are closed, and the processing on the side of the omnidirectional camera 110 ends.

The external storage 134 is taken out from the spherical camera 110 and attached to the image processing apparatus 150. Next, the description of the image processing device 150 side will be shifted. The reading unit 252 reads, from the external storage 160 mounted in the image processing apparatus 150, the two moving image files 230 and the tilt angle file 240 written on the omnidirectional camera 110 side. The two read moving image files 230 are passed to the image restoration unit 254. Further, the read tilt angle file 240 is passed to the tilt acquisition unit 256. In addition to attaching/detaching the external storage 134, the image processing apparatus 150 is connected to the spherical camera 110 from the external storage 134 provided in the spherical camera 110 through the USB interface provided in the spherical camera 110. The two moving image files 230 and the tilt angle file 240 that have been written may be read.

The image restoration unit 254 decodes each read moving image file 230 by using a predetermined codec corresponding to each read moving image file 230, restores a frame of a still image forming the moving image, and tilts the frame. Input to the correction unit 258. The tilt acquisition unit 256 acquires the tilt angle corresponding to each frame restored by the image restoration unit 254 from the read tilt angle file 240, and inputs the tilt angle to the tilt correction unit 258.

The tilt correction unit 258 performs tilt correction on each frame of the still image restored by the image restoration unit 254 based on the tilt angle corresponding to each frame input from the tilt acquisition unit 256, and also from the fisheye image. Convert to spherical image. More specifically, the tilt correction unit 258 includes a conversion table correction unit 260 and an omnidirectional image generation unit 262.

On the side of the image processing device 150, each of the two fisheye images captured by the two image pickup devices 130A and 130B (each of which is passed to the image processing device as a frame of a moving image file) is converted into a spherical coordinate system. , The conversion table for generating the spherical image is prepared in advance. The conversion table is data created in advance by a manufacturer or the like according to a predetermined projection model based on design data of each lens optical system, etc., and it is assumed that the direction directly above the omnidirectional camera 110 matches the vertical line. , Data for converting a fisheye image into a spherical image. In the embodiment to be described, when the omnidirectional camera 110 is in a tilted state and the direction directly above does not coincide with the vertical line, the zenith correction is reflected by correcting this conversion data according to the tilt.

The partial images included in each of the moving image format image data are image data captured by a two-dimensional image sensor having a light receiving region forming an area, and are image data represented by a plane coordinate system (x, y). (FIG. 5(B), and the first partial image and the second partial image shown in FIG. 11). On the other hand, the corrected image, which has been subjected to the zenith correction using the conversion table, is expressed in a spherical coordinate system (a polar coordinate system having a radius vector of 1 and two deviation angles θ and φ). The image data is in the spherical image format (the spherical image shown in FIGS. 10 and 11).

The conversion table may be stored in the moving image file and acquired together with the reading of the moving image file, or may be downloaded from the omnidirectional camera 110 to the image processing apparatus 150 in advance. Alternatively, a basic conversion table common to all individuals is prepared in advance on the image processing apparatus side, and the difference data for obtaining the conversion table specific to each individual celestial sphere camera 110 is written in the moving image file to create a moving image file. A unique conversion table may be restored together with the reading of

The conversion table correction unit 260 detects that the zenith direction of the image (the direction pointing to the point directly above the observer) is the detected vertical line (the straight line formed by the vertical direction in which gravity acts) according to the acquired tilt angle. The above-mentioned conversion table is corrected so that Therefore, if the fisheye image is converted to a spherical image using the corrected conversion table, the spherical image will be corrected in a manner that the zenith direction matches the vertical line according to the inclination. Is generated. The omnidirectional image generation unit 262 uses the conversion table corrected by the conversion table correction unit 260 to convert the two partial images corresponding to the restored frames into omnidirectional images in which the zenith correction is reflected. Each frame of the spherical image is converted and output to the moving image generation unit 264. The conversion table correction unit 260 and the celestial sphere image generation unit 262 correct the zenith direction of each of the corrected images so that the zenith direction matches the vertical line.

The moving image generation unit 264 encodes in a predetermined moving image compression format based on the omnidirectional images of each of the plurality of frames corrected by the tilt correction unit 258, generates final moving image data, and writes it as a moving image file 270. The moving image file 270 is moving image data of spherical images. The moving image compression format may be any format including the formats described above.

At the time of viewing, on the basis of the generated moving image data, an image of a specific field of view designated in the omnidirectional sphere is reproduced and displayed on the display application. In the generated moving image data, the zenith direction of the image is fixed with respect to the vertical line, so even if the display field of view is changed while viewing the moving image data, only the rotation as if the field of view was changed occurs. Is. As a result, it is possible to suppress the occurrence of complicated rotation accompanied by the change of the field of view and the change of the inclination, and it is possible to reduce the possibility of giving discomfort such as three-dimensional sickness.

Hereinafter, the details of the processing on the omnidirectional camera 110 side and the processing on the image processing apparatus 150 side according to the present embodiment will be separately described with reference to FIGS. 4 and 7. FIG. 4 is a flowchart showing the imaging process performed on the omnidirectional camera 110 side that constitutes the omnidirectional video imaging system 100 according to the present embodiment. The process shown in FIG. 4 is started from step S100 in response to an instruction to start imaging, such as pressing the moving image shooting button of the omnidirectional camera 110. In step S101, the omnidirectional camera 110 causes the image capturing unit 212 to read image data for one frame from the image sensor 130.

FIG. 5 is a diagram illustrating a projection relationship in the omnidirectional camera 110 using a fisheye lens. In the present embodiment, an image captured by one fish-eye lens is a hemispherical azimuth photographed from the photographing point. Further, as shown in FIG. 5A, the fisheye lens generates an image with an image height h corresponding to the incident angle φ with respect to the optical axis. The relationship between the image height h and the incident angle φ is determined by a projection function according to a predetermined projection model. The projection function differs depending on the property of the fisheye lens, but in the fisheye lens of the projection model called the equidistant projection method, f is the focal length and is expressed by the following equation (1).

Other than the above, the projection model includes a central projection method (h=f·tan φ), a stereoscopic projection method (h=2f·tan (φ/2)), and an equal solid angle projection method (h=2f·sin(φ/2). )) and the orthographic projection method (h=f·sin φ). In either method, the image height h of the formed image is determined according to the incident angle φ from the optical axis and the focal length f. Further, in the present embodiment, it is assumed that a so-called circumferential fisheye lens having a smaller image circle diameter than the diagonal line of the image is adopted, and the partial image obtained is approximately a hemisphere of the photographing range as shown in FIG. 5B. It becomes a planar image including the entire image circle onto which the minutes are projected.

In the embodiment to be described, the image data for one frame includes two plane images (FIG. 11 in FIG. 11) including the entire image circles captured by the two image sensors 130A and 130B, each of which is a substantially hemispherical portion of the shooting range. The first partial image and the second partial image).

Referring back to FIG. 4, in step S102, the omnidirectional camera 110 causes the image capturing unit 212 to perform image processing on the captured image data of one frame and store the image data in the DRAM 132. First, an optical black correction process, a defective pixel correction process, a linear correction process, a shading process, and the like are performed on the Bayer RAW image captured from the image sensor 130. Then, white balance processing, gamma correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement processing, and color correction processing are performed.

In step S103, the omnidirectional camera 110 detects the tilt angle in synchronization with the captured frame by the tilt recording unit 218, and stores the tilt angle in the memory.

FIG. 6 is a schematic diagram illustrating the tilt of the omnidirectional camera 110 according to the present embodiment. In FIG. 6, the vertical direction corresponds to the z axis in the Cartesian coordinates in the xyz three-dimensional direction of the global coordinate system. When this direction coincides with the direction directly above the spherical camera 110 shown in FIG. 6, the camera is not tilted. If they do not match, the omnidirectional camera 110 is tilted.

That is, the tilt angle from the gravity vector (hereinafter, tilt angle α) and the tilt angle β on the xy plane (hereinafter, tilt angle β) are obtained by the following formula (2) using the output of the acceleration sensor. .. Here, Ax is the value of the x0 axis direction component of the camera coordinate system of the acceleration sensor 136, Ay is the value of the y0 axis direction component of the camera coordinate system of the acceleration sensor 136, and Az is the camera coordinate system of the acceleration sensor 136. It is the value of the z0 axis direction component. In step S103, the tilt angles α and β are obtained from the values of the axial direction components of the acceleration sensor 136 and stored.

Referring again to FIG. 4, in step S104, the omnidirectional camera 110 inputs the captured two image data for one frame into the moving image compression unit 214, respectively, and writes the two image data in the moving image format into the memory. In step S105, the omnidirectional camera 110 determines whether it is necessary to write a moving image. When it is determined in step S105 that the predetermined writing size has been reached and the moving image needs to be written (YES), the process branches to step S106. In step S106, the omnidirectional camera 110 records the two image data in the moving image format on the DRAM 132 in the external storage 134 as moving image files by the output unit 220. On the other hand, if it is determined in step S105 that the predetermined writing size has not been reached and it is not necessary to write a moving image (NO), the process directly proceeds to step S107.

In step S107, the omnidirectional camera 110 determines whether tilt angle data needs to be written. When it is determined in step S107 that the predetermined writing size has been reached and the tilt angle data needs to be written (YES), the process branches to step S108. In step S108, the omnidirectional camera 110 records the tilt angle data on the DRAM 132 in the external storage 134 as a tilt angle file by the output unit 220. On the other hand, if it is determined in step S107 that the predetermined writing size has not been reached and it is not necessary to write the tilt angle data (NO), the process directly proceeds to step S109.

In step S109, the omnidirectional camera 110 determines whether or not the shooting has been completed. If it is determined in step S109 that the user has not issued a shooting end instruction and the shooting has not ended (NO), the process loops to step S101 to continue the process for the next frame. .. On the other hand, if it is determined in step S109 that the shooting end instruction has been received from the user and the shooting has ended (YES), the process proceeds to step S110.

In step S110, the omnidirectional camera 110 writes additional information necessary for converting an image on the image processing device 150 side into a moving image file, and in step S111, the imaging process on the omnidirectional camera 110 side ends. To do. Here, the additional information necessary for converting the image is data for restoring the conversion table suitable for the specific celestial sphere camera 110 that has captured the moving image, on the side of the image processing apparatus 150 that is the output destination. .. The additional information may be the conversion table as it is, may be difference data of the conversion table of each individual corresponding to the reference conversion table, or may be the corresponding conversion from a plurality of predefined conversion tables. It may be identification information that identifies the table.

FIG. 7 is a flowchart showing a recording process performed on the image processing device 150 side that constitutes the omnidirectional video image capturing system 100 according to the present embodiment. The recording process shown in FIG. 7 is performed in response to the fact that the moving image file and the tilt angle file are designated and the encode recording start instruction is performed in the application for generating the spherical moving image on the image processing device 150. It starts from S200.

In step S201, in the image processing apparatus 150, the reading unit 252 reads two moving image files from the external storage 160, and the image restoring unit 254 restores two image data of a predetermined number of frames. In step S202, the image processing apparatus 150 causes the reading unit 252 to read the tilt angle file from the external storage 160, and the tilt acquiring unit 256 acquires the tilt angle corresponding to each of the restored predetermined number of frames.

If all the frames that make up the moving image file 230 are expanded on the RAM 154, the memory will be consumed excessively. Therefore, in the preferred embodiment, in the correction, only the image data and the tilt angle data of a part of the frames forming the entire moving image file, which are based on the frame serving as the starting point, are expanded on the RAM 154. it can. The unit to be expanded is a unit referred to by a GOP (Group of Picture) or a concept equivalent to it as shown in FIG. 8 when a format that performs compression on the time axis such as MPEG-4/AVC is adopted. It is efficient to deploy it on memory with.

The GOP includes I pictures, P pictures, and B pictures. The I picture is a picture that is a starting point of reproduction and is a picture encoded by intraframe predictive encoding. A P picture is a picture encoded by interframe prediction from the previous I picture or P picture. A B picture is a picture that is coded by inter-frame prediction using a previous (forward direction), a subsequent (reverse direction), or a preceding/rearward (bidirectional) picture. Since a GOP includes at least one I picture, by expanding a frame in units of GOP and expanding a minimum number of frames, it is possible to restore a predetermined number of still images.

Referring to FIG. 7 again, in step S203, the image processing apparatus 150 reads additional information from within the moving image file and prepares a conversion table unique to the specific omnidirectional camera 110 that captured the moving image file. In step S204, the image processing apparatus 150 performs a conversion process on each of a predetermined number of frames of image data restored from each moving image file, based on the acquired tilt angle and the read additional information.

FIG. 9 is a flowchart showing a conversion process performed on the image processing device 150 side which constitutes the celestial sphere moving image capturing system 100 according to the present embodiment. The conversion process shown in FIG. 9 is called in step S204 shown in FIG. 7 and started from step S300. In the loop of steps S301 to S307, the processing of steps S302 to S306 is executed for each frame.

In step S302, the image processing apparatus 150 sets the tilt angles α and β corresponding to the frame. In step S303, the image processing apparatus 150 causes the conversion table correction unit 260 to correct the conversion table according to the acquired inclination angles α and β. The details of the conversion table correction process will be described later.

In step S304, the image processing apparatus 150 inputs the two image data of the restored frame. In step S305, the image processing apparatus 150 uses the corrected conversion table to convert the two image data of the restored frame (each including a fisheye image) by the omnidirectional image generation unit 262. , Two corrected image data (two spherical images corresponding to each hemisphere corresponding to each fisheye image) are generated. In step S306, the image processing apparatus 150 combines the two corrected image data of the frame (all-sphere image in charge of each hemisphere) to generate final image data (all-sphere image). ..

FIG. 10 is a diagram for explaining the spherical image format according to the present embodiment when it is represented by the (A) plane and when it is represented by the (B) spherical surface. In FIG. 10, the format of the omnidirectional image is, as shown in FIG. 10(A), when developed on a plane, pixels corresponding to angular coordinates with a horizontal angle of 0 to 360 degrees and a vertical angle of 0 to 180 degrees. It is an image with values. The angular coordinates are associated with each of the coordinate points on the spherical surface shown in FIG. 10B, and are like the latitude/longitude coordinates on the globe.

The relationship between the plane coordinate value of the image captured by the fisheye lens and the spherical coordinate value of the spherical image may be associated with each other by using the projection function f (h=f(θ)) as described in FIG. it can. As a result, by converting and combining (combining) the two partial images captured by the fisheye lens, it is possible to create a spherical image as shown in FIG.

Next, a process of generating a celestial sphere image using an image actually captured by a fisheye lens will be described. FIG. 11 is an image acquired by each of the image pickup devices 130A and 130B via the two fisheye lenses in the omnidirectional camera 110 according to the present embodiment, converted by the conversion table in the image processing device 150, and the two images after conversion are combined. It is a figure explaining each image generated by doing.

In FIG. 11, the partial image captured by each of the image sensors 130A and 130B via the two fisheye lenses is first subjected to the process of step S305 described in FIG. 9, that is, the image conversion process using the corrected conversion table. Converted into two spherical images. At this point, the image has an expression format that matches the spherical image format, that is, an expression corresponding to FIG.

The final one spherical image shown in FIG. 11 is generated by the process of step S306 shown in FIG. 9, that is, the image combining process after the conversion of the two images. That is, the two spherical images in charge of each hemisphere are superimposed and combined with the overlapping portion as a key to generate a final spherical image.

FIG. 11 also shows the correspondence relationship between the position of the pixel in each partial image and the position of the pixel on the corresponding omnidirectional image. When the omnidirectional camera 110 is correctly placed in the vertical direction and is photographed without tilting, and the direction directly above the omnidirectional camera 110 is on the vertical line, distortion correction is simply performed, as shown in FIG. , It is possible to obtain an image in which the zenith recognized by the photographer matches the horizon. At this time, the horizon is located at the center even in the partial image, and is arranged at the position corresponding to the equator of the globe in the spherical image as well.

Referring to FIG. 9 again, upon exiting the loop of steps S301 to S307, this processing ends in step S308 and returns to the processing of the calling source shown in FIG.

Referring to FIG. 7 again, in step S205, the image processing apparatus 150 causes the moving image generation unit 264 to encode the converted image data of the predetermined number of frames to generate the tilt-corrected moving image data, and to generate the spherical image. The image is written in the moving image file 270. In step S206, the image processing apparatus 150 determines whether or not the last frame of the moving image file 230 to be read has been reached. If it is determined in step S206 that the last frame has not been reached yet (NO), the process loops to step S201 to proceed to the next frame group. On the other hand, if it is determined in step S206 that the last frame has been reached (YES), the process branches to step S207 and the present recording process ends.

Note that in the preferred embodiment, between step S204 and step S205, noise suppression image processing in the time direction, image correction processing for noise and contour blurring, such as mosquito noise and block noise, which are derived from a moving image compression codec, and hue are performed. It is preferable to execute processing such as the correction processing of. As a result, it is expected that the final image quality will be improved.

Hereinafter, with reference to FIGS. 12 to 14, a process of correcting the conversion table of the omnidirectional image in the present embodiment according to the recorded tilt angle of the omnidirectional camera 110 will be described.

FIG. 12 is a diagram for explaining a conversion table for spherical images in the present embodiment. FIG. 12A is a diagram for explaining a conversion table showing a matrix of image coordinate values before and after conversion. In addition, FIG. 12B is a diagram illustrating the relationship between the coordinate values of the converted image and the coordinate values of the pre-converted image.

As shown in FIG. 12A, the conversion table used for the image conversion described in step S303 shown in FIG. 9 corresponds to the coordinate values of the converted image (θ, φ) (pix: pixel) and the corresponding values. The data set with the coordinate values (x, y) (pix) of the pre-conversion image to be obtained is included for all the coordinate values of the post-conversion images. Here, as the conversion table, a table-shaped data structure is shown in the embodiment to be described, but the conversion table does not necessarily have to be a table-shaped data structure. That is, it may be conversion data.

According to the conversion table shown in FIG. 12A, the converted image can be generated from the captured partial image (pre-conversion image). Specifically, as shown in FIG. 12B, from the correspondence relationship between the conversion table before conversion and the conversion table after conversion (FIG. 12A), each pixel of the converted image has a coordinate value (θ, φ). It can be generated by referring to the pixel value of the coordinate value (x, y) (pix) of the image before conversion corresponding to (pix).

The conversion table before the correction processing reflects the distortion correction, assuming that the direction directly above the omnidirectional camera 110 is aligned with the vertical line, and is corrected by performing the correction processing according to the tilt angle. The subsequent conversion table reflects the zenith correction for matching the direction directly above the omnidirectional camera 110 with the vertical line.

FIG. 13 is a flowchart illustrating the operation flow of the conversion table correction processing for the spherical image in the present embodiment. The correction process shown in FIG. 13 is called in step S303 shown in FIG. 9 and started from step S400.

In FIG. 13, in step S401, the image processing apparatus 150 causes the conversion table correction unit 260 to acquire the tilt angle (α, β) corresponding to the frame to be processed. Next, in step S402, the image processing apparatus 150 causes the conversion table correction unit 260 to set the input values (θ1, φ1) of the conversion table. Note that, in FIG. 13, the camera coordinate system is expressed as (θ0, φ0), and the global coordinate system is expressed as (θ1, φ1), and (θ, φ) that is a parameter value depending on the coordinate system is defined. Notated separately. That is, here, the global coordinate system (θ1, φ1) is set.

Next, in step S403, the conversion table correction unit 260 of the image processing apparatus 150 performs the vertical correction calculation to convert the global coordinate system (θ1, φ1) of the input value into the camera coordinate system (θ0, φ0). Convert to.

Here, the vertical correction calculation will be described. FIG. 14 is a diagram for explaining the vertical correction calculation of the omnidirectional image in the present embodiment using (A) camera coordinate system and (B) global coordinate system.

In FIG. 14, the three-dimensional Cartesian coordinates of the global coordinate system are represented by (x1, y1, z1), the spherical coordinates are represented by (θ1, φ1), and the three-dimensional Cartesian coordinates of the camera coordinate system are represented by (x0, y0, z0), The spherical coordinates are expressed as (θ0, φ0).

In the vertical correction calculation, conversion from spherical coordinates (θ1, φ1) to spherical coordinates (θ0, φ0) is performed using the following equations (3) to (8). First, in order to correct the inclination, it is necessary to perform rotation conversion using three-dimensional Cartesian coordinates. Therefore, using the following equations (3) to (5), the spherical coordinates (θ1, φ1) to three-dimensional Conversion to Cartesian coordinates (x1, y1, z1) is performed.

Next, using the tilt angles (α, β), the global coordinate system (x1, y1, z1) is changed to the camera coordinate system (x0, y0, z0) by the rotational coordinate conversion (Equation (6)). Convert. In other words, the above equation (6) gives the definition of the tilt angle (α, β).

This means that when the global coordinate system is first α-rotated around the z-axis and then β-rotated around the x-axis, it becomes the camera coordinate system. Finally, using the above equations (7) and (8), conversion is performed to return the three-dimensional orthogonal coordinates (x0, y0, z0) of the camera coordinate system to spherical coordinates (θ0, φ0).

Next, another embodiment of the vertical correction calculation will be described. In the vertical correction calculation according to another embodiment, in the vertical correction calculation, conversion from spherical coordinates (θ1, φ1) to spherical coordinates (θ0, φ0) can be performed using the following equations (9) to (16). it can. In another embodiment, the vertical correction calculation can be speeded up. The conversion formulas of the formulas (3) to (8) are rewritten as the formulas (9) to (16).

That is, since the rotations α and γ about the z axis are the rotations of θ of the spherical coordinates (θ, φ), the rotation transformation can be performed by simple addition and subtraction without converting to the Cartesian coordinate system. Therefore, the speed can be increased. Therefore, only the rotation conversion of the rotation β about the x-axis requires the conversion using the orthogonal coordinate system. Thereby, the calculation speed can be increased.

In the above-described embodiment, the coordinate conversion is performed by the vertical correction calculation. However, in still another embodiment, by holding a plurality of conversion tables so that the conversion tables have different values depending on the tilt angles (α, β), the processing of the vertical correction calculation is omitted, and the speed is further increased. Can be planned. That is, the tilt angle (α, β) is a three-dimensional real vector in principle, but a conversion table is prepared only for a specific (α, β), and the detected tilt angle ( By adopting the table closest to α, β), all cases can be dealt with. Alternatively, it is also effective to extract a plurality of tables close to the detected tilt angles (α, β) and perform an interpolation calculation such as weighting or difference according to the closeness. As a result, the conversion table can be corrected only by a relatively simple calculation, which is an interpolation calculation, and the processing required for the calculation can be suppressed.

Referring again to FIG. 13, subsequently, in step S404, the image processing apparatus 150 causes the conversion table correction unit 260 to convert (θ0, φ0), which is the camera coordinate system, into the conversion table before correction. Convert to the coordinate value (x, y) of the image before conversion. It is premised that a conversion table for generating an accurate spherical image created under the condition that the camera is not tilted is prepared in advance.

Next, in step S405, the conversion table correction unit 260 of the image processing apparatus 150 uses the input values (θ1, φ1) of the global coordinate system and the finally calculated coordinate values (x, y) before correction. Are stored as the corresponding coordinate sets in the modified conversion table.

Next, in step S406, the image processing apparatus 150 causes the unprocessed input value (θ1, φ1), that is, the corresponding input value (x, y) in the global coordinate system for which the uncorrected coordinate value (x, y) has not been calculated. It is determined whether or not θ1, φ1) remains. If it is determined in step S406 that the remaining values remain (YES), the process loops to step S402 so that the input value (θ1, φ1) of the global coordinate system is set as the next value.

On the other hand, if it is determined in step S406 that there is no remaining (NO), the process branches to step S407, the present process ends, and the process returns to the calling process shown in FIG. In this case, calculation of the uncorrected coordinate value (x, y) corresponding to each pixel of the celestial sphere image format having the input value (θ1, φ1) of the global coordinate system as the coordinate value is completed.

Hereinafter, with reference to FIGS. 11, 15 and 16, a frame of a moving image obtained by a process of correcting a specific inclination will be described. FIG. 15 shows a frame of a moving image before tilt correction (one spherical image is converted by using a conversion table that has not been corrected), and FIG. 16 shows a frame of a moving image after tilt correction (all spherical image). It is a figure which shows what was converted about the image using the corrected conversion table. Further, FIG. 11 exemplifies a case where the omnidirectional camera 110 is not tilted and the vertical direction coincides with the vertical line, and FIGS. 15 and 16 show the omnidirectional camera 110 tilted and the upward direction. The case where the vertical lines do not match is illustrated.

As described above, when the omnidirectional camera 110 is correctly placed in the vertical direction, as shown in FIG. 11, it is possible to capture an image such that the zenith and the horizontal line recognized by the person who captured the image match. When the omnidirectional camera 110 is mounted on a fixture and adjustment is possible using a spirit level or the like for image capturing, correct vertical image capturing is possible.

However, in general, it is difficult for a human hand to take a picture horizontally and vertically when the picture is taken by hand. Thus, if the main body camera is imaged while tilted, the zeniths of the images are displaced and the horizon is distorted, as shown in FIG. 15, unless the tilt is corrected. At this time, the center line in the partial image coincides with the equator of the spherical image as it is, but the horizontal line recognized by the photographer is distorted as a curve in the spherical image. If, as shown in FIG. 15, a moving image is viewed without correction, if the visual field is changed during viewing, three-dimensional motion sickness may occur due to the influence of the deviation between the vertical direction and the central axis of the spherical camera 110. It is accompanied by such discomfort.

On the other hand, in the present embodiment, the frames shown in FIG. 16 are obtained by converting the images of all the frames so that the vertically upward direction and the upward direction of the zenith coincide with each other and then performing encoding. .. That is, the horizon, which is a curve on the partial image and is recognized by the photographer, comes to coincide with the equator in the spherical image. By converting in this way, it becomes possible to create a spherical video that does not give the user a feeling of strangeness such as three-dimensional motion sickness when changing the field of view during viewing.

According to the embodiments described above, there are provided an imaging system, an imaging device, a program, and a system capable of detecting the tilt of the imaging means with respect to the quasi-direction and generating tilt-corrected moving image data according to the detected tilt. be able to.

According to the omnidirectional moving image capturing/recording function according to the present embodiment, tilt angle data in which the tilt angle of the omnidirectional camera 110 is associated with each frame of a moving image is recorded at the same frequency as that of recording each frame of the moving image. It is saved as a separate file. Then, while synchronizing the data of the tilt angle file with the frame, the tilt is corrected in each frame of the moving image while referring, and the moving image is displayed and re-encoded. For this reason, it is possible to obtain spherical video data with the correct vertical direction. Therefore, when the user changes the field of view, spherical video data that does not cause discomfort or discomfort such as three-dimensional motion sickness is obtained. Can be provided.

In the case of a still image, it is possible to cope with the display application side by forming the field of view in consideration of the correction of the tilt angle without performing the tilt correction. However, with moving image data, typically, the display content constantly changes at 15 to 30 frames/sec. As described above, by encoding a spherical video in which the tilt angle has been corrected in advance and generating a moving image file, it is preferable that the spherical image does not give the viewer an uncomfortable feeling such as three-dimensional motion sickness. It becomes possible to display a spherical moving image.

In the above-described embodiment, two partial images captured by the lens optical system having an angle of view larger than 180 degrees are overlapped and combined, but in another embodiment, one or a plurality of lens optical systems are used. It may be applied to superposition synthesis of three or more partial images taken by the system. Further, in the above-described embodiment, the image pickup system using the fisheye lens has been described as an example, but the present invention may be applied to the omnidirectional moving image pickup system using the super wide-angle lens.

Further, in the above-described embodiment, the images of a plurality of captured frames are temporarily compressed into a moving image format, and the compressed moving image format image data and time series data of the tilt are output from the omnidirectional camera 110. And Then, the image processing apparatus 150 side reads the output moving image format image data and the time series data of the inclination, restores the images of each of the plurality of frames from the read moving image format image data, and restores the restored plurality of frames. Inclination correction was performed on each image of each frame.

However, in another embodiment, it is possible to directly output the frames of continuous still images as a group without passing through the moving image format image data. That is, in another embodiment, the omnidirectional camera 110 outputs images of a plurality of frames and time series data of inclination. On the image processing device 150 side, it is possible to read the images of a plurality of frames output by the omnidirectional camera 110 and the time series data of the tilt, and perform tilt correction on the read images of each of the frames.

Further, in the above-described embodiment, the moving image file and the tilt angle file are passed from the omnidirectional camera 110 to the image processing apparatus 150 via the external storage. However, the method of transmitting the moving image data and the tilt angle data is not particularly limited, and the moving image data and the tilt angle data are transferred from the omnidirectional camera 110 to the image processing apparatus 150 by the above-mentioned communication means such as USB connection or wireless LAN connection. Good.

Further, in the above-described embodiment, moving image format image data is created for the images captured by each of the plurality of imaging elements 130A and 130B of the omnidirectional camera 110, and the image processing device 150 combines the images. However, it is not limited to this. That is, as described above, on the omnidirectional camera 110 side, images captured by the image pickup devices 130A and 130B are combined in advance, image data in a moving image format is generated from the combined images, and the data is processed by the image processing apparatus. You may make it process by 150. In this case, the process of synthesizing the images by the image processing device is unnecessary.

Further, in the above-described embodiment, the omnidirectional camera 110 and the image processing device 150 are described as separate bodies as the omnidirectional moving image capturing system, but the invention is not limited to this. That is, the omnidirectional video image capturing system may be configured by integrating the omnidirectional camera 110 and the image processing device 150. In this case, the CPU 112 and the CPU 152 may be configured by a common CPU. Further, the data transfer bus 142 and the bus 168 are configured by one line, and each hardware member is connected by the common bus. Therefore, it is not necessary to transfer the data using the external storage or the wired or wireless communication means. Furthermore, in this case, if the tilt angle and the corresponding image are obtained without recording the moving image file and the tilt angle file separately, the image data whose tilt is corrected (converted) according to the tilt angle May be generated, moving image data may be generated from these image data, and the moving image data may be recorded in the DRAM 132, the RAM 154, the HDD 156, or displayed on the display 162.

Furthermore, in the above embodiment, when the vertical direction and the direction directly above the celestial sphere camera 110 coincide with each other, there is no inclination, but the present invention is not limited to this. In addition to the vertical direction, for example, a horizontal direction or another desired direction is set as a reference direction, and the inclination of the image is corrected based on the inclination with respect to the reference direction with respect to a predetermined object such as the omnidirectional camera 110 and the image pickup devices 130A and 130B. You may do so. Although the acceleration sensor is used to detect the tilt in the above-described embodiment, the omnidirectional camera 110 or the image pickup device fixed to the omnidirectional camera 110 by another tilt sensor such as a combination of the acceleration sensor and the geomagnetic sensor. You may make it detect inclination of 130A, 130B, a sensor itself, etc.

The functional unit can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark) or an object-oriented programming language, such as ROM, EEPROM, or EPROM. , Flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, or other device-readable recording medium, or through an electric communication line. It can be distributed. Further, a part or all of the above-mentioned functional unit can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA), or as an ASIC (application specific integration). Circuit configuration data (bit stream data) to be downloaded to the PD in order to realize the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, and VHDL (Very High Speed Integrated Circuits). Hardware Description Language), Verilog-HDL, and the like can be distributed by a recording medium as data.

Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art can think of other embodiments, additions, changes, deletions, and the like. The present invention can be modified within the scope of the present invention and is included in the scope of the present invention as long as the effects and advantages of the present invention are exhibited in any of the embodiments.

(Appendix)
(Appendix 1)
An imaging system,
An image pickup means for picking up images of a plurality of frames;
Detection means for detecting the inclination of the imaging means with respect to the reference direction,
Recording means for recording the time series data of the inclination detected by the detecting means,
Correction means for performing tilt correction on the images of each of the plurality of frames imaged by the imaging means, based on the time-series data of the inclination recorded by the recording means and the additional information unique to the imaging means,
And a moving image generating unit that generates moving image data based on the images of each of the plurality of frames corrected by the correcting unit.
(Appendix 2)
The imaging system further comprises
A moving image compression unit that compresses images of a plurality of frames captured by the image capturing unit into a moving image format;
Output means for outputting the image data in the moving image format compressed by the moving image compression means and the time series data of the inclination recorded by the recording means;
Reading means for reading the moving image format image data output by the output means and the time series data of inclination;
Restoring means for restoring images of each of a plurality of frames from the moving image format image data read by the reading means, the correcting means based on the time series data of the inclination read by the reading means, The imaging system according to appendix 1, wherein tilt correction is performed on the images of each of the plurality of frames restored by the restoring unit.
(Appendix 3)
The imaging system further comprises
Output means for outputting the images of the plurality of frames imaged by the imaging means, and the time series data of the inclination recorded by the recording means,
A read unit for reading the images of the plurality of frames output by the output unit and the time series data of the tilt, and the correction unit based on the time series data of the tilt read by the read unit. The imaging system according to appendix 1, wherein tilt correction is performed on the images of each of the plurality of frames read by the reading unit.
(Appendix 4)
The image pickup device, the detection device, the recording device, and the output device are included in an image pickup device, and the reading device, the correction device, and the moving image generation device are included in an image processing device separated from the image pickup device. The imaging system according to appendix 2 or 3.
(Appendix 5)
The image pickup unit includes a first wide-angle image pickup optical system and a second wide-angle image pickup optical system, and an image of a frame picked up by the image pickup unit is a first image picked up by the first wide-angle image pickup optical system. And a second partial image captured by the second wide-angle imaging optical system, wherein the correction means includes:
Correction means for correcting the conversion data for converting the first partial image and the second partial image into a spherical coordinate system according to the inclination.
A celestial sphere image generation means for converting the first partial image and the second partial image into a spherical coordinate system using the conversion data corrected by the correction means, and generating a omnidirectional image, The imaging system according to any one of appendices 1 to 4, wherein the moving image data is moving image data of the spherical image.
(Appendix 6)
6. The image pickup system according to any one of appendices 1 to 5, wherein the correction unit develops an image of a part of frames, which is based on a frame serving as a starting point, of a frame forming the whole on a memory in correction. ..
(Appendix 7)
The reference direction is a vertical direction, and the correction unit corrects the zenith of the images of the plurality of frames corrected by the correction unit so that the zenith coincides with the vertical line. The imaging system according to any one of 1.
(Appendix 8)
The imaging system according to any one of appendices 1 to 7, wherein the detection unit includes an acceleration sensor.
(Appendix 9)
An imaging device for capturing an image,
An image pickup means for picking up images of a plurality of frames;
Detection means for detecting the inclination of the imaging means with respect to the reference direction,
Recording means for recording the time series data of the inclination detected by the detecting means,
First output means for outputting images of each of the plurality of frames imaged by the imaging means and additional information unique to the imaging means;
Second output means for outputting the time series data of the inclination recorded by the recording means, and the images of each of the plurality of frames output by the first output means are output by the second output means. An image pickup apparatus, wherein tilt correction is performed based on time series data of the tilt and the additional information.
(Appendix 10)
A program for realizing an image processing device for generating an image,
First reading means for reading image data including images of a plurality of frames captured by the image capturing device;
Second reading means for reading time series data of the inclination of the image pickup device with respect to the reference direction recorded by the image pickup device corresponding to the image data;
Based on the time-series data of the inclination read by the second reading means and the additional information unique to the image pickup apparatus, the inclination of each of the plurality of frames included in the image data read by the first reading means is tilted. A program that causes a correction unit that performs correction and a function as a moving image generation unit that generates moving image data based on images of each of a plurality of frames corrected by the correction unit.
(Appendix 11)
An image pickup means for picking up an image of the object and outputting it as a plurality of images;
Detection means for detecting the inclination when the object is imaged,
A moving image generating means for generating a moving image from the plurality of images,
Output means for outputting the additional information peculiar to the image pickup means for performing the inclination correction on the moving image by using the information about the inclination, the moving image, and the information about the inclination
An image pickup apparatus including.
(Appendix 12)
12. The image pickup apparatus according to appendix 11, wherein the output unit outputs the plurality of images forming the moving image and information regarding a tilt when each image of the plurality of images is picked up in association with each other.
(Appendix 13)
A plurality of the image pickup means,
The moving image generating means generates a moving image for each of the plurality of images output from the plurality of image capturing means,
The output means outputs each of the moving images,
The imaging device according to appendix 11 or 12.
(Appendix 14)
An image pickup means for picking up an image of the object and outputting it as a plurality of images;
Detection means for detecting the inclination when the object is imaged,
Image conversion means for converting the plurality of images according to the inclination and additional information specific to the image pickup means;
A moving image generating means for generating a moving image from the plurality of converted images;
System with.
(Appendix 15)
A plurality of the image pickup means,
The image conversion means converts a plurality of images output from the plurality of image pickup means,
The moving image generating unit generates a moving image from an image obtained by combining one of the converted plurality of images captured by the plurality of image capturing units at the same timing,
The system according to attachment 14.

Reference numeral 12... Image pickup body, 14... Housing, 18... Shutter button, 20... Imaging optical system, 22... Image pickup element, 100... Whole sphere moving image pickup system, 110... Whole sphere camera, 112... CPU, 114... ROM, 116... Image processing block, 118... Video compression block, 120... DRAM interface, 122... External storage interface, 124... External sensor interface, 126... USB interface, 128... Serial block, 130... Image sensor, 132... DRAM, 134... External storage, 136... Acceleration sensor, 138... USB connector, 140... Wireless NIC, 142... Bus, 150... Image processing device, 152... CPU, 154... RAM, 156... HDD, 158... Input device, 160... External Storage, 162... Display, 164... Wireless NIC, 166... USB connector, 168... Bus, 200... Functional block, 210... Functional block, 212... Imaging section, 214... Video compression section, 216... Tilt detection section, 218... Tilt Recording unit 220... Output unit 230... Video file 240... Tilt angle file 250... Functional block 252... Readout unit 254... Image restoration unit 256... Tilt acquisition unit 258... Tilt correction unit 260... Conversion Table correction unit, 262... Spherical image generation unit, 264... Movie generation unit, 270... Movie file

JP, 2013-214947, A

Claims (18)

A data generation device which is used in a computer including a control unit and a moving image generation unit, and which outputs data used for a process for generating a celestial sphere moving image capable of changing a field of view during viewing to the outside ,
The data is
Image data consisting of multiple frame data,
Time-series data of a tilt angle of the image pickup unit with respect to a reference direction of the image data picked up by an image pickup unit of an image pickup apparatus including a plurality of optical systems and an image pickup element , the tilt angle being an acceleration included in the image pickup apparatus. Based on a sensor, having a plurality of tilt angle data associated with the plurality of frame data,
The computer receives the data , and the control unit performs zenith correction on the plurality of frame data based on the plurality of tilt angle data associated with the plurality of frame data to generate the moving image. A data generation device that outputs data, wherein the unit is used in a process for generating a celestial sphere moving image based on the corrected plurality of frame data. The plurality of optical systems and the image pickup devices are two optical systems and two image pickup devices, and an image captured by each of the two image pickup devices is combined to generate a spherical image. The data generation device according to claim 1, wherein the moving image is compressed moving image format image data in which the generated spherical image is used as a frame. The tilt angle of the image pickup unit is detected by a tilt detection unit included in the image pickup apparatus, and the tilt detection unit gives a tilt to two images obtained from each of the two image pickup devices. Item 2. The data generation device according to Item 2. A method for outputting data used in a process for generating a celestial sphere moving image capable of changing a field of view during viewing , the computer comprising:
It is used in a computer having a control unit and a moving image generation unit, and executes a step of outputting data used for a process for generating a celestial sphere moving image to the outside .
Image data composed of a plurality of frame data, and time-series data of a tilt angle of the image capturing means with respect to a reference direction of the image data captured by the image capturing means of an image capturing apparatus including a plurality of optical systems and an image sensor , The tilt angle is based on an acceleration sensor included in the imaging device , and has a plurality of tilt angle data associated with the plurality of frame data,
In the step, a computer including the control unit and the moving image generation unit receives the input of the data , and the control unit controls the plurality of tilt angle data based on the plurality of tilt angle data associated with the plurality of frame data. A method of performing zenith correction on frame data, and outputting the data characterized in that the moving image generation unit is used in a process for generating a celestial sphere moving image based on the corrected plurality of frame data. The plurality of optical systems and the image pickup devices are two optical systems and two image pickup devices, and an image captured by each of the two image pickup devices is combined to generate a spherical image. The method according to claim 4, wherein the moving image is image data in a compressed moving image format in which the generated spherical image is used as a frame. The tilt angle of the image pickup unit is detected by a tilt detection unit included in the image pickup apparatus, and the tilt detection unit gives a tilt to two images obtained from each of the two image pickup devices. Item 5. The method according to Item 5. Computer,
A program that is used in a computer that includes a control unit and a moving image generation unit and that functions as a means for externally outputting the data used in the processing for generating a celestial sphere moving image that can change the field of view during viewing. , The data is
Image data composed of a plurality of frame data, and time-series data of a tilt angle of the image capturing means with respect to a reference direction of the image data captured by the image capturing means of an image capturing apparatus including a plurality of optical systems and an image sensor , The tilt angle is based on an acceleration sensor included in the imaging device , and has a plurality of tilt angle data associated with the plurality of frame data,
In the means, a computer including the control unit and a moving image generation unit receives an input of the data , and the control unit causes the plurality of frames based on the plurality of tilt angle data associated with the plurality of frame data. A program for performing zenith correction on data, and outputting the data, wherein the moving image generation unit outputs data that is used in a process for generating a celestial sphere moving image based on the corrected plurality of frame data. The plurality of optical systems and the image pickup devices are two optical systems and two image pickup devices, and an image captured by each of the two image pickup devices is combined to generate a spherical image. The program according to claim 7, wherein the moving image is image data in a compressed moving image format in which the generated spherical image is used as a frame. The tilt angle of the image pickup unit is detected by a tilt detection unit included in the image pickup apparatus, and the tilt detection unit gives a tilt to two images obtained from each of the two image pickup devices. The program according to item 8. An image processing device for generating a spherical video capable of changing the field of view during viewing ,
Image data composed of a plurality of frame data, and time-series data of a tilt angle of the image capturing means with respect to a reference direction of the image data captured by the image capturing means of an image capturing apparatus including a plurality of optical systems and an image sensor , The tilt angle is based on an acceleration sensor included in the imaging device, and has a plurality of tilt angle data associated with the plurality of frame data, and is used in a process for generating a spherical video. A control unit that receives an input from outside and performs zenith correction on the plurality of frame data based on the plurality of tilt angle data associated with the plurality of frame data,
An image processing apparatus, comprising: a moving image generating unit that generates a spherical spherical moving image based on the corrected plurality of frame data. The plurality of optical systems and the image pickup devices are two optical systems and two image pickup devices, and a omnidirectional image is generated by synthesizing images picked up by the two image pickup devices, respectively. The image processing apparatus according to claim 10, wherein the moving image is compressed moving image format image data in which the generated spherical image is used as a frame. The tilt angle of the image pickup unit is detected by a tilt detection unit included in the image pickup apparatus, and the tilt detection unit gives a tilt to two images obtained from each of the two image pickup devices. Item 12. The image processing device according to item 11. A method for generating a spherical video capable of changing the field of view during viewing , the computer
Image data composed of a plurality of frame data, and time-series data of a tilt angle of the image capturing means with respect to a reference direction of the image data captured by the image capturing means of an image capturing apparatus including a plurality of optical systems and an image sensor , The tilt angle is based on an acceleration sensor included in the imaging device, and has a plurality of tilt angle data associated with the plurality of frame data, and is used in a process for generating a spherical video. Receiving an input from the outside, and performing zenith correction on the plurality of frame data based on the plurality of tilt angle data associated with the plurality of frame data,
Generating a spherical movie based on the corrected plurality of frame data. The plurality of optical systems and the image pickup devices are two optical systems and two image pickup devices, and a omnidirectional image is generated by synthesizing images picked up by the two image pickup devices, respectively. The method according to claim 13, wherein the moving image is image data in a compressed moving image format in which the generated spherical image is used as a frame. The tilt angle of the image pickup unit is detected by a tilt detection unit included in the image pickup apparatus, and the tilt detection unit gives a tilt to two images obtained from each of the two image pickup devices. Item 15. The method according to Item 14. In order to generate a spherical video that allows you to change the field of view while watching ,
Image data composed of a plurality of frame data, and time-series data of a tilt angle of the image capturing means with respect to a reference direction of the image data captured by the image capturing means of an image capturing apparatus including a plurality of optical systems and an image sensor , The tilt angle is based on an acceleration sensor included in the imaging device, and has a plurality of tilt angle data associated with the plurality of frame data, and is used in a process for generating a spherical video. A control unit that receives an input from the outside and performs zenith correction on the plurality of frame data based on the plurality of tilt angle data associated with the plurality of frame data, and
A moving image generating unit that generates a spherical moving image based on the corrected plurality of frame data,
Program to function as. The plurality of optical systems and the image pickup devices are two optical systems and two image pickup devices, and a omnidirectional image is generated by synthesizing images picked up by the two image pickup devices, respectively. The program according to claim 16, wherein the moving image is image data in a compressed moving image format in which the generated spherical image is used as a frame. The tilt angle of the image pickup unit is detected by a tilt detection unit included in the image pickup apparatus, and the tilt detection unit gives a tilt to two images obtained from each of the two image pickup devices. The program according to item 17.