JP2020096349A - Image processing device, imaging device, moving image reproduction system, method, and program - Google Patents

Abstract

To provide an image processing device.SOLUTION: An image processing device 110 processes a moving image including continuous images represented by a coordinate system including at least angular coordinates around a predetermined axis. The image processing device 110 includes: moving image data acquisition means (212, 214, and 220) for acquiring moving image data; sensor data acquisition means (216, 218, 224, and 226) for acquiring sensor data corresponding to the moving image data; and rotation correction means 230 for correcting the image so as to cancel a rotational change around a reference axis so that the reference for displaying the image is fixed in a specific direction during photographing over a plurality of frames of the moving image based on the sensor data.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to an image processing device, an imaging device, a moving image reproduction system, a method, and a program.

2. Description of the Related Art In recent years, spherical cameras capable of shooting moving images of all directions at a shooting location have become popular. With such a omnidirectional camera, it is assumed that the user takes an image in a fixed state, or takes a moving image while the user holds it while moving. At that time, there is also known a correction technique that corrects camera shake and tilt of the omnidirectional camera to create a moving image with less blur.

The spherical video captured by the spherical camera usually has an initial frontal direction when a viewer views the spherical video. In general, the frontal direction during viewing follows the frontal direction of the omnidirectional camera when the omnidirectional camera photographs (for example, the frontal direction of one lens in the case of two eyes). However, when the front direction during viewing follows the movement of the omnidirectional camera at the time of image capturing, the video displayed may change significantly depending on the shooting conditions of the photographer, which may cause the viewer to get drunk. There is.

For example, when the photographer fixes the omnidirectional camera to his head and shoots, if the front direction during viewing follows the movement of the omnidirectional camera during imaging, the photographer shakes his or her head and the surroundings If you look around, the celestial sphere video will rotate significantly, causing viewers to get motion sick. Further, a large rotation of the spherical video may make the video itself very difficult to see. The same applies to the case where the omnidirectional camera is fixed to the head for photographing, and the case where the front direction changes greatly during photographing even if the user holds the camera.

Japanese Unexamined Patent Application Publication No. 2017-147682 (Patent Document 1) discloses a configuration in which, for image data, a shake in a horizontal plane in global coordinates is subjected to coordinate conversion by correction excluding a minute shake component to generate a spherical image. To do. However, although the high-frequency component of the shake is corrected for the rotation around the vertical axis, it follows the intentional action of the photographer shaking his head and looking around, so the viewer is sick at the time of viewing. It was not enough in terms of causing.

The present disclosure has been made in view of the above points, and the present disclosure provides an image processing device that can suppress motion sickness of a viewer caused by rotation around a reference axis during shooting when viewing a moving image. With the goal.

In order to address the above problems, the present disclosure provides an image processing device having the following features, which processes a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis. The image processing apparatus includes a moving image data acquisition unit that acquires moving image data and a sensor data acquisition unit that acquires sensor data corresponding to the moving image data. The image processing apparatus further corrects the image based on the sensor data by canceling the rotation change around the reference axis so that the reference of the image display is fixed in a specific direction during shooting over a plurality of frames of the moving image. It includes rotation correction means for performing the rotation correction.

With the above configuration, it is possible to suppress the motion sickness of the viewer caused by the rotation around the reference axis during shooting when watching a moving image.

Sectional drawing of the spherical camera which comprises the spherical video system by this embodiment. The figure explaining the case where the spherical camera by this embodiment is fixed to a head. The hardware block diagram of the spherical video system by this embodiment. FIG. 3 is a main functional block diagram related to a spherical video playback function realized on the spherical video system according to the present embodiment. (A) An image data flow diagram in the generation of a spherical image, and a diagram illustrating a case where the data structure of the spherical image is represented by a plane (B) and a case (C) represented by a spherical surface. The figure explaining zenith correction and rotation correction of the omnidirectional image by this embodiment. The figure explaining the omnidirectional image obtained by the zenith correction and rotation correction of the omnidirectional image according to this embodiment. FIG. 6 is a sequence diagram illustrating processing from shooting to viewing of a moving image in the spherical video system according to a specific embodiment. FIG. 11 is a sequence diagram illustrating processing from shooting to viewing of a moving image in the spherical video system according to another specific embodiment. 6 is a flowchart showing a rotation process for frame composite data executed by the omnidirectional camera which constitutes the omnidirectional video system according to the present embodiment. The figure explaining the relationship between the advancing direction and the front direction of photography when a photographer wears and moves the omnidirectional camera on his head. FIG. 12 is a diagram (1/2) illustrating a captured spherical image before rotation correction and a captured spherical image after rotation correction when the photographer performs the operation shown in FIG. 11. FIG. 12 is a diagram (2/2) illustrating a captured spherical image before rotation correction and a captured spherical image when the photographer performs the operation shown in FIG. 11. 9 is a flowchart showing a rotation process for frame composite data, which is executed by a omnidirectional camera that constitutes an omnidirectional video system according to another embodiment.

Hereinafter, the present embodiment will be described, but the embodiment is not limited to the embodiment described below. In the following embodiments, a spherical camera and a spherical moving image system will be described as examples of an image processing device (and an imaging device) and a moving image reproducing system, respectively.

Hereinafter, the overall configuration of the celestial sphere moving image system according to the present embodiment will be described with reference to FIGS. 1 to 3.

FIG. 1 is a cross-sectional view of a celestial sphere camera 110 that constitutes the celestial sphere video system according to the present embodiment. An omnidirectional camera 110 shown in FIG. 1 includes an image pickup body 12, a casing 14 that holds the image pickup body 12, components such as a controller 15 and a battery 16, and a shutter button 18 provided in the casing 14. Equipped with.

The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B, and two imaging elements 22A and 22B such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. It Each of the image forming optical systems 20 is configured as, for example, a fisheye lens with 7 elements in 6 groups. In the embodiment shown in FIG. 1, the fish-eye lens has a total angle of view larger than 180 degrees (=360 degrees/n; the number of optical systems n=2), and preferably has a field angle of 185 degrees or more. However, more preferably, the angle of view is 190 degrees or more. Such a combination of the wide-angle imaging optical system 20 and the image pickup device 22 is referred to as a wide-angle image pickup optical system. In the embodiment to be described, the case where the number of optical systems is 2 will be described as an example, but the number of optical systems may be 1 as long as it is possible to capture a wide-angle region. It may be 3 or more. Further, it is not limited to a fisheye lens having an angle of view of 180 degrees or more.

In the embodiment shown in FIG. 1, the image forming optical systems 20A and 20B have the same specifications, and they are combined in opposite directions so that their optical axes coincide with each other. The image pickup devices 22A and 22B convert the received light distribution into image signals and sequentially output image frames to the image processing means on the controller. As will be described in detail later, the images respectively captured by the image pickup devices 22A and 22B are combined to generate an image with a solid angle of 4π steradians (hereinafter referred to as a “spheroidal image”). .. The celestial sphere image is a photograph of all directions that can be viewed from the photographing point. Then, a spherical moving image is composed of continuous frames of the spherical image.

Here, in the embodiment to be described, it is assumed that a omnidirectional image and a omnidirectional moving image are generated. However, a omnidirectional image and a omnidirectional moving image, a so-called panoramic image and panorama obtained by photographing only 360 degrees in a horizontal plane It may be a moving image, or may be a hemispherical image or a semi-cylindrical image captured at an angle of view of 180 degrees, and may be image data having image information on a spherical surface or a cylinder. That is, the moving image and the image included in the moving image according to the present embodiment are images represented by a coordinate system including angular coordinates around a predetermined axis. In the case of the above-mentioned spherical image/moving image, hemispherical image/moving image, and partial spherical image, the radius vector is set as a constant (for example, 1), and the angular coordinate representing the angle between the predetermined axis and the radius vector and the predetermined axis It is represented by a spherical coordinate system that includes the angular coordinates of. In the case of the panoramic image/moving image or its partial image, the radius vector is represented by a cylindrical coordinate system including the coordinate in the axial direction of the cylinder and the angular coordinate around the axis, with the radius vector being a constant (for example, 1).

FIG. 2 illustrates a case where the omnidirectional camera 110 of FIG. 1 is fixed to the photographer's head 50. FIG. 2 is an example in which the omnidirectional camera 110 is fixed to the hat-type fixing device 52. By mounting the omnidirectional camera 110 on the photographer's head 50 in this way, the image capturing front direction of the image data imaged by the celestial sphere camera 110 follows the photographer's swinging motion H. Become. The shooting front direction is a preset direction and is a reference direction displayed when playing back moving image data. FIG. 2 also shows the moving direction T of the photographer. The photographer may proceed while looking around the area, and the front direction of photographing by the swinging motion H and the traveling direction T do not necessarily match. As shown in FIG. 2, in addition to fixing the omnidirectional camera 110 to the hat-type fixing device 52 and attaching it to the head 50, the omnidirectional camera 110 itself can be attached to the head 50. It may be (helmet or hat with embedded spherical camera function).

FIG. 3A shows a hardware configuration of the omnidirectional camera 110 that constitutes the omnidirectional video system according to the present embodiment. The omnidirectional camera 110 corresponds to the image processing device or the imaging device in the embodiments described.

The spherical camera 110 is connected via a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a moving image compression block 118, and a DRAM (Dynamic Random Access Memory) interface 120. The DRAM 132 and the sensor 136 connected via the external sensor interface 124 are included. Here, the CPU 112, the ROM 114, the image processing block 116, and the moving image compression block 118 are mounted on the controller 15 shown in FIG.

The CPU 112 controls the operation and overall operation of each unit of the spherical camera 110. The ROM 114 stores an operating system (OS), a control program, and various parameters described in a code that can be read by the CPU 112. The image processing block 116 is connected to the two image pickup devices 130A and 130B (the image pickup devices 22A and 22B in FIG. 1), and the image signals of the images picked up by the respective image pickup blocks 130A and 130B are input. The image processing block 116 is configured to include an ISP (Image Signal Processor) and the like, and performs shading correction, Bayer interpolation, white balance correction, gamma correction, and the like on the image signal input from the image sensor 130.

The moving picture compression block 118 is an MPEG-4 AVC/H. It is a codec block that performs video compression and decompression such as H.264. The DRAM 132 provides a storage area for temporarily storing data when performing various kinds of signal processing and image processing.

In the embodiment shown in FIG. 3A, the sensor 136 includes an angular velocity sensor 136A and an acceleration sensor 136B. The angular velocity sensor 136A is a sensor that detects triaxial angular velocity components. The acceleration sensor 136B is a sensor that detects triaxial acceleration components. The detected acceleration component and angular velocity component are used to perform zenith correction of the omnidirectional image in the direction of gravity and rotation correction around the direction of gravity, as will be described later. As the sensor 136, another sensor such as a geomagnetic sensor (also referred to as an electronic compass) for obtaining an azimuth angle or the like may be further provided, and the azimuth angle may be used for correcting the rotation around the gravity direction. ..

The spherical camera 110 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, and a serial block 128. An external storage 134 is connected to the external storage interface 122. The external storage interface 122 controls reading and writing with respect to the external storage 134 such as a memory card inserted in the memory card slot. A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with an external device such as a smartphone connected via the USB connector 138. The serial block 128 controls serial communication with an external device such as a smartphone and is connected to a wireless NIC (Network Interface Card) 140.

When the power is turned on by operating the power switch, the OS and the control program are loaded into the main memory. The CPU 112 controls the operation of each part of the device according to the program read into the main memory, and temporarily stores the data necessary for the control in the memory. As a result, the functional units and processes of the spherical camera 110 described later are realized.

FIG. 3B shows a hardware configuration of the information terminal 150 that constitutes the spherical video system according to the present embodiment. The information terminal 150 corresponds to the information processing device in the embodiment described.

The information terminal 150 illustrated in FIG. 3B includes a CPU 152, a RAM 154, an internal storage 156, an input device 158, an external storage 160, a display 162, a wireless NIC 164, and a USB connector 166. ..

The CPU 152 controls the operation of each unit of the information terminal 150 and the overall operation. The RAM 154 provides a work area for the CPU 152. The internal storage 156 stores a control program such as an operating system described in a code readable by the CPU 152 and an application responsible for processing on the information terminal 150 side according to the present embodiment. The CPU 152 may include a plurality of CPUs.

The input device 158 is an input device such as a touch screen and provides a user interface. The input device 158 receives various instructions from the operator such as correction of spherical video. The external storage 160 is a removable recording medium mounted in a memory card slot or the like, and records various data such as moving image data and still image data. The display 162 displays the corrected spherical video in response to a user operation. The wireless NIC 164 establishes a wireless communication connection with an external device such as the spherical camera 110. The USB connector 166 makes a USB connection with an external device such as the spherical camera 110.

When the information terminal 150 is turned on and turned on, the control program is read from the internal storage 156 and loaded into the RAM 154. The CPU 152 controls the operation of each part of the device according to the control program read in the RAM 154, and temporarily stores the data necessary for the control in the memory. Thereby, each functional unit and processing of the information terminal 150, which will be described later, are realized.

Hereinafter, with reference to FIG. 4 to FIG. 9, the omnidirectional video reproduction function provided in the omnidirectional video system according to the present embodiment will be described.

FIG. 4 shows the main functional blocks 200 related to the spherical video playback function realized on the spherical video system according to the present embodiment. As shown in FIG. 4, the functional block of the omnidirectional camera 110 is configured to include an image processing unit 210 and an output unit 240. The image processing unit 210 includes a first imaging unit 212, a second imaging unit 214, an angular velocity detection unit 216, an acceleration detection unit 218, an image composition unit 220, a zenith correction amount calculation unit 222, and a first rotation amount. The calculation unit 224, the second rotation amount calculation unit 226, the storage unit 228, and the image rotation unit 230 are included. The processing of each unit is performed by the CPU 112 and the image processing block 116 included in the controller 15 executing various programs.

On the other hand, the functional block of the information terminal 150 is configured to include the receiving unit 252 and the display control unit 254, as also shown in FIG. The processing of each unit is performed by the CPU 152 included in the information terminal 150 executing various programs.

Hereinafter, first, the functional blocks on the spherical camera 110 side will be described.

The first image capturing unit 212 sequentially captures images using the image sensor 130A and generates a fish-eye image A that is continuous in time series. The second image capturing unit 214 sequentially captures images using the image sensor 130B and generates a fish-eye image B that is continuous in time series. The image combining unit 220 performs stitching processing on the fisheye image A (first frame data) that is continuous in time series and the fisheye image B (second frame data) that is continuous in time series to combine frames. Create the data. The frame composite data is moving image data which is a main body of the spherical moving image data and includes a plurality of frames each of which forms a spherical moving image from the start to the end of shooting. The fish-eye image A and the fish-eye image B are partial images for forming the omnidirectional image, and the first and second frame data also indicate the frames of the partial images from the start to the end of shooting. This is the moving picture data that can compose a spherical video. The image composition unit 220 outputs the created frame composition data to the image rotation unit 230. The first image capturing section 212, the second image capturing section 214, and the image synthesizing section 220 can configure the image capturing means in this embodiment.

Hereinafter, generation of a spherical image and a generated spherical image will be described with reference to FIG. FIG. 5A illustrates the data structure of each image and the data flow of the image in the spherical image generation. First, the image directly captured by each of the image pickup devices 130 is an image in which the hemisphere of the whole celestial sphere is included in the visual field. The light incident on the imaging optical system is imaged on the light receiving area of the image sensor 130 according to a predetermined projection method. The image captured here is an image captured by a two-dimensional image sensor whose light-receiving area forms a planar area, and is image data expressed in a planar coordinate system. In addition, typically, the obtained image is a fish including the entire image circle in which each shooting range is projected, as shown by “fisheye image A” and “fisheye image B” in FIG. It is configured as an eye image.

The plurality of fish-eye images of each frame captured by the plurality of image sensors 130 are subjected to distortion correction and synthesis processing to form one spherical image corresponding to each frame. In the combining process, first, each hemispherical image including each complementary hemispherical portion is generated from each fisheye image configured as a planar image. Then, the two hemispherical images including the respective hemispherical portions are stitched (image combination) based on the matching of the overlapping regions, and a spherical image including the entire spherical image is generated.

FIG. 5B is a diagram illustrating the data structure of the image data of the omnidirectional image used in the present embodiment by showing it in a plane. FIG. 5C is a diagram illustrating the data structure of the image data of the omnidirectional image represented by a spherical surface. As shown in FIG. 5B, the image data of the spherical image has coordinates of a vertical angle φ with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the predetermined axis. It is expressed as an array of pixel values. The vertical angle φ is in the range of 0 degrees to 180 degrees (or -90 degrees to +90 degrees), and the horizontal angle θ is in the range of 0 degrees to 360 degrees (or -180 degrees to +180 degrees).

Each coordinate value (θ, φ) in the omnidirectional format is associated with each point on the spherical surface that represents the azimuth around the shooting point, as shown in FIG. It is associated with the spherical image. The plane coordinates of the fisheye image captured by the fisheye lens and the spherical coordinates of the spherical image are associated with each other in a predetermined conversion table. The conversion table is data created in advance by a manufacturer or the like according to a predetermined projection model based on design data of each lens optical system, etc., and is data for converting a fisheye image into a spherical image.

It should be noted that in the embodiment to be described, the flow is such that the fisheye image is converted into a celestial sphere image, and the frame synthesizing data after the stitching process is rotated by the image rotating unit 230. However, the order of conversion processing, combination processing, and rotation processing is not particularly limited. The fisheye image A and the fisheye image B that are partial images (two hemisphere images including complementary hemisphere parts obtained by converting the fisheye images) are subjected to rotation processing (correction) by the image rotation unit 230 and then stitched. It is also possible to use a flow for performing a ring processing. Further, in addition to performing rotation processing on the image in the spherical format, the rotation processing is reflected in the conversion table for converting the fisheye image into a spherical image, and based on the conversion table in which the rotation processing is reflected, It is also possible to obtain the omnidirectional image after the rotation processing from the fisheye image A and the fisheye image B. In the present embodiment, “correcting a moving image” refers to finally obtaining a moving image including an image in which a predetermined correction is reflected, regardless of the order of the conversion process, the combining process, and the rotation process. Further, in the present embodiment, the spherical image is created by stitching two fish-eye images, but the present invention is not limited to this, and in other embodiments, for example, stitching three or more images. It may be one that creates a spherical image or a panoramic image.

Here, refer to FIG. 4 again. The angular velocity detection unit 216 is an angular velocity detection mechanism including an angular velocity sensor 136A, and outputs angular velocity data obtained by measuring the angular velocity components in the three axis directions using the angular velocity sensor 136A during shooting. The angular velocity data is time-series data of angular velocities generated around the three axes of the omnidirectional camera 110 (the angular velocity sensor 136A therein) from the start to the end of photographing. The angular velocity data does not need to be recorded in a one-to-one correspondence with a frame, but typically can be recorded at a rate faster than the frame rate. In this case, the time stamp can be used as a clue to find the correspondence with each frame later. Alternatively, the angular velocity may be recorded in a one-to-one correspondence with each frame of the spherical video. In the present embodiment, the angular velocity detection unit 216 constitutes a recording unit that records the sensor data in association with each frame of the moving image, and the angular velocity data may be included in the sensor data in the present embodiment.

The acceleration detection unit 218 is an acceleration detection mechanism including an acceleration sensor 136B, and outputs acceleration data obtained by measuring acceleration components in the three axis directions using the acceleration sensor 136B during shooting. The acceleration data is time-series data of three-axis acceleration of the celestial sphere camera 110 (the acceleration sensor 136B therein) from the start to the end of shooting. Acceleration data need not be recorded in a one-to-one correspondence with frames, but typically can be recorded at a rate faster than the frame rate. In this case, the time stamp can be used as a clue to find the correspondence with each frame later. Alternatively, the acceleration may be recorded in a one-to-one correspondence with each frame of the spherical video. In the present embodiment, the acceleration detection unit 218 constitutes a recording unit that records the sensor data in association with each frame of the moving image, and the acceleration data may be included in the sensor data in the present embodiment.

The above-described acceleration data and angular velocity data directly obtained from the acceleration sensor 116B and the angular velocity sensor 136A are data based on the three axis directions of the sensor coordinate system (or the coordinate system of the omnidirectional camera 110). On the other hand, the acceleration sensor 136B outputs the sum of the acceleration based on the motion and the gravitational acceleration, and the above-described acceleration data and angular velocity data can be converted into an absolute coordinate system based on the gravitational acceleration axis. Therefore, the sensor data described above indicates the amount of displacement in the absolute coordinate system.

The zenith correction amount calculation unit 222 calculates the tilt angle at the time of imaging with respect to the reference axis in each frame based on the acceleration data output from the acceleration detection unit 218, and creates zenith correction data. The zenith correction data is time-series data of the tilt angle of the omnidirectional camera 110 with respect to the reference axis from the start to the end of shooting, which is recorded in association with each frame of the omnidirectional video. The tilt angle with respect to the reference axis is typically configured as a vector composed of acceleration dimension values. The reference axis typically corresponds to the direction of gravity in which gravitational acceleration acts, and the description will be continued below assuming that the reference axis is the direction of gravity. Since the acceleration sensor 136B does not distinguish between gravity and inertial force, the tilt angle obtained from the acceleration sensor 136B may be corrected, preferably based on the signal measured by the angular velocity sensor 136A.

The first rotation amount calculation unit 224 determines, based on the acceleration data output from the acceleration detection unit 218 and the angular velocity data output from the angular velocity detection unit 216, from the amount of change in the relative angle of the omnidirectional camera 110 in the direction of gravity. On the other hand, a vector in the shooting front direction of the omnidirectional camera 110 in the coordinate system of a plane (horizontal plane) in the vertical direction is calculated for each frame, and the first rotation correction data is recorded. The first rotation correction data is time-series data of the angle change amount in the front direction of shooting from the reference value from the start to the end of shooting, which is recorded in association with each frame of the celestial sphere moving image. The angle change amount in the image capturing front direction is the image capturing front direction in the horizontal plane based on the posture of the spherical camera 110 in the reference frame and the image capturing front direction in the horizontal plane based on the posture of the spherical camera 110 in the target frame. It is represented by the difference from the direction. Here, the reference frame can be, for example, a frame at the start of shooting or a frame at the start of applying the main correction (in the case of an embodiment in which application of the correction can be instructed after the start of shooting). The first rotation correction data is represented by, for example, a numerical value of the rotation angle or a vector. The first rotation correction data may also be included in the sensor data in the present embodiment, and the first rotation amount calculation unit 224 in the present embodiment acquires and records the sensor data in association with each frame of the moving image. The acquisition means and the recording means may be configured.

The second rotation amount calculation unit 226 is based on the acceleration data output from the acceleration detection unit 218 and the angular velocity data output from the angular velocity detection unit 216 in the plane coordinate system of a plane (horizontal plane) perpendicular to the gravity direction. A vector in the traveling direction of the omnidirectional camera 110 is calculated for each prescribed number of frames, and second rotation correction data is generated. The traveling direction in the frame at the start of shooting is equal to the shooting front direction of the spherical camera 110 in the frame at the start of shooting. The second rotation correction data is recorded in association with each frame of the celestial celestial sphere moving image or each frame of the specified number of frames and is recorded as an angle change amount in the traveling direction from the reference value from the start to the end of shooting. It is time series data. The angle change amount of the traveling direction is the difference between the traveling direction of the celestial sphere camera 110 in the horizontal plane in the reference frame (imaging front direction) and the traveling direction of the celestial sphere camera 110 in the horizontal plane in the target frame. It is represented by. The second rotation correction data is, for example, a numerical value of a rotation angle or a vector. The second rotation correction data may also be included in the sensor data in the present embodiment, and the second rotation amount calculation unit 226 acquires and records the sensor data in association with each frame of the moving image in the present embodiment. The acquisition means and the recording means may be configured.

Regarding the change in the traveling direction, if the acceleration of the omnidirectional camera 110 in the specific direction satisfies a predetermined condition in the acceleration data, the specific direction at that time is determined to be the traveling direction. The predetermined condition can be, for example, a case where the acceleration in the specific direction exceeds a predetermined threshold value when the vectors of a plurality of frames are combined. When the predetermined condition is not satisfied, the traveling direction can inherit the data of the previous frame. The definition of the traveling direction is not particularly limited, and may be the traveling direction of the photographer or the traveling direction of the camera itself. The second rotation amount calculation unit 226 can also function as the determination unit in this embodiment.

The image rotation unit 230 includes the frame synthesis data, the zenith correction data, and the first rotation correction data output by the image synthesis unit 220, the zenith correction amount calculation unit 222, the first rotation amount calculation unit 224, and the second rotation amount calculation unit 226. And the second rotation correction data, respectively. The image rotation unit 230 performs rotation processing (zenith correction and rotation correction) on each frame of the acquired frame composite data based on the acquired zenith correction data, the first rotation correction data, and the second rotation correction data, and performs correction. Output the omnidirectional spherical frame data.

Here, the zenith correction and the rotation correction will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram illustrating zenith correction and rotation correction of the omnidirectional image performed in the present embodiment. FIG. 7 is a diagram illustrating a omnidirectional image obtained by the zenith correction and the rotation correction of the omnidirectional image performed in the present embodiment. FIG. 7A shows a moving image frame before the zenith correction, and FIG. 7B shows a moving image frame after the zenith correction.

As described above, the image data in the spherical image format has a pixel value in which the vertical angle φ with respect to the predetermined axis z0 and the horizontal angle θ corresponding to the rotation angle around the predetermined axis z0 are coordinates. Expressed as an array of. Here, the predetermined axis is an axis defined with the omnidirectional camera 110 as a reference when no correction is made. For example, with the imaging body 12 side of the omnidirectional camera 110 shown in FIG. 1 as the head and the opposite side as the bottom, the central axis passing through the center of the housing from the bottom to the head is represented by the horizontal angle θ and the vertical angle φ. A spherical image can be defined as a predetermined axis z0 to be defined. Further, for example, the horizontal angle θ of the spherical image can be defined so that the optical axis direction of one optical element of the two imaging optical systems 20A and 20B is centered at the horizontal angle θ.

The zenith correction is a celestial sphere image (FIG. 7(A)) captured in a state where the central axis z0 is actually inclined with respect to the gravity direction as shown in the left diagram of FIG. As shown in the figure, it refers to a process (correction in Roll and Pitch directions) for correcting a celestial sphere image (FIG. 7B) as if the image was taken with the central axis z0 coinciding with the gravity direction Z. The image rotation unit 230 performs zenith correction on each frame based on the tilt angle with respect to the gravity direction in each frame included in the zenith correction data so that a predetermined axis (center axis) substantially matches the gravity direction. It functions as a correction means.

On the other hand, the rotation correction is an omnidirectional image corrected by the zenith correction so that the central axis z0 coincides with the reference direction, and further an angular change around the gravity direction Z (horizontal angle θ direction in FIG. 7). Correction for fixing the display reference (for example, the center at the horizontal angle θ in the celestial sphere format, which is the reference position displayed by default when reproducing moving image data) in a specific direction during shooting (Yaw direction). Correction). The reference position displayed by default is an area in the spherical format that is displayed when the user does not perform an operation to change the display direction for moving image data. The specific direction is, for example, the front direction of the spherical camera 110. The image rotation unit 230 serves as a rotation correction unit that performs a correction for canceling a rotation change around the reference axis (which is matched with the gravity direction by the zenith correction) based on the first rotation correction data and the second rotation correction data. Function.

Here, the specific direction during shooting in which the change in angle is canceled and is fixed to the display reference may be the shooting front direction in the horizontal plane of the omnidirectional camera 110 in the reference frame described above. Further, after the traveling direction of the omnidirectional camera 110 is obtained, the specific direction may be the traveling direction of the omnidirectional camera 110 in the horizontal plane.

More specifically, the image rotation unit 230 is based on the first rotation correction data and the second rotation correction data, and for each frame of the moving image, at least a plurality of frames for a certain period of time, the display reference indicates that the shooting is being performed. Correction is performed to cancel the rotation change around the reference axis so that it substantially matches the specific direction. Here, the amount of rotation change to be canceled is determined in a specific direction in a plane perpendicular to the reference axis (for example, the shooting front direction or the advancing direction at the start of shooting) and in the plane perpendicular to the reference axis in the correction target frame. It is based on the difference from the image capturing front direction. Further, the period fixed in the specific direction is from the timing when the specific direction is determined to the timing when the specific direction is determined next time.

In the embodiment to be described, at the beginning, the reference (center of the image) of the display is fixed by setting the shooting front direction of the omnidirectional camera 110 in the reference frame in the horizontal plane as the specific direction, and the omnidirectional camera 110 is started. After the traveling direction of is determined, the traveling direction in the horizontal plane is determined as the specific direction, and the display reference (the center of the image) is fixed to the traveling direction. In this case, regardless of the swinging motion H shown in FIG. 2, initially, the rotation correction is performed so that the shooting front direction at the start time comes to the center of the image, and when the traveling direction T is gradually determined, the traveling direction T is determined. The rotation will be corrected so that it will come to the center of the image.

However, the method of determining the specific direction is not particularly limited. In another embodiment, regardless of the traveling direction T, the reference position that is always displayed by default may be fixed in the shooting front direction of the omnidirectional camera 110 in the horizontal plane in the reference frame. Good. In such an embodiment, the rotation correction using the second rotation amount calculation unit 226 and the second rotation correction data can be omitted. Further, in this case, since the specific direction is not updated due to a change in the traveling direction during shooting, the period fixed in the specific direction is from the start time of the video or the time when the start of the fixation is instructed to the end time of the video or the fixed time. It is the entire period until the time when the end is instructed.

The storage unit 228 stores the above-described first frame data, second frame data, frame composite data, corrected frame composite data that has undergone zenith correction and rotation correction, acceleration data, angular velocity data, zenith correction data, first rotation correction data, and It is used to store the second rotation correction data. The storage unit 228 is provided as a part of a storage area such as the DRAM 132, the external storage 134, or another storage illustrated in FIG.

Note that the spherical moving image acquired by one shooting may be recorded in the storage unit 228 as one file, for example. In that case, various modes are conceivable as the recording method of the spherical video.

For example, the corrected frame composite data can be recorded as one spherical video data. In that case, the acceleration data, the angular velocity data, the zenith correction data, the first rotation correction data, and the second rotation correction data may be arbitrarily recorded as metadata together with the corrected frame synthesis data. In this embodiment, when the corrected data becomes necessary, the corrected frame composite data may be read.

In another embodiment, the frame composite data before correction can be recorded as one celestial sphere moving image data. In that case, the acceleration data, the angular velocity data, the zenith correction data, the first rotation correction data, and the second rotation correction data are recorded in a predetermined combination as metadata together with the frame combination data before correction. In this other embodiment, when the corrected data is needed, the zenith correction data, the first rotation correction data, and the second rotation correction data are prepared from the metadata (lacking data is calculated. .), and the image rotation unit 230 applies image rotation to the frame composite data before correction. For example, if acceleration data and angular velocity data are available, any of the zenith correction data, the first rotation correction data, and the second rotation correction data that are insufficient can be calculated.

The frame-combined data subjected to the zenith correction and the rotation correction by the image rotation unit 230 is output to the output unit 240. The output unit 240 can transmit and output the corrected data as moving image data to the external information terminal 150 via the wireless NIC 140 and the USB connector 138. Alternatively, the output unit 240 may output the corrected data as a moving image data to a file in a predetermined storage.

Although referred to as “moving image”, it may be recorded in any form as long as the moving image can be reproduced. For example, H.264. H.264/MPEG-4 AVC (Advanced Video Coding), H.264. It may be recorded as moving image data obtained by compressing a plurality of frames with a predetermined codec such as H.265/HEVC (High Efficiency Video Coding). Further, Motion JPEG (Joint Photographic Experts Group) is a format for expressing moving images as continuous still images, but moving image data may be recorded as a continuous series of still images of a plurality of frames in this way. However, the moving image may be recorded as a set of files of still images of a plurality of frames. The output unit 240 includes an appropriate codec.

Hereinafter, the functional blocks on the information terminal 150 side will be continuously described. The information terminal 150 is a terminal device in which an application for communicating with the omnidirectional camera 110 and viewing and reproducing an omnidirectional image is installed. The information terminal 150 may be a smartphone, a tablet computer, a personal computer, a head mounted display (HMD), or the like. The information terminal 150 receives various instructions from the operator via the application and issues various requests to the omnidirectional camera 110. For example, the information terminal 150 responds to the reception of a specified spherical video reproduction instruction (for example, a video reproduction instruction in which rotation correction is applied) from the operator, and then the information terminal 150 instructs the spherical camera 110 to perform a predetermined operation. Issue a request for the corrected video data of the spherical video.

The receiving unit 252 of the information terminal 150 receives the moving image data output from the spherical camera 110. The display control unit 254 of the information terminal 150 displays the spherical video on a display device such as the display 162 included in the information terminal 150 based on the received video data. In a specific embodiment, the display control unit 254 causes at least a part of the image of the corrected moving image data to be displayed on the display 162 based on the above-described display standard. Further, the display control unit 254 changes the direction displayed in the omnidirectional format by the user's operation, and changes the area displayed on the display 162.

It should be noted that what kind of image is displayed on the information terminal 150 side based on the corrected image data is arbitrary. For example, the display control unit 254 may display the entire celestial sphere image on the display device, or the celestial sphere image may be attached to the spherical object, and the spherical object may be displayed from a predetermined position with a virtual camera having a predetermined viewing angle. You may display a moving image as a frame with the image when you observe. In any case, in the spherical format, a predetermined display standard is defined, and an image is displayed in a specific visual field based on the display standard. For example, when the entire spherical image is displayed on the display 162, it can be displayed with the specific direction fixed at the center of the display 162 at all times. When displaying an image observed by a virtual camera on the display 162, the spherical image is attached to the spherical object based on the display standard. In this case, the specific direction during shooting is not fixed to the center of the display 162, but if the display range (position and direction of the virtual camera) is not changed, the predetermined direction (for example, the left hand in the traveling direction) will not be changed. It will always be fixed to the center of the display 162.

In the present embodiment, the zenith correction and the rotation correction are actually performed using the resources of the celestial sphere camera 110 side instead of the information terminal 150, and the correction result is output and displayed on the information terminal 150. Adopt a configuration. With this configuration, when sufficient resources are provided on the omnidirectional camera 110 side, regardless of the processing performance of the information terminal 150, it is possible to stably perform moving image reproduction while performing zenith correction and rotation correction. Become.

In addition, in the embodiment to be described, as the output mode, the image data of the spherical moving image is transmitted to the information terminal 150, but the present invention is not limited to this. When the omnidirectional camera 110 includes a display device, it is also possible to adopt a mode in which the display is performed on the display device.

In the embodiment shown in FIG. 4, the functional block of the omnidirectional camera 110 may be configured to further include an instruction receiving unit 242 in addition to the image processing unit 210 and the output unit 240. The instruction receiving unit 242 can receive an instruction to specify a specific direction during shooting of a moving image. In the above-described embodiment, the specific direction is the image capturing front direction or the advancing direction at a predetermined time point. However, by providing the instruction receiving unit 242, the direction desired to be fixed to the image display reference at an arbitrary timing during image capturing. It is possible to specify (such as the direction in which the subject of interest exists). The instruction receiving unit 242 constitutes a receiving unit in this embodiment.

FIG. 8 and FIG. 9 are sequence diagrams illustrating the processing from shooting to viewing of a moving image in the spherical spherical moving image system according to various specific embodiments.

FIG. 8A corresponds to the embodiment shown in FIG. In the embodiment shown in FIG. 8A, imaging and image processing (including image combination, zenith correction, and rotation correction) are performed on the omnidirectional camera 110 side, and the moving image data after the image processing is stored in the information terminal 150. Sent. Then, in the information terminal 150, display display is performed based on the moving image data after the image processing. More specifically, in the embodiment of FIG. 8(A), the omnidirectional camera 110 acquires moving image data by the image capturing process in S10, performs image processing on the moving image data in S11, and performs image processing in S12. The processed moving image data is transmitted to the information processing terminal. In S13, the information processing terminal displays the moving image data on the display.

In the embodiment shown in FIG. 4 and FIG. 8A, the omnidirectional camera 110 uses the moving image data acquisition means (the first image capturing unit 212, the second image capturing unit 214, and the image combining unit 220 as necessary), It operates as an image processing apparatus including a sensor data acquisition unit (angular velocity detection unit 216, acceleration detection unit 218, first rotation amount calculation unit 224, and first rotation amount calculation unit 224) and rotation correction unit (image rotation unit 230).

4 and 8A, an example in which image processing (including image synthesis, zenith correction, and rotation correction) is performed using the resources on the omnidirectional camera 110 side has been described, but the present invention is not limited to this. .. For example, a part or all of the first frame data, the second frame data, the frame composite data, the angular velocity data, the acceleration data, the zenith correction data, the first rotation correction data and the second rotation correction data are sent to the information terminal 150 side, Part or all of the image processing may be performed on the information terminal 150 side. In that case, the information terminal 150 side performs some functions on the omnidirectional camera 110 side shown in the functional block diagram of FIG. 4 by the CPU 152 on the information terminal 150 side executing various programs. As a result, for example, the CPU having a higher processing speed can perform a heavy load process such as a process of combining the first frame data (fisheye image A) and the second frame data (fisheye image B) by the image combining unit 220. It becomes possible to carry out the processing by using it, and it becomes possible to process even data having a large number of pixels in a short time or in real time.

For example, in the embodiment shown in FIG. 8B, only the omnidirectional camera 110 takes an image (S30), and the frame data before image processing and the rotation parameter are transmitted to the information terminal 150 (S31). Here, the rotation parameter is a general term for parameters related to correction of frame data such as the acceleration data, the angular velocity data, the zenith correction data, the first rotation correction data, and the second rotation correction data described above. Then, in the information terminal 150, image processing such as zenith correction and rotation correction is performed on the received frame data based on the received rotation parameter (S32), and display is performed based on the moving image data after the image processing. (S33).

In the embodiment shown in FIG. 8B, the information terminal 150 includes a moving image data acquisition unit (reception unit), a sensor data acquisition unit (reception unit), and a rotation correction unit (a unit corresponding to the image rotation unit 230). It operates as a provided image processing device.

In this case, the image combination may be performed by either the omnidirectional camera 110 or the information terminal 150. When images are combined on the omnidirectional camera 110 side, the frame data transmitted from the omnidirectional camera 110 to the information terminal 150 is frame combined data (omnidirectional format). When image synthesis is performed on the information terminal 150 side, the frame data transmitted from the omnidirectional camera 110 to the information terminal 150 is the first frame data (fisheye image A) and the second frame data (fisheye image B). Is. The first frame data and the second frame data may be transmitted as moving image data corresponding to each of the first frame data and the second frame data, or moving image data of one combined image obtained by combining two fisheye images (the fisheye image A and the fisheye image). The moving image data in the case where the images B are arranged side by side and joined to form one image) may be transmitted. In this case, the information terminal 150 is provided with a unit corresponding to the image synthesizing unit 220 as a moving image data acquisition unit.

Further, at which stage the rotation parameter data is transmitted to the information terminal 150 is arbitrary. For example, the acceleration data and the angular velocity data described above may be transmitted to the information terminal 150 as a rotation parameter. In this case, the zenith correction data, the first rotation correction data, and the second rotation correction data are calculated on the information terminal 150 side from the acceleration data and the angular velocity data. In this case, the information terminal 150 includes the first rotation amount calculation unit 224 and a unit corresponding to the first rotation amount calculation unit 224 as the sensor data acquisition unit. Alternatively, the zenith correction data, the first rotation correction data, and the second rotation correction data may be transmitted to the information terminal 150 as rotation parameters. In this case, the information terminal 150 only needs to include a receiving unit that receives the zenith correction data, the first rotation correction data, and the second rotation correction data.

In the embodiment described with reference to FIGS. 4, 8A and 8B described above, the omnidirectional video system includes two devices, the omnidirectional camera 110 and the information terminal 150. However, the present invention is not limited to this.

For example, as shown in FIG. 9A, the second information terminal 170 connected to the first information terminal 150 may be included. Here, the second information terminal 170 is, for example, a head mounted display or the like. In the embodiment shown in FIG. 9A, only the omnidirectional camera 110 takes an image (S50), and the frame data and the rotation parameter before image processing are transmitted to the first information terminal 150 (S51). .. The first information terminal 150 performs image processing such as zenith correction and rotation correction on the received frame data (S52), and the moving image data after the image processing is transmitted to the second information terminal 170 ( S53). On the second information terminal 170, display is performed based on the moving image data after the image processing (S54).

In the embodiment shown in FIG. 9A, the first information terminal 150 corresponds to the moving image data acquisition unit (reception unit), the sensor data acquisition unit (reception unit), and the rotation correction unit (image rotation unit 230), respectively. Means) and operates as an image processing apparatus. The image synthesizing unit 220, the first rotation amount calculation unit 224, and the means corresponding to the first rotation amount calculation unit 224 may be provided in either the omnidirectional camera 110 or the first information terminal 150.

In addition, as shown in FIG. 9B, the spherical moving image system may include a server device 190. In the embodiment shown in FIG. 9B, only the imaging is performed on the omnidirectional camera 110 side (S70), and the frame data before the image processing and the rotation parameter are transmitted to the information terminal 150 (S71). The information terminal 150 transfers the received frame data and the rotation parameter to the server device 190 (S72), and causes the server device 190 to perform image processing such as zenith correction and rotation correction on the frame data (S73). The information terminal 150 receives download or streaming distribution of the moving image data after the image processing from the server device 190 (S74), and performs display display based on the moving image data after the image processing (S75). Further, access to the server device 190 is not limited to the information terminal 150.

In the embodiment shown in FIG. 9B, the server device 190 includes a moving image data acquisition unit (reception unit), a sensor data acquisition unit (reception unit), and a rotation correction unit (a unit corresponding to the image rotation unit 230). It operates as a provided image processing device. The image synthesizing unit 220, the first rotation amount calculation unit 224, and the unit corresponding to the first rotation amount calculation unit 224 may be provided in either the omnidirectional camera 110 or the server device 190.

In addition, when the omnidirectional camera 110 includes a display device, the omnidirectional camera 110 alone may configure the omnidirectional video system. In that case, the omnidirectional camera 110 may display on a display device included in itself.

Hereinafter, the zenith correction and the rotation correction in the present embodiment will be described in more detail with reference to FIGS. 10 to 13. FIG. 10 is a rotation process for frame-combined data executed by the omnidirectional camera 110 (which may be the information terminal 150 or the server device 190 in a specific embodiment) included in the omnidirectional video system according to the present embodiment. It is a flowchart showing.

The process shown in FIG. 10 is started from step S100 in response to the omnidirectional camera 110 receiving a request for the corrected moving image data.

In step S101, the celestial sphere camera 110 acquires frame synthesis data and rotation parameters. Here, for convenience of description, the recording of the first frame data and the second frame data, the generation of the frame composite data by image composition, and the recording of the angular velocity data and the acceleration data are completed at the start of the rotation process. And However, in another embodiment, when performing real-time distribution or the like, the first frame data and the second frame data are recorded, the angular velocity data and the acceleration data are recorded, and the frame combined data is generated by the image combination, The rotation process shown in 10 can also be performed. In addition, here, it is assumed that the zenith correction data, the first rotation correction data, and the second rotation correction data have not been calculated.

In step S102, the omnidirectional camera 110 stores an initial value in the shooting front direction of the omnidirectional camera 110. The shooting front direction of the omnidirectional camera 110 at the start time is used when calculating the reference of the first rotation correction data and the second rotation correction data. Further, in step S102, the first frame is targeted for processing, and the process proceeds to step S103.

In step S103, the omnidirectional camera 110 calculates the tilt angle with respect to the gravity direction in the frame to be processed by the zenith correction amount calculation unit 222 based on at least the acceleration data, and writes it in the zenith correction data.

In step S104, the omnidirectional camera 110 detects the change amount of the rotation angle in the horizontal plane compared with the previous frame by the first rotation amount calculation unit 224. In step S105, the omnidirectional camera 110 uses the first rotation amount calculation unit 224 to integrate the angle change amount between the frames calculated in step S104, and calculate the angle change amount in the shooting front direction from the initial value. , Write the first rotation correction data. As described above, the first rotation correction data is time-series data of the angle change amount in the shooting front direction in the target frame from the initial value in the shooting front direction of the omnidirectional camera 110 in the horizontal coordinate system.

The processing of steps S103 to 105 can be executed for each frame. On the other hand, the processing of steps S106 to S108 can be executed for each specified number of frames. Here, the specified number of frames can be set to any value. Although it is possible to increase the followability by reducing the specified number of frames, if the specified number of frames is too small, the blur component at the time of progress and the blur component at the time of stillness are likely to be affected. Therefore, if the moving image is 60 frames per second (60 fps), the specified number of frames may be set to be, for example, about 10 to 30 frames.

In step S106, the celestial sphere camera 110 calculates the moving distance and the traveling direction vector of the celestial sphere camera 110 within the specified number of frames in the coordinate system of the horizontal plane by the second rotation amount calculation unit 226. It should be noted that the specified frame to be calculated here may be a predetermined number of frames before and after the central frame in the case of reproducing the stored spherical video data, or in the case of real-time distribution from the first frame. The frame may be up to a predetermined number of minutes.

In step S107, the omnidirectional camera 110 causes the second rotation amount calculation unit 226 to determine whether or not the movement distance calculated in step S106 is equal to or greater than a predetermined threshold. If it is determined in step S107 that the moving distance is equal to or greater than the predetermined threshold value (YES), the process proceeds to step S108. Here, the moving distance means the difference between the position of the first frame and the position of the last frame in the coordinate system of the horizontal plane in the specified number of frames.

In step S108, the omnidirectional camera 110 calculates the angle change amount of the traveling direction vector calculated in S105 from the initial value by the second rotation amount calculation unit 226, and writes the second rotation correction data. Here, the second rotation correction data is time-series data of the difference between the initial value of the traveling direction vector (equal to the initial value of the camera direction) and the traveling direction vector as an average for the specified number of frames as described above. is there.

On the other hand, if it is determined in step S107 that the moving distance is less than or equal to the threshold value (NO), the process directly proceeds to step S109.

In step S109, the omnidirectional camera 110 performs the zenith correction on the target frame of the frame composite data based on the zenith correction data (the tilt angle of the target frame therein) by the image rotation unit 230.

In step S110, the omnidirectional camera 110 uses the image rotation unit 230 to determine the rotation change amount (rotation correction amount) in the horizontal plane to be canceled in the target frame based on the first rotation correction data and the second rotation correction data. calculate. As described above, the first rotation correction data is a difference between the initial value in the shooting front direction of the reference frame and the shooting front direction at the time of the target frame in the horizontal plane, and the second rotation correction data is in the horizontal plane. In the moving direction of the reference frame (equal to the shooting front direction) and the moving direction at the time of the target frame. Therefore, by adding the first rotation correction data and the second rotation correction data, it is possible to finally calculate the rotation change amount to be canceled. In addition, when the specific direction is the initial value of the shooting front direction, the second rotation correction data is the default value (zero) until the moving distance becomes equal to or more than the threshold, and therefore the rotation change amount is the first rotation correction data. As defined by, the difference is based on the difference between the initial value in the shooting front direction and the shooting front direction in the horizontal plane of the correction target frame. On the other hand, when the specific direction is determined to be the traveling direction, the rotation change amount is based on the difference between the traveling direction and the shooting front direction in the correction target frame.

In step S111, the omnidirectional camera 110 performs rotation correction on the target frame based on the calculated rotation correction amount.

In step S112, the omnidirectional camera 110 determines whether or not there is a next frame. If it is determined in step S112 that there is a next frame (YES), the process loops to step S103. On the other hand, if it is determined in step S112 that there is no next frame (NO), the main rotation process ends in step S113.

By the above-described processing, the moving image is corrected so as to cancel the rotational change around the reference axis so that the display reference substantially coincides with the specific direction during shooting over at least a plurality of frames for a certain period. Become.

In the above-described embodiment, the rotation amount to be canceled is calculated based on the first rotation correction data and the second rotation correction data in step S110, and the rotation correction is collectively performed in step S111. However, the present invention is not limited to this, and in other embodiments, the rotation correction based on the first rotation correction data and the rotation correction based on the second rotation correction data may be sequentially applied. Further, in another embodiment, the zenith correction and the rotation correction may be collectively performed based on the zenith correction data, the first rotation correction data, and the second rotation correction data.

By the above rotation correction, it becomes possible to fix the reference for displaying an image in a specific direction such as the front direction of shooting of the camera or the advancing direction regardless of the low frequency rotation change due to swinging or the like.

Hereinafter, the relationship between the traveling direction and the shooting front direction and the rotation processing by the image rotation unit at that time will be described with reference to FIGS. 11 to 13. FIG. 11 is a diagram for explaining an example of the relationship between the traveling direction T and the shooting front direction H when the photographer wears the omnidirectional camera 110 on the head 50 and moves. FIG. 12 and FIG. 13 are diagrams illustrating a captured spherical image before rotation correction (left side) and a rotationally corrected spherical image (right side) when the photographer performs the operation illustrated in FIG. 11. is there.

FIG. 11 is a diagram showing the numbers 0, 1, 1 ′, and 1 to 10 in a time-series manner. In FIG. 11, all of the numbers fixed to the head 50 of the photographer at each time point indicated by the numbers. The state of the celestial sphere camera 110 is schematically shown together with the traveling direction T of the photographer (the celestial sphere camera 110) and the photographing front direction H of the celestial sphere camera 110 according to the swing motion.

In FIG. 11, focusing on the traveling direction T, in this example, it is shown that the photographer is traveling toward “North” in the period of numbers 0 to 5. Further, focusing on the shooting front direction H in FIG. 11, the photographer faces “North”, which is the same as the traveling direction at the time of numbers 0 to 1, and looks around by the swinging motion at the time of numbers 2 to 4. It is shown that the action is performed and at the time of number 5, the player is looking at “West” on the left side in the traveling direction. It should be noted that the numbers 1 and 1'represent the operation of returning to the original path after making a parallel translation to the right once, for example, in the case of yielding a passage when passing another person. Then, the photographer changes the traveling direction from "North" to "West" between the numbers 6 to 7 while looking at the "West", and at the time of the numbers 7 to 8, the same "West" as the traveling direction is set. It shows that you are progressing while watching. Also, at the time of number 9, while looking at the "west", you are looking at the "south" on the left side of the direction of travel, and at the time of number 10, you can see that you have changed the direction of travel to "south". There is. Note that the north, south, east, and west are drawn for convenience, but the north, south, east, and west are not necessarily identified.

12 and 13, the spherical image before rotation correction (left side) 300 and the spherical image after rotation correction (right side) corresponding to the numbers 0, 1, 1', 1 to 10 shown in FIG. ) 310 is shown schematically. Further, in the omnidirectional image (right side) 310 after the rotation correction, it is schematically shown that a predetermined range 316 set at the center of the omnidirectional image is displayed on the display screen of the information terminal 150. It represents.

In the periods of numbers 0, 1, 1', 1 to 5, the photographer is moving to the "north", but as shown on the left side of FIGS. 12 and 13, in the uncorrected spherical image 300, It can be seen that the image is rotated according to the person's swinging motion. More specifically, in the periods of numbers 0, 1, 1', and 1 to 5, the shooting front direction 302 is "North" → "Northwest (60 degrees from north to left)" → "Northeast (from north to right). "60 degrees)" → "northwest (45 degrees left from north)" → "west (90 degrees left from north)". On the other hand, as shown on the right side of FIGS. 12 and 13, in the corrected celestial sphere image 310, at the time points of numbers 0, 1, 1 ′, 1 to 5, in the shooting front direction and the traveling direction at the start time point. The display standard 314 is always fixed to a certain "north". Further, in the parallel movement at the time points 1 and 1′, the moving direction of the omnidirectional camera 110 changes momentarily, but since it is a short time change, in step S107 shown in FIG. It is determined that the moving distance in the frame is less than the threshold value, the traveling direction is not redetermined, and “North” remains.

To explain the rotation correction by extracting representative points, at the time of number 2 in FIG. 11, the celestial sphere camera 110 rotates 60 degrees to the left while the photographer is heading north. It is facing northwest. At this time, the photographer is different from the shooting front direction H of the omnidirectional camera 110 and the traveling direction T. Here, the rotation process of the image rotation unit 230 when the photographer changes to the state of the number 2 in FIG. 11 will be described. When the photographer changes the direction of the neck from the time point of number 1 to the time point of number 2, that is, when the direction of the imaging device is rotating leftward by 60 degrees, the image rotation unit 230 causes the first rotation correction. Based on the data, a rotation process of rotating the spherical image 60 degrees to the right is executed so as to cancel the rotation of the spherical image. This makes it possible to fix the default direction shown on the display to the traveling direction. Actually, the rotation process based on the first rotation correction data is sequentially executed for each frame that changes from the time point 1 to the time point 2. At this time, since there is no change in the traveling direction, the rotation process based on the second rotation correction data is not executed. The same applies to changes from time point 2 to time point 3, changes from time point 3 to time point 4, and changes from time point 4 to time point 5.

In Nos. 5 to 7, the photographer gradually changes the traveling direction to “West” which is the viewing direction, but as shown on the left side of FIG. The direction of the person is always looking at "west" and there is no change in the image. On the other hand, as shown on the right side of FIG. 13, in the corrected celestial sphere image 310, the reference 314 displayed in the updated traveling directions “northwest (number 6)” and “west (number 7)” respectively. It is fixed. Here, it is shown that the traveling direction at time point 6 is constant for a sufficiently long period of time, and the direction to be fixed once at the re-determined “northwest” is changing.

Here, the rotation process of the image rotation unit when the photographer changes to the state of the numbers 5, 6, and 7 in FIG. 11 will be described. When the photographer changes the traveling direction T without changing the photographing front direction H from the time point 5 to the time point 6, based on the first rotation correction data, the rotation of the spherical image with respect to the photographing front direction is the initial value. A rotation process of rotating 90 degrees to the right so as to be canceled by is executed. On the other hand, based on the second rotation correction data, the rotation processing of rotating the spherical image to the left by 45 degrees is executed based on the difference in the traveling direction from the initial traveling direction (the initial shooting front direction) of the spherical image. Become. Then, when the first rotation correction data and the second rotation correction data are summed up, a rotation process of rotating 45 degrees to the right is executed.

Further, in the state at the time of number 7, based on the first rotation correction data, a rotation process of rotating 90 degrees to the right is executed so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. Become. On the other hand, based on the second rotation correction data, a rotation process of rotating the celestial sphere image 90 degrees to the left is executed based on the difference in the direction of travel from the initial direction of travel of the celestial sphere image (initial shooting front direction). Then, when the first rotation correction data and the second rotation correction data are added together, there is no rotation correction.

The change in the traveling direction may be interpolated so that a smooth transition is made from the previous traveling direction to the next traveling direction. That is, it is possible to apply a value obtained by interpolating the angle change of 45 degrees in the periods of time points 5 and 6 and the periods of time points 6 and 7 in each frame therebetween. Alternatively, the change in the traveling direction may be an instantaneous change. For example, the screen display may be once blacked out, and the state in which the display reference is fixed in the previous traveling direction may be switched to the state in which the display reference is fixed in the new traveling direction.

On the other hand, in the change from the number 8 to the number 9, the photographer changes the viewing direction from “west” to “south” without changing the traveling direction. In this case, as shown on the left side of FIG. 13, at the time points 8 and 9, in the uncorrected spherical image 300, the orientation of the photographer changes from “west” to “south”, but Is fixed to "west", the display reference 314 is fixed to "west" as shown on the right side of FIG. On the other hand, with the numbers 9 to 10, the photographer changes the traveling direction from “west” to “south” without changing the viewing direction. In this case, as shown on the left side of FIG. 13, at the time points 9 and 10, in the uncorrected spherical image 300, the orientation of the photographer is always looking at “south”, but shown on the right side of FIG. As described above, the traveling direction is changed from “west” to “south” by the direction in which the reference 314 displayed is fixed.

Here, the rotation process of the image rotation unit when the photographer changes to the state at the numbers 8, 9, and 10 in FIG. 11 will be described. When the photographer changes the shooting front direction H without changing the traveling direction T from the time point of number 8 to the time point of number 9, based on the first rotation correction data, the rotation of the spherical image with respect to the shooting front direction is changed. A rotation process of rotating 180 degrees to the right so as to be canceled from the initial value is executed. On the other hand, based on the second rotation correction data, the rotation processing of rotating 90 degrees to the left based on the difference in the traveling direction from the initial traveling direction of the celestial sphere image (the initial shooting front direction) is performed. Become. Then, when the first rotation correction data and the second rotation correction data are added together, a rotation process of rotating 90 degrees to the right is executed. Further, in the state at the time of the number 10, based on the first rotation correction data, the rotation process of rotating 180 degrees to the right is executed so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. Become. On the other hand, based on the second rotation correction data, a rotation process of rotating 180 degrees to the left based on the difference in the traveling direction from the initial traveling direction of the celestial sphere image (the initial shooting front direction) is executed. Become. Then, when the first rotation correction data and the second rotation correction data are added together, there is no rotation correction.

In this way, the default direction displayed on the display can be fixed to either the shooting front direction at the start of shooting or the traveling direction.

Hereinafter, the zenith correction and the rotation correction in another embodiment will be described in more detail with reference to FIG. FIG. 14 is a rotation with respect to the frame composite data executed by the omnidirectional camera 110 (which may be the information terminal 150 or the server device 190 in a specific embodiment) which constitutes the omnidirectional video system according to another embodiment. It is a flowchart which shows a process. Note that FIG. 14 shows an embodiment in which the advancing direction is not determined, and the display reference is always fixed in the shooting front direction at the start of shooting.

The process shown in FIG. 14 is started from step S200 in response to the omnidirectional camera 110 accepting the request for the corrected moving image data.

In step S201, the omnidirectional camera 110 acquires the frame composite data and the rotation parameter. Similar to the embodiment shown in FIG. 10, for convenience of description, at the start of the rotation process, recording of the first frame data and the second frame data, generation of frame composite data by image composition, recording of angular velocity data and acceleration data. Have been completed.

In step S202, the omnidirectional camera 110 stores the initial value in the shooting front direction of the omnidirectional camera 110. In step S203, in the omnidirectional camera 110, the zenith correction amount calculation unit 222 calculates the tilt angle with respect to the gravity direction in the frame to be processed based on at least the acceleration data, and writes the tilt angle in the zenith correction data.

In step S204, the omnidirectional camera 110 causes the first rotation amount calculation unit 224 to integrate the angle change amount of the coordinate system of the horizontal plane for each clock and calculate the change amount of the rotation angle in the horizontal plane compared to the previous frame. To detect. In step S205, the omnidirectional camera 110 uses the first rotation amount calculation unit 224 to integrate the angle change amounts between the frames calculated in step S204, and calculate the angle change amount in the shooting front direction from the initial value. , Write the first rotation correction data.

In step S206, the omnidirectional camera 110 performs the zenith correction on the target frame of the frame composite data based on the zenith correction data (the tilt angle of the target frame therein) by the image rotation unit 230.

In step S207, the omnidirectional camera 110 causes the image rotation unit 230 to perform rotation correction on the target frame based on the first rotation correction data. As described above, the first rotation correction data is the difference between the initial value in the shooting front direction of the reference frame and the shooting front direction at the time of the target frame in the horizontal plane, and the first rotation correction data in the horizontal plane to be canceled in the target frame. The amount of rotation change is expressed as it is.

In step S208, the omnidirectional camera 110 determines whether there is a next frame. If it is determined in step S208 that there is a next frame (YES), the process loops to step S203. On the other hand, if it is determined in step S208 that there is no next frame (NO), the main rotation processing ends in step S209.

By the above process, the rotation change around the reference axis is canceled so that the display reference substantially coincides with the shooting front direction at the start of shooting over a plurality of frames from the start to the end of shooting for the moving image. Such correction will be applied.

As described above, according to the present embodiment, an image processing device, an imaging device, a moving image reproducing system, a method, and a method that can suppress motion sickness of a viewer caused by rotation around a reference axis when capturing an image when viewing a moving image. It becomes possible to provide the program.

In a specific embodiment, since the display standard at the time of viewing is fixed in the traveling direction of the photographer, it is possible to suppress the motion sickness of the viewer and to give the viewer a realistic sensation close to the scenery seen by the photographer. It will be possible.

In addition, in the above-described embodiment, the spherical camera 110 is fixed to the head 50 of the photographer. The rotation correction function according to the present embodiment can be suitably applied to such a case, but the application scene is not particularly limited, and it is also applicable to the case where the user holds the image by hand. Needless to say.

Moreover, in the above-described embodiment, the first rotation correction data and the second rotation correction data are calculated using the angular velocity sensor 136A and the acceleration sensor 136B. However, in other embodiments, the first rotation correction data and the second rotation correction data may be calculated using the geomagnetic sensor when calculating the angle change amount. In that case, the initial value of the geomagnetic sensor is stored, the rotation angle of the coordinate system of the horizontal plane is calculated based on the difference between the initial value of the geomagnetic sensor and the direction indicated by the geomagnetic sensor in the target frame, and the first rotation correction data and The second rotation correction data can be calculated. Further, in the above description, the frame synthesis data, the acceleration data, and the angular velocity data are described as being stored in advance. It can also be applied to a stream of data.

Furthermore, in the above-described embodiment, the zenith correction is performed on the moving image of the spherical coordinate system whose coordinate system includes two angular coordinates. However, depending on the embodiment, there is a case in which the inclination in the direction of gravity that does not require the zenith correction does not occur, and in such a case, the zenith correction can be omitted. In this case, the reference axis may be an axis of the angular coordinate if the coordinate system is a cylindrical coordinate system including one angular coordinate, and two angular coordinates if the coordinate system is a spherical coordinate system including two angular coordinates. May be used as the axis for giving.

The functional unit can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark), or an object-oriented programming language, and can be ROM, EEPROM, EPROM. , Flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, or other device-readable recording medium, or through an electric communication line. It can be distributed. Further, a part or all of the above-mentioned functional units can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA), or as an ASIC (application specific integration). Circuit configuration data (bit stream data) to be downloaded to the PD in order to realize the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, and VHDL (Very High Speed Integrated Circuits). Hardware Description Language), Verilog-HDL, and the like can be distributed by a recording medium as data described.

Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art can conceive of other embodiments, additions, changes, deletions, and the like. The present invention can be modified within the scope of the present invention, and is included in the scope of the present invention as long as the effects and advantages of the present invention are exhibited in any of the aspects.

12... Imaging body, 14... Housing, 18... Shutter button, 20... Imaging optical system, 22, 130... Imaging element, 110... Spherical camera, 112, 152... CPU, 114... ROM, 116... Image Processing block, 118... Video compression block, 120, 126... Interface, 122... External storage interface, 124... External sensor interface, 126... USB interface, 128... Serial block, 132... DRAM, 134, 160... External storage, 136A... Angular velocity sensor 136B... Acceleration sensor, 138, 166... USB connector, 150... Information terminal, 154... RAM, 156... Internal storage, 158... Input device, 162... Display, 164... Wireless NIC, 200... Functional block, 212... 1st imaging part, 214... 2nd imaging part, 216... Angular velocity detection part, 218... Acceleration detection part, 220... Image composition part, 222... Zenith correction amount calculation part, 224... 1st rotation amount calculation part, 226... 2 rotation amount calculation unit, 228... storage unit, 230... image rotation unit, 240... output unit, 242... instruction receiving unit, 252... receiving unit, 254... display control unit

JP, 2017-147682, A

Claims (16)

An image processing device for processing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis,
Video data acquisition means for acquiring video data,
Sensor data acquisition means for acquiring sensor data corresponding to the moving image data,
Based on the sensor data, a rotation correction unit that performs correction to cancel the rotation change around the reference axis on the image so that the display reference of the image is fixed in a specific direction during shooting over a plurality of frames of the moving image. An image processing apparatus including: A direction within a plane perpendicular to the reference axis based on the attitude of the camera that captures the moving image at the reference frame at a predetermined time point is defined as the specific direction during the shooting, and the amount of the rotational change is the reference frame. The image processing apparatus according to claim 1, wherein the image processing apparatus is based on a difference between a direction based on the orientation of the image and a direction within a plane perpendicular to the reference axis based on the orientation of the camera in the correction target frame. The image processing apparatus according to claim 2, wherein the predetermined time point is a time point when the shooting of the moving image is started or a time point when an instruction during shooting the moving image is received. The certain period corresponding to the plurality of frames is a period from a first time point when the start or fixing of the moving image is instructed to a second time point when the end or fixing of the moving image is instructed. The image processing apparatus according to claim 2. The rotation direction when the traveling direction is determined as the specific direction, the determining unit determining the traveling direction of the camera moving in a plane perpendicular to the reference axis as the specific direction during the photographing. 4. The amount of change is based on a difference between the traveling direction and a direction in a plane perpendicular to the reference axis based on the posture of the camera in a correction target frame, according to claim 1. Image processing device. The determining means determines, based on the sensor data, the traveling direction obtained again when the moving distance in a predetermined number of frames satisfies a predetermined condition, as the specific direction during the photographing. The image processing apparatus according to claim 5, wherein the fixed period corresponding to the plurality of frames corresponding to the specific direction during the photographing is a period from the determination to the next determination. The coordinate system further includes angular coordinates with respect to the predetermined axis, the image processing device,
And a zenith correction means for performing zenith correction on each frame of the moving image based on an inclination angle of each frame of the moving image with respect to the reference axis when the image is captured. The image processing apparatus according to claim 1, further comprising: The moving image includes a spherical image represented by a coordinate system including two angular coordinates as each frame, the moving image is captured by a plurality of image pickup elements, and the spherical image is The moving image data includes, as each frame, the omnidirectional image or the plurality of partial images, and the correction is performed on the omnidirectional image. The image processing apparatus according to claim 1, wherein the reference axis is applied to a spherical image or the plurality of partial images, and the reference axis is a gravity direction. The sensor data corresponding to the moving image is at least one of data output by the acceleration sensor, data output by the angular velocity sensor, data output by the geomagnetic sensor, and rotation correction data calculated based on the data output by these sensors. The image processing device according to claim 1, comprising one. An imaging device for capturing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis,
Imaging means for imaging each frame of the moving image,
Recording means for recording sensor data indicating a displacement amount of the imaging device in an absolute coordinate system;
Based on the sensor data, a rotation correction unit that performs correction to cancel the rotation change around the reference axis on the image so that the display reference of the image is fixed in a specific direction during shooting over a plurality of frames of the moving image. An imaging device comprising: The image pickup apparatus according to claim 10, further comprising a receiving unit that receives an instruction to specify the specific direction during shooting of the moving image. A moving image reproducing system for reproducing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis,
Video data acquisition means for acquiring video data,
Sensor data acquisition means for acquiring sensor data corresponding to the moving image data,
Based on the sensor data, a rotation correction unit that performs correction to cancel the rotation change around the reference axis on the image so that the display reference of the image is fixed in a specific direction during shooting over a plurality of frames of the moving image. When,
And a display control unit that causes a display unit to display at least a part of the corrected image of the moving image on the basis of the display standard. 13. The moving image reproduction system according to claim 12, further comprising a rotation amount calculation means for calculating a specific direction during shooting in a plane coordinate system perpendicular to the reference axis. The moving image data acquisition unit includes an image capturing unit that captures each frame of a moving image, and the sensor data acquisition unit outputs output data of a sensor provided together with the image capturing unit or correction data calculated from the output data. Includes a recording means for recording in association with each frame of the moving image, or
The moving image data acquisition means and the sensor data acquisition means receive, together with each frame of the moving image captured by the image capturing means, output data of a sensor provided with the image capturing means or correction data calculated from the output data. Including means,
The video playback system according to claim 12 or 13. A method for processing a moving image including a series of images represented by a coordinate system including at least angular coordinates around a predetermined axis, the computer comprising:
A step of obtaining video data,
Obtaining sensor data corresponding to the moving image data,
Applying a correction to the image based on the sensor data so as to cancel the rotational change about the reference axis so that the display reference of the image is fixed in a specific direction during shooting over a plurality of frames of the moving image. How to do. A program for realizing an image processing device for processing a moving image including a continuous image, which is represented by a coordinate system including at least angular coordinates around a predetermined axis, the computer comprising:
Video data acquisition means for acquiring video data,
A sensor data acquisition unit that acquires sensor data corresponding to the moving image data, and based on the sensor data, so that the display reference of the image is fixed in a specific direction during shooting over a plurality of frames of the moving image, A program for functioning as a rotation correction means for performing a correction for canceling a rotation change around a reference axis on the image.