JP2020096350A - Image processing device, imaging device, moving image reproduction system, method, and program - Google Patents

Abstract

To provide an image processing device.SOLUTION: An image processing device 150 processes a moving image including continuous images represented by a coordinate system including at least angular coordinates around a predetermined axis. The moving image is a rotation-corrected image. The image processing device 150 includes: data acquisition means 252 for acquiring data indicating a predetermined direction during photographing for each frame of the moving image; and output control means 254 for performing control so as to output a screen including an image range of at least part of the moving image. In response to establishment of a predetermined condition, the output control means 254 performs control to add an expression indicating the predetermined direction during photographing in a screen based on the data.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to an image processing device, an imaging device, a moving image reproduction system, a method, and a program.

2. Description of the Related Art Conventionally, a technique of shooting a moving image in all directions at a shooting point is known. Further, there is known a technique of displaying a part of an omnidirectional moving image on a display, a head mounted display (HMD), or the like in a format in which a predetermined direction is viewed from a shooting point. Further, in recent years, a fixture for fixing the image pickup device to the head and photographing the image has been known. By fixing the image pickup device to the head, it is possible to take an image in all directions at a position close to the user's line of sight, and the hand for holding the normal image pickup device is also released.

When the image pickup device is fixed to the head and is being taken, the center direction of the image pickup of the image pickup device is located in the center of the taken moving image. For this reason, when viewing a moving image captured in such a manner, the viewer shares the direction in which the photographer is looking in accordance with the frame of the moving image. However, when the change in the center direction of the camera due to a swing or the like is large, the change in the image is also large, which may cause a sickness to the viewer.

Japanese Patent Laid-Open No. 2016-149752 (Patent Document 1) is known as a countermeasure against the motion sickness when viewing the spherical image. In the related art of Patent Document 1, a moving image is captured so that the zenith direction of the captured moving image is oriented in the direction of the gravity vector in order to prevent motion sickness that occurs when the moving image is captured while the camera body is tilted. It is for rotating coordinate conversion. According to the conventional technique of Patent Document 1, it is possible to suppress the viewer's motion sickness due to the inclination of the video to a certain extent, but it has not dealt with the motion sickness caused by the rapid rotation of the video around the direction of gravity.

The present disclosure has been made in view of the above point, and the present disclosure can share a predetermined direction during shooting with a viewer while suppressing motion sickness of the viewer caused by rotation around a reference axis in a moving image. An object of the present invention is to provide an image processing device.

In order to address the above problems, the present disclosure provides an image processing device having the following features, which processes a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis. Here, the moving image is an image whose image has been rotationally corrected. The image processing device includes a data acquisition unit that acquires data indicating a predetermined direction at the time of capturing an image in each frame of a moving image, and an output control unit that controls to output a screen including an image range of at least a part of the moving image. .. The output control means is characterized in that, in response to the satisfaction of a predetermined condition, the output control means controls to add an expression indicating a predetermined direction at the time of the image pickup in the screen based on the data.

With the above configuration, it is possible to share the predetermined direction during shooting with the viewer while suppressing motion sickness of the viewer caused by rotation around the reference axis in the moving image.

Sectional drawing of the spherical camera which comprises the spherical video system by this embodiment. The figure explaining the case where the spherical camera by this embodiment is fixed to a head. The hardware block diagram of the spherical video system by this embodiment. FIG. 3 is a main functional block diagram related to a spherical video playback function realized on the spherical video system according to the present embodiment. (A) An image data flow diagram in the generation of a spherical image, and a diagram illustrating a case where the data structure of the spherical image is represented by a plane (B) and a case (C) represented by a spherical surface. The figure explaining zenith correction and rotation correction of the omnidirectional image by this embodiment. The figure explaining the omnidirectional image obtained by the zenith correction and rotation correction of the omnidirectional image according to this embodiment. FIG. 6 is a sequence diagram illustrating processing from shooting to viewing of a moving image in the spherical video system according to a specific embodiment. FIG. 11 is a sequence diagram illustrating processing from shooting to viewing of a moving image in the spherical video system according to another specific embodiment. 6 is a flowchart showing a rotation process for frame composite data executed by the omnidirectional camera which constitutes the omnidirectional video system according to the present embodiment. The figure explaining the relationship between the advancing direction and the front direction of photography when a photographer wears and moves the omnidirectional camera on his head. FIG. 12 is a diagram (1/2) illustrating a captured spherical image before rotation correction and a captured spherical image after rotation correction when the photographer performs the operation shown in FIG. 11. FIG. 12 is a diagram (2/2) illustrating a captured spherical image before rotation correction and a captured spherical image when the photographer performs the operation shown in FIG. 11. 9 is a flowchart showing a rotation process for frame composite data, which is executed by a omnidirectional camera that constitutes an omnidirectional video system according to another embodiment. The flowchart which shows the spherical video reproduction process which the information terminal which comprises the spherical video system by this embodiment performs. In the present embodiment, the corrected celestial sphere moving image and the image range set therein are in the traveling direction and the photographing front direction when the photographer wears the celestial sphere camera and moves, and FIG. 4 is a diagram showing in time series in association with an uncorrected spherical image captured at a time point. The figure explaining the display method of the instruction icon in a specific embodiment.

Hereinafter, the present embodiment will be described, but the embodiment is not limited to the embodiment described below. In the following embodiments, an information processing device, an omnidirectional camera connected to the information terminal, and an omnidirectional imaging system will be described as examples of an image processing device, an imaging device, and a moving image reproduction system, respectively.

Hereinafter, the overall configuration of the celestial sphere moving image system according to the present embodiment will be described with reference to FIGS. 1 to 3.

FIG. 1 is a cross-sectional view of a celestial sphere camera 110 that constitutes the celestial sphere video system according to the present embodiment. An omnidirectional camera 110 shown in FIG. 1 includes an image pickup body 12, a casing 14 that holds the image pickup body 12, components such as a controller 15 and a battery 16, and a shutter button 18 provided in the casing 14. Equipped with.

The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B, and two imaging elements 22A and 22B such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. It Each of the image forming optical systems 20 is configured as, for example, a fisheye lens with 7 elements in 6 groups. In the embodiment shown in FIG. 1, the fish-eye lens has a total angle of view larger than 180 degrees (=360 degrees/n; the number of optical systems n=2), and preferably has a field angle of 185 degrees or more. However, more preferably, the angle of view is 190 degrees or more. Such a combination of the wide-angle imaging optical system 20 and the image pickup device 22 is referred to as a wide-angle image pickup optical system. In the embodiment to be described, the case where the number of optical systems is 2 will be described as an example, but the number of optical systems may be 1 as long as it is possible to capture a wide-angle region. It may be 3 or more. Further, it is not limited to a fisheye lens having an angle of view of 180 degrees or more.

In the embodiment shown in FIG. 1, the image forming optical systems 20A and 20B have the same specifications, and they are combined in opposite directions so that their optical axes coincide with each other. The image pickup devices 22A and 22B convert the received light distribution into image signals and sequentially output image frames to the image processing means on the controller. As will be described in detail later, the images respectively captured by the image pickup devices 22A and 22B are combined to generate an image with a solid angle of 4π steradians (hereinafter referred to as a “spheroidal image”). .. The celestial sphere image is a photograph of all directions that can be viewed from the photographing point. Then, a spherical moving image is composed of continuous frames of the spherical image.

Here, in the embodiment to be described, it is assumed that a omnidirectional image and a omnidirectional moving image are generated. However, a omnidirectional image and a omnidirectional moving image, a so-called panoramic image and panorama obtained by photographing only 360 degrees in a horizontal plane It may be a moving image, or may be a hemispherical image or a semi-cylindrical image captured at an angle of view of 180 degrees, and may be image data having image information on a spherical surface or a cylinder. That is, the moving image and the image included in the moving image according to the present embodiment are images represented by a coordinate system including angular coordinates around a predetermined axis. In the case of the above-mentioned spherical image/moving image, hemispherical image/moving image, and partial spherical image, the radius vector is set as a constant (for example, 1), and the angular coordinate representing the angle between the predetermined axis and the radius vector and the predetermined axis It is represented by a spherical coordinate system that includes the angular coordinates of. In the case of the panoramic image/moving image or its partial image, the radius vector is represented by a cylindrical coordinate system including the coordinate in the axial direction of the cylinder and the angular coordinate around the axis, with the radius vector being a constant (for example, 1).

FIG. 2 illustrates a case where the omnidirectional camera 110 of FIG. 1 is fixed to the photographer's head 50. FIG. 2 is an example in which the omnidirectional camera 110 is fixed to the hat-type fixing device 52. By mounting the omnidirectional camera 110 on the photographer's head 50 in this way, the image capturing front direction of the image data imaged by the celestial sphere camera 110 follows the photographer's swinging motion H. Become. The shooting front direction is a preset direction and is a reference direction displayed when playing back moving image data. FIG. 2 also shows the moving direction T of the photographer. The photographer may proceed while looking around the area, and the front direction of photographing by the swinging motion H and the traveling direction T do not necessarily match. As shown in FIG. 2, in addition to fixing the omnidirectional camera 110 to the hat-type fixing device 52 and attaching it to the head 50, the omnidirectional camera 110 itself can be attached to the head 50. It may be (helmet or hat with embedded spherical camera function).

FIG. 3A shows a hardware configuration of the omnidirectional camera 110 that constitutes the omnidirectional video system according to the present embodiment. The omnidirectional camera 110 corresponds to the image processing device or the imaging device in the embodiments described.

The spherical camera 110 is connected via a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a moving image compression block 118, and a DRAM (Dynamic Random Access Memory) interface 120. The DRAM 132 and the sensor 136 connected via the external sensor interface 124 are included. Here, the CPU 112, the ROM 114, the image processing block 116, and the moving image compression block 118 are mounted on the controller 15 shown in FIG.

The CPU 112 controls the operation and overall operation of each unit of the spherical camera 110. The ROM 114 stores an operating system (OS), a control program, and various parameters described in a code that can be read by the CPU 112. The image processing block 116 is connected to the two image pickup devices 130A and 130B (the image pickup devices 22A and 22B in FIG. 1), and the image signals of the images picked up by the respective image pickup blocks 130A and 130B are input. The image processing block 116 is configured to include an ISP (Image Signal Processor) and the like, and performs shading correction, Bayer interpolation, white balance correction, gamma correction, and the like on the image signal input from the image sensor 130.

The moving picture compression block 118 is an MPEG-4 AVC/H. It is a codec block that performs video compression and decompression such as H.264. The DRAM 132 provides a storage area for temporarily storing data when performing various kinds of signal processing and image processing.

In the embodiment shown in FIG. 3A, the sensor 136 includes an angular velocity sensor 136A and an acceleration sensor 136B. The angular velocity sensor 136A is a sensor that detects triaxial angular velocity components. The acceleration sensor 136B is a sensor that detects triaxial acceleration components. The detected acceleration component and angular velocity component are used to perform zenith correction of the omnidirectional image in the direction of gravity and rotation correction around the direction of gravity, as will be described later. As the sensor 136, another sensor such as a geomagnetic sensor (also referred to as an electronic compass) for obtaining an azimuth angle or the like may be further provided, and the azimuth angle may be used for correcting the rotation around the gravity direction. ..

The spherical camera 110 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, and a serial block 128. An external storage 134 is connected to the external storage interface 122. The external storage interface 122 controls reading and writing with respect to the external storage 134 such as a memory card inserted in the memory card slot. A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with an external device such as a smartphone connected via the USB connector 138. The serial block 128 controls serial communication with an external device such as a smartphone and is connected to a wireless NIC (Network Interface Card) 140.

When the power is turned on by operating the power switch, the OS and the control program are loaded into the main memory. The CPU 112 controls the operation of each part of the device according to the program read into the main memory, and temporarily stores the data necessary for the control in the memory. As a result, the functional units and processes of the spherical camera 110 described later are realized.

FIG. 3B shows a hardware configuration of the information terminal 150 that constitutes the spherical video system according to the present embodiment. The information terminal 150 corresponds to the information processing device in the embodiment described.

The information terminal 150 illustrated in FIG. 3B includes a CPU 152, a RAM 154, an internal storage 156, an input device 158, an external storage 160, a display 162, a wireless NIC 164, and a USB connector 166. ..

The CPU 152 controls the operation of each unit of the information terminal 150 and the overall operation. The RAM 154 provides a work area for the CPU 152. The internal storage 156 stores a control program such as an operating system described in a code readable by the CPU 152 and an application responsible for processing on the information terminal 150 side according to the present embodiment. The CPU 152 may include a plurality of CPUs.

The input device 158 is an input device such as a touch screen and provides a user interface. The input device 158 receives various instructions from the operator such as correction of spherical video. The external storage 160 is a removable recording medium mounted in a memory card slot or the like, and records various data such as moving image data and still image data. The display 162 displays the corrected spherical video in response to a user operation. The wireless NIC 164 establishes a wireless communication connection with an external device such as the spherical camera 110. The USB connector 166 makes a USB connection with an external device such as the spherical camera 110.

When the information terminal 150 is turned on and turned on, the control program is read from the internal storage 156 and loaded into the RAM 154. The CPU 152 controls the operation of each part of the device according to the control program read in the RAM 154, and temporarily stores the data necessary for the control in the memory. Thereby, each functional unit and processing of the information terminal 150, which will be described later, are realized.

Hereinafter, with reference to FIG. 4 to FIG. 9, the omnidirectional video reproduction function provided in the omnidirectional video system according to the present embodiment will be described.

FIG. 4 shows the main functional blocks 200 related to the spherical video playback function realized on the spherical video system according to the present embodiment. As shown in FIG. 4, the functional block of the omnidirectional camera 110 is configured to include an image processing unit 210 and an output unit 240. The image processing unit 210 includes a first imaging unit 212, a second imaging unit 214, an angular velocity detection unit 216, an acceleration detection unit 218, an image composition unit 220, a zenith correction amount calculation unit 222, and a first rotation amount. The calculation unit 224, the second rotation amount calculation unit 226, the storage unit 228, and the image rotation unit 230 are included. The processing of each unit is performed by the CPU 112 and the image processing block 116 included in the controller 15 executing various programs.

On the other hand, the functional block of the information terminal 150 is configured to include the receiving unit 252 and the display control unit 254, as also shown in FIG. The processing of each unit is performed by the CPU 152 included in the information terminal 150 executing various programs.

Hereinafter, first, the functional blocks on the spherical camera 110 side will be described.

The first image capturing unit 212 sequentially captures images using the image sensor 130A and generates a fish-eye image A that is continuous in time series. The second image capturing unit 214 sequentially captures images using the image sensor 130B and generates a fish-eye image B that is continuous in time series. The image combining unit 220 performs stitching processing on the fisheye image A (first frame data) that is continuous in time series and the fisheye image B (second frame data) that is continuous in time series to combine frames. Create the data. The frame composite data is moving image data which is a main body of the spherical moving image data and includes a plurality of frames each of which forms a spherical moving image from the start to the end of shooting. The fish-eye image A and the fish-eye image B are partial images for forming the omnidirectional image, and the first and second frame data also indicate the frames of the partial images from the start to the end of shooting. This is the moving picture data that can compose a spherical video. The image composition unit 220 outputs the created frame composition data to the image rotation unit 230. The first image capturing section 212, the second image capturing section 214, and the image synthesizing section 220 can configure the image capturing means in this embodiment.

Hereinafter, generation of a spherical image and a generated spherical image will be described with reference to FIG. FIG. 5A illustrates the data structure of each image and the data flow of the image in the spherical image generation. First, the image directly captured by each of the image pickup devices 130 is an image in which the hemisphere of the whole celestial sphere is included in the visual field. The light incident on the imaging optical system is imaged on the light receiving area of the image sensor 130 according to a predetermined projection method. The image captured here is an image captured by a two-dimensional image sensor whose light-receiving area forms a planar area, and is image data expressed in a planar coordinate system. In addition, typically, the obtained image is a fish including the entire image circle in which each shooting range is projected, as shown by “fisheye image A” and “fisheye image B” in FIG. It is configured as an eye image.

The plurality of fish-eye images of each frame captured by the plurality of image sensors 130 are subjected to distortion correction and synthesis processing to form one spherical image corresponding to each frame. In the combining process, first, each hemispherical image including each complementary hemispherical portion is generated from each fisheye image configured as a planar image. Then, the two hemispherical images including the respective hemispherical portions are stitched (image combination) based on the matching of the overlapping regions, and a spherical image including the entire spherical image is generated.

FIG. 5B is a diagram illustrating the data structure of the image data of the omnidirectional image used in the present embodiment by showing it in a plane. FIG. 5C is a diagram illustrating the data structure of the image data of the omnidirectional image represented by a spherical surface. As shown in FIG. 5B, the image data of the spherical image has coordinates of a vertical angle φ with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the predetermined axis. It is expressed as an array of pixel values. The vertical angle φ is in the range of 0 degrees to 180 degrees (or -90 degrees to +90 degrees), and the horizontal angle θ is in the range of 0 degrees to 360 degrees (or -180 degrees to +180 degrees).

Each coordinate value (θ, φ) in the omnidirectional format is associated with each point on the spherical surface that represents the azimuth around the shooting point, as shown in FIG. It is associated with the spherical image. The plane coordinates of the fisheye image captured by the fisheye lens and the spherical coordinates of the spherical image are associated with each other in a predetermined conversion table. The conversion table is data created in advance by a manufacturer or the like according to a predetermined projection model based on design data of each lens optical system, etc., and is data for converting a fisheye image into a spherical image.

It should be noted that in the embodiment to be described, the flow is such that the fisheye image is converted into a celestial sphere image, and the frame synthesizing data after the stitching process is rotated by the image rotating unit 230. However, the order of conversion processing, combination processing, and rotation processing is not particularly limited. The fisheye image A and the fisheye image B that are partial images (two hemisphere images including complementary hemisphere parts obtained by converting the fisheye images) are subjected to rotation processing (correction) by the image rotation unit 230 and then stitched. It is also possible to use a flow for performing a ring processing. Further, in addition to performing rotation processing on the image in the spherical format, the rotation processing is reflected in the conversion table for converting the fisheye image into a spherical image, and based on the conversion table in which the rotation processing is reflected, It is also possible to obtain the omnidirectional image after the rotation processing from the fisheye image A and the fisheye image B. In the present embodiment, “correcting a moving image” refers to finally obtaining a moving image including an image in which a predetermined correction is reflected, regardless of the order of the conversion process, the combining process, and the rotation process. Further, in the present embodiment, the spherical image is created by stitching two fish-eye images, but the present invention is not limited to this, and in other embodiments, for example, stitching three or more images. It may be one that creates a spherical image or a panoramic image.

Here, refer to FIG. 4 again. The angular velocity detection unit 216 is an angular velocity detection mechanism including an angular velocity sensor 136A, and outputs angular velocity data obtained by measuring the angular velocity components in the three axis directions using the angular velocity sensor 136A during shooting. The angular velocity data is time-series data of angular velocities generated around the three axes of the omnidirectional camera 110 (the angular velocity sensor 136A therein) from the start to the end of photographing. The angular velocity data does not need to be recorded in a one-to-one correspondence with a frame, but typically can be recorded at a rate faster than the frame rate. In this case, the time stamp can be used as a clue to find the correspondence with each frame later. Alternatively, the angular velocity may be recorded in a one-to-one correspondence with each frame of the spherical video. In the present embodiment, the angular velocity detection unit 216 constitutes a recording unit that records the sensor data in association with each frame of the moving image, and the angular velocity data may be included in the sensor data in the present embodiment.

The acceleration detection unit 218 is an acceleration detection mechanism including an acceleration sensor 136B, and outputs acceleration data obtained by measuring acceleration components in the three axis directions using the acceleration sensor 136B during shooting. The acceleration data is time-series data of three-axis acceleration of the celestial sphere camera 110 (the acceleration sensor 136B therein) from the start to the end of shooting. Acceleration data need not be recorded in a one-to-one correspondence with frames, but typically can be recorded at a rate faster than the frame rate. In this case, the time stamp can be used as a clue to find the correspondence with each frame later. Alternatively, the acceleration may be recorded in a one-to-one correspondence with each frame of the spherical video. In the present embodiment, the acceleration detection unit 218 constitutes a recording unit that records the sensor data in association with each frame of the moving image, and the acceleration data may be included in the sensor data in the present embodiment.

The above-described acceleration data and angular velocity data directly obtained from the acceleration sensor 116B and the angular velocity sensor 136A are data based on the three axis directions of the sensor coordinate system (or the coordinate system of the omnidirectional camera 110). On the other hand, the acceleration sensor 136B outputs the sum of the acceleration based on the motion and the gravitational acceleration, and the above-described acceleration data and angular velocity data can be converted into an absolute coordinate system based on the gravitational acceleration axis. Therefore, the sensor data described above indicates the amount of displacement in the absolute coordinate system.

The zenith correction amount calculation unit 222 calculates the tilt angle at the time of imaging with respect to the reference axis in each frame based on the acceleration data output from the acceleration detection unit 218, and creates zenith correction data. The zenith correction data is time-series data of the tilt angle of the omnidirectional camera 110 with respect to the reference axis from the start to the end of shooting, which is recorded in association with each frame of the omnidirectional video. The tilt angle with respect to the reference axis is typically configured as a vector composed of acceleration dimension values. The reference axis typically corresponds to the direction of gravity in which gravitational acceleration acts, and the description will be continued below assuming that the reference axis is the direction of gravity. Since the acceleration sensor 136B does not distinguish between gravity and inertial force, the tilt angle obtained from the acceleration sensor 136B may be corrected, preferably based on the signal measured by the angular velocity sensor 136A.

The first rotation amount calculation unit 224 determines, based on the acceleration data output from the acceleration detection unit 218 and the angular velocity data output from the angular velocity detection unit 216, from the amount of change in the relative angle of the omnidirectional camera 110 in the direction of gravity. On the other hand, a vector in the shooting front direction of the omnidirectional camera 110 in the coordinate system of a plane (horizontal plane) in the vertical direction is calculated for each frame, and the first rotation correction data is recorded. The first rotation correction data is time-series data of the angle change amount in the front direction of shooting from the reference value from the start to the end of shooting, which is recorded in association with each frame of the celestial sphere moving image. The angle change amount in the image capturing front direction is the image capturing front direction in the horizontal plane based on the posture of the spherical camera 110 in the reference frame and the image capturing front direction in the horizontal plane based on the posture of the spherical camera 110 in the target frame. It is represented by the difference from the direction. Here, the reference frame can be, for example, a frame at the start of shooting or a frame at the start of applying the main correction (in the case of an embodiment in which application of the correction can be instructed after the start of shooting). The first rotation correction data is represented by, for example, a numerical value of the rotation angle or a vector. The first rotation correction data may also be included in the sensor data in the present embodiment, and the first rotation amount calculation unit 224 constitutes a recording unit that records the sensor data in association with each frame of the moving image in the present embodiment. You can

The second rotation amount calculation unit 226 is based on the acceleration data output from the acceleration detection unit 218 and the angular velocity data output from the angular velocity detection unit 216 in the plane coordinate system of a plane (horizontal plane) perpendicular to the gravity direction. A vector in the traveling direction of the omnidirectional camera 110 is calculated for each prescribed number of frames, and second rotation correction data is generated. The traveling direction in the frame at the start of shooting is equal to the shooting front direction of the spherical camera 110 in the frame at the start of shooting. The second rotation correction data is recorded in association with each frame of the celestial celestial sphere moving image or each frame of the specified number of frames and is recorded as an angle change amount in the traveling direction from the reference value from the start to the end of shooting. It is time series data. The angle change amount of the traveling direction is the difference between the traveling direction of the celestial sphere camera 110 in the horizontal plane in the reference frame (imaging front direction) and the traveling direction of the celestial sphere camera 110 in the horizontal plane in the target frame. It is represented by. The second rotation correction data is, for example, a numerical value of a rotation angle or a vector. The second rotation correction data may also be included in the sensor data in the present embodiment, and the second rotation amount calculation unit 226 constitutes a recording unit that records the sensor data in association with each frame of the moving image in the present embodiment. You can

Regarding the change in the traveling direction, if the acceleration of the omnidirectional camera 110 in the specific direction satisfies a predetermined condition in the acceleration data, the specific direction at that time is determined to be the traveling direction. The predetermined condition can be, for example, a case where the acceleration in the specific direction exceeds a predetermined threshold value when the vectors of a plurality of frames are combined. When the predetermined condition is not satisfied, the traveling direction can inherit the data of the previous frame. The definition of the traveling direction is not particularly limited, and may be the traveling direction of the photographer or the traveling direction of the camera itself. The second rotation amount calculation unit 226 can also function as the determination unit in this embodiment.

The image rotation unit 230 includes the frame synthesis data, the zenith correction data, and the first rotation correction data output by the image synthesis unit 220, the zenith correction amount calculation unit 222, the first rotation amount calculation unit 224, and the second rotation amount calculation unit 226. And the second rotation correction data, respectively. The image rotation unit 230 performs rotation processing (zenith correction and rotation correction) on each frame of the acquired frame composite data based on the acquired zenith correction data, the first rotation correction data, and the second rotation correction data, and performs correction. Output the omnidirectional spherical frame data.

Here, the zenith correction and the rotation correction will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram illustrating zenith correction and rotation correction of the omnidirectional image performed in the present embodiment. FIG. 7 is a diagram illustrating a omnidirectional image obtained by the zenith correction and the rotation correction of the omnidirectional image performed in the present embodiment. FIG. 7A shows a moving image frame before the zenith correction, and FIG. 7B shows a moving image frame after the zenith correction.

As described above, the image data in the spherical image format has a pixel value in which the vertical angle φ with respect to the predetermined axis z0 and the horizontal angle θ corresponding to the rotation angle around the predetermined axis z0 are coordinates. Expressed as an array of. Here, the predetermined axis is an axis defined with the omnidirectional camera 110 as a reference when no correction is made. For example, with the imaging body 12 side of the omnidirectional camera 110 shown in FIG. 1 as the head and the opposite side as the bottom, the central axis passing through the center of the housing from the bottom to the head is represented by the horizontal angle θ and the vertical angle φ. A spherical image can be defined as a predetermined axis z0 to be defined. Further, for example, the horizontal angle θ of the spherical image can be defined so that the optical axis direction of one optical element of the two imaging optical systems 20A and 20B is centered at the horizontal angle θ.

The zenith correction is a celestial sphere image (FIG. 7(A)) captured in a state where the central axis z0 is actually inclined with respect to the gravity direction as shown in the left diagram of FIG. As shown in the figure, it refers to a process (correction in Roll and Pitch directions) for correcting a celestial sphere image (FIG. 7B) as if the image was taken with the central axis z0 coinciding with the gravity direction Z. The image rotation unit 230 performs zenith correction on each frame based on the tilt angle with respect to the gravity direction in each frame included in the zenith correction data so that a predetermined axis (center axis) substantially matches the gravity direction. It functions as a correction means.

On the other hand, the rotation correction is an omnidirectional image corrected by the zenith correction so that the central axis z0 coincides with the reference direction, and further an angular change around the gravity direction Z (horizontal angle θ direction in FIG. 7). Correction) to fix the display reference (for example, the center at the horizontal angle θ in the celestial sphere format, which is the reference position displayed by default when playing back video data) in a specific direction during shooting. (Correction in the Yaw direction). The reference position displayed by default is an area in the spherical format that is displayed when the user does not perform an operation to change the display direction for moving image data. The specific direction is, for example, the front direction of the spherical camera 110. The image rotation unit 230 serves as a rotation correction unit that performs a correction for canceling a rotation change around the reference axis (which is matched with the gravity direction by the zenith correction) based on the first rotation correction data and the second rotation correction data. Function.

Here, the specific direction during shooting in which the change in angle is canceled and is fixed to the display reference may be the shooting front direction in the horizontal plane of the omnidirectional camera 110 in the reference frame described above. Further, after the traveling direction of the omnidirectional camera 110 is obtained, the specific direction may be the traveling direction of the omnidirectional camera 110 in the horizontal plane.

More specifically, the image rotation unit 230 is based on the first rotation correction data and the second rotation correction data, and for each frame of the moving image, at least a plurality of frames for a certain period of time, the display reference indicates that the shooting is being performed. Correction is performed to cancel the rotation change around the reference axis so that it substantially matches the specific direction. Here, the amount of rotation change to be canceled is determined in a specific direction in a plane perpendicular to the reference axis (for example, the shooting front direction or the advancing direction at the start of shooting) and in the plane perpendicular to the reference axis in the correction target frame. It is based on the difference from the image capturing front direction. Further, the period fixed in the specific direction is from the timing when the specific direction is determined to the timing when the specific direction is determined next time.

In the embodiment to be described, at the beginning, the reference (center of the image) of the display is fixed by setting the shooting front direction of the omnidirectional camera 110 in the reference frame in the horizontal plane as the specific direction, and the omnidirectional camera 110 is started. After the traveling direction of is determined, the traveling direction in the horizontal plane is determined as the specific direction, and the display reference (the center of the image) is fixed to the traveling direction. In this case, regardless of the swinging motion H shown in FIG. 2, initially, the rotation correction is performed so that the shooting front direction at the start time comes to the center of the image, and when the traveling direction T is gradually determined, the traveling direction T is determined. The rotation will be corrected so that it will come to the center of the image.

However, the method of determining the specific direction is not particularly limited. In another embodiment, regardless of the traveling direction T, the reference position that is always displayed by default may be fixed in the shooting front direction of the omnidirectional camera 110 in the horizontal plane in the reference frame. Good. In such an embodiment, the rotation correction using the second rotation amount calculation unit 226 and the second rotation correction data can be omitted. Further, in this case, since the specific direction is not updated due to a change in the traveling direction during shooting, the period fixed in the specific direction is from the start time of the video or the time when the start of the fixation is instructed to the end time of the video or the fixed time. It is the entire period until the time when the end is instructed.

The storage unit 228 stores the above-described first frame data, second frame data, frame composite data, corrected frame composite data that has undergone zenith correction and rotation correction, acceleration data, angular velocity data, zenith correction data, first rotation correction data, and It is used to store the second rotation correction data. The storage unit 228 is provided as a part of a storage area such as the DRAM 132, the external storage 134, or another storage illustrated in FIG.

Note that the spherical moving image acquired by one shooting may be recorded in the storage unit 228 as one file, for example. In that case, various modes are conceivable as the recording method of the spherical video.

For example, the corrected frame composite data can be recorded as one spherical video data. In that case, the acceleration data, the angular velocity data, the zenith correction data, the first rotation correction data, and the second rotation correction data may be recorded as metadata together with the corrected frame combination data. In this embodiment, when the corrected data becomes necessary, the corrected frame composite data and its metadata should be read.

In another embodiment, the frame composite data before correction can be recorded as one celestial sphere moving image data. In that case, the acceleration data, the angular velocity data, the zenith correction data, the first rotation correction data, and the second rotation correction data are recorded in a predetermined combination as metadata together with the frame combination data before correction. In this other embodiment, when the corrected data is needed, the zenith correction data, the first rotation correction data, and the second rotation correction data are prepared from the metadata (lacking data is calculated. .), and the image rotation unit 230 applies image rotation to the frame composite data before correction. For example, if acceleration data and angular velocity data are available, any of the zenith correction data, the first rotation correction data, and the second rotation correction data that are insufficient can be calculated.

The frame-combined data subjected to the zenith correction and the rotation correction by the image rotation unit 230 is output to the output unit 240. The output unit 240 can transmit and output the corrected data as moving image data to the external information terminal 150 via the wireless NIC 140 and the USB connector 138. Alternatively, the output unit 240 may output the corrected data as a moving image data to a file in a predetermined storage. Here, the data output by the output unit 240 may include the moving image data after the correction, and may further include the above-described metadata.

Although referred to as “moving image”, it may be recorded in any form as long as the moving image can be reproduced. For example, H.264. H.264/MPEG-4 AVC (Advanced Video Coding), H.264. It may be recorded as moving image data obtained by compressing a plurality of frames with a predetermined codec such as H.265/HEVC (High Efficiency Video Coding). Further, Motion JPEG (Joint Photographic Experts Group) is a format for expressing moving images as continuous still images, but moving image data may be recorded as a continuous series of still images of a plurality of frames in this way. However, the moving image may be recorded as a set of files of still images of a plurality of frames. The output unit 240 includes an appropriate codec.

Hereinafter, the functional blocks on the information terminal 150 side will be continuously described. The information terminal 150 is a terminal device in which an application for communicating with the omnidirectional camera 110 and viewing and reproducing an omnidirectional image is installed. The information terminal 150 may be a smartphone, a tablet computer, a personal computer, a head mounted display (HMD), or the like. The information terminal 150 receives various instructions from the operator via the application and issues various requests to the omnidirectional camera 110. For example, the information terminal 150 responds to the reception of a specified spherical video reproduction instruction (for example, a video reproduction instruction in which rotation correction is applied) from the operator, and then the information terminal 150 instructs the spherical camera 110 to perform a predetermined operation. Issue a request for the corrected video data of the spherical video.

The receiving unit 252 of the information terminal 150 receives the moving image data output from the spherical camera 110. Here, the received moving image data has been subjected to the zenith correction and the rotation correction as described above, and is further accompanied by metadata indicating the predetermined direction at the time of image capturing in each frame. Here, the metadata includes the first rotation correction data and the second rotation correction data, and the predetermined direction at the time of image capturing in each frame is the image capturing front direction in each frame. However, the acceleration data and the angular velocity data may be received as metadata on the assumption that the information terminal 150 side calculates the first rotation correction data and the second rotation correction data. The reception unit 252 constitutes a data acquisition unit that acquires data indicating a predetermined direction at the time of image capturing in the present embodiment.

The display control unit 254 of the information terminal 150 performs control based on the received moving image data to output a display screen including at least a part of the image range of the corrected moving image data. Preferably, the display screen is displayed on the display device such as the display 162 included in the information terminal 150 based on the above-described display standard. The display control unit 254 constitutes the output control means in this embodiment. Further, the display control unit 254 changes the direction displayed in the omnidirectional format by the user's operation, and changes the area displayed on the display 162.

In the embodiment to be described, the display control unit 254 attaches the spherical image to the spherical object and displays a moving image as an image when the spherical object is observed from a predetermined position with a virtual camera at a predetermined viewing angle. At this time, the range observed from the virtual camera is the image range that can be included in the display screen. As a result, the viewer can watch the moving image in a format as if he or she were looking in a predetermined direction from the shooting point.

In the spherical format, a predetermined display standard is defined, and the spherical image is attached to the spherical object based on the display standard. Here, since the received moving image data has been subjected to the zenith correction and the rotation correction as described above, regardless of the shooting front direction at the time of shooting, a specific direction during shooting (shooting front direction at the start of shooting) And the direction of travel) will be fixed to the display standard. On this display screen, the specific direction (for example, the traveling direction) is not always fixed in the center, but unless the image range (the position and the direction of the virtual camera) is changed, a certain direction (for example, the left hand in the traveling direction). If the display range is set to, the left hand in the traveling direction) is always fixed to the center of the display screen.

In the described embodiment, the display control unit 254 further controls, in response to the satisfaction of the predetermined condition, to add an expression indicating the shooting front direction in the display screen based on the metadata described above. Is configured as follows. Here, the expression indicating the shooting front direction is an instruction icon that indicates a position in the shooting front direction on the display screen. The instruction icon is an image, a figure, a symbol, or a character that indicates a position in the image where the viewer pays attention to the viewer. As described above, on the display screen, regardless of the shooting front direction at the time of shooting, the specific direction during shooting (such as the traveling direction) is fixed as the display reference, and the instruction icon indicates This is to urge the viewer's attention in the shooting front direction at the time of shooting.

Further, in general, when a photographer often looks at an object of interest for a few seconds when he or she has an object of interest, the predetermined condition is that the shooting front direction remains for a predetermined number of frames or more. It can be a condition. In this case, the display control unit 254 can determine whether or not the predetermined condition is satisfied based on the metadata.

As shown in FIG. 4, the functional block of the information terminal 150 may further include a display range change instruction accepting unit 256. The display range change instruction receiving unit 256 is a change instruction receiving unit that receives an instruction to change the image range (position and direction of the virtual camera) included in the display screen.

In a specific embodiment, the display control unit 254 responds to the fact that the center of the image range on the display screen is located within a certain range centered on the coordinate indicating the shooting front direction on the display screen, The displayed instruction icon can be controlled to be hidden or semi-transparently displayed.

As shown in FIG. 4, the functional block of the information terminal 150 may further include a line-of-sight data receiving unit 258. The line-of-sight data receiving unit 258 detects the point of sight of the viewer on the display screen, which is detected by a line-of-sight detection unit such as an eye tracker when the information terminal 150 is an HMD or is connected to the HMD. It is a receiving means for receiving the line-of-sight data indicating. In a specific embodiment, the display control unit 254 displays the above in response to the gazing point within the display screen being located within a certain range centered on the coordinates indicating the shooting front direction within the display screen. The instruction icon can be controlled to be hidden or semi-transparent.

Further, in a preferred embodiment, the display control unit 254 may stop the reproduction of the moving image or perform the slow reproduction for a predetermined period from the time when the instruction icon is added to the time when the instruction icon is hidden or semi-transparently displayed. Can be controlled.

In the present embodiment, the zenith correction and the rotation correction are actually performed using the resources of the celestial sphere camera 110 side instead of the information terminal 150, and the correction result is output and displayed on the information terminal 150. Adopt a configuration. With this configuration, when sufficient resources are provided on the omnidirectional camera 110 side, regardless of the processing performance of the information terminal 150, it is possible to stably perform moving image reproduction while performing zenith correction and rotation correction. Become.

In addition, in the embodiment to be described, as the output mode, the image data of the spherical moving image is transmitted to the information terminal 150, but the present invention is not limited to this. When the omnidirectional camera 110 includes a display device, it is also possible to adopt a mode in which the display is performed on the display device.

In the embodiment shown in FIG. 4, the functional block of the omnidirectional camera 110 is configured to further include an instruction receiving unit 242 in addition to the image processing unit 210 and the output unit 240. The instruction receiving unit 242 gives an instruction to specify a specific direction in which the display reference is fixed by rotation correction during shooting of a moving image, or an instruction to indicate a direction in which a photographer wants to show an instruction icon during shooting of a moving image. Can be accepted. The instruction is given by operating a hardware key provided in the omnidirectional camera 110, a terminal to which the omnidirectional camera 110 is connected, in which a shooting application is installed, a dedicated controller, or the like.

In the above description, the specific direction has been the shooting front direction or the traveling direction at a predetermined time. However, in the specific embodiment, by providing the instruction receiving unit 242 to receive the instruction to specify the specific direction, the direction to be fixed to the image display reference at any timing during shooting (the direction in which the subject of interest exists) Etc.) can be specified.

Further, in the above description, the instruction icon is displayed on condition that the shooting front direction remains for a predetermined number of frames or more. However, in the specific embodiment, by providing the instruction receiving unit 242 to receive the instruction indicating the direction of interest, the photographer can explicitly determine the timing and the direction in which the instruction icon is displayed during viewing. .. The received information instructing the specific direction and the instruction information instructing the attention direction are attached to the moving image data by the output unit 240, output in association with a predetermined frame, and received by the receiving unit 252 of the information terminal 150. Will be. The instruction receiving unit 242 constitutes an attention instruction receiving unit in this embodiment.

FIG. 8 and FIG. 9 are sequence diagrams illustrating the processing from shooting to viewing of a moving image in the spherical spherical moving image system according to various specific embodiments.

FIG. 8A corresponds to the embodiment shown in FIG. In the embodiment shown in FIG. 8A, imaging and image processing (including image combination, zenith correction, and rotation correction) are performed on the omnidirectional camera 110 side, and moving image data after image processing is accompanied by metadata. It is transmitted to the information terminal 150. Then, in the information terminal 150, a display is performed based on the moving image data and the metadata after the image processing. More specifically, in the embodiment of FIG. 8(A), the omnidirectional camera 110 acquires moving image data by the image capturing process in S10, performs image processing on the moving image data in S11, and performs image processing in S12. The processed moving image data is transmitted to the information processing terminal. In S13, the information processing terminal displays the moving image data on the display.

In the embodiment shown in FIGS. 4 and 8A, the information terminal 150 uses the data acquisition unit (reception unit 252), the moving image acquisition unit (reception unit 252), the output control unit (display control unit 254), and the change instruction. It operates as an image processing apparatus provided with a reception unit (display range change instruction reception unit 256).

4 and 8A, an example in which image processing (including image synthesis, zenith correction, and rotation correction) is performed using the resources on the omnidirectional camera 110 side has been described, but the present invention is not limited to this. .. For example, a part or all of the first frame data, the second frame data, the frame composite data, the angular velocity data, the acceleration data, the zenith correction data, the first rotation correction data and the second rotation correction data are sent to the information terminal 150 side, Part or all of the image processing may be performed on the information terminal 150 side. In that case, the information terminal 150 side performs some functions on the omnidirectional camera 110 side shown in the functional block diagram of FIG. 4 by the CPU 152 on the information terminal 150 side executing various programs. As a result, for example, the CPU having a higher processing speed can perform a heavy load process such as a process of combining the first frame data (fisheye image A) and the second frame data (fisheye image B) by the image combining unit 220. It becomes possible to carry out the processing by using it, and it becomes possible to process even data having a large number of pixels in a short time or in real time.

For example, in the embodiment shown in FIG. 8B, only the omnidirectional camera 110 takes an image (S30), and the frame data before image processing and the rotation parameter are transmitted to the information terminal 150 (S31). Here, the rotation parameter is a general term for parameters related to correction of frame data such as the acceleration data, the angular velocity data, the zenith correction data, the first rotation correction data, and the second rotation correction data described above. Then, in the information terminal 150, image processing such as zenith correction and rotation correction is performed on the received frame data based on the received rotation parameter (S32), and the moving image data after the image processing and the metadata attached thereto are processed. Display is performed based on the display (S33).

In the embodiment shown in FIG. 8B, the information terminal 150 includes a data acquisition unit (reception unit 252), a moving image acquisition unit (a unit corresponding to the reception unit 252 and the image rotation unit 230), and an output control unit (display control). Unit 254) and a change instruction receiving unit (display range change instruction receiving unit 256).

In this case, the image combination may be performed by either the omnidirectional camera 110 or the information terminal 150. When images are combined on the omnidirectional camera 110 side, the frame data transmitted from the omnidirectional camera 110 to the information terminal 150 is frame combined data (omnidirectional format). When image synthesis is performed on the information terminal 150 side, the frame data transmitted from the omnidirectional camera 110 to the information terminal 150 is the first frame data (fisheye image A) and the second frame data (fisheye image B). Is. The first frame data and the second frame data may be transmitted as moving image data corresponding to each of the first frame data and the second frame data, or moving image data of one combined image obtained by combining two fisheye images (the fisheye image A and the fisheye image). The moving image data in the case where the images B are arranged side by side and joined to form one image) may be transmitted. In this case, the information terminal 150 is equipped with a unit corresponding to the image synthesizing unit 220 as a moving image acquisition unit.

Further, at which stage the rotation parameter data is transmitted to the information terminal 150 is arbitrary. For example, the acceleration data and the angular velocity data described above may be transmitted to the information terminal 150 as a rotation parameter. In this case, the zenith correction data, the first rotation correction data, and the second rotation correction data are calculated on the information terminal 150 side from the acceleration data and the angular velocity data. In this case, the information terminal 150 includes the first rotation amount calculation unit 224 and a unit corresponding to the first rotation amount calculation unit 224 as the data acquisition unit. Alternatively, the zenith correction data, the first rotation correction data, and the second rotation correction data may be transmitted to the information terminal 150 as rotation parameters. In this case, the information terminal 150 may include a receiving unit that receives the zenith correction data, the first rotation correction data, and the second rotation correction data as the data acquisition unit.

In the embodiment described with reference to FIGS. 4, 8A and 8B described above, the omnidirectional video system includes two devices, the omnidirectional camera 110 and the information terminal 150. However, the present invention is not limited to this.

For example, as shown in FIG. 9A, the second information terminal 170 connected to the first information terminal 150 may be included. Here, the second information terminal 170 is, for example, a head mounted display connected to a terminal such as a smartphone, a tablet, a personal computer. In the embodiment shown in FIG. 9A, only the omnidirectional camera 110 takes an image (S50), and the frame data and the rotation parameter before image processing are transmitted to the first information terminal 150 (S51). .. In the first information terminal 150, the received frame data is subjected to image processing such as zenith correction and rotation correction (S52), and the moving image data after the image processing is transmitted to the second information terminal 170 together with the metadata. (S53). On the second information terminal 170, display is performed based on the moving image data and the metadata after the image processing (S54).

In the embodiment shown in FIG. 9A, the second information terminal 170 uses the data acquisition unit (reception unit 252), the moving image acquisition unit (reception unit 252), the output control unit (display control unit 254), and the change instruction. It operates as an image processing apparatus provided with a reception unit (display range change instruction reception unit 256).

In addition, as shown in FIG. 9B, the spherical moving image system may include a server device 190. In the embodiment shown in FIG. 9B, only the imaging is performed on the omnidirectional camera 110 side (S70), and the frame data before the image processing and the rotation parameter are transmitted to the information terminal 150 (S71). The information terminal 150 transfers the received frame data and the rotation parameter to the server device 190 (S72), and causes the server device 190 to perform image processing such as zenith correction and rotation correction on the frame data (S73). The information terminal 150 receives the moving image data and the metadata after the image processing from the server device 190 or receives the streaming distribution (S74), and displays the display based on the moving image data and the metadata after the image processing (S75). Further, access to the server device 190 is not limited to the information terminal 150.

In the embodiment shown in FIG. 9B, the information terminal 150 uses the data acquisition unit (reception unit 252), the moving image acquisition unit (reception unit 252), the output control unit (display control unit 254), and the change instruction reception unit ( It operates as an image processing apparatus including a display range change instruction receiving unit 256).

In addition, when the omnidirectional camera 110 includes a display device, the omnidirectional camera 110 alone may configure the omnidirectional video system. In this case, the omnidirectional camera 110 may be configured to display on a display device included in itself as an image processing device.

Hereinafter, the zenith correction and the rotation correction in the present embodiment will be described in more detail with reference to FIGS. 10 to 13. FIG. 10 is a rotation process for frame-combined data executed by the omnidirectional camera 110 (which may be the information terminal 150 or the server device 190 in a specific embodiment) included in the omnidirectional video system according to the present embodiment. It is a flowchart showing.

The process shown in FIG. 10 is started from step S100 in response to the omnidirectional camera 110 receiving a request for the corrected moving image data.

In step S101, the celestial sphere camera 110 acquires frame synthesis data and rotation parameters. Here, for convenience of description, the recording of the first frame data and the second frame data, the generation of the frame composite data by image composition, and the recording of the angular velocity data and the acceleration data are completed at the start of the rotation process. And However, in another embodiment, when performing real-time distribution or the like, the first frame data and the second frame data are recorded, the angular velocity data and the acceleration data are recorded, and the frame combined data is generated by the image combination, The rotation process shown in 10 can also be performed. In addition, here, it is assumed that the zenith correction data, the first rotation correction data, and the second rotation correction data have not been calculated.

In step S102, the omnidirectional camera 110 stores an initial value in the shooting front direction of the omnidirectional camera 110. The shooting front direction of the omnidirectional camera 110 at the start time is used when calculating the reference of the first rotation correction data and the second rotation correction data. Further, in step S102, the first frame is targeted for processing, and the process proceeds to step S103.

In step S103, the omnidirectional camera 110 calculates the tilt angle with respect to the gravity direction in the frame to be processed by the zenith correction amount calculation unit 222 based on at least the acceleration data, and writes it in the zenith correction data.

In step S104, the omnidirectional camera 110 detects the change amount of the rotation angle in the horizontal plane compared with the previous frame by the first rotation amount calculation unit 224. In step S105, the omnidirectional camera 110 uses the first rotation amount calculation unit 224 to integrate the angle change amount between the frames calculated in step S104, and calculate the angle change amount in the shooting front direction from the initial value. , Write the first rotation correction data. As described above, the first rotation correction data is time-series data of the angle change amount in the shooting front direction in the target frame from the initial value in the shooting front direction of the omnidirectional camera 110 in the horizontal coordinate system.

The processing of steps S103 to 105 can be executed for each frame. On the other hand, the processing of steps S106 to S108 can be executed for each specified number of frames. Here, the specified number of frames can be set to any value. Although it is possible to increase the followability by reducing the specified number of frames, if the specified number of frames is too small, the blur component at the time of progress and the blur component at the time of stillness are likely to be affected. Therefore, if the moving image is 60 frames per second (60 fps), the specified number of frames may be set to be, for example, about 10 to 30 frames.

In step S106, the celestial sphere camera 110 uses the second rotation amount calculation unit 226 to calculate the moving distance and the traveling direction vector of the celestial sphere camera 110 within the prescribed number of prescribed frames in the coordinate system of the horizontal plane. It should be noted that the specified frame to be calculated here may be a predetermined number of frames before and after the central frame in the case of reproducing the stored spherical video data, or in the case of real-time distribution from the first frame. The frame may be up to a predetermined number of minutes.

In step S107, the omnidirectional camera 110 causes the second rotation amount calculation unit 226 to determine whether or not the movement distance calculated in step S106 is equal to or greater than a predetermined threshold. If it is determined in step S107 that the moving distance is equal to or greater than the predetermined threshold value (YES), the process proceeds to step S108. Here, the moving distance means the difference between the position of the first frame and the position of the last frame in the coordinate system of the horizontal plane in the specified number of frames.

In step S108, the omnidirectional camera 110 calculates the angle change amount of the traveling direction vector calculated in S105 from the initial value by the second rotation amount calculation unit 226, and writes the second rotation correction data. Here, the second rotation correction data is time-series data of the difference between the initial value of the traveling direction vector (equal to the initial value of the camera direction) and the traveling direction vector as an average for the specified number of frames as described above. is there.

On the other hand, if it is determined in step S107 that the moving distance is less than or equal to the threshold value (NO), the process directly proceeds to step S109.

In step S109, the omnidirectional camera 110 performs the zenith correction on the target frame of the frame composite data based on the zenith correction data (the tilt angle of the target frame therein) by the image rotation unit 230.

In step S110, the omnidirectional camera 110 uses the image rotation unit 230 to determine the rotation change amount (rotation correction amount) in the horizontal plane to be canceled in the target frame based on the first rotation correction data and the second rotation correction data. calculate. As described above, the first rotation correction data is a difference between the initial value in the shooting front direction of the reference frame and the shooting front direction at the time of the target frame in the horizontal plane, and the second rotation correction data is in the horizontal plane. In the moving direction of the reference frame (equal to the shooting front direction) and the moving direction at the time of the target frame. Therefore, by adding the first rotation correction data and the second rotation correction data, it is possible to finally calculate the rotation change amount to be canceled. In addition, when the specific direction is the initial value of the shooting front direction, the second rotation correction data is the default value (zero) until the moving distance becomes equal to or more than the threshold, and therefore the rotation change amount is the first rotation correction data. As defined by, the difference is based on the difference between the initial value in the shooting front direction and the shooting front direction in the horizontal plane of the correction target frame. On the other hand, when the specific direction is determined to be the traveling direction, the rotation change amount is based on the difference between the traveling direction and the shooting front direction in the correction target frame.

In step S111, the omnidirectional camera 110 performs rotation correction on the target frame based on the calculated rotation correction amount.

In step S112, the omnidirectional camera 110 determines whether or not there is a next frame. If it is determined in step S112 that there is a next frame (YES), the process loops to step S103. On the other hand, if it is determined in step S112 that there is no next frame (NO), the main rotation process ends in step S113.

By the above-described processing, the moving image is corrected so as to cancel the rotational change around the reference axis so that the display reference substantially coincides with the specific direction during shooting over at least a plurality of frames for a certain period. Become.

In the above-described embodiment, the rotation amount to be canceled is calculated based on the first rotation correction data and the second rotation correction data in step S110, and the rotation correction is collectively performed in step S111. However, the present invention is not limited to this, and in other embodiments, the rotation correction based on the first rotation correction data and the rotation correction based on the second rotation correction data may be sequentially applied. Further, in another embodiment, the zenith correction and the rotation correction may be collectively performed based on the zenith correction data, the first rotation correction data, and the second rotation correction data.

By the above rotation correction, it becomes possible to fix the reference for displaying an image in a specific direction such as the front direction of shooting of the camera or the advancing direction regardless of the low frequency rotation change due to swinging or the like.

Hereinafter, the relationship between the traveling direction and the shooting front direction and the rotation processing by the image rotation unit at that time will be described with reference to FIGS. 11 to 13. FIG. 11 is a diagram for explaining an example of the relationship between the traveling direction T and the shooting front direction H when the photographer wears the omnidirectional camera 110 on the head 50 and moves. FIG. 12 and FIG. 13 are diagrams illustrating a captured spherical image before rotation correction (left side) and a rotationally corrected spherical image (right side) when the photographer performs the operation illustrated in FIG. 11. is there.

FIG. 11 is a diagram showing the numbers 0, 1, 1 ′, and 1 to 10 in a time-series manner. In FIG. 11, all of the numbers fixed to the head 50 of the photographer at each time point indicated by the numbers. The state of the celestial sphere camera 110 is schematically shown together with the traveling direction H of the photographer (the celestial sphere camera 110) and the photographing front direction H of the celestial sphere camera 110 according to the swing motion.

In FIG. 11, focusing on the traveling direction T, in this example, it is shown that the photographer is traveling toward “North” in the period of numbers 0 to 5. Further, focusing on the shooting front direction H in FIG. 11, the photographer faces “North”, which is the same as the traveling direction at the time of numbers 0 to 1, and looks around by the swinging motion at the time of numbers 2 to 4. It is shown that the action is performed and at the time of number 5, the player is looking at “West” on the left side in the traveling direction. It should be noted that the numbers 1 and 1'represent the operation of returning to the original path after making a parallel translation to the right once, for example, in the case of yielding a passage when passing another person. Then, the photographer changes the traveling direction from "North" to "West" between the numbers 6 to 7 while looking at the "West", and at the time of the numbers 7 to 8, the same "West" as the traveling direction is set. It shows that you are progressing while watching. Also, at the time of number 9, while looking at the "west", you are looking at the "south" on the left side of the direction of travel, and at the time of number 10, you can see that you have changed the direction of travel to "south". There is. Note that the north, south, east, and west are drawn for convenience, but the north, south, east, and west are not necessarily identified.

12 and 13, the spherical image before rotation correction (left side) 300 and the spherical image after rotation correction (right side) corresponding to the numbers 0, 1, 1', 1 to 10 shown in FIG. ) 310 is shown schematically. Further, in the omnidirectional image (right side) 310 after the rotation correction, it is schematically shown that the image range 316 set at the center of the omnidirectional image is displayed on the display screen of the information terminal 150. It represents.

In the periods of numbers 0, 1, 1', 1 to 5, the photographer is moving to the "north", but as shown on the left side of FIGS. 12 and 13, in the uncorrected spherical image 300, It can be seen that the image is rotated according to the person's swinging motion. More specifically, in the periods of numbers 0, 1, 1', and 1 to 5, the shooting front direction 302 is "North" → "Northwest (60 degrees from north to left)" → "Northeast (from north to right). "60 degrees)" → "northwest (45 degrees left from north)" → "west (90 degrees left from north)". On the other hand, as shown on the right side of FIGS. 12 and 13, in the corrected celestial sphere image 310, at the time points of numbers 0, 1, 1 ′, 1 to 5, in the shooting front direction and the traveling direction at the start time point. The display standard 314 is always fixed to a certain "north". Further, in the parallel movement at the time points 1 and 1′, the moving direction of the omnidirectional camera 110 changes momentarily, but since it is a short time change, in step S107 shown in FIG. It is determined that the moving distance in the frame is less than the threshold value, the traveling direction is not redetermined, and “North” remains.

To explain the rotation correction by extracting representative points, at the time of number 2 in FIG. 11, the celestial sphere camera 110 rotates 60 degrees to the left while the photographer is heading north. It is facing northwest. At this time, the photographer is different from the shooting front direction H of the omnidirectional camera 110 and the traveling direction T. Here, the rotation process of the image rotation unit 230 when the photographer changes to the state of the number 2 in FIG. 11 will be described. When the photographer changes the direction of the neck from the time point of number 1 to the time point of number 2, that is, when the direction of the omnidirectional camera 110 is rotated leftward by 60 degrees, the image rotation unit 230 moves the A rotation process of rotating 60 degrees to the right is executed so as to cancel the rotation of the omnidirectional image based on the one rotation correction data. This makes it possible to fix the default direction shown on the display to the traveling direction. Actually, the rotation process based on the first rotation correction data is sequentially executed for each frame that changes from the time point 1 to the time point 2. At this time, since there is no change in the traveling direction, the rotation process based on the second rotation correction data is not executed. The same applies to changes from time point 2 to time point 3, changes from time point 3 to time point 4, and changes from time point 4 to time point 5.

In Nos. 5 to 7, the photographer gradually changes the traveling direction to “West” which is the viewing direction, but as shown on the left side of FIG. The direction of the person is always looking at "west" and there is no change in the image. On the other hand, as shown on the right side of FIG. 13, in the corrected celestial sphere image 310, the reference 314 displayed in the updated traveling directions “northwest (number 6)” and “west (number 7)” respectively. It is fixed. Here, it is shown that the traveling direction at time point 6 is constant for a sufficiently long period of time, and the direction to be fixed once at the re-determined “northwest” is changing.

Here, the rotation process of the image rotation unit when the photographer changes to the state of the numbers 5, 6, and 7 in FIG. 11 will be described. When the photographer changes the traveling direction T without changing the photographing front direction H from the time point 5 to the time point 6, based on the first rotation correction data, the rotation of the spherical image with respect to the photographing front direction is the initial value. A rotation process of rotating 90 degrees to the right so as to be canceled by is executed. On the other hand, based on the second rotation correction data, the rotation processing of rotating the spherical image to the left by 45 degrees is executed based on the difference in the traveling direction from the initial traveling direction (the initial shooting front direction) of the spherical image. Become. Then, when the first rotation correction data and the second rotation correction data are summed up, a rotation process of rotating 45 degrees to the right is executed.

Further, in the state at the time of number 7, based on the first rotation correction data, a rotation process of rotating 90 degrees to the right is executed so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. Become. On the other hand, based on the second rotation correction data, a rotation process of rotating the celestial sphere image 90 degrees to the left is executed based on the difference in the direction of travel from the initial direction of travel of the celestial sphere image (initial shooting front direction). Then, when the first rotation correction data and the second rotation correction data are added together, there is no rotation correction.

The change in the traveling direction may be interpolated so that a smooth transition is made from the previous traveling direction to the next traveling direction. That is, it is possible to apply a value obtained by interpolating the angle change of 45 degrees in the periods of time points 5 and 6 and the periods of time points 6 and 7 in each frame therebetween. Alternatively, the change in the traveling direction may be an instantaneous change. For example, the screen display may be once blacked out, and the state in which the display reference is fixed in the previous traveling direction may be switched to the state in which the display reference is fixed in the new traveling direction.

On the other hand, in the change from the number 8 to the number 9, the photographer changes the viewing direction from “west” to “south” without changing the traveling direction. In this case, as shown on the left side of FIG. 13, at the time points 8 and 9, in the uncorrected spherical image 300, the orientation of the photographer changes from “west” to “south”, but Is fixed to "west", the display reference 314 is fixed to "west" as shown on the right side of FIG. On the other hand, with the numbers 9 to 10, the photographer changes the traveling direction from “west” to “south” without changing the viewing direction. In this case, as shown on the left side of FIG. 13, at the time points 9 and 10, in the uncorrected spherical image 300, the orientation of the photographer is always looking at “south”, but shown on the right side of FIG. As described above, the traveling direction is changed from “west” to “south” by the direction in which the reference 314 displayed is fixed.

Here, the rotation process of the image rotation unit when the photographer changes to the state at the numbers 8, 9, and 10 in FIG. 11 will be described. When the photographer changes the shooting front direction H without changing the traveling direction T from the time point of number 8 to the time point of number 9, based on the first rotation correction data, the rotation of the spherical image with respect to the shooting front direction is changed. A rotation process of rotating 180 degrees to the right so as to be canceled from the initial value is executed. On the other hand, based on the second rotation correction data, the rotation processing of rotating 90 degrees to the left based on the difference in the traveling direction from the initial traveling direction of the celestial sphere image (the initial shooting front direction) is performed. Become. Then, when the first rotation correction data and the second rotation correction data are added together, a rotation process of rotating 90 degrees to the right is executed. Further, in the state at the time of the number 10, based on the first rotation correction data, the rotation process of rotating 180 degrees to the right is executed so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. Become. On the other hand, based on the second rotation correction data, a rotation process of rotating 180 degrees to the left based on the difference in the traveling direction from the initial traveling direction of the celestial sphere image (the initial shooting front direction) is executed. Become. Then, when the first rotation correction data and the second rotation correction data are added together, there is no rotation correction.

In this way, the default direction displayed on the display can be fixed to either the shooting front direction at the start of shooting or the traveling direction.

Hereinafter, the zenith correction and the rotation correction in another embodiment will be described in more detail with reference to FIG. FIG. 14 is a rotation with respect to the frame composite data executed by the omnidirectional camera 110 (which may be the information terminal 150 or the server device 190 in a specific embodiment) which constitutes the omnidirectional video system according to another embodiment. It is a flowchart which shows a process. Note that FIG. 14 shows an embodiment in which the advancing direction is not determined, and the display reference is always fixed in the shooting front direction at the start of shooting.

The process shown in FIG. 14 is started from step S200 in response to the omnidirectional camera 110 accepting the request for the corrected moving image data.

In step S201, the omnidirectional camera 110 acquires the frame composite data and the rotation parameter. Similar to the embodiment shown in FIG. 10, for convenience of description, at the start of the rotation process, recording of the first frame data and the second frame data, generation of frame composite data by image composition, recording of angular velocity data and acceleration data. Have been completed.

In step S202, the omnidirectional camera 110 stores the initial value in the shooting front direction of the omnidirectional camera 110. In step S203, in the omnidirectional camera 110, the zenith correction amount calculation unit 222 calculates the tilt angle with respect to the gravity direction in the frame to be processed based on at least the acceleration data, and writes the tilt angle in the zenith correction data.

In step S204, the omnidirectional camera 110 causes the first rotation amount calculation unit 224 to integrate the angle change amount of the coordinate system of the horizontal plane for each clock and calculate the change amount of the rotation angle in the horizontal plane compared to the previous frame. To detect. In step S205, the omnidirectional camera 110 uses the first rotation amount calculation unit 224 to integrate the angle change amounts between the frames calculated in step S204, and calculate the angle change amount in the shooting front direction from the initial value. , Write the first rotation correction data.

In step S206, the omnidirectional camera 110 performs the zenith correction on the target frame of the frame composite data based on the zenith correction data (the tilt angle of the target frame therein) by the image rotation unit 230.

In step S207, the omnidirectional camera 110 causes the image rotation unit 230 to perform rotation correction on the target frame based on the first rotation correction data. As described above, the first rotation correction data is the difference between the initial value in the shooting front direction of the reference frame and the shooting front direction at the time of the target frame in the horizontal plane, and the first rotation correction data in the horizontal plane to be canceled in the target frame. The amount of rotation change is expressed as it is.

In step S208, the omnidirectional camera 110 determines whether there is a next frame. If it is determined in step S208 that there is a next frame (YES), the process loops to step S203. On the other hand, if it is determined in step S208 that there is no next frame (NO), the main rotation processing ends in step S209.

By the above process, the rotation change around the reference axis is canceled so that the display reference substantially coincides with the shooting front direction at the start of shooting over a plurality of frames from the start to the end of shooting for the moving image. Such correction will be applied.

Hereinafter, the display processing on the information terminal 150 side for reproducing and displaying the zenith-corrected and rotation-corrected spherical video in the present embodiment will be described in more detail with reference to FIGS. 15 to 17. FIG. 15 is a flowchart showing the spherical video reproducing process executed by the information terminal 150 which constitutes the spherical video system according to the present embodiment.

The process illustrated in FIG. 15 is started from step S300 in response to the information terminal 150 receiving a request for reproduction of a corrected moving image from an operator who is a viewer.

In step S301, the information terminal 150 causes the receiving unit 252 to receive the corrected moving image data and metadata from the omnidirectional camera 110, and causes the display control unit 254 to start reproducing the omnidirectional moving image.

Here, for convenience of explanation, at the time of starting the reproduction of the moving image, recording of the first frame data and the second frame data, generation of frame combined data by image combination, recording of angular velocity data and acceleration data, zenith correction data, It is assumed that the calculation of the first rotation correction data and the second rotation correction data and the zenith correction and the rotation correction for the frame composite data have been completed. However, in another embodiment, when performing real-time distribution, etc., recording of the first frame data and the second frame data, recording of angular velocity data and acceleration data, generation of frame composite data by image composition, zenith correction data, FIG. 15 is shown while sequentially performing the calculation of the first rotation correction data and the second rotation correction data, the zenith correction and the rotation correction for the frame composite data by the omnidirectional camera 110 (including the case where a part is performed by the information terminal 150). It is also possible to execute the spherical video reproduction processing shown. In addition, here, it is assumed that the zenith correction data, the first rotation correction data, and the second rotation correction data are attached to the moving image data as metadata.

In step S302, the information terminal 150 acquires or calculates, from the metadata, data indicating the shooting front direction at the time of shooting the frame currently being processed. As described above, in the moving image data, the rotation correction has already been performed based on the first rotation correction data and the second rotation correction data, and the display reference is set in a specific direction such as the shooting front direction or the traveling direction at the start of shooting. It is fixed. The first rotation correction data and the second rotation correction data give a rotation change amount canceled by adding them up, and give a difference between the specific direction and the shooting front direction in the correction target frame. In the embodiment, the data is data indicating the shooting front direction in each frame. If there is no second rotation correction data, the rotation change amount canceled by the first rotation correction data is directly given. Furthermore, the zenith correction data may be used in order to obtain the front direction of photographing not only in the direction within the horizontal plane perpendicular to the gravity direction but also in the inclination with respect to the gravity direction.

In step S303, the display terminal 254 of the information terminal 150 determines whether or not the position in the shooting front direction is within the predetermined range. If it is determined in step S303 that the position in the shooting front direction is not within the predetermined range (NO), the elapsed time is reset in step S305, and then the process loops to step S302. In the case of the first time, it is determined in step S303 that it is not within the predetermined range. When it is determined in step S303 that the position in the shooting front direction remains within the predetermined range (YES), the process proceeds to step S304.

In step S304, the information terminal 150 further determines, by the display control unit 254, whether or not a predetermined time has elapsed with the position in the shooting front direction remaining within the predetermined range. If it is determined in step S304 that the predetermined time has not elapsed (NO), the process loops to step S302. On the other hand, if it is determined in step S304 that the predetermined time has elapsed (YES), the process proceeds to step S306. In this case, it means that the predetermined condition is satisfied.

In step S306, the information terminal 150 causes the display control unit 254 to display an instruction icon indicating the shooting front direction on the display screen based on the data indicating the shooting front direction. In a specific embodiment, the instruction icon may be displayed in step S306, and further, the moving image may be stopped or slow-played. By this processing, it becomes possible to share the place where the photographer is paying attention in real time.

In step S307, the information terminal 150 uses the display control unit 254 to acquire the center position of the image range displayed on the display screen. In step S308, the display terminal 254 of the information terminal 150 determines whether or not the center position has moved within the predetermined range indicated by the instruction icon. If it is determined in step S308 that the center position of the display screen has not moved within the predetermined range (NO), the process branches to step S309. In step S309, the display terminal 254 of the information terminal 150 further determines whether or not a time-out has occurred after displaying the instruction icon. If it is determined in step S309 that the timeout has not occurred (NO), the process loops to step S307.

During the loop of steps S307 to S309, the display range change instruction receiving unit 256 receives an instruction to change the image range (position and direction of the virtual camera) included in the display screen. When the information terminal 150 is a personal computer, for example, the user gives an instruction to change the image range to be displayed by rotating the viewing area by an operation such as dragging with a mouse. When the information terminal 150 is a smartphone or a tablet terminal, the viewing area is rotated by an operation such as a flick on the touch panel. When the information terminal 150 is an HMD, it is performed by rotating the viewing area by the operation of rotating the HMD. As a result of the display range change instruction receiving unit 256 receiving the instruction, if it is determined in step S308 that the center position of the display image has already moved within the predetermined range (YES), the process branches to step S310.

In step S310, the information terminal 150 causes the display control unit 254 to delete the displayed instruction icon, and loops the processing to step S302. On the other hand, if it is determined in step S309 that the time-out has occurred (YES), the process proceeds to step S310, the displayed instruction icon is deleted, and the process loops to step S302. At this time, when the instruction icon is displayed and the reproduction of the moving image is stopped or slow-reproduced, the reproduction of the moving image may be started or changed to the normal reproduction. In the embodiment to be described, only one instruction icon is displayed until a predetermined display condition (for example, staying for a certain period of time) is satisfied and then a deletion condition (for example, the image range to be displayed is moved in the front direction of shooting) is satisfied. As described above, between the satisfaction of the predetermined display condition and the deletion condition, a plurality of instruction icons are displayed as a result in response to the satisfaction of the display condition at another place. It does not prevent things.

Hereinafter, a method of displaying icons will be described in more detail with reference to FIGS. 16 and 17. FIG. 16 is an image of the corrected spherical moving image and the image range set in the corrected moving spherical direction T and the moving direction T when the photographer wears the spherical camera 110 on the head 50 and moves in this embodiment. It is a figure shown in time series linked|related with the state with frontal direction H, and the spherical image before correction image|photographed at each time. In FIG. 16, the left side shows the spherical image 300 before rotation correction, and the right side shows the spherical image 310 after rotation correction. In the spherical image 310 after the rotation correction, the image range 316 set therein and various icons 318 and 320 added thereto are also shown.

In FIG. 16, the number 0 indicates that the specific direction (the shooting front direction or the traveling direction at the start in the horizontal plane coordinate system perpendicular to the gravity direction) and the shooting front direction in the frame (also in the horizontal plane coordinate system) match. It shows the state of being. The shooting front direction is displayed as a white circle, and this white circle is referred to as a center icon. In this case, the white circle representing the shooting front direction is located at the center of the spherical image format, and when the image range 316 is set at the center of the spherical image, a center icon is displayed at the center of the image range 316. It will be. The center icon may be an arbitrary image, figure, symbol or character other than a white circle.

If the direction of travel does not change, even if the photographer changes his/her viewpoint (when he/she moves his/her neck to look around), the specific direction does not change from the viewer's viewpoint, but the shooting front direction changes. Become. In FIG. 16, numbers 1 to 4 represent states in which the shooting front direction in each frame is changed by the photographer's swinging motion without changing the specific direction. In the corrected omnidirectional image 310, since the specific direction is fixed to "North", no large change is seen in the image, but as is apparent from the uncorrected omnidirectional image, the omnidirectional image is corrected. In reality, the image changes greatly due to the change in the shooting front direction when the ball camera 110 takes an image. In FIG. 16, numbers 1 to 4 show that the center icon 318 indicated by a white circle moves to the left and right in response to the change in the front direction of photographing, compared to the case of number 0.

Generally, when a photographer has an object of interest, the photographer often looks at the place for several seconds. The number 5 represents a state in which the imaging front direction has not changed from the position indicated by the number 4 for a predetermined period. In number 5, in the corrected celestial sphere image 310, an instruction icon 320 that is an arrow pointing in the shooting front direction is shown. The display control unit 254 allows the photographer to determine whether a pixel indicated by the shooting front direction exists in a predetermined area centered on a pixel indicated by the shooting front direction of the camera in a certain frame from a certain frame to a predetermined frame. It is determined that the user is paying attention to the place, and the instruction icon 320 for notifying the viewer is displayed. By displaying the instruction icon 320 on the display, it is possible to notify the viewer of the direction in which the photographer is paying attention.

Further, the instruction icon 320 is controlled so that the display is erased when the center of the image range 316 displayed on the display is moved to near the center icon 318 by the user. The number 5'represents a state in which the center of the image range 316 has been moved to the vicinity of the center icon 318 by the user by, for example, dragging, tapping, and rotating the HMD.

In addition, in the example shown in FIG. 16, the center icon 318 is always displayed in addition to the instruction icon 320 when the predetermined condition is satisfied, but the center icon 318 is not essential. In another embodiment, it is possible to display only the instruction icon 320 without displaying the center icon 318. Further, in another embodiment, the center icon 318 is not always displayed, the front direction of shooting is displayed on the display screen, and if the center icon 318 does not change for a predetermined period, the center icon 318 is temporarily used as the instruction icon. It may be displayed as a message.

FIG. 17 is a diagram illustrating a method of displaying the instruction icon 320 by the display control unit 254. In FIGS. 17A and 17B, the grid represents coordinates. In FIG. 17A and FIG. 17B, the blackened coordinates indicate the position 350 in the shooting front direction. FIG. 17A shows a state in which the instruction icon 320 is displayed without changing the position 350 in the shooting front direction for a predetermined period, and the coordinates indicated by a star are the coordinates 354 of the center point of the screen display. It represents. The instruction icon 320 is displayed in the form of an arrow indicating the direction of the coordinate 350 at the position indicating the shooting front direction on the line connecting the coordinates 354 of the center point of the screen display indicated by the star and the coordinates 350 indicating the shooting front direction.

FIG. 17B shows how the instruction icon 320' is displayed when the center 354' of the display screen moves from the coordinates shown by the dotted star to the coordinates shown by the solid star. As the center 354 of the screen display moves, the display position of the instruction icon 320' is updated on the line connecting the center 354' of the screen display after the movement and the position 350 indicating the front direction of shooting. Note that, in FIG. 17, a gray area 352 around the blacked coordinates 350 indicating the front direction of photographing is an area indicating a predetermined range for deleting the above-described instruction icon. In response to the center 354 of the display screen moving into the gray area 352, the indicator icon 320' is erased. The size and shape of the predetermined area 352 shown in gray can be arbitrarily set. In addition, as described above, when eye tracking is available, the viewer's line of sight is read from the line-of-sight data, and in response to the detected coordinates of the gazing point moving within the gray area 352, Alternatively, the display of the instruction icon 320 may be erased.

As described above, according to the embodiments described above, an image processing apparatus capable of sharing a predetermined direction during image capturing with a viewer while suppressing motion sickness of the viewer caused by rotation around a reference axis in a moving image. It is possible to provide an imaging device, a moving image reproduction system, a method and a program.

In a specific embodiment, the rotation correction fixes the display standard during viewing in the traveling direction of the photographer, and thus it is possible to suppress the motion sickness of the viewer. Also, while the display reference is fixed in the reference direction (starting direction or traveling direction), the operation of rotating the direction of the moving image for the user can be performed by displaying the instruction icon under specific conditions. It is configured to prompt and move the image range displayed by the moving image according to the user's own operation or action. Therefore, it is possible for the viewer to share the direction in which the photographer was paying attention while reducing motion sickness at the time of viewing.

It should be noted that in the above-described embodiment, as the icon, the instruction icon indicating the shooting front direction and the center icon are illustrated. In still another embodiment, a traveling direction icon indicating the traveling direction may be displayed instead of the instruction icon indicating the shooting front direction. This is effective in the case of the embodiment in which the display reference is fixed only in the front direction of shooting at the start of shooting as the specific direction and not fixed in the traveling direction.

In addition, in the above-described embodiment, the spherical camera 110 is fixed to the head 50 of the photographer. The rotation correction function according to the present embodiment can be suitably applied in such a case, but the application scene is not particularly limited, and the user holds the image by hand and directs the spherical camera to the attention area. It goes without saying that it can be applied to any case.

Moreover, in the above-described embodiment, the first rotation correction data and the second rotation correction data are calculated using the angular velocity sensor 136A and the acceleration sensor 136B. However, in other embodiments, the first rotation correction data and the second rotation correction data may be calculated using the geomagnetic sensor when calculating the angle change amount. In that case, the initial value of the geomagnetic sensor is stored, the rotation angle of the coordinate system of the horizontal plane is calculated based on the difference between the initial value of the geomagnetic sensor and the direction indicated by the geomagnetic sensor in the target frame, and the first rotation correction data and The second rotation correction data can be calculated. Further, in the above description, the frame synthesis data, the acceleration data, and the angular velocity data are described as being stored in advance. It can also be applied to a stream of data.

Furthermore, in the above-described embodiment, the zenith correction is performed on the moving image of the spherical coordinate system whose coordinate system includes two angular coordinates. However, depending on the embodiment, there is a case in which the inclination in the direction of gravity that does not require the zenith correction does not occur, and in such a case, the zenith correction can be omitted. In this case, the reference axis may be an axis of the angular coordinate if the coordinate system is a cylindrical coordinate system including one angular coordinate, and two angular coordinates if the coordinate system is a spherical coordinate system including two angular coordinates. May be used as the axis for giving.

The functional unit can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark), or an object-oriented programming language, and can be ROM, EEPROM, EPROM. , Flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, or other device-readable recording medium, or through an electric communication line. It can be distributed. Further, a part or all of the above-mentioned functional units can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA), or as an ASIC (application specific integration). Circuit configuration data (bit stream data) to be downloaded to the PD in order to realize the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, and VHDL (Very High Speed Integrated Circuits). Hardware Description Language), Verilog-HDL, and the like can be distributed by a recording medium as data described.

Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art can conceive of other embodiments, additions, changes, deletions, and the like. The present invention can be modified within the scope of the present invention, and is included in the scope of the present invention as long as the effects and advantages of the present invention are exhibited in any of the aspects.

12... Imaging body, 14... Housing, 18... Shutter button, 20... Imaging optical system, 22, 130... Imaging element, 110... Spherical camera, 112, 152... CPU, 114... ROM, 116... Image Processing block, 118... Video compression block, 120, 126... Interface, 122... External storage interface, 124... External sensor interface, 126... USB interface, 128... Serial block, 132... DRAM, 134, 160... External storage, 136A... Angular velocity sensor 136B... Acceleration sensor, 138, 166... USB connector, 150... Information terminal, 154... RAM, 156... Internal storage, 158... Input device, 162... Display, 164... Wireless NIC, 200... Functional block, 212... First imaging unit, 214... Second imaging unit, 216... Angular velocity detection unit, 218... Acceleration detection unit, 220... Image combining unit, 222... Zenith correction amount calculation unit, 224... First rotation amount calculation unit, 226... 2 rotation amount calculation unit, 228... storage unit, 230... image rotation unit, 240... output unit, 242... instruction receiving unit, 252... receiving unit, 254... display control unit, 256... display range change instruction receiving unit, 258... Line-of-sight data receiver

JP, 2016-149752, A

Claims (16)

An image processing apparatus for processing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis, wherein the moving image is an image whose rotation is corrected. Is
Data acquisition means for acquiring data indicating a predetermined direction at the time of image capturing in each frame of the moving image,
Output control means for controlling to output a screen including at least a part of the image range of the moving image, wherein the output control means is responsive to a predetermined condition being satisfied, based on the data, An image processing apparatus, characterized in that control is performed so as to add an expression indicating a predetermined direction at the time of image pickup within a screen. The data includes sensor data corresponding to the moving image, and the output control unit determines, based on the sensor data, that the predetermined direction at the time of imaging remains for a predetermined number of frames or more. The image processing apparatus according to claim 1, wherein it is determined that the condition is satisfied. The data includes instruction information for instructing a direction of interest made during the shooting of the moving image, and the output control unit satisfies the predetermined condition on condition that the instruction information is associated with a frame. The image processing apparatus according to claim 1, wherein the image processing apparatus determines. The output control means further includes: a change instruction receiving means for receiving an instruction to change the image range included in the screen or a receiving means for receiving line-of-sight data indicating a gazing point of the viewer on the screen. In response to the center of the image range or the gazing point on the screen being located within a certain range centered on the coordinate indicating the predetermined direction at the time of image capturing on the screen, the predetermined direction at the time of image capturing The image processing apparatus according to claim 1, wherein the image processing apparatus is controlled so as to be hidden or semi-transparently displayed. The output control means controls to stop the reproduction of the moving image or to perform the slow reproduction for a predetermined period after the expression indicating the predetermined direction at the time of the image capturing is added. The image processing apparatus according to any one of items. The expression indicating the predetermined direction at the time of imaging includes an image, a figure, a symbol, or a character indicating the coordinates of the predetermined direction at the time of imaging in the screen from the center coordinates of the image range included in the screen, The image processing apparatus according to claim 1. The moving image is set around the reference axis so that the zenith correction for making the original predetermined axis of the image substantially coincide with the reference axis and the display reference of the image are fixed in the shooting front direction at the predetermined time point or the traveling direction at the predetermined time point. The image processing apparatus according to any one of claims 1 to 6, wherein rotation correction is performed to cancel a rotation change, and the predetermined direction at the time of image capturing is a shooting front direction at a time point of each frame. .. The output control means further includes an image, a figure, a symbol or a character arranged on the screen at a coordinate indicating a predetermined direction at the time of the image capturing, and an image, a figure, a symbol or a character indicating a traveling direction on the screen, or these. The image processing apparatus according to any one of claims 1 to 7, wherein the image processing apparatus is controlled so that any one of them is added. An imaging device for capturing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis,
Imaging means for imaging each frame of the moving image,
Recording means for recording data indicating a predetermined direction at the time of image capturing in each frame of the moving image in association with each frame of the moving image;
With respect to the moving image, rotation correction means for correcting the rotation of the image,
And an output unit that outputs, in addition to the rotation-corrected moving image, information that conditions addition of an expression indicating a predetermined direction at the time of image capturing within a screen including at least a part of the image range of the moving image. , Imaging device. The condition information is the data indicating a predetermined direction at the time of image capturing, the data includes sensor data corresponding to the moving image, and the predetermined direction at the time of image capturing is retained for a predetermined number of frames or more. The image pickup apparatus according to claim 9, wherein the expression is added to the screen as a condition. The apparatus further includes an attention instruction receiving unit that receives an instruction to set a predetermined direction at the time of image capturing while capturing the moving image, and the condition information is instruction information according to the instruction, and the frame includes the instruction information. 11. The image pickup apparatus according to claim 9, wherein the expression is added in the screen on the condition that is associated with. A moving image reproducing system for reproducing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis,
A video acquisition unit that acquires a video whose image has been rotation corrected,
Data acquisition means for acquiring data indicating a predetermined direction at the time of image capturing in each frame of the moving image,
Output control means for controlling to output a screen including at least a part of the image range of the moving image, wherein the output control means is responsive to a predetermined condition being satisfied, based on the data, A moving image reproducing system, characterized in that control is performed so as to add an expression indicating a predetermined direction at the time of image capturing within a screen. Imaging means for imaging each frame of the moving image,
For each frame of the moving image, a rotation correction unit that performs image rotation correction and data output from a sensor provided together with the image pickup unit or correction data calculated from the data is associated with each frame of the moving image. Recording means for recording,
The video playback system according to claim 12, further comprising: During the shooting of the moving image, further includes an attention instruction receiving unit that receives an instruction to set a predetermined direction at the time of image capturing as an attention direction and records the instruction as the data, wherein the output control unit associates the instruction information with a frame. 14. The moving image reproduction system according to claim 12, wherein the predetermined condition is determined to be satisfied under the condition. A method for processing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates around a predetermined axis, wherein the moving image is a rotation-corrected image. Computer
Acquiring data indicating a predetermined direction at the time of imaging in each frame of the moving image,
And a step of controlling to output a screen including at least a part of the image range of the moving image, and in the controlling step, the computer,
A method comprising controlling to add an expression indicating a predetermined direction at the time of image capturing in the screen based on the data, in response to a predetermined condition being satisfied. A program for realizing an image processing device for processing a moving image including continuous images, which is represented by a coordinate system including at least angular coordinates about a predetermined axis, wherein the moving image is a rotation-corrected image. Yes, the program is a computer,
A program for functioning as a data acquisition unit that acquires data indicating a predetermined direction at the time of imaging in each frame of the moving image, and an output control unit that controls to output a screen including at least a part of the image range of the moving image. The output control means controls in response to the satisfaction of a predetermined condition, based on the data, to add an expression indicating the predetermined direction at the time of image capturing in the screen. Yes, the program.