JP2014165763A - Editing device, editing method, and editing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable efficient editing even when editing a panoramic moving image.SOLUTION: An editing device 12 comprises a display control unit 58, an acquisition unit 60, and a storage control unit 68. The display control unit 58 displays, on a display unit, an extracted moving image obtained by extracting, from a panoramic moving image to be edited, an angle-of-view area having an angle of view narrower than that of the panoramic moving image. The acquisition unit 60 acquires operation information related to the angle of view during reproduction of the extracted moving image. The storage control unit 68 stores the operation information in a storage unit in association with an elapsed time from the start of reproduction of the extracted moving image each time the operation information is acquired.

Description

  The present invention relates to an editing device, an editing method, and an editing program.

  2. Description of the Related Art In recent years, editing apparatuses that perform non-linear editing that read various materials stored on a recording medium and generate a series of images by connecting desired portions of the materials are known.

  As a material to be edited by nonlinear editing, a moving image, a still image, sound, and the like are used. It is also known to use a panoramic still image as a material to be edited. When editing a panoramic still image, an area having a size corresponding to the aspect ratio of the display device is extracted at the time of final output. The position of the extraction area is designated by displaying a panoramic still image on the display device and receiving the position of the extraction area in the panoramic still image from the operation unit or the like. Specifically, after receiving an instruction to select a frame to be edited (panorama generated image) from the operation unit or the like according to an operation instruction from the user, a process of receiving designation of a position coordinate of an angle of view as an extraction region in the frame. , Went every frame.

  Conventionally, the extraction range is determined in advance, and the trimming process is performed even when the extraction range is adjusted.

  A technique for improving the operability of adjusting the angle of view such as the position and size of a region extracted from a moving image as the material is also disclosed. For example, instead of numerical input of coordinates, a technique for moving a CG (computer graphic) image following the mouse pointer, or a direction of the rotation direction of the panorama image is set, and the rotation direction region set from the panorama image is set. A method for cutting out a video from a panoramic image is disclosed.

  However, when a moving image is used as an editing target at the time of final output, that is, as an image having a wider angle of view than the aspect ratio of the display device, the position and size of the region extracted from the moving image that is the material is the conventional method. The angle of view adjustment and the time setting such as the frame for adjusting the angle of view in the moving image cannot be executed at the same time. Therefore, conventionally, when a panoramic moving image is an editing target, it has been difficult to perform efficient editing.

  The present invention has been made in view of the above, and provides an editing apparatus, an editing method, and an editing program capable of performing efficient editing even when a panoramic moving image is edited. For the purpose.

  In order to solve the above-described problems and achieve the object, an editing apparatus according to the present invention includes a display control unit, an acquisition unit, and a storage control unit. The display control unit displays, on the display unit, an extracted moving image obtained by extracting an angle-of-view area having an angle of view narrower than the panoramic moving image from the panoramic moving image to be edited. The acquisition unit acquires operation information related to the angle of view during reproduction of the extracted moving image. The storage control unit stores the operation information in the storage unit in association with the elapsed time from the start of reproduction of the extracted moving image every time the operation information is acquired.

  The editing method according to the present invention includes a step of displaying, on a display unit, an extracted moving image obtained by extracting an angle-of-view area having a smaller angle of view than the panoramic moving image from the panoramic moving image to be edited, and during the reproduction of the extracted moving image. Acquiring operation information relating to the angle of view; storing the operation information in a storage unit in association with an elapsed time from the start of reproduction of the extracted moving image for each time when the operation information is acquired; , Including editing methods.

  An editing program according to the present invention displays, on a display unit, an extracted moving image obtained by extracting an angle-of-view region having a narrower angle of view than a panoramic moving image from a panoramic moving image to be edited on a computer; The step of acquiring operation information relating to the angle of view during reproduction, and storing the operation information in the storage unit in association with the elapsed time from the start of reproduction of the extracted moving image for each time when the operation information is acquired. Editing step.

  According to the present invention, it is possible to provide an editing apparatus, an editing method, and an editing program that can perform efficient editing even when editing a panoramic video.

FIG. 1 is a schematic diagram of an editing system according to the present embodiment. FIG. 2 is a schematic diagram showing another editing system. FIG. 3 is a schematic diagram showing another editing system. FIG. 4 is a schematic external view of the photographing apparatus. FIG. 5 is a diagram illustrating an example of a usage state of the imaging apparatus. FIG. 6 is an explanatory diagram of an image photographed by the photographing apparatus. FIG. 7 is a schematic diagram illustrating an example of one frame of the omnidirectional panoramic video. FIG. 8 is a schematic diagram showing the UI unit. FIG. 9 is an explanatory diagram of the hardware configuration of the photographing apparatus. FIG. 10 is an explanatory diagram of the hardware configuration of the UI unit. FIG. 11 is an explanatory diagram of the hardware configuration of the editing apparatus. FIG. 12 is a schematic diagram illustrating a functional configuration of the editing apparatus. FIG. 13 is a schematic diagram illustrating an example of an editing screen. FIG. 14 is an image diagram specifically illustrating an example of the editing screen. FIG. 15 is an explanatory diagram showing a state in which an omnidirectional panoramic moving image assumed to be a three-dimensional solid sphere is photographed by the virtual photographing device from the center of the solid sphere. FIG. 16 is an explanatory diagram showing a state in which an omnidirectional panoramic moving image assumed to be a three-dimensional solid sphere is photographed by the virtual photographing device from the center of the solid sphere. FIG. 17 is a schematic diagram illustrating an example of a media control area displayed on the UI unit. FIG. 18 is an explanatory diagram of an example of an operation instruction using the UI unit. FIG. 19 is a flowchart illustrating a procedure of editing processing executed by the control unit. FIG. 20 is a schematic diagram illustrating an example of the data structure of the operation information stored in the HD. FIG. 21 is a schematic diagram illustrating an example of a user operation.

  Hereinafter, embodiments of the present invention will be described.

  FIG. 1 is a schematic diagram of an editing system 10 according to the present embodiment.

  The editing system 10 includes an editing device 12, a photographing device 14, a display device 16, and an operation unit 18. The editing device 12, the photographing device 14, the display device 16, and the operation unit 18 are connected to be communicable via a wireless or wired communication line such as a network.

  For the communication line, for example, a short-range wireless technology, a wireless communication network using a mobile communication system, the Internet, or the like is used. Examples of the short-range wireless technology include Bluetooth (registered trademark). Examples of the wireless communication network using the mobile communication system include 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and the like.

  The editing device 12 is a computer that executes a material editing process. Examples of the material to be edited include moving images, still images, and audio. Details of the editing process will be described later.

  The photographing device 14 is a photographing device that obtains a panoramic moving image. A panoramic video is a video having a wider angle of view than the aspect ratio at the time of final output. In the present embodiment, the imaging device 14 obtains a panoramic video. The editing device 12 obtains the panoramic moving image obtained by the photographing device 14 as an editing target. In the present embodiment, the case where the imaging device 14 obtains an omnidirectional panoramic moving image as a panoramic moving image will be described. The omnidirectional panoramic moving image is a panoramic moving image obtained by obtaining an image within a solid angle of 4π radians and photographing the entire celestial sphere.

  Note that the material to be edited in the editing apparatus 12 may include at least a panoramic video, and other materials may also be subject to editing. It should be noted that suitable editing is also possible for the omnidirectional panoramic video.

  The display device 16 is a general display device that displays an image. For example, a liquid crystal display or an organic EL display. In the present embodiment, the display device 16 displays an edit screen 74 and the like which will be described later.

  The operation unit 18 has a function of accepting various operation instructions from the user. In the example illustrated in FIG. 1, the operation unit 18 includes a keyboard 18A and a mouse 18B.

  Note that the editing system 10 may be configured to include other types of operation units other than the keyboard 18A and the mouse 18B, or may further include a plurality of imaging devices 14.

  FIG. 2 is a schematic diagram showing another editing system 10A. As shown in FIG. 2, the editing system 10 </ b> A is configured to further include a UI unit 18 </ b> C in addition to a keyboard 18 </ b> A and a mouse 18 </ b> B as the operation unit 18 in the editing system 10. The UI unit 18C and the editing device 12 are connected in a wired manner in FIG. 2, but may be wirelessly connected. In particular, in the case of wireless connection, the editing apparatus 12 may be placed on the Internet, and the UI unit 18C may operate the editing apparatus 12 over the network to perform editing.

  The UI unit 18C is a device having a touch panel function that accepts an operation input from a user and displays various images. For the UI unit 18C, for example, a computer such as a smartphone, a tablet terminal, a notebook personal computer, a desktop personal computer, or a PDA (Personal Data Assistance) is used.

  In the present embodiment, a case where a smartphone or a tablet terminal is used as the UI unit 18C will be described, but any configuration having a touch panel function may be used.

FIG. 3 is a schematic diagram showing another editing system 10B. As illustrated in FIG. 3, the editing system 10B includes a plurality of photographing devices 14 ₁ , photographing devices 14 ₂ , and photographing devices 14 ₃ as the photographing devices 14 in the editing system 10. 3 shows a case where the editing system 10B includes three photographing devices 14, but the configuration may include two or four or more.

In the editing system 10B, the editing device 12 is connected to a plurality of imaging devices 14 (imaging device 14 ₁ , imaging device 14 ₂ , imaging device 14 ₃ ) via a multi-video receiver 13. By adopting a configuration in which a plurality of imaging devices 14 are connected via the multi-video receiver 13, even if the communication interface capability of the imaging device 14 is limited, the panoramic panorama can be efficiently transmitted from the plurality of imaging devices 14. A moving image can be acquired. The multi video receiver 13 may be a video capture device and is not limited to a multi video receiver.

The editing device 12 of the editing system 10B includes a material (global celestial panoramic video in the present embodiment) acquired from any of the plurality of imaging devices 14 (imaging device 14 ₁ , imaging device 14 ₂ , imaging device 14 ₃ ). It is good also as a structure provided with the switching function which switches whether editing is performed. Note that the editing device 12 may perform switching control so that the material acquired by the imaging device 14 that has switched the material to be displayed on the display device 16 or the UI unit 18C each time the imaging device 14 is switched.

In addition, the editing device 12 preferably includes a plurality of UI units 18C (18C _{1 to} 18C ₃ ) when data input from the multi-video receiver 13 is executed in real time.

  In the following description, it is assumed that the editing system 10 is the editing system 10A having the configuration shown in FIG.

  Next, the external appearance of the imaging device 14 will be described with reference to FIG. FIG. 4 is a schematic external view of the photographing apparatus 14. FIG. 4A is a side view of the photographing apparatus 14. FIG. 4B is a side view of the imaging device 14 on the side opposite to FIG. 4A. FIG. 4C is a plan view of the imaging device 14.

  As shown in FIG. 4A, the photographing device 14 has a size that a human can hold with one hand, for example. Note that the size of the imaging device 14 is not limited to such a size.

  As shown in FIG. 4, a lens 14 </ b> A is provided on the front side (one side) and a lens 14 </ b> B is provided on the back side (the other side) on the upper part of the photographing apparatus 14. An image is guided through each of these wide-angle lenses having an angle of view of 180 degrees or more, and an image is formed on each image sensor. Examples of the image sensor include a CCD and a CMOS. 4B, an operation unit 14C such as a shutter button is provided on the front side of the photographing apparatus 14.

  Next, an example of a usage state of the photographing apparatus 14 will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of a usage state of the imaging device 14. As shown in FIG. 5, the photographing device 14 is used for photographing a subject around the user with the hand held by the user. In this case, a subject around the user is imaged by the lens 14A and the lens 14B, and two hemispherical images are obtained, and an image within a solid angle of 4π radians is obtained.

  Next, an image photographed by the photographing device 14 will be described with reference to FIG. FIG. 6 is an explanatory diagram of an image photographed by the photographing device 14. 6A is a hemispheric image (front side) photographed by the lens 14A, FIG. 6B is a hemispheric image photographed by the lens 14B (rear side), and FIG. 6C is an image represented by Mercator projection. (Hereinafter, referred to as “Mercatl image”).

  As shown in FIG. 6A, the image obtained by the lens 14A is a curved hemispherical image (front side) because there is a fisheye lens that is one of the wide-angle lenses. Further, as shown in FIG. 6B, the image obtained by the lens 14B is a fish-eye lens that is one of the wide-angle lenses, and thus becomes a curved hemispherical image (rear side). Then, the hemispherical image (front side) and the hemispherical image (rear side) are combined by, for example, the photographing device 14 to create a Mercator image shown in FIG. As described above, a Mercator moving image is generated by continuously generating one Mercator image.

  Returning to FIG. 2, the Mercator moving image obtained by the photographing device 14 is transmitted from the photographing device 14 to the editing device 12 when connected to the editing device 12. In the present embodiment, the imaging device 14 transmits a Mercator moving image to the editing device 12. In addition, two hemispherical images photographed by the photographing device 14 may be transmitted to the editing device 12. In this case, the editing device 12 generates a Mercator image shown in FIG. 6C by using two hemispherical images, and when the Mercator images are continuously connected, an omnidirectional panoramic moving image is obtained.

  The editing device 12 acquires the Mercator moving image from the photographing device 14 as a panoramic moving image. FIG. 7 is a schematic diagram illustrating an example of one frame of the omnidirectional panoramic video.

  In the present embodiment, the editing device 12 executes editing processing of the omnidirectional panoramic video (details will be described later).

  The editing device 12 displays the acquired Mercator image on the display device 16, a UI unit 18 </ b> C, which will be described later, and the like as an omnidirectional panoramic moving image by using OpenGL (Open Graphics Library).

  Next, the configuration of the UI unit 18C will be described.

  FIG. 8 is a schematic diagram showing the UI unit 18C. In the present embodiment, a case where a terminal such as a smartphone is used as the UI unit 18C will be described. The UI unit 18C includes a display 315. The display 315 is a device having a touch panel function for displaying various images and receiving various operation instructions from the user.

  Next, the hardware configuration of the imaging device 14, the editing device 12, and the UI unit 18C will be described.

  FIG. 9 is an explanatory diagram of a hardware configuration of the imaging device 14.

  As shown in FIG. 9, the imaging device 14 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and an SRAM (Static Random Access Memory) 113. , A DRAM (Dynamic Random Access Memory) 114, an operation unit 14C, a network I / F 116, and a communication unit 117.

  The imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 14A and 14B each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging elements 15A provided corresponding to the wide-angle lenses. 15B. The image sensors 15A and 15B are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by a fisheye lens into image data of an electric signal and output the image data. A timing generation circuit for generating a vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 15A and 15B of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 15A and 15B of the imaging unit 101 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 14C, network I / F 116, and the like are also connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image pickup devices 15A and 15B through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. Then, data of a Mercator image as shown in FIG. 6C is created. The image data output by the imaging elements 15A and 15B is continuously performed. When a plurality of Mercator images are connected by these image data, an omnidirectional panoramic moving image is obtained.

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 15A and 15B using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 15A and 15B as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture the status data of the register groups of the imaging elements 15A and 15B and send it to the CPU 111.

  Further, the imaging control unit 105 instructs the imaging elements 15A and 15B to output image data at the timing when the shutter button of the operation unit 14C is pressed. Some imaging devices 14 may have a preview display function using a display or a function corresponding to moving image display. In this case, output of image data from the image sensors 15A and 15B is continuously performed at a predetermined frame rate (frame / min).

  The imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of the image data of the imaging elements 15A and 15B in cooperation with the CPU 111. In the present embodiment, the imaging device 14 is not provided with a display unit, but a display unit may be provided.

  The CPU 111 controls the overall operation of the photographing apparatus 14 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed Mercator image data.

  The operation unit 14C is a generic name for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user inputs various shooting modes and shooting conditions by operating the operation buttons.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. Further, the network I / F 116 may be a network interface regardless of wireless or wired. The data of the Mercator image stored in the DRAM 114 is recorded on an external medium via the network I / F 116 or, if necessary, the editing device 12 via the network I / F 116 serving as the network I / F. To the external device.

  The communication unit 117 communicates with an external device such as the editing device 12 through an antenna by a short-range wireless technology such as WiFi (wireless fidelity) or Bluetooth (registered trademark). The communication unit 117 can also transmit Mercator moving image data as an omnidirectional panoramic moving image to an external device such as the editing device 12.

  FIG. 10 is an explanatory diagram of the hardware configuration of the UI unit 18C.

  As shown in FIG. 10, the UI unit 18C includes a CPU 301 that controls the operation of the entire UI unit 18C, a ROM 302 that stores basic input / output programs, a RAM (Random Access Memory) 303 that is used as a work area for the CPU 301, An EEPROM (Electrically Erasable and Programmable ROM) 304 that reads or writes data according to control, a CMOS 305 that captures an image of a subject under the control of the CPU 301 and obtains image data, an electromagnetic compass or gyro compass that detects geomagnetism, and acceleration Control reading and writing (memory) of data to various acceleration / direction sensors 306 such as sensors and recording media 307 such as flash memory. It is equipped with a media drive 308.

  The EEPROM 104 stores an operating system (OS) executed by the CPU 301, other programs, and various data. A CCD may be used instead of the CMOS 305.

  In addition, the UI unit 18C includes a voice input unit 311 that converts voice into a voice signal, a voice output unit 312 that converts voice signals into voice, an antenna 313a, and a radio communication signal that uses the antenna 313a to obtain the nearest base station. A communication unit 313 that communicates with the GPS, a GPS receiving unit 314 that receives GPS signals including position information (latitude, longitude, and altitude) of the UI unit 18C from a GPS (Global Positioning System) satellite, and a display 315.

  The display 315 includes a display unit such as liquid crystal or organic EL, and a pressure-sensitive or electrostatic panel. Therefore, the display 315 displays various images and accepts various operation instructions from the user as described above.

  Next, the hardware configuration of the editing device 12 will be described.

  FIG. 11 is an explanatory diagram of the hardware configuration of the editing device 12.

  As shown in FIG. 11, the editing device 12 includes a CPU 30 that controls the operation of the editing device 12 as a whole, a ROM 32 that stores a program used to drive the CPU 30 such as an IPL, a RAM 34 that is used as a work area for the CPU 30, and an editing device. 12 includes an HD 38 for storing various data such as a program for 12 and an HDD (Hard Disk Drive) 36 for controlling reading or writing of various data to the HD 38 according to the control of the CPU 30.

  Also, the editing device 12 includes a media drive 507 that controls reading or writing (storage) of data with respect to a recording medium 506 such as a flash memory, a display 508 that displays various images, a photographing device 14, and an operation unit 18 (a keyboard 18A, a mouse). 18B, I / F 509 for communicating with the UI unit 18C), a CD-ROM drive 514 for controlling reading or writing of various data with respect to a compact disc read only memory (CD-ROM) 513 as an example of a removable recording medium. And a bus line 510 that electrically connects the above-described components.

  Next, a functional configuration of the editing device 12 will be described.

  FIG. 12 is a schematic diagram illustrating a functional configuration of the editing device 12. As illustrated in FIG. 12, the editing device 12 includes a control unit 52, an operation unit 18 such as a UI unit 18 </ b> C, and an HD (hard disk) 38. The control unit 52 is electrically connected to the imaging device 14, the display device 16, the operation unit 18 such as the UI unit 18 </ b> C, and the HD 38.

  The control unit 52 is a computer that controls the entire editing apparatus 12 and includes the above-described CPU 30, ROM 32, RAM 34, and the like.

  The control unit 52 includes a connection unit 54, a capture unit 56, a display control unit 58, an acquisition unit 60, a determination unit 62, a trimming unit 64, a storage control unit 68, and a writing unit 70.

  The connection unit 54 establishes a connection with the imaging device 14. For example, the connection unit 54 performs communication control for establishing a connection with the photographing apparatus 14 according to a predetermined communication protocol in a state where the photographing apparatus 14 is connected to the editing apparatus 12 by wire or wirelessly. As a result, the connection unit 54 establishes a connection with the imaging device 14.

  The capturing unit 56 captures an omnidirectional panoramic moving image from the imaging device 14. Further, the capturing unit 56 imports materials other than the omnidirectional panoramic video from the imaging device 14 and various storage media. The display control unit 58 performs control to display various materials on the display device 16 and the UI unit 18C.

  In the present embodiment, the display control unit 58 performs control to display the editing screen 74 on the display device 16. In addition, the display control unit 58 performs control to display the media control area 76 on the UI unit 18C.

  The editing screen 74 is a display screen for editing that is displayed on the display device 16 when the editing device 12 executes editing processing of the omnidirectional panoramic video.

  FIG. 13 is a schematic diagram illustrating an example of the edit screen 74. FIG. 14 is an image diagram specifically showing an example of the edit screen 74.

  As shown in FIGS. 13 and 14, the edit screen 74 is provided with a title area 74A, a menu area 74B, a media browser area 74C, a video monitor area 74D, a media control area 74E, and a timeline area 74F.

  The title area 74A is an area for displaying an application title and the like. The menu area 74B is an area for displaying a menu item list. In the example shown in FIG. 14, “file”, “edit”, “display”, “tool”, “effect”, “window”, and “help” are shown as menu items.

  The user can select any of these menu items by operating the operation unit 18 and perform various settings indicated by the menu items.

  The media browser area 74C is an area for displaying a list of materials that can be edited in the editing apparatus 12. The video monitor area 74D is an area for displaying an image of a material selected as an object to be edited or a moving image of the material. A timeline area 74F is an editable material displayed in the media browser area 74C in chronological order. It is the arranged image. In addition, the arrangement position of each material in time series is adjusted by an operation instruction of the operation unit 18 by the user.

  In the present embodiment, the edit screen 74 includes a media control area 74E. The media control area 74E is an area for displaying an omnidirectional panoramic image to be edited selected by the user via the operation unit 18. Specifically, the media control area 74E displays an extracted moving image extracted from the omnidirectional panoramic moving image to be edited (details will be described later).

  The user gives an operation instruction for image editing using the extracted moving image of the omnidirectional panoramic moving image displayed in the media control area 74E. The editing apparatus 12 receives operation information indicated by an operation instruction instructed via the media control area 74E, and executes an editing process described later.

  The media control area 76 displayed on the UI unit 18C is the same image as the media control area 74E. That is, the user issues an operation instruction for image editing while confirming the extracted image of the omnidirectional panoramic moving image to be edited displayed on the UI unit 18C.

  Returning to FIG. 12, in the present embodiment, the display control unit 58 extracts an extracted moving image in which an angle-of-view area having a narrower angle of view than the omnidirectional panoramic moving image is extracted from the omnidirectional panoramic moving image to be edited. Control is performed to display the image on the display device 16 and the UI unit 18C.

  That is, the display control unit 58 extracts an extracted moving image obtained by extracting a field angle region having a narrower angle of view than the omnidirectional panoramic moving image from the omnidirectional panoramic moving image to be edited. The moving image for editing showing the image is displayed on the media control area 74E and the UI unit 18C of the display device 16.

  Specifically, the display control unit 58 uses the extracted moving image extracted from the omnidirectional panoramic moving image to be edited, the omnidirectional panoramic moving image assuming a three-dimensional solid sphere as the center of the three-dimensional sphere. The moving image captured at the angle of view by the virtual imaging device is displayed on the media control area 74E and the UI unit 18C.

  FIGS. 15 and 16 are explanatory diagrams showing a state in which a panoramic moving image assumed to be a three-dimensional solid sphere is photographed from the center of the solid sphere by a virtual photographing device.

  It is assumed that the omnidirectional panoramic video is a three-dimensional solid sphere as shown in FIG. In this case, as shown in FIGS. 15 and 16, the virtual photographing device is positioned at the center P of the solid sphere CS. Then, a coordinate system with this center P as the origin is constructed. The angle of view area S is an area corresponding to a shooting angle of view when the virtual imaging device installed at the center P captures the stereoscopic spherical panoramic moving image with a narrower angle of view than the panoramic moving image. is there.

  The angle-of-view area S depends on the orientation of the virtual photographing device in the omnidirectional panoramic moving image to be edited, the photographing magnification (zoom) of the virtual photographing device, and the viewpoint position of the virtual photographing device in the omnidirectional panoramic moving image. Identified. The photographing magnification indicates magnification information (zoom) of the angle of view.

  The attitude of the virtual imaging device is specified by the rotation angles (yaw, pitch, roll) of each of the three axes that constitute the coordinate system with the center P as the origin.

  The orientation of the virtual photographing device, the magnification information of the angle of view, and the viewpoint position of the virtual photographing device are instructed by an operation instruction of the operation unit 18 by the user. The operation unit 18 transmits the operation information input by the operation instruction to the editing device 12.

  Thereby, the editing device 12 acquires operation information. The operation information is information related to the angle of view as the shooting angle of view of the virtual shooting device. As described above, at least one of the attitude of the virtual shooting device, the magnification information of the angle of view, and the viewpoint position of the virtual shooting device is selected. Including.

  The acquisition unit 60 receives operation information regarding the angle of view from the operation unit 18 during reproduction of the extracted moving image on at least one of the display device 16 and the UI unit 18C.

  The determination unit 62 determines whether or not the material to be edited is an omnidirectional panoramic video. The discriminating unit 62 discriminates whether or not the material to be edited instructed by the operation instruction of the operation unit 18 by the user is an omnidirectional panoramic moving image by the discrimination method. Examples of the determination method include determination based on the format of a moving image, attribute information included in the moving image, and the like.

  The storage control unit 68 edits the orientation of the virtual imaging device, the magnification information of the angle of view, and the viewpoint position of the virtual imaging device included in the operation information acquired by the acquisition unit 60 for each time when the operation information is acquired. In accordance with the elapsed time from the start of playback of the omnidirectional panoramic video during playback, control for sequentially storing in the HD 38 is performed.

  Note that the storage control unit 68 stores the operation information in the HD 38 in association with the above-mentioned elapsed time every time when the instruction time in which the operation information is instructed by the operation unit 18 is the time when the operation information is acquired. Also good. In this case, the operation unit 18 may be configured to transmit the operation information including the instruction time for which the operation information is instructed to the editing device 12. The storage control unit 68 stores the elapsed time and the operation information in the HD 38 in association with the identification information of the omnidirectional panoramic moving image to be edited.

  The trimming unit 64 performs a trimming process on the omnidirectional panoramic moving image to be edited by a predetermined method. Examples of the predetermined method include a trimming method performed on a still image.

  The writing unit 70 executes a moving image writing process. The writing unit 70 includes a determination unit 70A, a cutting unit 70B, a conversion unit 70C, and a generation unit 70D. The discriminating unit 70A discriminates whether or not the writing target is an omnidirectional panoramic video.

  The cutout unit 70B uses the operation information of the omnidirectional panoramic moving image to be written stored in the HD 38, and displays the image of the angle of view specified by the operation information regarding the angle of view for each elapsed time from the start of reproduction. A corner area is cut out from the omnidirectional panoramic moving image to be written at every corresponding elapsed time.

  Specifically, the cutout unit 70B determines the angle of view from the attitude of the virtual photographing device, the magnification information of the angle of view, and the viewpoint position of the virtual photographing device included in the operation information of the omnidirectional panoramic moving image to be written. And the angle-of-view area S of the specified angle of view is cut out from the omnidirectional panoramic moving image for each corresponding elapsed time.

  The conversion unit 70C converts the extracted moving image extracted from the omnidirectional panoramic moving image into a predetermined file format or a file format instructed by an operation instruction by the user. The generation unit 70D generates the cutout moving image converted by the conversion unit 70C as a moving image to be written.

  FIG. 17 is a schematic diagram illustrating an example of the media control area 76 displayed on the UI unit 18C. The UI unit 18C displays a display field 76A and various instruction buttons such as a REC button 76B together with the media control area 76.

  For the media control area 76 displayed on the UI unit 18C, the user performs an operation instruction and inputs operation information by combining one or more operations such as pinch-in / pinch-out and various pressing instructions.

  FIG. 18 is an explanatory diagram of an example of an operation instruction using the UI unit 18C.

  For example, when adjusting the yaw and pitch in the posture of the virtual photographing device and the viewpoint position of the virtual photographing device, the user extracts the panoramic panoramic video displayed in the media control area 76 of the UI unit 18C. An operation instruction is given while referring to the moving image V1, and operation information is input.

 Specifically, the user gives an operation instruction to adjust the yaw, pitch, and viewpoint position as the posture by instructing the touch panel in the media control area 76 in the vertical and horizontal directions on the panel (see FIG. 18 (A)). The UI unit 18C receives operation information including yaw, pitch, and viewpoint position by receiving these operation instructions.

  For example, when adjusting the roll of the virtual photographing device, the user moves the two fingers in the direction in which the virtual photographing device is to be rotated with the two fingers placed on the UI unit 18C. (See FIG. 18B). The UI unit 18C receives operation information including a role by receiving these operation instructions.

  Further, for example, the user may adjust the roll of the virtual photographing device by moving the finger of one hand against the UI unit 18C so as to draw a circle on the UI unit 18C with the other finger. Good (see FIG. 18C). The UI unit 18C receives operation information including a role by receiving these operation instructions.

  Further, for example, the user performs pinch-in / pinch-out when performing a zoom instruction of the virtual photographing apparatus (FIG. 18D). The UI unit 18C receives zoom (magnification information) of the virtual photographing device by receiving these operation instructions.

  Next, editing processing executed by the control unit 52 of the editing device 12 will be described.

  FIG. 19 is a flowchart illustrating a procedure of editing processing executed by the control unit 52.

  First, the acquisition unit 60 of the editing device 12 receives an instruction to execute an editing program from the operation unit 18. The control unit 52 executes the procedure shown in FIG. 19 by executing an editing program stored in advance in the ROM 32 or the like. When the control unit 52 executes the editing program, the display control unit 58 performs control for displaying the editing screen 74 on the display device 16 and performs control for displaying the media browser area 74C on the UI unit 18C.

  In the state before executing the following processing, nothing is stored in the media browser area 74C, video monitor area 74D, media control area 74E, and timeline area 74F in the edit screen 74 displayed on the display device 16. It is not displayed, or the data at the time of the previous editing is displayed. Similarly, nothing is displayed in the media control area 76 displayed on the UI unit 18C.

  First, the connection unit 54 establishes a connection with the imaging device 14 (step S200). By the processing in step S200, the editing device 12 is in a state where signals can be exchanged with the photographing device 14.

  Next, the control unit 52 repeats the processing of step S202 to step S206 for the number of materials instructed from the operation unit 18.

  Specifically, the capturing unit 56 captures, that is, captures a panoramic moving image as a material instructed as a capturing target by an operation instruction of the operation unit 18 by the user (step S202).

  Next, the display control unit 58 arranges the material instructed by the operation instruction of the operation unit 18 by the user in the omnidirectional panoramic moving image captured in step S202 in the timeline region 74F on the editing screen 74. (Step S204). In step S204, the display control unit 58 places the material specified by the information indicating the material acquired by the capturing unit 56 from the operation unit 18 in the timeline area 74F.

  Next, the capturing unit 56 imports other materials (step S206). In step S206, the other material instructed by the operation instruction of the operation unit 18 by the user is imported. The display control unit 58 places the imported material in the timeline area 74F.

  Next, the acquisition unit 60 determines whether or not any selection instruction from among the plurality of timelines displayed in the timeline area 74F has been received from the operation unit 18 (step S208).

  In step S208, a negative determination (step S208: No) is repeated until an affirmative determination (step S208: Yes). If an affirmative determination is made in step S208 (step S208: Yes), the process proceeds to step S210.

  In step S210, it is determined whether or not the timeline selected in step S208 is the timeline of the omnidirectional panoramic video (step S210).

  If a negative determination is made in step S210 (step 210: No), the process proceeds to step S224 described later. If an affirmative determination is made in step S210 (step S210: Yes), the process proceeds to step S212.

  In step S212, the display control unit 58 extracts the view angle region S from the omnidirectional panoramic moving image of the timeline selected in step S208 (step S212). In step S212, the display control unit 58 specifies the view angle region S by reading the operation information indicated by the latest time stamp corresponding to the identification information of the omnidirectional panoramic moving image stored in the HD 38. ,Extract. When the operation information corresponding to the HD 38 is not stored, the display control unit 58 reads the operation information as an initial value stored in advance in the HD 38 to identify and extract the view angle region S. This initial value may be determined in advance.

  Next, the display control unit 58 uses the extracted moving image of the angle-of-view area S extracted in step S212 to display the media control area 74E of the editing screen 74 displayed on the display device 16 and the media control area 76 of the UI unit 18C. (Step S214).

  Next, the acquisition unit 60 determines whether or not there is a reproduction instruction (step S216). The determination in step S216 is determined by determining whether the acquisition unit 60 has received a signal indicating a reproduction instruction from the operation unit 18. In the present embodiment, a signal indicating a playback instruction is transmitted from the UI unit 18C to the editing apparatus 12 when the user instructs to operate the REC button 76B of the UI unit 18C. The acquisition unit 60 determines whether or not this reproduction instruction has been accepted, thereby performing the determination in step S216.

  If there is no reproduction instruction (step S216: No), the process proceeds to step S224. If there is a reproduction instruction (step S216: Yes), the display control unit 58 reproduces the extracted moving image extracted in step S212 (step S218). Through the processing in step S218, the playback of the extracted moving image displayed in the media control area 76C of the display device 16 and the media control area 76 of the UI unit 18C is started.

  Next, when the acquisition unit 60 acquires the operation information from the operation unit 18, the acquired operation information includes the elapsed time from the start of reproduction of the extracted moving image, the identification information of the omnidirectional panoramic moving image to be edited, Are stored in the HD 38 in association with (step S220, step S222).

  FIG. 20 is a schematic diagram illustrating an example of a data structure of operation information stored in the HD 38. FIG. 20 shows an example in which the operation information includes elapsed time.

  As shown in FIG. 20, the operation information includes the orientation of the virtual photographing device specified by yaw, pitch, roll, magnification information (zoom), and the elapsed time from the start of playback. And a video time frame shown.

  For this reason, the operation information as the control information of the virtual photographing device can be applied only to the material arranged in the timeline region 74F. For example, it is possible to place the same material in a plurality of timeline regions 74F, input different operation information for the virtual photographing device, and simultaneously display them on the display device 16 and the UI unit 18C.

  In the example shown in FIG. 20, the upper limit value and the lower limit value of the rotation magnification of the roll are set in a data structure so that a larger range can be set than yaw and pitch. A data area corresponding to each item is defined. This is because the number of rotations is considered to be larger when the virtual photographing device is rolled than when the virtual photographing device is rotated up and down. Note that the size of the data area corresponding to each item is not limited to the form shown in FIG.

  In the example illustrated in FIG. 20, the case where each item is indicated in units of 8 bits has been described, but is not limited to 8 bits.

  Returning to FIG. 19, the acquisition unit 60 then executes a known trimming process (time editing process) in accordance with the trimming instruction from the operation unit 18 (step S <b> 224).

  Next, the control unit 52 determines whether or not to end the editing process (step S226). The determination in step S226 can be made by determining whether the acquisition unit 60 has received a signal indicating the end of the editing process from the operation unit 18.

  If a negative determination is made in step S226 (step S226: No), the process proceeds to step S208. If an affirmative determination is made in step S226 (step S226: Yes), the process proceeds to step S228.

  In step S228, the acquisition unit 60 determines whether a writing instruction has been received from the operation unit 18. If a negative determination is made in step S228 (step S228: No), this routine ends. On the other hand, if an affirmative determination is made in step S228 (step S228: Yes), the process proceeds to step S230.

  In step S230, the determination unit 62 determines whether the material for which writing has been instructed is an omnidirectional panoramic video (step S230).

  If a negative determination is made in step S230 (step S230: No), the process proceeds to step S236. If an affirmative determination is made in step S230 (step S230: Yes), the process proceeds to step S232.

  In step S232, the cutout unit 70B reads the operation information corresponding to the omnidirectional panoramic moving image instructed to be written from the HD 38 for each corresponding elapsed time, and the angle of view area S specified by each read operation information. Is cut out from the frame of the corresponding elapsed time of the omnidirectional panoramic video (step S232).

  Next, the conversion unit 70C performs file conversion on the extracted moving image extracted by the process of step S232 (step S234). For example, the conversion unit 70C converts the cutout moving image into a predetermined file format or a file format instructed by an operation instruction by the user.

  Next, the generation unit 70D generates the cut out moving image converted by the conversion unit 70C as a video file (moving image) to be written (step S236), and ends this routine.

  19 is executed, for example, in the media control area 76 of the UI unit 18C and the media control area 74E of the editing screen 74 displayed on the display device 16, the entire sky to be edited is displayed. An extracted moving image obtained by extracting the view angle region S from the spherical panoramic moving image is displayed. When the extracted moving image is reproduced, the operation information such as the attitude of the virtual photographing device, the photographing magnification of the virtual photographing device, the viewpoint position of the virtual photographing device, and the like, which are elements for specifying the view angle region S, is operated from the user. , The operation information corresponding to the elapsed time from the start of reproduction is stored in the HD 38.

  FIG. 21 is a schematic diagram illustrating an example of a user operation.

  As shown in FIG. 21A, for example, an angle of view area S1 of the extracted moving image displayed in the media control area 76 of the UI unit 18C is an angle of view area in the omnidirectional panoramic moving image in accordance with a user operation instruction. It is assumed that an operation is performed so as to move from S1 to the view angle region S2. By this operation, the operation locus of the viewpoint position of the virtual photographing device in the omnidirectional panoramic video is input as the instruction information.

  In this case, the control unit 52 stores, in the HD 38, operation information corresponding to an operation trajectory in which the user has operated the view angle area S1 to the view angle area S2 in association with the elapsed time from the start of reproduction.

  For this reason, when the generation unit 70D generates a cutout moving image for reproduction in accordance with the operation information, the moving image shown in FIG. 21B is obtained.

  As described above, in the editing system 10 according to the present embodiment, the display control unit 58 of the editing device 12 has an angle of view narrower than the omnidirectional panoramic moving image from the omnidirectional panoramic moving image to be edited. The extracted moving image from which the area S has been extracted is displayed in the media control area 74E on the editing screen 74 of the display device 16 and the media control area 76 of the UI unit 18C. During the reproduction of the extracted moving image, the acquisition unit 60 receives operation information related to the angle of view from the operation unit 18. The operation information is input in response to an operation instruction from the operation unit 18 by the user and transmitted from the operation unit 18 to the editing device 12. The storage control unit 68 stores the operation information in the HD 38 in association with the elapsed time from the start of reproduction of the extracted moving image every time the operation information is acquired.

  For this reason, in the editing system 10 according to the present embodiment, editing of the time axis of the panoramic panoramic video and editing of the field angle region S to be extracted in the panoramic panoramic video can be performed simultaneously. .

  That is, the editing system 10 according to the present embodiment can simultaneously control the angle of view and time specialized for the omnidirectional panoramic video.

  Therefore, the editing system 10 according to the present embodiment can perform efficient editing even when editing a panoramic moving image such as an omnidirectional panoramic moving image.

  In the present embodiment, the editing device 12, the photographing device 14, and the UI unit 18C include a control device such as a CPU, a storage device such as a ROM and a RAM, an external storage device such as an HDD and a CD drive device, And a hardware configuration using a normal computer.

  The editing program for executing the editing process executed by the editing device 12 is provided by being incorporated in advance in NV-RAM, ROM, or other nonvolatile storage medium. An editing program for executing the editing process executed by the editing device 12 is a file in an installable format or an executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD ( It is also possible to record and provide in a computer-readable recording medium such as Digital Versatile Disk).

  The editing program for executing the editing process executed by the editing device 12 is stored on a computer connected to a network such as the Internet, and is provided or distributed by being downloaded via the network. May be.

  Further, an editing program for executing the editing process executed by the editing device 12 may be provided by being incorporated in advance in a ROM or the like.

  Further, the program for executing the editing process executed by the editing device 12 has a module configuration including the above-described units. As actual hardware, a CPU (processor) stores the editing program from the storage medium. Are read out and executed, the above-described units are loaded onto the main storage device, and the above-described units are generated on the main storage device.

  While various embodiments for carrying out the invention have been described above, the present invention is not limited to the above-described embodiments. Modifications can be made without departing from the spirit of the present invention.

10, 10A, 10B Editing system 12 Editing device 14 Imaging device 16 Display device 18 Operation unit 18C UI unit 58 Display control unit 60 Acquisition unit 68 Storage control unit 70B Extraction unit 70C Conversion unit

Japanese Patent No. 3465892 Japanese Patent No. 4610156

Claims (9)

A display control unit for displaying on the display unit an extracted moving image obtained by extracting an angle-of-view area having a narrower angle of view than the panoramic moving image to be edited;
An acquisition unit that acquires operation information related to the angle of view during reproduction of the extracted moving image;
A storage control unit that stores the operation information in a storage unit in association with an elapsed time from the start of reproduction of the extracted moving image for each time when the operation information is acquired;
Editing device equipped with.   The editing apparatus according to claim 1, wherein the panoramic moving image is an omnidirectional panoramic moving image obtained by photographing an omnidirectional range.   The editing apparatus according to claim 1, wherein the acquisition unit acquires the operation information including magnification information of the angle of view. The display control unit displays the extracted moving image on the display unit as a moving image obtained by capturing the panoramic moving image, which is assumed to be a three-dimensional solid sphere, from the center of the three-dimensional sphere by the virtual photographing device at the angle of view. And
The editing apparatus according to claim 1, wherein the acquisition unit acquires the operation information including posture information of the virtual photographing apparatus.   The editing apparatus according to claim 4, wherein the display control unit acquires the operation information including a viewpoint position of the virtual imaging device in the panoramic video.   The editing apparatus according to claim 1, wherein the display unit and the operation unit are integrally configured. A cutout unit that cuts out the angle-of-view area of the angle of view according to the operation information from the panoramic moving image at each corresponding elapsed time;
A conversion unit for converting the data format of the cut out moving image,
The editing apparatus according to any one of claims 1 to 5, further comprising: Displaying an extracted moving image obtained by extracting an angle-of-view area having a narrower angle of view from the panoramic moving image to be edited on the display unit;
Acquiring operation information relating to the angle of view during reproduction of the extracted moving image;
Storing the operation information in the storage unit in association with an elapsed time from the start of reproduction of the extracted moving image every time the operation information is acquired;
Editing method including. On the computer,
Displaying an extracted moving image obtained by extracting an angle-of-view area having a narrower angle of view from the panoramic moving image to be edited on the display unit;
Acquiring operation information relating to the angle of view during reproduction of the extracted moving image;
Storing the operation information in the storage unit in association with an elapsed time from the start of reproduction of the extracted moving image every time the operation information is acquired;
An editing program that executes

Images