JP2019140667A - Information processing unit, information processing system, data generation method and program - Google Patents

Abstract

To generate 2D sound data incident to 2D image data, when reproducing a prescribed range of entire-celestial-sphere image as a 2D image.SOLUTION: A communication terminal 5 for displaying the entire-celestial-sphere image 210 acquired by an imaging apparatus 1, and reproducing 3D sound data 230 acquired incident to the entire-celestial-sphere image data 210 acquired by the imaging apparatus 1 accepts crop designation operation in the displayed entire-celestial-sphere image. The communication terminal 5 generates crop image data 610 corresponding to a crop range 6100 from the entire-celestial-sphere image data 210, and generates sound data for reproduction 650 corresponding to the crop range 6100 from the 3D sound data 230.SELECTED DRAWING: Figure 11

Description

  The present invention relates to an information processing apparatus, an information processing system, a data generation method, and a program.

  A special digital camera that obtains a 360 ° panoramic image at one time is provided. In such a digital camera, not only a still image but also an omnidirectional video including stereophonic sound data (three-dimensional sound data) can be taken.

  For example, Patent Document 1 discloses the content of reproducing and outputting three-dimensional audio corresponding to the display range of a video image from the three-dimensional audio data corresponding to the all-round video data captured by the all-round camera.

  In addition, by playing back a two-dimensional moving image that is a part of the omnidirectional moving image in an information processing apparatus such as a smartphone, the user can view the same flat moving image as a moving image shot with a conventional digital camera. .

  When reproducing a predetermined range of the omnidirectional video as a two-dimensional video, an information processing device such as a smartphone is adapted to the display range of the two-dimensional image data so that the user can view the two-dimensional video without a sense of incongruity. It is necessary to output sound data. However, in the conventional method, when a predetermined range of the omnidirectional image is displayed as a two-dimensional image, there is a problem that the two-dimensional sound data accompanying the two-dimensional image data cannot be generated.

  An information processing apparatus according to the present invention is an information processing apparatus that displays an omnidirectional image acquired by an imaging device and reproduces three-dimensional sound data acquired accompanying the omnidirectional image, Receiving means for accepting designation of a predetermined display direction in the displayed omnidirectional image; and generating two-dimensional image data corresponding to the designated display direction from the data of the omnidirectional image, and the tertiary Generating means for generating two-dimensional sound data corresponding to the designated display direction from the original sound data.

  According to the present invention, when a predetermined region of the omnidirectional image is displayed as a two-dimensional image, two-dimensional sound data accompanying the two-dimensional image data can be generated.

(A) is a left side view of the photographing apparatus, (b) is a rear view of the photographing apparatus, (c) is a plan view of the photographing apparatus, and (d) is a bottom view of the photographing apparatus. It is a usage image figure of an imaging device. (A) is a hemisphere image (front) photographed by the photographing apparatus, (b) is a hemispheric image photographed by the photographing apparatus (rear), and (c) is a diagram showing an image represented by equirectangular projection. is there. (A) is the conceptual diagram which showed the state which covered the sphere with an equirectangular projection image, (b) is the figure which showed the omnidirectional image. It is the figure which showed the position of the virtual camera and predetermined area at the time of making a omnidirectional image into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, (b) is a figure which shows the state in which the image of the predetermined area is displayed on the display of a communication terminal. FIG. 6 is a diagram illustrating a relationship between predetermined area information and an image of a predetermined area T. It is a figure which shows an example of the system configuration | structure of the information processing system which concerns on 1st Embodiment. It is a figure which shows an example of the hardware constitutions of the imaging device which concerns on 1st Embodiment. It is a figure which shows an example of the hardware constitutions of the communication terminal which concerns on 1st Embodiment. It is a figure which shows an example of a function structure of the information processing system which concerns on 1st Embodiment. It is a figure which shows an example of the imaging | photography data which concern on 1st Embodiment. It is a figure which shows an example of the detailed functional structure of the image and sound process part which concerns on 1st Embodiment. It is a figure which shows an example of the data for reproduction | regeneration based on 1st Embodiment. 6 is a flowchart illustrating an example of shooting data recording processing in the shooting apparatus according to the first embodiment. It is a flowchart which shows an example of the process from the collection of sound data to reproduction | regeneration in the case of employ | adopting ambisonics. It is a figure explaining the coordinate axis of three-dimensional sound data. It is a sequence diagram which shows an example of the production | generation process of the data for reproduction | regeneration in the information processing system which concerns on 1st Embodiment. It is the schematic for demonstrating the display process of the omnidirectional image data in the communication terminal which concerns on 1st Embodiment. It is the schematic for demonstrating the crop process with respect to the display image in the communication terminal which concerns on 1st Embodiment. It is the schematic for demonstrating the calculation process of the position of the crop image in the omnidirectional image in the communication terminal which concerns on 1st Embodiment. It is a figure which shows an example of the coordinate information of the crop image in 1st Embodiment. It is the schematic for demonstrating an example of the sound source direction of the sound data for reproduction | regeneration which concerns on 1st Embodiment. It is the schematic which shows an example of the production | generation method of the sound data for reproduction | regeneration in 1st Embodiment. It is a figure which shows an example of the parameter for producing | generating the sound data for reproduction | regeneration in 1st Embodiment. It is a figure which shows an example of the system configuration | structure of the information processing system which concerns on 2nd Embodiment. It is a figure which shows an example of the hardware constitutions of the image processing server which concerns on 2nd Embodiment. It is a figure which shows an example of a function structure of the information processing system which concerns on 2nd Embodiment. It is a figure which shows an example of the detailed functional structure of the image and sound process part which concerns on 2nd Embodiment. It is a sequence diagram which shows an example of the production | generation process of the data for reproduction | regeneration in the information processing system which concerns on 2nd Embodiment. It is a figure which shows an example of the imaging | photography data which concern on the modification 1 of 1st Embodiment. It is a figure which shows an example of the data for reproduction | regeneration which concerns on the modification 1 of 1st Embodiment.

  Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted.

● Summary of embodiment ●
Hereinafter, an outline of the embodiment will be described. First, a method for generating an omnidirectional image will be described with reference to FIGS.

  First, the external appearance of the photographing apparatus 1 will be described with reference to FIG. The photographing apparatus 1 is a digital camera for obtaining a photographed image that is the basis of a panoramic image of a celestial sphere (360 °). 1A is a left side view of the photographing apparatus, FIG. 1B is a rear view of the photographing apparatus, FIG. 1C is a plan view of the photographing apparatus, and FIG. It is a bottom view of an apparatus.

  As shown in FIGS. 1 (a), 1 (b), 1 (c), and 1 (d), a fish-eye lens 102a on the front side (front side) and the back side are arranged on the upper part of the photographing apparatus 1. A fish-eye lens 102b is provided on the side (rear side). Inside the photographing apparatus 1, imaging elements (image sensors) 103a and 103b, which will be described later, are provided. The photographing apparatus 1 can obtain a hemispherical image (angle of view of 180 ° or more) by photographing a subject or a landscape through the lenses 102a and 102b, respectively. A shutter button 115 a is provided on the surface opposite to the front side of the photographing apparatus 1. Further, a power button 115b, a Wi-Fi (Wireless Fidelity (registered trademark)) button 115c, and a shooting mode switching button 115d are provided on the side surface of the photographing apparatus 1. Each time the power button 115b and the Wi-Fi button 115c are pressed, they are switched on and off. The shooting mode switching button 115d switches between a still image shooting mode and a moving image shooting mode each time the button is pressed. The shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are part of the operation unit 115. The operation unit 115 is not limited to these buttons.

  A tripod screw hole 151 for attaching the photographing device 1 to a camera tripod is provided in the center of the bottom 150 of the photographing device 1. A micro USB (Universal Serial Bus (registered trademark)) terminal 152 is provided on the left end side of the bottom 150. An HDMI (High-Definition Multimedia Interface (registered trademark)) terminal 153 is provided on the right end side of the bottom 150.

  Next, the usage situation of the imaging device 1 will be described with reference to FIG. FIG. 2 is a usage image diagram of the photographing apparatus. As shown in FIG. 2, the photographing apparatus 1 is used, for example, for photographing a subject around the user by holding it in his hand. In this case, two hemispherical images can be obtained by imaging the subject around the user by the imaging device 103a and the imaging device 103b shown in FIG.

  Next, an outline of processing until an equirectangular projection image EC and an omnidirectional image CE are created from an image photographed by the photographing apparatus 1 will be described with reference to FIGS. 3 and 4. 3A is a hemispheric image (front side) photographed by the photographing apparatus, FIG. 3B is a hemispheric image photographed by the photographing apparatus (rear side), and FIG. 3C is represented by equirectangular cylindrical projection. FIG. 6 is a diagram showing a captured image (hereinafter referred to as an “equal-distance cylindrical projection image”). FIG. 4A is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing an omnidirectional image.

  As shown in FIG. 3A, an image obtained by the image sensor 103a is a hemispherical image (front side) curved by a lens 102a described later. Further, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by a lens 102b described later. Then, the photographing apparatus 1 combines the hemispherical image (front side) and the hemispherical image inverted by 180 degrees (rear side) to create an equirectangular projection image EC as shown in FIG.

  Then, the photographing apparatus 1 uses OpenGLES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image so as to cover the spherical surface as shown in FIG. An omnidirectional image CE as shown in FIG. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGLES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The omnidirectional image CE may be a still image or a moving image.

  As described above, since the omnidirectional image CE is an image that is pasted so as to cover the spherical surface, it is uncomfortable when viewed by a human. Therefore, the photographing apparatus 1 displays a predetermined area (hereinafter referred to as “predetermined area image”) of the whole celestial sphere image CE on a predetermined display as a flat image with less curvature, thereby giving a sense of incongruity to humans. Can not display. This will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. FIG. 6A shows the omnidirectional image CE shown in FIG. 4 as a three-dimensional solid sphere CS. When the omnidirectional image CE generated in this way is a solid sphere CS, the virtual camera IC is located inside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is a shooting area of the virtual camera IC, and is specified by predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. The

  Then, the predetermined area image Q shown in FIG. 6A is displayed as an image of the photographing area of the virtual camera IC on a predetermined display as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined (default) predetermined area information. Hereinafter, description will be made using the shooting direction (ea, aa) and the angle of view (α) of the virtual camera IC.

  The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined region information and the relationship between the images of the predetermined region T. “Ea” represents an elevation angle, “aa” represents an azimuth angle, and “α” represents an angle of view. That is, the attitude of the virtual camera IC is changed so that the gazing point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP of the predetermined area T that is the shooting area of the virtual camera IC. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is a distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function represented by the following (formula 1) is generally established.

● First embodiment ●
Subsequently, a first embodiment of the present invention will be described with reference to FIGS. In the following description, an example in which a moving image is shot by the shooting device 1 will be described, but the same applies to a case where a still image is shot by the shooting device 1. In this case, sound data of about several seconds is acquired with respect to the still image photographed by the photographing apparatus 1.

System Configuration First, an outline of the configuration of the information processing system according to the present embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of a system configuration of the information processing system according to the first embodiment. The information processing system shown in FIG. 8 generates two-dimensional sound data associated with two-dimensional image data when a predetermined region of the omnidirectional image photographed by the photographing device 1 is displayed on the communication terminal 5 as a two-dimensional image. It is a system that can do.

  The information processing system according to the first embodiment includes an imaging device 1 and a communication terminal 5. As described above, the photographing apparatus 1 is a special digital camera for photographing a subject, a landscape, and the like to obtain two hemispherical images that are the basis of an omnidirectional (panoramic) image.

  The communication terminal 5 performs wireless communication with the photographing apparatus 1 using a short-range wireless communication technology such as Wi-Fi (Wireless Fidelity (registered trademark)), Bluetooth (registered trademark), NFC (Near field communication), or the like. It is a smartphone that can. In addition, the communication terminal 5 can display an image (still image or moving image) acquired from the imaging device 1 on a display 517 (described later) provided in the own device. The communication terminal 5 is an example of an information processing device.

  Note that the communication terminal 5 may communicate with the photographing apparatus 1 through a wired cable without using the short-range wireless communication technology. The communication terminal 5 may be not only a smartphone but also a tablet PC (Personal Computer), a notebook PC, a desktop PC, or the like.

Hardware Configuration Next, the hardware configuration of the imaging device 1 and the communication terminal 5 in the first embodiment will be described with reference to FIGS. 9 and 10. The hardware configurations shown in FIGS. 9 and 10 may have the same configuration in each embodiment, and components may be added or deleted as necessary.

O Hardware configuration of the imaging device O First, the hardware configuration of the imaging device 1 will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of a hardware configuration of the photographing apparatus according to the first embodiment. Hereinafter, the imaging device 1 is an omnidirectional (omnidirectional) imaging device using two imaging elements, but the number of imaging elements may be two or more. The photographing device 1 does not necessarily have to be a dedicated device for omnidirectional photographing. By attaching a retrofit omnidirectional imaging unit to a normal digital camera, a smartphone, or the like, substantially the same function as the photographing device 1 is obtained. You may make it have.

  The imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108a to a microphone 108d, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random). It includes an access memory (113), a dynamic random access memory (DRAM) 114, an operation unit 115, a network I / F 116, a communication I / F 117, an antenna 117a, and an electronic compass 118.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by the fish-eye lenses 102a and 102b into electric signal image data and output the image data. A timing generation circuit for generating a horizontal or vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The image sensors 103a and 103b are respectively connected to the image processing unit 104 by a parallel I / F bus. The imaging elements 103a and 103b are connected to the imaging control unit 105 via a serial I / F bus (I2C bus or the like). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. Furthermore, the bus 110 is connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication I / F 117, electronic compass 118, and the like.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus. Then, the image processing unit 104 performs predetermined processing on the respective image data, and then combines these image data to obtain equirectangular cylindrical projection image data as shown in FIG. Create

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data and the like of the register groups of the imaging elements 103a and 103b and send them to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. The photographing apparatus 1 may have a preview display function by a display (for example, the display 517 of the communication terminal 5) and a function corresponding to the display of a still image or a moving image. In the case of a moving image, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / minute).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the photographing apparatus 1 is not provided with a display, but may be provided with a display unit.

  Each of the microphones 108a to 108d collects sound from the surrounding environment of the photographing apparatus 1, and converts the collected sound into sound (signal) data. The sound processing unit 109 takes in sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data. Note that the microphone 108 includes four microphones 108a to 108d having a predetermined arrangement configuration, and is preferably an ambisonics microphone. In the present embodiment, the microphones 108 constitute sound collecting means for collecting sound from the surrounding environment. FIG. 2 shows only the case where the microphone 108 is built in the photographing apparatus 1, but the microphone 108 may be externally attached to the photographing apparatus 1. Further, the number of microphones 108 is not limited to four.

  The CPU 111 controls the overall operation of the photographing apparatus 1 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111 and data being processed. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed equirectangular projection image data.

  The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user operates the operation unit 115 to input various shooting modes, shooting conditions, and the like.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. Data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or an external terminal (device) such as the communication terminal 5 via the network I / F 116 as necessary. ).

  The communication I / F 117 is connected to an external terminal such as the communication terminal 5 by a short-range wireless communication technology such as Wi-Fi (registered trademark), NFC, or Bluetooth (registered trademark) via an antenna 117 a provided in the photographing apparatus 1. Communicate with (device). The communication I / F 117 can transmit data of the equirectangular projection image to an external terminal (apparatus) such as the communication terminal 5.

  The electronic compass 118 calculates the azimuth of the photographing apparatus 1 from the earth's magnetism and outputs azimuth information. The azimuth information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. Note that the related information includes each data of the shooting date and time of the image and the data capacity of the image data.

O Hardware configuration of communication terminal O Next, the hardware configuration of the communication terminal 5 will be described with reference to FIG. FIG. 10 is a diagram illustrating an example of a hardware configuration of the communication terminal according to the first embodiment. The communication terminal 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM (Electrically Erasable Programmable Read-Only Memory) 504, a CMOS sensor 505, an image sensor I / F 513a, an acceleration / direction sensor 506, a media I / F 508, and a GPS receiver 509. ing.

  The CPU 501 controls the overall operation of the communication terminal 5. The ROM 502 stores a program used for driving the CPU 501 such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as a communication terminal program under the control of the CPU 501.

  The CMOS sensor 505 captures a subject (mainly a self-portrait) under the control of the CPU 501 and obtains image data. The image sensor I / F 513 a is a circuit that controls driving of the CMOS sensor 512. The acceleration / direction sensor 506 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. A media I / F 508 controls reading or writing (storage) of data with respect to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

  The communication terminal 5 includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I / F 513b, a microphone 514, a speaker 515, a sound input / output I / F 516, a display 517, an external device connection I / F 518, A distance communication circuit 519, an antenna 519a of the near field communication circuit 519, and a touch panel 521 are provided.

  The long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network 100 described later. The CMOS sensor 512 is a kind of built-in imaging unit that captures an image of a subject under the control of the CPU 501 and obtains image data. The image sensor I / F 513b is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 516 is a circuit that processes input / output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501.

  A display 517 is a kind of display means such as a liquid crystal display or an organic EL display that displays an image of a subject, various icons, and the like. The external device connection I / F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 521 is a kind of input means for operating the communication terminal 5 when the user presses the display 517.

  The communication terminal 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

  Note that any recording medium such as an HD (Hard Disk) or a CD-ROM in which the above programs are stored can be provided as a program product domestically or abroad.

Functional Configuration Next, the functional configuration of the device and the terminal according to the first embodiment will be described. FIG. 11 is a diagram illustrating an example of a functional configuration of the information processing system according to the first embodiment.

O Functional Configuration of Imaging Device O First, the functional configuration of the imaging device 1 will be described with reference to FIG. The functions realized by the photographing apparatus 1 include a communication unit 11, a reception unit 12, an imaging unit 13, a sound collection unit 14, a sensor information acquisition unit 15, a determination unit 16, an image / sound processing unit 17, a storage / reading unit 18, and A storage unit 1000 is included.

  The communication unit 11 has a function of communicating with an external device such as the communication terminal 5 using a short-range wireless communication technology such as Wi-Fi (registered trademark). The communication unit 11 is mainly realized by the processing of the CPU 111 illustrated in FIG. 9, the communication I / F 117, and the antenna 117a.

  The accepting unit 12 is a function that accepts an operation input from a user. The receiving unit 12 is mainly realized by the processing of the operation unit 115 and the CPU 111 illustrated in FIG.

  The imaging unit 13 has a function of capturing a subject, a landscape image, and the like to obtain captured image data. As shown in FIGS. 3A and 3B, the captured image data is two hemispherical image data that is the basis of the omnidirectional image data. The captured image data may be not only a still image but also a moving image. The imaging unit 13 is mainly realized by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 illustrated in FIG.

  The sound collection unit 14 has a function of collecting sounds around the photographing apparatus 1. The sound collecting unit 14 is mainly realized by the processing of the microphone 108 and the sound processing unit 109 and the CPU 111 shown in FIG. The sound collection unit 14 acquires sound collection data collected using the plurality of microphones 108 (108a to 108d) illustrated in FIG.

  The sensor information acquisition unit 15 has a function of acquiring sensor detection result information such as the direction of each azimuth (azimuth angle, magnetic north) from a sensor such as the electronic compass 118. The sensor detection result information such as the direction of each azimuth measured indicates the posture of the photographing apparatus 1 at a predetermined time point. The sensor information acquisition unit 15 is mainly realized by the processing of the electronic compass 118 and the CPU 111 shown in FIG.

  The determination unit 16 is realized by the processing of the CPU 111 shown in FIG.

  The image / sound processing unit 17 has a function of performing various processes on the captured image data obtained by the imaging unit 13 or the sound data obtained by the sound collecting unit 14. The image / sound processing unit 17, for example, equirectangular projection image data (based on the two hemispherical image data (FIGS. 3A and 3B) obtained by the two image sensors 103 a and 103 b, respectively). FIG. 3C is created. Further, the image / sound processing unit 17 creates the three-dimensional sound data 230 based on the sound data obtained by the sound collection unit 14, for example. The image / sound processing unit 17 is mainly realized by a command from the CPU 111 shown in FIG.

  The storage / reading unit 18 has a function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000. The storage / reading unit 18 is realized mainly by the processing of the CPU 111 shown in FIG. The storage unit 1000 is mainly realized by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. In addition, the storage unit 1000 stores a shooting data file 200.

O Captured data file Here, the detail of the data memorize | stored in the memory | storage part 1000 is demonstrated. FIG. 12 is a diagram illustrating an example of the data structure of the shooting data file according to the first embodiment. The shooting data file 200 is a file that stores image data and sound data acquired by shooting a subject by the shooting apparatus 1. 12 includes a channel of omnidirectional image data 210, a channel of inclination angle data 250, a channel of sound data 220, and a channel of three-dimensional sound data 230. The photographic data file 200 stores each data for each channel as a data transmission path inside the photographic device 1, so that the omnidirectional image data 210, the sound data 220, the three-dimensional sound data 230, and the tilt angle data 250 Can be stored in a single file.

  The omnidirectional image data 210 is recorded in the MPEG system, which is one of the moving image formats, and is encoded in a unit called GOP (Group Of Picture). Here, GOP refers to a unit of a group of frames configured to include at least one reference frame (I picture in MPEG). The omnidirectional image data 210 is, for example, MPEG4 AVC / H. H.264 media data.

  The sound data 220, the three-dimensional sound data 230, and the inclination angle data 250 are recorded by being divided by a time interval corresponding to the GOP. The sound data 220, the three-dimensional sound data 230, and the tilt angle data 250 are related so that the recorded times of the sound data 220, the three-dimensional sound data 230, and the tilt angle data 250 coincide with each other on the basis of the recording start. . Therefore, the inclination angle data 250, the sound data 220, and the three-dimensional sound data 230 can match the elapsed time from the start of recording.

  The sound data 220 is sound data generated based on the sound signal collected by the microphone 108. The sound data 220 is recorded for each channel of the plurality of microphones 108a to 108d. The sound data 220 is generated based on, for example, an ambisonic A format described later.

  The three-dimensional sound data 230 is stereophonic data generated based on the Ambisonics B format described later. The three-dimensional sound data 230 is generated using the sound data 220 stored for each channel of the plurality of microphones 108a to 108d. Details of the method of generating the three-dimensional sound data 230 will be described later.

  The sound data 220 and the three-dimensional sound data 230 are recorded in an uncompressed audio format such as AAC-LC (AAC Low Complexity) and Linear PCM (Pulse Code Modulation). Note that the sound data 220 and the three-dimensional sound data 230 may be recorded as a compressed audio format such as MP3 (MPEG Layer 3). The captured data file 200 only needs to include at least one of the sound data 220 and the three-dimensional sound data 230. This is because the three-dimensional sound data 230 is generated from the sound data 220 as described above. For example, when only the sound data 220 is recorded, the three-dimensional sound data 230 may be generated from the sound data 220 as necessary, which contributes to a reduction in the data amount. Further, when only the three-dimensional sound data 230 is recorded, it is not necessary to generate the sound data 220, which further contributes to the reduction of the data amount.

  The tilt angle data 250 is metadata including posture information of the photographing apparatus 1. The posture information of the photographing device 1 is data indicating the posture of the photographing device 1 at a predetermined time. The tilt angle data 250 describes the direction of the photographing apparatus 1 in the coordinate system of the omnidirectional image as three-axis data indicated by pitch, roll, and Yaw.

  Note that the omnidirectional image data 210, the sound data 220, the three-dimensional sound data 230, and the tilt angle data 250 are stored as a single file 200, but are not particularly limited and are separate files. May be stored as Further, the shooting data file 200 may have a configuration in which the omnidirectional image data 210, the sound data 220, the three-dimensional sound data 230, and the tilt angle data 250 are associated with each other in units of frames.

O Functional configuration of communication terminal O Next, the functional configuration of the communication terminal 5 will be described with reference to FIG. Functions realized by the communication terminal 5 include a communication unit 51, a reception unit 52, a display control unit 53, a sound reproduction unit 54, a determination unit 55, an image / sound processing unit 56, a storage / reading unit 57, and a storage unit 5000. . The communication terminal 5 is installed with a dedicated application program for reproducing the omnidirectional video. The communication terminal 5 realizes the data generation method according to the present invention, for example, by the CPU 501 executing the installed application program.

  The communication unit 51 has a function of performing communication with an external device such as the photographing device 1 using a short-range wireless communication technology such as Wi-Fi (registered trademark). The communication unit 51 is mainly realized by the processing of the CPU 501 shown in FIG. 10, the short-range communication circuit 519, and the antenna 519a.

  The accepting unit 52 is a function that accepts various selections or inputs from the user. The accepting unit 52 is mainly realized by processing of the touch panel 521 and the CPU 501. Note that the touch panel 521 may be shared with the display 517. Moreover, the reception part 52 may be implement | achieved by input means other than a touch panel. The reception unit 52 is an example of a reception unit.

  The display control unit 53 is a function that performs control for displaying (reproducing) various images on the display 517 of the communication terminal 5. The image display method is not particularly limited, and the omnidirectional image may be displayed as it is, or an image range corresponding to a predetermined angle of view of the omnidirectional image may be cut out and displayed. For example, the display control unit 53 reproduces the crop image data 610 generated by the reproduction data generation unit 63. The display control unit 53 is mainly realized by the processing of the CPU 501 shown in FIG. The display control unit 53 is an example of a reproducing unit that reproduces two-dimensional image data. The display control unit 53 is an example of a playback unit that plays back a two-dimensional video.

  The sound reproducing unit 54 has a function of performing control for reproducing sound data from the speaker 515 of the communication terminal 5. The sound reproduction unit 54 reproduces the reproduction sound data 650 generated by the reproduction data generation unit 63. The sound reproducing unit 54 is realized mainly by the processing of the CPU 501 shown in FIG. The sound reproducing unit 54 is an example of a reproducing unit that reproduces two-dimensional sound data. The sound reproducing unit 54 is an example of a reproducing unit that reproduces a two-dimensional moving image.

  The determination unit 55 is realized mainly by the processing of the CPU 501 illustrated in FIG. 10 and performs various determinations.

  The image / sound processing unit 56 performs various processes for reproducing the omnidirectional image data 210 and the sound data (three-dimensional sound data 230 or reproduction sound data 650 described later) acquired from the photographing apparatus 1 on the communication terminal 5. It is a function to perform. The image / sound processing unit 56 is mainly realized by a command from the CPU 501 shown in FIG.

  The storage / reading unit 57 has a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000. The storage / reading unit 57 is mainly realized by the processing of the CPU 501 shown in FIG. The storage unit 5000 is mainly realized by the ROM 502, the EEPROM 504, and the recording medium 507 shown in FIG. The storage unit 5000 stores a reproduction data file 600 described later. The storage unit 500 is an example of a storage unit.

Detailed Functional Configuration of Image / Sound Processing Unit Here, the functional configuration of the image / sound processing unit 56 will be described in detail with reference to FIG. FIG. 13 is a diagram illustrating an example of a detailed functional configuration of the image / sound processing unit according to the first embodiment. The image / sound processing unit 56 includes an image data management unit 61, a sound data management unit 62, and a reproduction data generation unit 63.

  The image data management unit 61 has a function of managing, controlling, and holding the display range of the omnidirectional image data 210. The image data management unit 61, for example, the position or display direction of the crop image data 610 (an example of two-dimensional image data) that is a predetermined area included in the omnidirectional image data 210 designated by the user of the communication terminal 5. Is identified. The position or display direction of the crop image data 610 is coordinate information of the crop image data 610, for example. The image data management unit 61 is realized mainly by the processing of the CPU 501 shown in FIG. The image data management unit 61 is an example of a display direction specifying unit.

  The sound data management unit 62 has a function of managing, controlling, and holding reproduction sound data 650 corresponding to the crop image data 610. The sound data management unit 62 specifies the sound source direction in the three-dimensional sound data 230 based on the position or display direction of the crop image data 610 specified by the image data management unit 61. The sound data management unit 62 is realized mainly by the processing of the CPU 501 shown in FIG. The sound data management unit 62 is an example of a sound source direction specifying unit.

  The reproduction data generation unit 63 has a function of generating a reproduction data file 600 using the photographing data file 200 received from the photographing apparatus 1. For example, the reproduction data generation unit 63 generates crop image data 610 from the omnidirectional image data 210 and also generates reproduction sound data 650 from the three-dimensional sound data 230. The reproduction data generation unit 63 is mainly realized by the processing of the CPU 501 shown in FIG. The reproduction data generation unit 63 is an example of a generation unit.

Here, the details of the data stored in the storage unit 5000 will be described. FIG. 14 is a diagram illustrating an example of reproduction data according to the first embodiment. The reproduction data file 600 shown in FIG. 14 is media data for reproducing the omnidirectional video as a two-dimensional video in the communication terminal 5.

  The reproduction data file 600 stores the crop image data 610 and the reproduction sound data 650 in association with each other for each frame. The crop image data 610 is image data cropped by the user from the omnidirectional image displayed on the communication terminal 5. The cropping process is a process of cutting out (crop) a predetermined area that is a part of the omnidirectional image data. The predetermined area is a partial area of the omnidirectional image data designated by a specific input operation by the user. The crop image data 610 is an example of two-dimensional image data.

  The reproduction sound data 650 is sound data corresponding to a position in the omnidirectional image data of the crop image data 610. The reproduction sound data 650 includes sound data of two channels, left and right (L and R). The reproduction sound data 650 is stereo sound data for stereophonic reproduction generated by synthesizing sound data (L and R) of these two channels. The reproduction sound data 650 is an example of two-dimensional sound data.

Shooting Data Recording Process FIG. 15 is a flowchart illustrating an example of shooting data recording process in the shooting apparatus according to the first embodiment. The process illustrated in FIG. 15 is started in response to a specific operation for giving an instruction to start recording, such as pressing the operation unit 115 provided in the housing of the photographing apparatus 1.

  In step S101, the imaging unit 13 of the imaging apparatus 1 acquires image data captured using the imaging elements 103a and 103b. The image data acquired by the imaging unit 13 is, for example, the two hemispherical image data illustrated in FIGS.

  In step S102, the image / sound processing unit 17 of the photographing apparatus 1 performs image processing on the image data acquired in step S101. For example, the image / sound processing unit 17 generates equirectangular projection image data (FIG. 3C) based on the acquired two hemispherical image data (FIGS. 3A and 3B). Note that the photographing apparatus 1 performs image data acquisition and image processing in units of frame sets.

  When the processing illustrated in FIG. 15 is started, the imaging device 1 executes the processing of step S103 and step S104 in parallel with the processing of step S101 and step S102.

  In step S <b> 103, the sound collection unit 14 of the photographing apparatus 1 acquires sound collection data collected for each microphone 108 from the microphones 108 a to 108 d via the sound processing unit 109. In step S104, the image / sound processing unit 17 of the photographing apparatus 1 performs sound processing on the collected sound data acquired in step S103. For example, the image / sound processing unit 17 converts the collected sound data collected for each microphone 108 into sound data 220 corresponding to the A-format of Ambisonics using the sound processing unit 109. Here, it is assumed that the photographing apparatus 1 performs acquisition and sound processing of sound collection data in a time interval corresponding to a frame set unit.

  In step S <b> 105, the sensor information acquisition unit 15 of the photographing apparatus 1 acquires sensor detection result information from the electronic compass 118 when recording image data and sound data in steps S <b> 101 and S <b> 103. In step S <b> 106, the imaging apparatus 1 calculates the tilt angle and orientation of the imaging apparatus 1 at the time of recording based on sensor detection result information acquired using the electronic compass 118.

  In step S <b> 107, the image / sound processing unit 17 of the photographing apparatus 1 generates three-dimensional sound data 230 using the sound data 220. Details of the method of generating the three-dimensional sound data 230 will be described later.

  In step S <b> 108, the storage / reading unit 18 of the photographing apparatus 1 associates the omnidirectional image data 210, the three-dimensional sound data 230, and the tilt angle data 250 with the storage unit 1000 and stores them as the photographing data file 200. The photographic data file 200 records omnidirectional image data 210, three-dimensional sound data 230, and tilt angle data 250 for a set of frames in association with each other. Note that, as shown in FIG. 12, the shooting data file 200 may store sound data 220.

  In step S109, the imaging apparatus 1 determines whether an instruction to end recording has been received. If it is determined that the recording end instruction has not yet been received (NO), the imaging apparatus 1 repeats the processing of step S101 and step S103, and proceeds with the processing for the next frame set. On the other hand, when it is determined that the recording end instruction has been received (YES), the photographing apparatus 1 closes the file and ends the process.

3D Sound Data Generation Processing The flow from sound collection to reproduction when ambisonics is adopted as the 3D sound data 230 will be described below with reference to FIGS. 16 and 17. FIG. 16A is a flowchart illustrating an example of processing from sound collection to reproduction in the first embodiment.

  In step S151a, the sound collection unit 14 of the photographing apparatus 1 acquires sound collection data for each microphone 108 illustrated in FIG. In step S152a, the image / sound processing unit 17 of the photographing apparatus 1 converts the collected sound data for each collected channel into ambisonics A format sound data 220 (LF, LB, RF, RB). To do. The converted sound data 220 (LF, LB, RF, RB) is recorded in the photographing data file 200 in association with the inclination angle data 250. Note that the processing in step S152a is the same as the processing in step S104 shown in FIG.

  In step S153a, the image / sound processing unit 17 of the photographing apparatus 1 performs zenith correction on the sound data 220 (LF, LB, RF, RB). In step S154a, the image / sound processing unit 17 of the photographing apparatus 1 encodes the zenith corrected sound data (A-format LF ′, LB ′, RF ′, RB ′) with an ambisonic encoder. , Three-dimensional sound data 230 (W, X, Y, Z) of B format of Ambisonics is generated. The encoding can be expressed by Equation 2, for example. The photographing apparatus 1 collects sound using four directional microphones arranged at the apex of a regular tetrahedron, and the omnidirectional signal W and the bi-directional signals X, Y, Z are obtained from the collected sound data. Is generated.

  As a result of the conversion process to the B format, the omnidirectional signal W and the bidirectional signals X, Y, and Z are handled as collected by a virtual omnidirectional microphone and a bidirectional microphone.

  FIG. 17A is a diagram for explaining the definition of axes in the photographing apparatus 1. When the user holds the photographing apparatus 1 as shown in FIG. 17A, the vertical direction of the photographing apparatus 1 is associated with the Z axis, and the left and right direction of the photographing apparatus 1 is associated with the X axis. In addition, the front-rear direction of the photographing apparatus 1 is associated with the Y axis. Moreover, FIG.17 (b)-FIG.17 (e) are the figures explaining the sound collection directivity characteristic in a stereophonic sound. The B-format W channel corresponds to a sound collection signal from the omnidirectional microphone 108 as shown in FIG. Each of the B-format X, Y, and Z channels corresponds to a sound collection signal with a bidirectional microphone as shown in FIGS. 17 (c) to 17 (e). As shown in Expression 2, the three-dimensional sound data 230 is generated by a simple calculation using sound collection data collected for each microphone 108.

  In step S155a, the image / sound processing unit 17 of the photographing apparatus 1 decodes the generated three-dimensional sound data 230 (W, X, Y, Z) by an ambisonic decoder. The image / sound processing unit 17 generates a speaker drive signal corresponding to the configuration of the speaker 515 of the communication terminal 5 by the decoding process. The imaging device 1 transmits the decoded three-dimensional sound data 230 (generated speaker drive signal) to the communication terminal 5. Note that the decoding process of the three-dimensional sound data 230 may be performed by the image / sound processing unit 56 of the communication terminal 5.

  In step S156a, the sound reproduction unit 54 of the communication terminal 5 reproduces the decoded three-dimensional sound data 230 (sounds the generated speaker drive signal). Thereby, the sound field including directionality is reproduced.

  FIG. 16B is a flowchart illustrating an example of processing from sound collection to reproduction of sound data according to another embodiment. In another embodiment shown in FIG. 16B, the imaging device 1 encodes the sound data 220 acquired from the plurality of microphones 108 to generate the three-dimensional sound data 230 once. The photographing apparatus 1 is subjected to zenith correction on the three-dimensional sound data 230.

  In step S151b, the sound collection unit 14 of the photographing apparatus 1 acquires sound collection data for each microphone 108 illustrated in FIG. In step S152b, the image / sound processing unit 17 of the photographing apparatus 1 converts the collected sound data for each collected channel into ambisonic A format sound data 220 (LF, LB, RF, RB). To do. Note that the processing in step S152b is the same as the processing in step S104 shown in FIG.

  In step S153b, the image / sound processing unit 17 of the photographing apparatus 1 encodes the sound data 220 (LF, LB, RF, RB) to generate three-dimensional sound data 230 (B-format W, X, Y, Z). ) Is generated. The generated three-dimensional sound data 230 is recorded in the photographing data file 200 in association with the tilt angle data 250.

  In step S154b, the image / sound processing unit 17 of the photographing apparatus 1 performs zenith correction on the three-dimensional sound data 230 (W, X, Y, Z). As shown in FIG. 17A, the zenith correction corresponding to the rotation by γ on the horizontal plane can be realized by, for example, projective transformation represented by the following Expression 3.

  In step S155b, the image / sound processing unit 17 of the photographing apparatus 1 decodes the zenith corrected three-dimensional sound data (B-format W ′, X ′, Y ′, Z ′) by an ambisonic decoder. The image / sound processing unit 17 generates a speaker drive signal corresponding to the configuration of the speaker 515 of the communication terminal 5 of the communication terminal 5 by this decoding process. The imaging device 1 transmits the three-dimensional sound data 230 (generated speaker drive signal) subjected to zenith correction and decoding processing to the communication terminal 5. Note that the decoding process of the three-dimensional sound data (W ′, X ′, Y ′, Z ′) corrected for zenith may be performed by the image / sound processing unit 56 of the communication terminal 5.

  In step S156b, the sound reproduction unit 54 of the communication terminal 5 reproduces the three-dimensional sound data 230 (sounds the generated speaker drive signal). Thereby, the sound field including directionality is reproduced.

  As described above, the photographing apparatus 1 records the tilt angle data 250 at the corresponding time in association with the sound data 220 or the three-dimensional sound data 230 at the predetermined time. The imaging apparatus 1 can perform zenith correction according to the tilt angle data 250 on the sound data 220 or the three-dimensional sound data 230. Therefore, the user can shoot the omnidirectional video while moving the imaging device 1 without worrying about the state of the microphone 108.

  16A and 16B have been described on the assumption that the communication terminal 5 includes a plurality of loudspeakers. However, the 3D sound data 230 may be viewed using headphones. In that case, the image / sound processor 17 of the photographing apparatus 1 once decodes the signal for a loudspeaker having a predetermined configuration, and then convolves a predetermined head-related transfer function (HRTF). And adding them together, it outputs to the headphones connected to the communication terminal 5 as a binaural signal.

  FIGS. 16A and 16B show sound data 220 (A-format LF, LB, RF, RB) and three-dimensional sound data 230 (B-format W, X, Y, Z). Is described as being recorded in association with the tilt angle data 250. However, the format of the recorded sound data 220 and the three-dimensional sound data 230 is not limited to this.

  Further, FIG. 16A and FIG. 16B have been described on the assumption that processing related to sound data (conversion to sound data 220, zenith correction, encoding, decoding) is performed by the photographing apparatus 1. However, a configuration in which at least a part of the processing related to the sound data is performed by the communication terminal 5 may be used. In this case, for example, the communication terminal 5 may be configured to perform zenith correction according to the inclination angle data 250 on the sound data 220 acquired from the imaging device 1, for example, when viewing the omnidirectional video.

Processing or Operation of the First Embodiment Next, a method for generating and playing back a two-dimensional moving image according to the first embodiment will be described with reference to FIGS. FIG. 18 is a sequence diagram illustrating an example of reproduction data generation processing in the information processing system according to the first embodiment.

  First, in step S <b> 201, the communication unit 11 of the photographing apparatus 1 transmits the photographing data file 200 generated by the image / sound processing unit 17 to the communication terminal 5. The image data file 200 is generated by the image / sound processor 17 by the method described with reference to FIGS. The photographic data file 200 is omnidirectional video data obtained by photographing the subject by the photographic device 1, and as shown in FIG. 12, omnidirectional image data 210, sound data 220, three-dimensional sound data. 230 and tilt angle data 250.

In step S <b> 202, the image / sound processing unit 56 of the communication terminal 5 acquires reference information from the shooting data file 200 received by the communication unit 51. The reference information is metadata for causing the omnidirectional image data 210 included in the captured data file 200 to be displayed as a planar image on the display 517 of the communication terminal 5. The reference information includes, for example, the angle θ _{S in} the display range direction of the image data, the angle of view α, the depth z of the display image, and the like.

FIG. 19A is a diagram for explaining an outline of an image displayed on the communication terminal 5. The communication terminal 5 causes the display 517 to display an image in the visual field range shown in FIG. The visual field range is a shooting area of the virtual camera IC, and is specified by the display range direction and the angle of view α. The visual field range corresponds to the predetermined region T shown in FIG. The display range direction is the shooting direction (reference position of the shooting direction) of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image data 210 as described with reference to FIG. The angle θ _{s in the} display range direction is an angle from the reference position in the shooting direction of the omnidirectional image in the virtual camera IC to the display range direction. In FIG. 19A, the angle θ _s in the display range direction is 0 °.

  In step S <b> 203, the display control unit 53 of the communication terminal 5 displays an image on the display 517. FIG. 19B is an example of a display image displayed on the communication terminal 5. A display screen 6000 shown in FIG. 19B is an image in the visual field range shown in FIG.

  In step S204, the reception unit 52 of the communication terminal 5 receives a crop range designation operation for the display screen 6000 (an example of a reception step). Specifically, the accepting unit 52 accepts selection of a crop image icon displayed at the bottom of the display screen 6000 shown in FIG. When the selection of the crop image icon is received by the reception unit 52, the display control unit 53 displays the crop range 6100 shown in FIG. 20A on the display screen 6000. Then, the accepting unit 52 designates the crop range by accepting a crop range designation operation for the crop range 6100 shown in FIG.

  The crop range designation operation is, for example, a specific input operation such as tap, drag, swipe, pinch-in, and pinch-out for the crop range 6100. The crop range is a predetermined region in the field of view of the omnidirectional image data as shown in FIG. The user of the communication terminal 5 can move the crop range 6100 and enlarge or reduce the crop range 6100 by performing a crop range designation operation on the crop range 6100.

  In step S205, the image data management unit 61 of the communication terminal 5 specifies the display direction of the cropped image in the omnidirectional image. The display direction of the crop image indicates the position of the crop image 6200 that is an image in the specified crop range 6100 in the omnidirectional image.

  Here, the specifying process of the display direction of the cropped image will be described with reference to FIGS. 21 and 22. First, the image data management unit 61 calculates coordinate information of the cropped image 6200 as shown in FIG. The coordinate information of the crop image 6200 is coordinate data on a planar image (XY plane) in the visual field area displayed on the communication terminal 5. The image data management unit 61 calculates the maximum value (Xmax) and minimum value (Xmin) of the X coordinate of the cropped image 6200, and the maximum value (Ymax) and minimum value (Ymin) of the Y coordinate.

Next, the image data management unit 61 calculates the center coordinates C (X, Y) of the crop image 6200 using the calculated coordinate information of the crop image 6200. FIG. 22 is a diagram illustrating an example of parameters of the crop image 6200 illustrated in FIG. The parameters shown in FIG. 22 are parameters for specifying the crop image 6200, and include the reference information of the display screen 6000 acquired in step S201 and the coordinate information of the crop image 6200. A display screen 6000 shown in FIG. 21A has a horizontal field angle α of 120 °, a depth z of 0.5, and a display range direction angle θ _s of 0 °. Further, the coordinate information of the cropped image 6200 shown in FIG. 21A includes X coordinate (Xmax, Xmin) = (0.95, −0.45), Y coordinate (Ymax, Ymin) = (− 0.20, 0.20). Furthermore, the coordinate information C (X, Y) of the center position of the cropped image shown in FIG. 21A is (0.25, 0).

  Then, the image data management unit 61 calculates the angle θ in the display direction of the cropped image using the calculated coordinate information of the cropped image 6200. The crop image display direction angle θ is an angle from the reference position in the shooting direction of the omnidirectional image in the image virtual camera IC to the crop image to the display direction of the crop image.

  FIG. 21B is a diagram for explaining an outline of a method for calculating the angle θ in the display direction of the cropped image. As shown in FIG. 21B, a is the distance from the reference position in the shooting direction of the omnidirectional image in the virtual camera IC on the XZ plane to the center position of the cropped image 6200. z is the depth of the display screen 6000. The display angle θ of the crop image 6200 is calculated using Equation 4.

  In the case of the cropped image 6200 shown in FIG. 21A, since a = 0.25 and z = 0.5, θ is 22.5 °. In this manner, the image data management unit 61 specifies the display direction of the crop image 6200 by calculating the angle θ of the display direction of the crop image 6200. )

  Note that the crop range designation operation in step S204 may be an operation for designating the center position as well as an operation for designating the crop range 6100 as shown in FIG. In other words, a configuration may be used in which a predetermined display direction (position) in the omnidirectional image displayed on the display screen 6000 is designated instead of showing the range in shape. The accepting unit 52 of the communication terminal 5 accepts designation of a predetermined display direction (position) in the omnidirectional image. In this case, the range is designated by a preset shape and size (for example, a size suitable for two-dimensional display such as 4: 3, 16: 9, etc. may be set). . Thereby, the processing of step S204 and step S205 shown in FIG. 18 is integrated, and processing for designating a predetermined display direction (position) in the omnidirectional image by the user is performed.

In step S <b> 206, the sound data management unit 62 of the communication terminal 5 identifies the sound source direction in the three-dimensional sound data 230 based on the identified display direction of the cropped image 6200. Specifically, the sound data management unit 62 calculates a sound source direction angle θ ₀ corresponding to the calculated crop image 6200 display direction angle θ. The angle θ _{0 in the} sound source direction is a _value represented by “θ _s (angle in the display range direction) + θ (angle in the display direction of the cropped image 6200)”. In this case, since the angle θ _s in the display range direction is 0 °, the angle θ _{0 in the} sound source direction corresponding to the display direction of the cropped image 6200 is θ ₀ = θ.

Then, the sound data management unit 62 calculates the angles θ ₁ and θ ₂ in the left and right channel directions with reference to the calculated sound source direction. The sound data management unit 62 specifies two directions separated at equal intervals with reference to the sound source direction corresponding to the display direction of the crop image 6200 as the sound source direction of the reproduction sound data 650. Specifically, as shown in FIG. 23, the sound data management unit 62, for example, the angle θ _{1 in} the channel direction of the left ch (channel) in the −45 ° direction from the sound source direction corresponding to the display direction of the crop image 6200. Then, the angle θ ₂ in the channel direction of the right ch (channel) in the + 45 ° direction from the sound source direction corresponding to the display direction of the crop image 6200 is calculated. In the case of the cropped image 6200 shown in FIG. 21A, θ ₁ is θ ₀ −45 °, and θ ₁ = −22.5 °. On the other hand, since θ ₂ is θ ₀ + 45 °, θ ₂ = 67.5 °.

The sound data management unit 62 calculates the sum of the angle from the sound source direction corresponding to the display direction of the crop image 6200 to the left channel direction and the angle from the sound source direction corresponding to the display direction of the crop image 6200 to the right channel direction is 90 °. Θ ₁ and θ ₂ are calculated so that Note that the sum of the angle from the sound source direction corresponding to the display direction of the crop image 6200 to the left channel direction and the angle from the sound source direction corresponding to the display direction of the crop image 6200 to the right channel direction is not limited to 90 °. The angles from the sound source direction corresponding to the display direction of the cropped image 6200 to the left and right channel directions may be substantially the same. Thus, the sound data management unit 62 specifies the sound source direction of the reproduction sound data 650 as the sound source direction in the three-dimensional sound data 230 by calculating the angles θ ₁ and θ ₂ of the sound source direction for two channels. To do.

In step S207, the sound data management unit 62 of the communication terminal 5 determines the channel angle of the reproduction sound data 650. Specifically, the sound data management unit 62 determines the channel angles θ _L and θ _R from the calculated angles θ ₁ and θ _{2 of} the reproduction sound data 650 in the channel direction. As shown in FIG. 24, θ _L is the angle of the left channel of the two-dimensional stereo sound, and θ _R is the angle of the right channel of the two-dimensional stereo sound. The sound data management unit 62 determines the channel angle θ _{L obtained} by converting the angle θ ₁ in the sound source direction of the left channel into a counterclockwise angle from the reference position in the shooting direction. Similarly, the sound data management unit 62 determines the channel angle θ _{R obtained} by converting the angle θ ₂ in the sound source direction of the right channel into a counterclockwise angle from the reference position in the shooting direction.

  In step S208, the reproduction data generation unit 63 generates reproduction data for reproduction by the communication terminal 5 (an example of a generation step). Specifically, the reproduction data generation unit 63 generates reproduction sound data corresponding to the determined channel angle of the reproduction sound data.

FIG. 25 is a diagram illustrating an example of parameters for generating sound data for reproduction in the first embodiment. The parameters shown in FIG. 25 are parameters for creating the reproduction sound data 650 for one frame included in the reproduction data file 600 shown in FIG. The parameters shown in FIG. 25 include data on the above-described θ, θ ₀ , angle θ ₁ , angle θ ₂ , left ch angle θ _L and right ch angle θ _R.

The left channel angle θ _L is 22.5 °, and the right channel angle θ _R is 292.5 °. The reproduction data generation unit 63 generates reproduction sound data 650 using Expression 5.

W, X and Y are parameters of the B format of Ambisonics. The W, X, and Y parameters can be calculated using the sound data 220 included in the photographic data file 200 by the method shown in FIG. p is a parameter for obtaining a predetermined primary pole pattern (omnidirectional, cardioid, hypercardioid, figure-eight, etc.) indicating a predetermined direction. The primary pole pattern corresponding to the parameter of p is listed in Table 1. Therefore, the values of W, X, Y, Z, and p used for the right ch sound data M _L (θ _L , p) and the left ch sound data M _R (θ _R , p) are the same value. Note that the value of p can be set or changed as appropriate according to the sound pattern desired to be acquired (generated).

The reproduction data generation unit 63 generates sound data for two channels of right-channel sound data M _L (θ _L , p) and left-channel sound data M _R (θ _R , p) according to Equation 5. As described above, the reproduction data generation unit 63 can generate the reproduction sound data 650 based on the product of the angle information based on the display direction of the cropped image 6200 and the arrangement of the three-dimensional sound data 230.

Then, the reproduction data generation unit 63 generates reproduction data by associating the generated reproduction sound data 650 with the crop image data 610 of the corresponding frame. The reproduction sound data M _L (θ _L , p) and M _R (θ _R , p) correspond to sound data (L and R) for one frame of the reproduction sound data 650 shown in FIG.

  In step S209, the storage / readout unit 57 causes the storage unit 5000 to store the reproduction data generated by the reproduction data generation unit 63 as the reproduction data file 600.

  In step S <b> 210, the display control unit 53 and the sound reproduction unit 54 of the communication terminal 5 display the crop image data 610 included in the reproduction data file 600 and reproduce the reproduction sound data 650. Thus, the communication terminal 5 can reproduce the predetermined range of the omnidirectional moving image as a two-dimensional moving image by reproducing the generated reproduction data for each frame.

  In the above description, the example in which the omnidirectional image is acquired by the imaging process by the imaging device 1 has been described. However, the image acquired by the imaging device 1 is not limited to the omnidirectional image, and an image of a predetermined value or more. Any wide-angle image having a corner may be used. In this case, the wide-angle image is acquired by the photographing apparatus 1 such as a wide-angle camera or a stereo camera. That is, the imaging device 1 may be an imaging unit that can acquire an image (omnidirectional image, wide-angle image) captured using a lens having a focal length shorter than a predetermined value.

Effect of First Embodiment Accordingly, when the information processing system according to the first embodiment converts a predetermined area of the omnidirectional video into a two-dimensional video, the two-dimensional sound data attached to the two-dimensional image data. Can be generated.

● Modification 1 of the first embodiment ●
Subsequently, Modification 1 of the first embodiment will be described. In the first embodiment, from the omnidirectional video obtained by photographing the subject by the photographing device 1. Although the process of generating a two-dimensional moving image including a part of the subject has been described, the image captured by the imaging device 1 or the image reproduced by the communication terminal 5 may be a still image. In this case, a shooting data file 200a shown in FIG. 31 is stored instead of the shooting data file 200 (see FIG. 12) stored in the storage unit 1000 of the shooting apparatus 1. FIG. 31 is a diagram illustrating an example of shooting data according to the first modification of the first embodiment. The photographic data file 200a shown in FIG. 31 stores omnidirectional image data 210a as a still image in a predetermined format, and the sound data 220a, the three-dimensional sound data 230a, and the tilt angle data 250a are based on the photographing time. Remember each. Thereby, the imaging device 1 can record sound data for an appropriate time while capturing an omnidirectional image as a still image. As the omnidirectional image data 210a, the same image may be recorded as shown in FIG. 31, or a different image may be recorded every time.

  The storage unit 5000 of the communication terminal 5 stores a reproduction data file 600a shown in FIG. 32 instead of the reproduction data file 600 (see FIG. 14). In the reproduction data file 600a shown in FIG. 32, crop image data 610a as a still image is stored in a predetermined format, and reproduction sound data 650a is stored for each frame. Thereby, the communication terminal 5 can reproduce sound data for an appropriate time while reproducing a still image. Note that the crop image data 610a may be the same image as shown in FIG. 32, or may be an image that changes with the passage of time.

  The display control unit 53 of the communication terminal 5 may be configured to reproduce a two-dimensional image as a still image generated based on the omnidirectional image as a still image acquired by the imaging device 1. . The display control unit 53 of the communication terminal 5 may be configured to reproduce a two-dimensional image as a still image generated based on the omnidirectional video acquired by the imaging device 1.

● Modification 2 of the first embodiment ●
Then, the modification 2 of 1st Embodiment is demonstrated. In the first embodiment, crop image data 610 (an example of two-dimensional image data) and reproduction sound data 650 (an example of two-dimensional sound data) in which a sound source direction is specified are generated as reproduction data. However, the communication terminal 5 may generate only the reproduction sound data 650 without generating the crop image data 610. The two-dimensional image data cropped by the crop range specified by the user is a part of the omnidirectional image data captured and acquired by the imaging device 1. Therefore, the communication terminal 5 stores the cropping range in advance, and generates only the reproduction sound data 650 in which the sound source direction is specified by the process shown in FIG. When reproducing the reproduction data, the communication terminal 5 does not accept the change of the display range from the display direction (position) specified as the crop range, and reproduces it together with the generated reproduction sound data 650. Thereby, since the communication terminal 5 does not need to perform the process which produces | generates two-dimensional image data, it can reduce processing time and data amount.

● Second embodiment ●
Next, an information processing system according to the second embodiment will be described. The same configurations and functions as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted. In the information processing system according to the second embodiment, the image processing server 7 generates two-dimensional sound data. Therefore, the information processing system according to the second embodiment can generate two-dimensional sound data attached to the two-dimensional image data while reducing the processing burden on the communication terminal 5A.

System Configuration First, an outline of the configuration of the information processing system according to the second embodiment will be described with reference to FIG. FIG. 26 is a diagram illustrating an example of a system configuration of an information processing system according to the second embodiment. As shown in FIG. 26, in the information processing system according to the second embodiment, an image processing server 7 is further added to the configuration according to the first embodiment. The communication terminal 5A and the image processing server 7 can communicate with each other via a communication network 100 such as the Internet or an intranet.

  The image processing server 7 is a server computer, and includes a case where image processing is performed in a distributed manner by a plurality of server computers. The image processing server 7 stores omnidirectional video data captured by the imaging device 1 and two-dimensional video data displayed (reproduced) by the communication terminal 5A. Further, the image processing server 7 performs image and sound processing on the omnidirectional video data in response to a request from the communication terminal 5A, and provides the processing data to the communication terminal 5A. The image processing server 7 is an example of an external device.

Hardware Configuration Next, the hardware configuration of the image processing server 7 in the second embodiment will be described with reference to FIG. Note that the hardware configurations of the imaging device 1 and the communication terminal 5A in the second embodiment are the same as those in the first embodiment, and thus the description thereof is omitted.

O Hardware Configuration of Image Processing Server FIG. 27 is a diagram illustrating an example of a hardware configuration of the image processing server according to the second embodiment. The image processing server 7 is constructed by a general computer. The image processing server 7 includes a CPU 701, ROM 702, RAM 703, HDD (Hard Disk Drive) 705, media I / F 707, display 708, network I / F 709, keyboard 711, mouse 712, CD-RW (Compact Disc-ReWritable) drive 714. And a bus line 710. Since the image processing server 7 functions as a server, the image processing server 7 may not include an input device such as the keyboard 711 and the mouse 712 and an output device such as the display 708.

  The CPU 701 controls the overall operation of the image processing server 7. The ROM 702 stores a program used for driving the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HDD 705 controls reading or writing of various data with respect to the HD 704 according to the control of the CPU 701. The HD 704 stores various data such as programs. The media I / F 707 controls reading or writing (storage) of data with respect to a recording medium 706 such as a flash memory.

  The display 708 displays various information such as a cursor, menu, window, character, or image. A network I / F 709 is an interface for performing data communication using the communication network 100. The keyboard 711 is a kind of input means provided with a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 714 controls reading of various data with respect to a CD-RW 713 as an example of a removable recording medium.

  Further, the image processing server 7 includes a bus line 710. The bus line 710 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 701 shown in FIG.

Functional Configuration FIG. 28 is a diagram illustrating an example of a functional configuration of the information processing system according to the second embodiment. The functions realized by the communication terminal 5 </ b> A include a server communication unit 58 in addition to the functions included in the communication terminal 5.

  The server communication unit 58 has a function of transmitting / receiving various data or information to / from other devices (for example, another communication terminal 5A or the image processing server 7) via the communication network 100 such as the Internet. The server communication unit 58 is realized mainly by the processing of the long-range communication circuit 511 and the CPU 501 shown in FIG. The server communication unit 58 is an example of a transmission unit.

  The functions realized by the image processing server 7 include a communication unit 71, an image / sound processing unit 72, a determination unit 73, a storage / reading unit 74, and a storage unit 7000.

  The communication unit 71 has a function of transmitting / receiving various data or information to / from another device (for example, another server or the communication terminal 5A) via the communication network 100. The communication unit 71 is mainly realized by the processing of the network I / F 709 and the CPU 701 illustrated in FIG.

  The image / sound processing unit 72 is a function for performing various processes on various data transmitted from the communication terminal 5A. The image / sound processor 72 is mainly realized by a command from the CPU 701 shown in FIG.

  The determination unit 73 is realized by the processing of the CPU 701 shown in FIG.

  The storage / reading unit 74 has a function of storing various data in the storage unit 7000 or reading various data from the storage unit 7000. The storage / reading unit 74 is mainly realized by the processing of the CPU 701 shown in FIG. The storage unit 7000 is mainly realized by the ROM 702, the HD 704, and the recording medium 706 shown in FIG. Further, the storage unit 7000 stores the shooting data file 200 shown in FIG. 12 and the reproduction data file 600 shown in FIG.

  Here, the functional configuration of the image / sound processing unit 72 will be described in detail with reference to FIG. FIG. 29 is a diagram illustrating an example of a detailed functional configuration of the image / sound processing unit according to the second embodiment. The image / sound processing unit 72 includes an image data management unit 81, a sound data management unit 82, and a reproduction data generation unit 83.

  The image data management unit 81 has a function of managing, controlling, and holding the display range of the omnidirectional image data 210. For example, the image data management unit 81 displays the display direction (position) of crop image data 610 (an example of two-dimensional image data) that is a predetermined area included in the omnidirectional image data 210 specified by the user of the communication terminal 5A. ). The image data management unit 81 is mainly realized by the processing of the CPU 701 shown in FIG. The image data management unit 81 is an example of a display direction specifying unit.

  The sound data management unit 82 is a function for managing, controlling, and holding reproduction sound data 650 corresponding to the crop image data 610. The sound data management unit 82 specifies the sound source direction in the three-dimensional sound data 230 based on the display direction (position) of the crop image data 610 specified by the image data management unit 81. The sound data management unit 82 is realized mainly by the processing of the CPU 701 shown in FIG. The sound data management unit 82 is an example of a sound source direction specifying unit.

  The reproduction data generation unit 83 has a function of generating a reproduction data file 600 using the shooting data file 200 stored in the storage unit 7000 in response to a request from the communication terminal 5A. For example, the reproduction data generation unit 83 generates crop image data 610 from the omnidirectional image data 210 and also generates reproduction sound data 650 from the three-dimensional sound data 230. The reproduction data generation unit 83 is realized mainly by the processing of the CPU 701 shown in FIG. The reproduction data generation unit 83 is an example of a generation unit.

Processing or Operation of Second Embodiment Next, a two-dimensional moving image generation and playback method according to the second embodiment will be described with reference to FIG. FIG. 30 is a sequence diagram illustrating an example of reproduction data generation processing in the information processing system according to the second embodiment.

  In step S301, the communication unit 11 of the photographing apparatus 1 transmits the photographing data file 200 generated by the image / sound processing unit 17 to the communication terminal 5A using short-range wireless communication. The image data file 200 is generated by the image / sound processor 17 by the method described with reference to FIGS. The photographic data file 200 is omnidirectional video data imaged by the photographic device 1, and as shown in FIG. 12, omnidirectional image data 210, sound data 220, three-dimensional sound data 230, and tilt angle data. 250.

  In step S <b> 302, the server communication unit 58 of the communication terminal 5 </ b> A transmits the captured data file 200 received by the communication unit 51 to the image processing server 7 via the communication network 100.

  In step S <b> 303, the storage / reading unit 74 of the image processing server 7 stores the shooting data file 200 received by the communication unit 71 in the storage unit 7000.

  In step S <b> 304, the image / sound processing unit 56 of the communication terminal 5 </ b> A acquires reference information from the captured data file 200 received by the communication unit 51. In step S305, the display control unit 53 of the communication terminal 5A causes the display 517 to display an image (for example, the display screen 6000). In step S306, the accepting unit 52 of the communication terminal 5A accepts a crop range designation operation for the display screen 6000. Note that the processing from step S304 to step S306 is the same as the processing from step S202 to step S204 shown in FIG.

  In step S307, the server communication unit 58 of the communication terminal 5A transmits the display range information of the cropped image 6200 to the image processing server 7 via the communication network 100. The display range information includes coordinate information of the crop image 6200. The display range information is an example of direction information. Note that in step S307, the server communication unit 58 of the communication terminal 5A may transmit the captured data file 200 corresponding to the crop image 6200 to the image processing server 7 together with the display range information of the crop image 6200. In this case, the process of step S302 may not be performed.

  In step S308, the image data management unit 81 of the image processing server 7 specifies the display direction of the cropped image 6200 in the omnidirectional image. The processing in step S308 is the same as the processing in step S205 shown in FIG.

  In the information processing system according to the second embodiment, the image data management unit 61 of the communication terminal 5A calculates the coordinate information of the crop image 6200 in step S307. In step S308, the image data management unit 81 of the image processing server 7 calculates the center coordinates C (X, Y) of the crop image 6200, and uses the calculated coordinate information of the crop image 6200 to display the crop image display direction. θ is calculated.

  In step S309, the sound data management unit 82 of the image processing server 7 specifies the sound source direction in the three-dimensional sound data 230 based on the specified display direction of the cropped image 6200. In step S310, the sound data management unit 82 of the image processing server 7 determines the channel angle of the reproduction sound data 650. In step S311, the reproduction data generation unit 83 of the image processing server 7 generates reproduction data for reproduction by the communication terminal 5A. Note that the processing from step S309 to step S311 is the same as the processing from step S206 to step S208 shown in FIG.

  In step S <b> 312, the storage / reading unit 74 of the image processing server 7 stores the reproduction data generated by the reproduction data generation unit 83 in the storage unit 7000 as the reproduction data file 600. In step S313, the communication unit 71 of the image processing server 7 transmits the reproduction data generated by the reproduction data generation unit 83 to the communication terminal 5A via the communication network 100. In addition, the order of the process of step S312 and the process of step S313 may be followed, or may be performed in parallel.

  In step S314, the display control unit 53 and the sound reproduction unit 54 of the communication terminal 5A display the crop image data 610 included in the reproduction data received by the server communication unit 58 and reproduce the reproduction sound data 650. . In this way, the communication terminal 5A can display the predetermined range of the omnidirectional video as a two-dimensional video by reproducing the data for reproduction generated by the image processing server 7 for each frame. it can.

Effect of Second Embodiment Accordingly, since the information processing system according to the second embodiment performs reproduction data generation processing by the image processing server 7, the processing load on the communication terminal 5A is reduced and two-dimensional. Two-dimensional sound data accompanying the image data can be generated.

● Modification of the second embodiment ●
Subsequently, a modification of the second embodiment will be described. In the second embodiment described above, the communication terminal 5A transmits the shooting data transmitted from the shooting device 1 to the image processing server 7. However, the shooting data file 200 is stored in the storage unit 5000 of the communication terminal 5A. If it is, the communication terminal 5 </ b> A does not have to transmit the shooting data to the image processing server 7. In this case, the process of step S302 shown in FIG. 30 is omitted, and the sound data 220, the three-dimensional sound data 230, and the inclination angle stored in the shooting data file 200 in time series together with the display range information in the process of step S307. Data 250 is transmitted from the communication terminal 5 </ b> A to the image processing server 7. Then, when reproducing the reproduction data, the communication terminal 5A displays the photographing data in the designated display range and reproduces the reproduction data (reproduction sound data 650) generated by the image processing server 7. it can.

● Summary ●
As described above, the communication terminal 5 (an example of an information processing apparatus) according to an embodiment of the present invention displays the omnidirectional image data 210 acquired by the imaging apparatus 1 and is acquired by the imaging apparatus 1. The communication terminal 5 that reproduces the three-dimensional sound data 230 acquired along with the omnidirectional image data 210, and performs a crop designation operation on the displayed omnidirectional image (an example of designation of a predetermined display direction). Accept. Then, the communication terminal 5 generates crop image data 610 (an example of two-dimensional image data) corresponding to the crop range 6100 (an example of a designated display direction) from the omnidirectional image data 210, and also generates a three-dimensional sound. From the data 230, reproduction sound data 650 (an example of two-dimensional sound data) corresponding to the cropping range 6100 is generated. Thereby, the communication terminal 5 can produce | generate the two-dimensional sound data accompanying the two-dimensional image data, when displaying the predetermined area | region of an omnidirectional image as a two-dimensional image.

  In addition, the communication terminal 5 (an example of an information processing apparatus) according to an embodiment of the present invention is based on the omnidirectional video including the omnidirectional image data 210 and the three-dimensional sound data 230 captured by the imaging apparatus 1. A communication terminal 5 that generates a three-dimensional moving image, and accepts a crop designation operation (an example of designation of a predetermined area) in an omnidirectional moving image. Then, the communication terminal 5 generates crop image data 610 (an example of two-dimensional image data) corresponding to the crop range 6100 (an example of a designated area) from the omnidirectional video, and from the three-dimensional sound data 230. , Reproduction sound data 650 (an example of two-dimensional sound data) corresponding to the cropping range 6100 is generated. Therefore, the communication terminal 5 can generate two-dimensional sound data accompanying the two-dimensional image data when reproducing a predetermined area of the omnidirectional video as a two-dimensional video.

  Furthermore, the communication terminal 5 (an example of an information processing apparatus) according to an embodiment of the present invention includes a reproduction sound data 650 (an example of two-dimensional sound data), and crop image data 610 (an example of two-dimensional image data). Is used to generate reproduction data (an example of a two-dimensional moving image), and the generated reproduction data is stored. Then, the communication terminal 5 reproduces the stored reproduction data. Therefore, when the predetermined area of the omnidirectional video is reproduced as a two-dimensional video, the communication terminal 5 reduces the sense of discomfort felt by the user by reproducing the two-dimensional sound data attached to the two-dimensional image data. Can do.

  In addition, the communication terminal 5 (an example of an information processing apparatus) according to an embodiment of the present invention specifies the position of the crop range 6100 (an example of a designated area) in the omnidirectional video. Then, the communication terminal 5 specifies the sound source direction in the three-dimensional sound data 230 included in the omnidirectional video based on the position of the specified crop range 6100, and specifies the specified sound source direction and the three-dimensional sound data 230. Based on, reproduction sound data 650 (an example of two-dimensional sound data) corresponding to the cropping range 6100 is generated. Therefore, the communication terminal 5 can generate two-dimensional sound data in the sound source direction corresponding to the position of the two-dimensional image data when reproducing a predetermined region of the omnidirectional moving image as a two-dimensional moving image.

  Furthermore, the three-dimensional sound data 230 according to an embodiment of the present invention is stereophonic data generated based on the Ambisonics B format. Therefore, the communication terminal 5 (an example of an information processing device) can generate 2D sound data associated with 2D image data by using the parameters of the 3D sound data 230 included in the omnidirectional video. .

  In addition, the information processing system according to the embodiment of the present invention displays the omnidirectional image data 210 acquired by the imaging device 1 and is attached to the omnidirectional image data 210 acquired by the imaging device 1. Information including a communication terminal 5A (an example of an information processing apparatus) that reproduces the acquired three-dimensional sound data 230, and an image processing server 7 (an example of an external apparatus) connected to the communication terminal 5A via a communication network. It is a processing system. The communication terminal 5A receives a crop designation operation (an example of designation of a predetermined display direction) in the displayed omnidirectional image, and shows the three-dimensional sound data 230 and the crop range 6100 (an example of the designated display direction). Display range information (an example of direction information) is transmitted to the image processing server 7. Then, the image processing server 7 generates reproduction sound data 650 (an example of two-dimensional sound data) corresponding to the cropping range 6100 from the three-dimensional sound data 230. Therefore, since the information processing system according to the embodiment of the present invention performs the two-dimensional sound data generation process by the image processing server 7, the processing load on the communication terminal 5A is reduced and the two-dimensional image data attached to the two-dimensional image data is reduced. Dimensional sound data can be generated.

  Furthermore, the information processing system according to an embodiment of the present invention is a two-dimensional moving image generated based on a omnidirectional moving image including the omnidirectional image data 210 and the three-dimensional sound data 230 that is captured by the image capturing device 1. Is an information processing system including a communication terminal 5A (an example of an information processing apparatus) that reproduces the image and an image processing server 7 (an example of an external apparatus) connected to the communication terminal 5A via a communication network 100. The communication terminal 5A receives a crop designation operation (an example of designation of a predetermined area) in the omnidirectional video, and displays display area information (area information) indicating the omnidirectional video and the crop range 6100 (an example of the designated area). Is transmitted to the image processing server 7. Then, the image processing server 7 generates crop image data 610 (an example of two-dimensional image data) corresponding to the crop range 6100 from the omnidirectional video, and also corresponds to the crop range 6100 from the three-dimensional sound data 230. Reproduction sound data 650 (an example of two-dimensional sound data) is generated. For this reason, the information processing system according to the embodiment of the present invention performs the two-dimensional video generation process by the image processing server 7, so that the two-dimensional image attached to the two-dimensional image data is reduced while reducing the processing burden on the communication terminal 5 </ b> A. Sound data can be generated.

  In addition, the data generation method according to the embodiment of the present invention displays the omnidirectional image data 210 acquired by the imaging device 1 and accompanies the omnidirectional image data 210 acquired by the imaging device 1. Is a data generation method executed by the communication terminal 5 (an example of an information processing device) that reproduces the three-dimensional sound data 230 acquired in this way, and includes a crop designation operation (designation of a predetermined display direction) in the displayed omnidirectional image And an omnidirectional image data 210, and crop image data 610 (an example of two-dimensional image data) corresponding to the crop range 6100 (an example of a designated display direction) is generated from the celestial sphere image data 210. A generation step for generating reproduction sound data 650 (an example of two-dimensional sound data) corresponding to the cropping range 6100 from the original sound data 230. And up, to run. Accordingly, the data generation method according to the embodiment of the present invention can generate two-dimensional sound data associated with two-dimensional image data when a predetermined region of the omnidirectional image is displayed as a two-dimensional image. .

  Furthermore, the moving image generating method according to an embodiment of the present invention is a communication terminal that generates a two-dimensional moving image from an omnidirectional moving image including the omnidirectional image data 210 and the three-dimensional sound data 230 captured by the photographing apparatus 1. 5 (an example of an information processing device) is a moving image generation method executed by a reception step of accepting a crop designation operation (an example of designation of a predetermined area) in an omnidirectional video, and a crop range 6100 from the omnidirectional video. Crop image data 610 (an example of two-dimensional image data) corresponding to (an example of a designated area) is generated, and reproduction sound data 650 (a two-dimensional sound) corresponding to the crop range 6100 is generated from the three-dimensional sound data 230. Generating an example of data). Therefore, the moving image generating method according to an embodiment of the present invention can generate two-dimensional sound data associated with two-dimensional image data when a predetermined area of the omnidirectional moving image is reproduced as a two-dimensional moving image.

● Supplement ●
The functions of the embodiments can be realized by a computer-executable program written in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark), an object-oriented programming language, or the like. The program for executing the function can be distributed through a telecommunication line.

  In addition, programs for executing the functions of the embodiments are ROM, EEPROM, EPROM (Erasable Programmable Read-Only Memory), flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM. , DVD-RW, Blu-ray disc, SD card, MO (Magneto-Optical disc) and the like can be stored and distributed.

  Furthermore, part or all of the functions of each embodiment can be mounted on a programmable device (PD) such as an FPGA (Field Programmable Gate Array), or can be mounted as an ASIC. Circuit configuration data (bitstream data) downloaded to the PD in order to realize the above functions on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, VHDL (Very High Speed Integrated Circuits Hardware Description Language), It can be distributed on a recording medium as data described in Verilog-HDL or the like.

  The information processing apparatus, the information processing system, the data generation method, and the program according to an embodiment of the present invention have been described so far, but the present invention is not limited to the above-described embodiment, and other embodiments are added. Any change can be made within the range that can be conceived by those skilled in the art, such as changes or deletions, and any embodiment is included in the scope of the present invention as long as the effects and advantages of the present invention are exhibited.

1 photographing device 5 communication terminal (an example of information processing device)
7 Image processing server (an example of an external device)
52 reception part (an example of reception means)
53 Display control unit (an example of playback means)
54 sound playback unit (an example of playback means)
61 Image data management unit (an example of display direction specifying means)
62 Sound data management unit (an example of sound source direction specifying means)
63 Data generator for reproduction (an example of generating means)
5000 storage unit (an example of storage means)

JP 2015-220745 A

Claims (10)

An information processing device that displays the omnidirectional image acquired by the imaging device and reproduces the three-dimensional sound data acquired accompanying the omnidirectional image,
Accepting means for accepting designation of a predetermined display direction in the displayed omnidirectional image;
Two-dimensional image data corresponding to the specified display direction is generated from the omnidirectional image data, and two-dimensional sound data corresponding to the specified display direction is generated from the three-dimensional sound data. Generating means;
An information processing apparatus comprising: The information processing apparatus according to claim 1, further comprising:
Storage means for storing the generated two-dimensional image data and the two-dimensional sound data;
An information processing apparatus comprising: reproduction means for reproducing the stored two-dimensional image data and the two-dimensional sound data. The information processing apparatus according to claim 1, further comprising:
Display direction specifying means for specifying the specified display direction;
Sound source direction specifying means for specifying a sound source direction in the three-dimensional sound data based on the specified display direction,
The information processing apparatus generates the two-dimensional sound data corresponding to the designated display direction based on the specified sound source direction and the three-dimensional sound data. The sound source direction specifying means specifies two directions that are equidistant from the display direction as the sound source direction,
The generating means generates the two-dimensional sound data using the two specified sound source directions.
The information processing apparatus according to any one of claims 1 to 3. The generating means generates a two-dimensional moving image using the two-dimensional sound data and the two-dimensional image data,
The storage means stores the two-dimensional video,
The information processing apparatus according to any one of claims 2 to 4, wherein the reproduction unit reproduces the stored two-dimensional moving image.   The information processing apparatus according to any one of claims 1 to 5, wherein the three-dimensional sound data is stereophonic sound data generated based on an Ambisonics B format. An information processing device that displays the omnidirectional image acquired by the imaging device and reproduces the three-dimensional sound data acquired accompanying the omnidirectional image, and the information processing device via the communication network An information processing system comprising a connected external device,
The information processing apparatus includes:
Accepting means for accepting designation of a predetermined display direction in the displayed omnidirectional image;
Transmitting means for transmitting the three-dimensional sound data and the direction information indicating the designated display direction to the external device;
The external device is
Generating means for generating two-dimensional sound data corresponding to the designated display direction from the three-dimensional sound data;
An information processing system comprising: The information processing system according to claim 7,
The transmission means transmits the three-dimensional sound data, the direction information and the omnidirectional image data to the external device,
The generating means generates two-dimensional image data corresponding to the specified display direction from the data of the omnidirectional image, and also generates a two-dimensional image corresponding to the specified display direction from the three-dimensional sound data. An information processing system that generates sound data. A data generation method executed by an information processing apparatus that displays an omnidirectional image acquired by an imaging device and reproduces three-dimensional sound data acquired accompanying the omnidirectional image,
An accepting step of accepting designation of a predetermined display direction in the displayed omnidirectional image;
Two-dimensional image data corresponding to the specified display direction is generated from the omnidirectional image data, and two-dimensional sound data corresponding to the specified display direction is generated from the three-dimensional sound data. Generation step;
Data generation method to execute.   The program which makes a computer perform the method of Claim 9.