JP2018157314A - Information processing system, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To output sounds matching an image to enhance reality.SOLUTION: An information processing system that has an information processing device connected with a photographing device for taking a plurality of images, comprises: a sound input unit to which a plurality of sounds are inputted; an image output unit that outputs a display image on the basis of the plurality of images; a setting unit that sets a predetermined region outputted on the display image; a first conversion unit that converts the plurality of sounds inputted to the sound input unit, on the basis of the predetermined region; and a sound output unit that outputs the plurality of sounds converted by the first conversion unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing method, and a program.

  Conventionally, a method of displaying an image showing a wide range, a so-called panoramic image is known.

  For example, the image processing system first pastes a target image into a three-dimensional shape to generate a three-dimensional model. Next, the image processing system determines the position of the viewpoint and the viewing angle based on the input value. In determining the viewpoint position and the viewing angle, the image processing system determines whether to change the viewing angle preferentially or to change the viewpoint position preferentially based on the input value. In this way, there is known a method for reducing the display of an uncomfortable image such as a display in which a subject is stretched in a wide field of view (for example, Patent Document 1).

  However, in the conventional method, there is a problem that the sound matching the image is not output and the presence is insufficient.

  An object of the present invention is to increase the sense of reality by outputting sound in accordance with an image.

In order to solve the above-described problem, an information processing system including an information processing device connected to an imaging device that captures a plurality of images in one embodiment of the present invention is provided.
A voice input unit for inputting a plurality of voices;
An image output unit that outputs a display image based on the plurality of images;
A setting unit for setting a predetermined area to be output to the display image;
A first conversion unit that converts a plurality of voices input to the voice input unit based on the predetermined area;
An audio output unit that outputs a plurality of audios converted by the first conversion unit.

  Sound that matches the image can be output to enhance the sense of presence.

It is a figure explaining an example of the whole information processing system composition concerning one embodiment of the present invention. It is a figure explaining an example of the imaging device concerning one embodiment of the present invention. It is a figure explaining an example of the image image | photographed with the imaging device which concerns on one Embodiment of this invention. It is a block diagram explaining an example of the hardware constitutions of the imaging device which concerns on one Embodiment of this invention. It is a block diagram explaining an example of the hardware constitutions of the information processing apparatus which concerns on one Embodiment of this invention. It is a sequence diagram explaining an example of the whole process by the information processing system which concerns on one Embodiment of this invention. It is a figure explaining an example of the omnidirectional image which concerns on one Embodiment of this invention. It is a figure explaining an example of the omnidirectional panoramic image which concerns on one Embodiment of this invention. It is a figure for demonstrating an example of the initial image which concerns on one Embodiment of this invention. It is a figure for demonstrating an example of another zoom process which concerns on one Embodiment of this invention. It is a table | surface for demonstrating an example of another zoom process which concerns on one Embodiment of this invention. It is a figure for demonstrating an example of the "range" of another zoom process which concerns on one Embodiment of this invention. It is a schematic diagram which shows the example of arrangement | positioning of the virtual speaker which concerns on one Embodiment of this invention. It is a schematic diagram which shows the 1st example which changed arrangement | positioning of the virtual speaker which concerns on one Embodiment of this invention. It is a schematic diagram which shows the 2nd example of arrangement | positioning of the virtual speaker which concerns on one Embodiment of this invention. It is a schematic diagram which shows the 3rd example of arrangement | positioning of the virtual speaker which concerns on one Embodiment of this invention. It is a schematic diagram which shows the 4th example of arrangement | positioning of the virtual speaker which concerns on one Embodiment of this invention. It is a schematic diagram which shows the 5th example of arrangement | positioning of the virtual speaker which concerns on one Embodiment of this invention. It is a functional block diagram which shows the function structural example of the information processing system which concerns on one Embodiment of this invention.

  Embodiments of the present invention will be described below. The sound of the embodiment of the present invention is not limited to a voice uttered by a person, but is referred to as a general term for music, mechanical sound, operation sound, and other sounds that are propagated by vibration of air.

<Example of overall configuration of information processing system>
FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system according to an embodiment of the present invention. The information processing system 10 includes a photographing device 1 and a smartphone 2 that is an example of the information processing device.

  The imaging device 1 is a camera having at least a plurality of optical systems. For example, the imaging device 1 generates an image showing a wide range such as an omnidirectional direction (hereinafter referred to as an “omnidirectional image”) based on a plurality of images captured using a plurality of optical systems. Next, the imaging device 1 transmits an omnidirectional image or the like to the smartphone 2. Then, the smartphone 2 performs image processing on the transmitted image and outputs a display image. Hereinafter, an example in which the first image is an omnidirectional image will be described as an input image. The panoramic image is, for example, an omnidirectional image.

  In this example, the photographing device 1 and the smartphone 2 are connected by wire or wirelessly. Then, the smartphone 2 downloads data such as an omnidirectional image from the imaging device 1. The connection may be via a network or the like. Note that the information processing system 10 transmits a plurality of images captured using a plurality of optical systems from the imaging device 1 to the smartphone 2, combines the plurality of images with the smartphone 2, and generates an omnidirectional image. Good.

  Furthermore, the overall configuration is not limited to the configuration shown in FIG. For example, the photographing device 1 and the smartphone 2 may be an integrated device. In addition, the information processing apparatus may be a PC (Personal Computer) or a tablet. The information processing system 10 may further include a photographing device or an information processing device in addition to the photographing device 1 and the smartphone 2.

<Photographing device example>
FIG. 2 is a diagram for explaining an example of a photographing apparatus according to an embodiment of the present invention. Specifically, FIG. 2A is an example of a front view of the photographing apparatus 1. FIG. 2B is an example of a left side view of the photographing apparatus 1. Further, FIG. 2C is an example of a plan view of the photographing apparatus 1.

  The photographing apparatus 1 includes a front photographing element 1H1, a rear photographing element 1H2, and a switch 1H3. In this example, optical systems such as the front photographing element 1H1 and the rear photographing element 1H2 are used for photographing. And the imaging device 1 produces | generates an omnidirectional image based on each image image | photographed using each optical system.

  Furthermore, the imaging device 1 has microphones at a plurality of locations. For example, the microphone 1 is arranged in the photographing apparatus 1. The photographing apparatus 1 collects sounds around the photographing apparatus 1 at four locations, and a sound signal is input to the photographing apparatus 1.

  Specifically, in FIG. 2, the microphone 1 HM 1, the microphone 1 HM 2, and the microphone 1 HM 3 are arranged on the front side of the photographing apparatus 1. Further, a microphone 1HM4 is disposed on the rear surface side of the photographing apparatus 1. The number of microphones and the positions where they are arranged are not limited to the arrangement shown in FIG. Multiple microphones can output more realistic sound. However, it is desirable that the number of microphones is four or more, that is, the number of places where voice is input is four or more.

  The microphone may be a directional microphone or an omnidirectional microphone. When a directional microphone is used, the photographing apparatus 1 can acquire sound in a specific direction with each microphone. On the other hand, when an omnidirectional microphone is used, the photographing apparatus 1 can easily calibrate each microphone. Further, the microphone may be a combination of a directional microphone and an omnidirectional microphone. When at least one of the microphones is omnidirectional, calibration can be easily performed, and the cost is low and individual variation is small.

  The switch 1H3 is a shutter button. The switch 1H3 is an example of an input device for the user to instruct the photographing apparatus 1 to perform photographing.

  As shown in FIG. 2A, when the switch 1H3 is pressed by the user, control to release the shutter is performed, and the photographing apparatus 1 performs photographing. In addition, the information processing system 10 may have a configuration in which an operation to remotely release a shutter is input from an information processing device such as the smartphone 2. All directions of the photographing apparatus 1 are photographed by the front photographing element 1H1 and the rear photographing element 1H2.

  FIG. 3 is a diagram for explaining an example of an image photographed by the photographing apparatus according to the embodiment of the present invention. Specifically, FIG. 3A is an example of an image photographed by the front photographing element 1H1. On the other hand, FIG. 3B is an example of an image photographed by the rear photographing element 1H2.

  FIG. 3C shows an image generated based on the image photographed by the front photographing element 1H1 in FIG. 3A and the image photographed by the rear photographing element 1H2 in FIG. It is an example.

  First, an image photographed by the front photographing element 1H1 is an image having a wide range on the front side of the photographing device 1, for example, a range of 180 ° or more in view angle. Similarly, the image captured by the rear imaging element 1H2 is an image having a wide range on the rear side, for example, a range of 180 ° in angle of view within the range captured by the imaging device 1. When a fisheye lens is used as the optical system, it often has distortion. That is, the images of FIG. 3A and FIG. 3B are one (in this example, the front side) and the other (in this example, the rear side) of the range captured by the imaging device 1. Each of which is a hemispherical image (hereinafter referred to as a “hemispherical image”).

  The angle of view of each optical system is preferably in the range of 180 ° to 200 °. In particular, when each angle of view exceeds 180 ° or more, there is an image area to be superimposed when the hemispherical image and the hemispherical image are combined, so that the photographing apparatus can generate an omnidirectional image even when there is parallax.

  Next, the photographing apparatus 1 performs processing such as distortion correction processing and composition processing, and based on the front hemispherical image shown in FIG. 3A and the rear hemispherical image in FIG. The image of FIG. 3C is generated. 3C is an example of an image generated by a method such as a so-called Mercator projection or equirectangular projection, that is, an omnidirectional image. A hemispherical image may be transmitted from the imaging device 1 to the information processing device, and the omnidirectional image may be generated by the information processing device.

  Note that the first image is not limited to an image generated by the photographing apparatus 1. For example, the first image may be an image captured by another camera or the like, or an image generated based on an image captured by another camera. The first image is preferably an image obtained by photographing a wide viewing angle range with an omnidirectional camera or a so-called wide-angle lens camera.

  In the following description, the first image is described as an example of an omnidirectional image, but the first image is not limited to an omnidirectional image. For example, the first image may be an image taken with a compact camera, a single-lens reflex camera, a smartphone, or the like. The image may be a panoramic image that extends horizontally or vertically.

<Example of hardware configuration of photographing apparatus>
FIG. 4 is a block diagram for explaining an example of the hardware configuration of the photographing apparatus according to an embodiment of the present invention. The photographing apparatus 1 includes a photographing unit 1H4, an image processing unit 1H7, a photographing control unit 1H8, a CPU (Central Processing Unit) 1H9, and a ROM (Read-Only Memory) 1H10. In addition, the photographing apparatus 1 includes an SRAM (Static Random Access Memory) 1H11, a DRAM (Dynamic Random Access Memory) 1H12, and an operation I / F (Interface) 1H13. Furthermore, the imaging device 1 includes a network I / F 1H14, a wireless I / F 1H15, and an antenna 1H16.

  Further, the photographing apparatus 1 includes microphones 1HM1, 1HM2, 1HM3, and 1HM4. With these microphones, the photographing apparatus 1 inputs sound at a plurality of locations.

  The photographing apparatus 1 preferably has an attitude sensor 1H18.

  Each hardware included in the photographing apparatus 1 is connected by a bus 1H17, and inputs or outputs data or signals through the bus 1H17.

  The photographing unit 1H4 includes a front photographing element 1H1 and a rear photographing element 1H2. Further, a lens 1H5 is installed corresponding to the front imaging element 1H1, and a lens 1H6 is installed corresponding to the rear imaging element 1H2. The lens 1H5 and the lens 1H6 are preferably fisheye lenses or wide-angle lenses.

  The front photographing element 1H1 and the rear photographing element 1H2 are so-called camera units. Specifically, the front imaging element 1H1 and the rear imaging element 1H2 each have an optical sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). Then, the front photographing element 1H1 converts light incident through the lens 1H5 and generates image data indicating a hemispherical image or the like. Similarly, the rear imaging element 1H2 converts light incident through the lens 1H6 and generates image data indicating a hemispherical image or the like.

  Next, the photographing unit 1H4 outputs the respective image data generated by the front photographing element 1H1 and the rear photographing element 1H2 to the image processing unit 1H7. The output image data is, for example, the front hemispherical image in FIG. 3A and the rear hemispherical image in FIG.

  Further, the front imaging element 1H1 and the rear imaging element 1H2 may further include other optical elements such as a diaphragm or a low-pass filter in order to perform high-quality imaging. Further, the front imaging element 1H1 and the rear imaging element 1H2 may perform defective pixel correction or camera shake correction in order to perform high-quality imaging.

  The image processing unit 1H7 generates the omnidirectional image of FIG. 3C based on the image data input from the photographing unit 1H4.

  The photographing control unit 1H8 is a control device that controls hardware included in the photographing device 1.

  The CPU 1H9 is a calculation device that performs calculation and data processing for realizing each process, and a control device that controls hardware. For example, the CPU 1H9 executes each process based on a program installed in advance.

  ROM1H10, SRAM1H11, and DRAM1H12 are examples of storage devices. For example, the ROM 1H10 stores programs, data, parameters, and the like for causing the CPU 1H9 to execute processing. The SRAM 1H11 and the DRAM 1H12 store a program used by the CPU 1H9 to execute processing based on the program, data used by the program, data generated by the program, and the like. Note that the photographing apparatus 1 may further include an auxiliary storage device such as a hard disk.

  The operation I / F 1H13 is an interface that is connected to an input device such as the switch 1H3 and performs processing for inputting a user operation on the photographing apparatus 1. For example, the operation I / F 1H13 is an input device such as a switch, a connector for connecting the input device, a cable, a circuit that processes a signal input from the input device, a driver, and a control device. The operation I / F 1H13 may further include an output device such as a display. The operation I / F 1H13 may be a so-called touch panel in which an input device and an output device are integrated. Further, the operation I / F 1H 13 may have an interface such as a USB (Universal Serial Bus), and may connect a recording medium such as a flash memory to the photographing apparatus 1. Accordingly, the operation I / F 1H13 may input / output data from the photographing apparatus 1 to the recording medium.

  The switch 1H3 may be a power switch, a parameter input switch, or the like for performing an operation other than the operation related to the shutter.

  The network I / F 1H14, the wireless I / F 1H15, and the antenna 1H16 connect the external apparatus and the photographing apparatus 1 by wireless or wired. For example, the imaging device 1 is connected to the network by the network I / F 1H 14 and transmits data to the smartphone 2. Note that the network I / F 1H14, the wireless I / F 1H15, and the antenna 1H16 may be hardware that is connected to another external device via a wired connection such as a USB. That is, the network I / F 1H14, the wireless I / F 1H15, and the antenna 1H16 may be connectors and cables.

  The bus 1H17 is used to input / output data and the like between hardware included in the photographing apparatus 1. That is, the bus 1H17 is a so-called internal bus. For example, the bus 1H17 is PCI Express (Peripheral Component Interconnect Bus Express) or the like.

  The attitude sensor 1H18 detects the attitude of the photographing apparatus 1. For example, the posture sensor 1H18 may be a combination of a plurality of sensors such as a triaxial acceleration sensor or an angular velocity sensor.

  Note that the photographing apparatus 1 is not limited to the case where there are two photographing elements. For example, you may have three or more imaging elements. Further, the photographing apparatus 1 may photograph a plurality of partial images by changing the photographing angle of one photographing element.

  Note that the processing performed by the photographing apparatus 1 may be performed by another apparatus. For example, part or all of the processing may be performed by the smartphone 2 or another information processing apparatus connected via the network, with the imaging apparatus 1 transmitting data and parameters. The information processing system 10 may include a plurality of information processing apparatuses and perform processing in a distributed, redundant, or parallel manner.

<Example of hardware configuration of information processing apparatus>
FIG. 5 is a block diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment of the present invention. A smartphone 2 that is an example of an information processing apparatus includes an auxiliary storage device 2H1, a main storage device 2H2, an input / output device 2H3, a state sensor 2H4, a CPU 2H5, and a network I / F 2H6. Furthermore, the smartphone 2 includes a speaker 2HS1 and a speaker 2HS2.

  The hardware included in the smartphone 2 is connected via a bus 2H7, and inputs / outputs data or signals via the bus 2H7.

  The auxiliary storage device 2H1 stores data, parameters, programs, and the like. Specifically, the auxiliary storage device 2H1 is, for example, a hard disk, a flash SSD (Solid State Drive), or the like. The data stored in the auxiliary storage device 2H1 may be partially or entirely stored redundantly or alternatively by a file server or the like connected by the network I / F 2H6.

  The main storage device 2H2 is a so-called memory that becomes a storage area used by a program for executing processing. That is, the main storage device 2H2 stores data, programs, parameters, and the like. For example, the main storage device 2H2 is an SRAM (Static Random Access Memory), a DRAM, or the like. The main storage device 2H2 may further include a control device that performs storage and retrieval.

  The input / output device 2H3 is an output device that displays an image or a processing result, and an input device that inputs an operation by a user. Specifically, the input / output device 2H3 is a so-called touch panel, a peripheral circuit, a driver, and the like. The input / output device 2H3 displays, for example, a predetermined GUI (Graphical User Interface), an image processed image, and the like to the user. On the other hand, for example, when the user operates a displayed GUI or image, the input / output device 2H3 inputs an operation by the user.

  The state sensor 2H4 is a sensor that detects the state of the smartphone 2. Specifically, the state sensor 2H4 is a gyro sensor, a triaxial acceleration sensor, or the like. For example, the state sensor 2H4 determines whether one side of the sides of the smartphone 2 is greater than or equal to a predetermined angle with respect to the horizontal. That is, the state sensor 2H4 detects whether the smartphone 2 is in a vertical posture state or a horizontal posture state.

  The CPU 2H5 is a calculation device that performs calculation and data processing for realizing each process, and a control device that controls hardware. The CPU 2H5 may include a plurality of CPUs, devices, or a plurality of cores in order to perform processing in parallel, redundancy, or distribution. The smartphone 2 may have a GPU (Graphics Processing Unit) or the like inside or outside in order to perform image processing.

  The network I / F 2H6 is connected to an external device via a network in a wireless or wired manner. Specifically, the network I / F 2H6 includes an antenna, a peripheral circuit, a driver, and the like for inputting and outputting data and the like. For example, the smartphone 2 inputs image data from the imaging device 1 or the like by the CPU 2H5 and the network I / F 2H6. On the other hand, the smartphone 2 outputs data and the like to the photographing apparatus 1 and the like by the CPU 2H5 and the network I / F 2H6.

  The speaker 2HS1 and the speaker 2HS2 output sound. The speaker 2HS1 and the speaker 2HS2 perform stereo output. Note that an external device such as an earphone or an external speaker may be connected to the speaker 2HS1 and the speaker 2HS2, and sound may be output from the connected external device. With these speakers, the smartphone 2 outputs sound from a plurality of locations. Further, it is desirable that the number of speakers is two or more, that is, there are two or more locations where sound is output.

  Note that the information processing apparatus is not limited to a smartphone. The information processing apparatus may be a computer other than a smartphone. For example, the information processing apparatus may be an HMD (Head Mounted Display), a PC, a PDA (Personal Digital Assistance), a tablet, a mobile phone, or a combination thereof.

<Example of overall processing by information processing system>
FIG. 6 is a sequence diagram illustrating an example of overall processing by the information processing system according to the embodiment of the present invention.

<Generation example of omnidirectional image> (step S01)
In step S01 of FIG. 6, the imaging device 1 generates an omnidirectional image. Note that the omnidirectional image is generated from the hemispherical image of FIGS. 3A and 3B by the processing of FIG.

  FIG. 7 is a diagram illustrating an example of an omnidirectional image according to an embodiment of the present invention. FIG. 7A is a diagram showing the hemispherical image of FIG. 3A by connecting the places where the incident angles in the horizontal direction and the vertical direction are equal to the optical axis by lines. The incident angle in the horizontal direction with respect to the optical axis is referred to as “θ”, and the incident angle in the direction perpendicular to the optical axis is referred to as “φ”. Further, in FIG. 7B, as in FIG. 7A, the hemispherical image of FIG. 3B is connected by a line at the positions where the incident angles in the horizontal and vertical directions are equal to the optical axis. It is a figure shown by.

  Moreover, FIG.7 (c) is a figure explaining an example of the image processed by Mercator projection. Specifically, the image of FIG. 7C is an image generated by equirectangular projection by associating the images of FIGS. 7A and 7B with a LUT (LookUpTable) generated in advance. It is. Then, after the state shown in FIG. 7C, the images of FIGS. 7A and 7B are combined as shown in FIG. 7D to generate an omnidirectional image. . The composition process is a process for generating an omnidirectional image using two hemispherical images in the state shown in FIG. Note that the composition process in FIG. 7D is not limited to a process in which hemispherical images in the state in FIG. For example, when the horizontal center of the omnidirectional image is not θ = 180 °, in the synthesis process, the imaging device first preprocesses the hemispherical image of FIG. 3A and places it at the center of the omnidirectional image. . Next, the imaging apparatus divides the image obtained by pre-processing the hemispherical image of FIG. 3B into a size that can be arranged in the left and right parts, and synthesizes the hemispherical images into the left and right parts of the generated image, and then combines them in FIG. ) Spherical image may be generated.

  In addition, the process which produces | generates an omnidirectional image is not restricted to the process by equirectangular projection. For example, in the φ direction, the pixel arrangement of the hemispherical image of FIG. 7B and the pixel arrangement of the hemispherical image of FIG. 7A are upside down, and each pixel in the θ direction is In some cases, the top-and-bottom reversal is reversed. In this case, the imaging apparatus performs a process of rotating 180 ° Roll in order to align the hemispherical image in FIG. 7B with the arrangement of the pixels in the φ direction and the θ direction in FIG. Also good.

  Moreover, the process which produces | generates an omnidirectional image may perform the distortion correction process etc. which correct | amend each distortion aberration which the hemispherical image of Fig.7 (a) and FIG.7 (b) has. Furthermore, the processing for generating the omnidirectional image may be performed by shading correction, gamma correction, white balance, camera shake correction, optical black correction processing, defective pixel correction processing, edge enhancement processing, linear correction processing, or the like. Note that if the shooting range of the hemisphere image and the shooting range of the other hemisphere image overlap, the compositing process can be performed with high accuracy by performing correction using the pixels of the subject shot in the overlapping shooting range. Can be synthesized.

  Through the above processing, the imaging device 1 generates an omnidirectional image from a plurality of hemispheric images to be captured. Note that the omnidirectional image may be generated by another process.

<Example of transmission of omnidirectional image> (step S02)
In step S02 in FIG. 6, the smartphone 2 acquires the omnidirectional image generated in step S01 via a network or the like. A case where the smartphone 2 acquires the omnidirectional image of FIG. 7D will be described as an example.

<Generation example of omnidirectional panoramic image> (step S03)
In step S03 of FIG. 6, the smartphone 2 generates an omnidirectional panoramic image from the omnidirectional image acquired in step S02.

  FIG. 8 is a diagram illustrating an example of an omnidirectional panoramic image according to an embodiment of the present invention. For example, in step S03, the smartphone 2 generates the omnidirectional panoramic image of FIG. 8 from the omnidirectional image of FIG. The omnidirectional panoramic image is an image obtained by pasting the omnidirectional image into a spherical shape (3D model).

  The process of generating the omnidirectional panoramic image is realized by an API (Application Programming Interface) such as OpenGL ES (OpenGL (registered trademark) for Embedded Systems). Specifically, in the omnidirectional panoramic image, the pixels of the omnidirectional image are divided into triangles. Then, the vertices P of the triangles (hereinafter referred to as “vertices P”) are connected and generated as a polygon.

<Example of selecting spherical panoramic image> (Step S04)
In step S04 of FIG. 6, the smartphone 2 inputs an operation for selecting an omnidirectional panoramic image from the user. Specifically, in step S04, the smartphone 2 displays a plurality of thumbnail images in a thumbnail image format as a reduced image of the omnidirectional panoramic image generated in step S03.

  For example, when a plurality of omnidirectional panoramic images are stored in the smartphone 2, the smartphone 2 outputs a list of thumbnail images from the plurality of omnidirectional panoramic images. Then, a user operation for selecting one thumbnail image from the list of thumbnail images is input to the smartphone 2. The omnidirectional panoramic image selected in step S04 is processed, and processing is performed.

  Note that step S04 may be omitted when there is only one type of omnidirectional image, or depending on settings or the like. Alternatively, the omnidirectional image may be output as a thumbnail image in a list first. Then, one of the thumbnail images may be selected, and an omnidirectional panoramic image may be generated in step S03 based on the selected omnidirectional image.

<Setting of Predetermined Area and Output Example of Display Image> (Step S05)
In step S05 of FIG. 6, the smartphone 2 outputs an area (hereinafter referred to as “predetermined area”) to be output as an image from the user in the range indicated by the omnidirectional image (in this example, all directions). Set. An image that the smartphone 2 outputs to the user is referred to as an “output image”. The output image is an image showing a predetermined area. In addition, a display image that is set with a predetermined area and is first output by the smartphone 2 is referred to as an “initial image”.

  In step S05, the smartphone 2 generates an initial image.

  FIG. 9 is a diagram for explaining an example of an initial image according to an embodiment of the present invention. FIG. 9A is a diagram illustrating an XYZ axis three-dimensional coordinate system as an example of an initial image.

  The smartphone 2 generates a display image from the viewpoint from the virtual camera 3 with the predetermined area T as a range where the “virtual camera 3” captures. The initial position of the virtual camera is the position of the origin (0, 0, 0) of the coordinate system. Furthermore, the omnidirectional panoramic image is expressed as a solid sphere CS. In the initial state, the virtual camera 3 corresponds to the viewpoint of the user who views the panoramic image from the origin with respect to the panoramic image of the solid sphere CS.

  Next, FIG. 9B is a three-view diagram illustrating an example of the predetermined region T. In the initial state, the virtual camera 3 is located at the origin. Further, FIG. 9C is a projection view of an example of the predetermined region T. The virtual camera 3 projects the predetermined area T onto the solid sphere CS.

  FIG. 9D is a diagram illustrating an example of a position and a viewing angle for specifying a predetermined region. The predetermined area T is determined by the viewpoint position (X, Y, Z) corresponding to the three-dimensional coordinates of the virtual camera 3 and the viewing angle α of the virtual camera 3. When the predetermined area T is determined from the viewing angle α, the two-dimensional coordinates of the center point CP of the predetermined area T are determined as the midpoint of the diagonal field angle 2L.

  Next, the distance from the virtual camera 3 to the center point CP is expressed by the following equation (1).

The predetermined area T is determined by the initial setting. Then, based on the predetermined area T, an initial image is generated. For example, the initial setting of the viewpoint position (X, Y, Z) and the viewing angle α is set by a user or the like such as (X, Y, Z, α) = (0, 0, 0, 34). .

  When an operation for changing the angle of view, a so-called zoom operation, is input, the smartphone 2 performs zoom processing. The zoom process is a process for enlarging or reducing a predetermined area based on an operation by the user and generating a display image based on the changed predetermined area.

  The operation amount input by the zoom operation by the user is referred to as “change amount dz”. First, when a zoom operation is input, the smartphone 2 acquires a change amount dz. And the smart phone 2 calculates following (2) Formula based on the variation | change_quantity dz.

Note that “α” in the above equation (2) is the viewing angle α shown in FIG. Further, “m” shown in the above equation (2) is a coefficient for adjusting the zoom amount, and is a value set in advance. Furthermore, “α0” in the above equation (2) is the initial value of the viewing angle α in the initial state, the so-called viewing angle α.

  Next, the smartphone 2 determines the predetermined region T using the viewing angle α calculated based on the above equation (2) as a projection matrix.

  Note that after the operation of inputting the change amount dz is performed, when the user further performs a zoom operation that becomes the change amount dz2, the smartphone 2 calculates the following expression (3).

The viewing angle α in the above equation (3) is calculated based on the total value of the amounts of change input by each operation. Even if a plurality of operations are performed, the smartphone 2 can maintain a consistent operability by calculating the viewing angle α.

  The zoom process is not limited to the process based on the above formula (2) or the above formula (3). For example, the zoom process may be realized by combining the change of the viewing angle α and the viewpoint position of the virtual camera 3. Specifically, the following zoom processing may be performed.

  FIG. 10 is a diagram for explaining an example of another zoom process according to an embodiment of the present invention. The solid sphere CS shown in FIG. 10 is the same as the solid sphere CS shown in FIG. 9, and the radius of the solid sphere CS will be described as “1”.

  First, the origin shown in FIG. 10 is the initial position of the virtual camera 3. The virtual camera 3 changes the position by moving the optical axis. The optical axis is the same as the Z axis shown in FIG. The moving amount d of the virtual camera 3 is indicated by the distance moved from the origin. For example, in the initial state where the virtual camera 3 is located at the origin, the movement amount d is “0”.

  Based on the movement amount d of the virtual camera 3 and the viewing angle α, a range that becomes the predetermined region T shown in FIG. The angle of view ω is an angle of view of d = 0 when the virtual camera 3 is located at the origin. When d = 0, the angle of view ω and the viewing angle α coincide.

  On the other hand, when the virtual camera 3 is away from the origin and the value of d is larger than “0”, the angle of view ω and the viewing angle α are in different ranges. Another zoom process is a process for changing the range of the angle of view ω.

  FIG. 11 is a table for explaining an example of another zoom process according to the embodiment of the present invention. The explanatory table 4 shows an example in which the range of the angle of view ω is 60 ° to 300 °. The smartphone 2 determines which of the viewing angle α and the movement amount d of the virtual camera 3 is preferentially changed based on the zoom designation value ZP.

  The “range” is a range determined based on the zoom designation value ZP. The “output magnification” is an output magnification of an image calculated based on an image parameter determined by another zoom process. Further, the “zoom designation value ZP” is a value corresponding to the angle of view to be output.

  In another zoom process, the process for determining the movement amount d and the viewing angle α is changed based on the zoom designation value ZP. Specifically, the process of performing another zoom process is determined as one of the four methods in the explanatory table 4 based on the zoom designation value ZP. The range of the zoom designation value ZP is divided into four ranges “A to B”, “B to C”, “C to D”, and “D to E”.

  The “view angle ω” is the view angle ω corresponding to the image parameter determined by another zoom process. Furthermore, “parameter to be changed” is a description for explaining parameters to be changed by four methods based on the zoom designation value ZP. “Remarks” is a note on “parameters to be changed”.

  “ViewWH” is a value indicating the width or height of the output area. For example, when the output area is horizontally long, “viewWH” indicates a width value. On the other hand, when the output area is vertically long, “viewWH” indicates a height value. That is, “viewWH” is a value indicating the size of the output area in the longitudinal direction.

  “ImgWH” is a value indicating the width or height of the output image. For example, when the output area is horizontally long, “viewWH” indicates the value of the width of the output image. On the other hand, when the output area is vertically long, “viewWH” indicates the height value of the output image. That is, “viewWH” is a value indicating the size of the output image in the longitudinal direction.

  “ImgDeg” is a value indicating the angle of the display range of the output image. Specifically, when indicating the width of the output image, “imgDeg” is 360 °. On the other hand, when indicating the height of the output image, “imgDeg” is 180 °.

  FIG. 12 is a diagram for explaining an example of a “range” of another zoom process according to an embodiment of the present invention. An example of a “range” displayed on an image and an image when another zoom process is performed is shown. The zoom-out will be described using the example shown in FIG. In addition, the left figure in each figure of FIG. 12 shows an example of the image output. The right diagram in each figure of FIG. 12 is a diagram showing an example of the state of the virtual camera 3 when it is output, in a model diagram similar to FIG.

  FIG. 12A shows an example of an output image and “range” when a zoom designation value ZP in which “range” in the explanatory table 4 is “A to B” is input. The viewing angle α of the virtual camera 3 is fixed as α = 60 °. Furthermore, it is assumed that the zoom designation value ZP is “A to B” and the movement amount d of the virtual camera 3 is changed in a state where the viewing angle α is fixed. An example in which the movement amount d of the virtual camera 3 is changed so as to increase in a state where the viewing angle α is fixed will be described. When the movement amount d increases, the angle of view ω increases. That is, zoom-out processing can be realized by setting the zoom designation value ZP to “A to B”, fixing the viewing angle α, and increasing the movement amount d of the virtual camera 3. When the zoom designation value ZP is “A to B”, the moving amount d of the virtual camera 3 is from “0” to the radius of the solid sphere CS. Specifically, since the radius of the solid sphere CS is “1”, the movement amount d of the virtual camera 3 has a value of “0 to 1”. Further, the moving amount d of the virtual camera 3 is a value corresponding to the zoom designation value ZP.

  Next, FIG. 12B shows an example of an output image and “range” when a zoom designation value ZP in which “range” in the explanatory table 4 is “B to C” is input. Note that “B to C” is a value for which the zoom designation value ZP is larger than “A to B”. The zoom designation value ZP is “B to C”, and the movement amount d of the virtual camera 3 is fixed to a value at which the virtual camera 3 is positioned on the outer edge of the solid sphere CS. The movement amount d of the virtual camera 3 in FIG. 12B is fixed to “1” which is the radius of the solid sphere CS. Further, it is assumed that the viewing angle α is changed in a state where the zoom designation value ZP is “B to C” and the movement amount d of the virtual camera 3 is fixed. As shown in FIGS. 12A to 12B, the angle of view ω increases. That is, zoom-out processing can be realized by setting the zoom designation value ZP to “B to C”, fixing the movement amount d of the virtual camera 3 and increasing the viewing angle α. When the zoom designation value ZP is “B to C”, the viewing angle α is calculated as “ω / 2”. When the zoom designation value ZP is “B to C”, the viewing angle α ranges from “60 °”, which is a fixed value when “A to B”, to “120 °”. Become.

  When the zoom designation value ZP is “A to B” or “B to C”, the angle of view ω matches the zoom designation value ZP. When the zoom designation value ZP is “A to B” and “B to C”, the angle of view ω increases.

  FIG. 12C shows an example of an output image and “range” when a zoom designation value ZP in which “range” in the explanatory table 4 is “C to D” is input. Note that “C to D” is a value for which the zoom designation value ZP is larger than “B to C”. The zoom designation value ZP is “C to D”, and the viewing angle α is fixed to α = 120 °. When the zoom designation value ZP is “C to D” and the movement amount d of the virtual camera 3 is changed while the viewing angle α is fixed, the angle of view ω is widened. The movement amount d of the virtual camera 3 is calculated by an expression based on the zoom designation value ZP in the explanatory table 4. When the zoom designation value ZP is “C to D”, the moving amount d of the virtual camera 3 is changed to the maximum display distance dmax1. The maximum display distance dmax1 is an output area in the smartphone 2 and is a distance at which the solid sphere CS can be displayed to the maximum. The output area is the size of the screen on which the smartphone 2 outputs an image or the like. Further, the maximum display distance dmax1 is calculated by the following equation (4) in the state shown in FIG.

Note that “viewW” and “viewH” in the above formula (4) are values indicating the width and height of the output area in the smartphone 2, respectively. The maximum display distance dmax1 is calculated based on the output area in the smartphone 2, that is, the values of “viewW” and “viewH”.

  FIG. 12D shows an example of an output image and “range” when a zoom designation value ZP in which “range” in the explanatory table 4 is “D to E” is input. Note that “D to E” is a value for which the zoom designation value ZP is larger than “C to D”. The zoom designation value ZP is “D to E”, and the viewing angle α is fixed to α = 120 °. As shown in FIG. 12D, it is assumed that the zoom designation value ZP is “C to D” and the movement amount d of the virtual camera 3 is changed in a state where the viewing angle α is fixed. Further, the moving amount d of the virtual camera 3 is changed to the limit display distance dmax2. The limit display distance dmax2 is a distance where the solid sphere CS is inscribed in the output area of the smartphone 2. Specifically, the limit display distance dmax2 is calculated by the following equation (5). The limit display distance dmax2 is the state shown in FIG.

The limit display distance dmax2 in the above equation (5) is calculated based on the values of “viewW” and “viewH” that are output areas in the smartphone 2. The limit display distance dmax2 indicates a limit value that can increase the moving amount d of the virtual camera 3 in the maximum range that the smartphone 2 can output. The smartphone 2 limits the input value so that the zoom designation value ZP falls within the range of the explanatory table 4, that is, the value of the movement amount d of the virtual camera 3 is equal to or less than the limit display distance dmax2. Also good. Due to this limitation, the smartphone 2 is in a state where the output image is fitted to the screen which is the output area, or in a state where the image can be output to the user at a predetermined output magnification, and zoom-out can be realized. And the process of "DE" can make the smart phone 2 recognize that the image currently output to the user is a spherical panorama.

  When the zoom designation value ZP is “C to D” or “D to E”, the angle of view ω is different from the zoom designation value ZP. In addition, the angle of view ω is continuous between the ranges shown in the explanatory table 4 and FIG. 12, but the angle of view ω does not need to increase uniformly by zooming out to the wide angle side. For example, when the zoom designation value ZP is “C to D”, the angle of view ω increases with the movement amount d of the virtual camera 3. On the other hand, when the zoom designation value ZP is “D to E”, the angle of view ω decreases with the movement amount d of the virtual camera 3. This decrease is because the outer area of the solid sphere CS is reflected. When designating a wide field of view where the zoom designation value ZP is 240 ° or more, the smartphone 2 outputs an image with less discomfort to the user by changing the movement amount d of the virtual camera 3, and sets the angle of view ω. Can be changed.

  In addition, when the zoom designation value ZP is changed in the wide-angle direction, the angle of view ω is often widened. When the angle of view ω is wide, the smartphone 2 fixes the viewing angle α of the virtual camera 3 and increases the movement amount d of the virtual camera 3. By fixing the viewing angle α of the virtual camera 3, the smartphone 2 can reduce an increase in the viewing angle α of the virtual camera 3 and output an image with less distortion.

  When the viewing angle α of the virtual camera 3 is fixed and the smartphone 2 increases the movement amount d of the virtual camera 3, that is, moves the virtual camera 3 away from the smartphone 2, the smartphone 2 provides the user with a wide-angle display feeling of opening. Can be given. In addition, when moving the virtual camera 3 away, it is similar to the movement when a human confirms a wide range, and thus the smartphone 2 can realize zoom-out with little discomfort.

  When the zoom designation value ZP is “D to E”, the angle of view ω decreases as the zoom designation value ZP changes in the wide-angle direction. By reducing the angle of view ω, the smartphone 2 can give the user a sense of moving away from the solid sphere CS, and can output an image with less discomfort.

  By another zoom process of the explanatory table 4 illustrated in FIG. 11, the smartphone 2 can output an image with less discomfort to the user.

  Note that the smartphone 2 is not limited to changing only the movement amount d or the viewing angle α of the virtual camera 3 described in the explanatory table 4. That is, the smartphone 2 only needs to change the movement amount d or the viewing angle α of the virtual camera 3 preferentially in the explanatory table 4, and even if the fixed value is changed to a sufficiently small value for adjustment. Good. The smartphone 2 is not limited to zooming out. The smartphone 2 may zoom in.

<Example of voice input at multiple locations> (step S06)
In step S06 in FIG. 6, the photographing apparatus 1 inputs sound with a plurality of microphones. For example, voices are input at four locations of the microphones 1HM1, 1HM2, 1HM3, and 1HM4 of the photographing apparatus 1. Desirably, the imaging device 1 processes audio input at a plurality of locations, and associates each input audio with the direction in which the input audio is generated, such as the Ambisonics B format. (Hereinafter referred to as “voice input data”). That is, when the voice input data is generated in the Ambisonics B format or the like, the smartphone 2 or the like can know the direction in which each input voice is generated by referring to the voice input data.

<Transmission example of voice input data> (step S07)
In step S <b> 07 of FIG. 6, the imaging device 1 transmits voice input data to the smartphone 2. Hereinafter, an example in which the smartphone 2 performs each process will be described.

<Example of Converting Audio Input Data to Generate Virtual Speaker Data> (Step S08)
In step S08 in FIG. 6, the smartphone 2 converts the voice input data and generates data (hereinafter referred to as “virtual speaker data”) for outputting voice to a plurality of virtual speakers that are virtually arranged. To do.

  FIG. 13 is a schematic diagram illustrating an arrangement example of virtual speakers according to an embodiment of the present invention. There is a user UR who becomes a listener at the viewpoint position (X, Y, Z). The user UR has the same concept as the virtual camera 3 shown in FIG.

  Then, the processing is performed assuming that speakers are virtually arranged one by one at a plurality of locations around the user UR as the viewpoint. Although the speaker arranged in FIG. 9 is a virtual speaker, the processing is performed on the assumption that the speaker is virtually arranged around the user UR, not an actually installed apparatus.

  When the photographing apparatus 1 is in a horizontal state, the virtual speakers are arranged at four positions on the front, back, left, and right of the user UR. When the photographing apparatus is used in a state other than the horizontal state, one virtual speaker may be added at the top and bottom in consideration of the vertical direction (Y-axis direction). That is, the virtual speakers are arranged at six locations. An example of 4 channels (channels) in which virtual speakers are arranged at four positions on the front, back, left, and right of the user UR will be described.

  Specifically, virtual speakers VSF, VSR, VSL, and VSB are respectively arranged in front, right hand direction, left hand direction, and rear of the user UR. With this arrangement, sound is output from the front, back, left and right of the user UR, and the information processing system can output realistic output sound.

  The method of outputting audio from a 4ch virtual speaker is described in “Ryuichi Nishimura (2014), 5, Ambisonics (<Special Issue> Stereophonic Technology) The Journal of the Institute of Television Information and Television Engineers, 68 (8). 620, http://ci.ni.ac.jp/naid/110009844051 ".

Specifically, First order Ambisonics is an example of virtual speaker data (L, R, F, B) as the sound output from the virtual speakers VSL, VSR, VSF, and VSB. Is calculated as follows.

L = W + k1 × Y
R = W−k1 × Y
F = W + k1 × X
B = W−k1 × X (6)

In the above equation (6), “k1” is a coefficient set in advance, and “X”, “Y”, and “W” are four data (X, Y, Z, and W) indicated by the B format of Ambisonics. W). Note that X, Y, and Z in the above equation (6) are data different from the viewpoint position (X, Y, Z).

  That is, audio input data such as the Ambisonics B format is converted into virtual speaker data according to the above equation (6).

  FIG. 14 is a schematic diagram illustrating a first example in which the arrangement of virtual speakers according to an embodiment of the present invention is changed. Compared to FIG. 13, in FIG. 14, the user UR is rotated 90 ° in the left direction RL with the origin at the center. The virtual speaker VSF is also arranged at a position rotated 90 ° in the left direction RL in accordance with the rotation of the user UR, and is arranged in front of the user UR after the rotation. Similarly, the virtual speakers VSR, VSL, and VSB are arranged in the right hand direction, the left hand direction, and the rear side of the user UR after the rotation. For example, the sound output from the virtual speaker VSF in FIG. 13 is output from the virtual speaker VSR in FIG.

  In addition, when zooming in or zooming out is performed, audio may be output in accordance with zoom processing.

  FIG. 15 is a schematic diagram illustrating a second example of the arrangement of virtual speakers according to an embodiment of the present invention. Compared to FIG. 13, in FIG. 14, the user UR moves to the rear SZ in the optical axis direction (Z axis) with the origin at the center. Accordingly, the position (X, Y, Z) of the visual field in FIG. 14 is changed from FIG. 13 only in the coordinate value of Z.

  If the initial state in the case shown in FIG. 13 is a state corresponding to FIG. 12A, the changed state shown in FIG. 15 corresponds to, for example, FIG. 12B or FIG. State.

  When the predetermined area T is changed, even if the viewing angle α is constant, the range of the predetermined area T is widened, and the angle of view is ω <ω2. Therefore, the display image is an image showing a wider range than the setting of the angle of view ω.

  The virtual speakers VSF, VSR, VSL, and VSB are arranged at the same position before and after the change. Then, the information processing system 10 decreases the volume of the sound output from the virtual speaker VSF and increases the volume of the sound output from the virtual speaker VSR. Note that the information processing system 10 may also change the volume of sound output from the virtual speakers VSB and VSL.

  In this case, the user UR can hear a large amount of sound generated in the rear and a small amount of sound generated in the front. Therefore, the information processing system 10 can output the same sound as when the user UR actually moves in the backward SZ, and can enhance the sense of reality.

  Further, the information processing system 10 may limit the sound to be output based on the predetermined area T. That is, the information processing system 10 may output a sound input from a direction corresponding to the range of the predetermined region T. In this case, the information processing system 10 can output a sound according to the subject shown in the display image, and can enhance the sense of reality.

  FIG. 16 is a schematic diagram illustrating a third example of the arrangement of virtual speakers according to an embodiment of the present invention. The change of the predetermined area T shown in FIG. 16 is an example when the change shown in FIG. 14 is made after the change shown in FIG. The change of the predetermined region T may be a change combining rotation and position change. In FIG. 16, the arrangement of the virtual speakers is changed as in FIG. 14, and the volume of the sound output from the virtual speakers is changed as in FIG.

  FIG. 17 is a schematic diagram illustrating a fourth example of the arrangement of virtual speakers according to an embodiment of the present invention. FIG. 17 is different from FIG. 13 in that the user UR moves the first distance SZ1 to the rear SZ in the optical axis direction (Z axis) with the origin at the center. Accordingly, the position (X, Y, Z) of the visual field in FIG. 16 is changed from FIG. 13 only in the coordinate value of Z.

  If the initial state shown in FIG. 13 is a state corresponding to FIG. 12A, the changed state shown in FIG. 15 is, for example, the state shown in FIG.

  The virtual speakers VSF, VSR, VSL, and VSB are arranged at the same position before and after the change. However, the virtual speakers VSF, VSR, VSL, and VSB are arranged in front of the user UR.

  In this case, since the virtual speaker is arranged in front of the user UR, the user UR can hear sound from the front. Therefore, the information processing system 10 can output the same sound as when the user UR has actually moved backward by the first distance SZ1 and can give a sense of realism.

  FIG. 18 is a schematic diagram illustrating a fifth example of the arrangement of virtual speakers according to an embodiment of the present invention. Compared with the example shown in FIG. 17, FIG. 18 shows that the user UR moves from the position of the first distance SZ1 to the second distance SZ2 that is the rear SZ in the optical axis direction (Z axis) with the origin at the center. ing. Accordingly, the position (X, Y, Z) of the visual field in FIG. 18 is changed from FIG.

If the initial state shown in FIG. 13 is a state corresponding to FIG. 12A, the changed state shown in FIG. 15 is, for example, the state shown in FIG.
And the arrangement | positioning of virtual speaker VSF, VSR, VSL, and VSB is made into the same position before and behind a change, for example. However, the virtual speakers VSF, VSR, VSL, and VSB are arranged in front of the user UR.

  Based on the predetermined region T, that is, the position of the viewpoint (X, Y, Z) and the viewing angle α, the information processing system 10 uses the above-mentioned formula (6) as virtual speaker data for outputting audio input data to the virtual speaker. And so on.

<Example of Converting Virtual Speaker Data to Generate Audio Output Data> (Step S09)
In step S09 in FIG. 6, the smartphone 2 converts the virtual speaker data to generate audio output data. For example, the virtual speaker data (L, R, F, B) generated by the above equation (6) is converted as shown in the following equation (7) and becomes audio output data (L2, R2).

L2 = k4 × (L + k2 × F + k3 × B)
R2 = k4 × (R + k2 × F + k3 × B) (7)

In the above equation (7), “k2,” “k3,” and “k4” are coefficients determined based on the position (X, Y, Z) of the viewpoint.

  Note that the virtual speaker data (L, R, F, B) may be converted by an acoustic transfer function or the like.

  For example, when the speaker 2HS1 and the speaker 2HS2 output sound to earphones or headphones, the virtual speaker data (L, R, F, B) is converted by a head-related transfer function (HRTF (Head-Related Transfer Function)) or the like. It is desirable to be done.

  On the other hand, when audio is output to the speakers 2HS1 and 2HS2, the virtual speaker data (L, R, F, B) is converted by a room transfer function (RTF (Room Transfer Function)) or the like. Is desirable.

  The sound may be deformed under the influence of reflection by a wall or diffraction by an obstacle. Therefore, when the room transfer function is used, the information processing system 10 can reflect the characteristics of deformation due to reflection or the like in the output sound.

  Further, in the above equation (7) and the like, even if a delay is set in order to give a phase difference between “F” and “B”, between “L” and “R”, or both of them, Good. When the delay is set, the sense of distance can be emphasized.

<Example of Output Audio Based on Audio Output Data> (Step S10)
In step S10 of FIG. 6, the smartphone 2 outputs an output sound based on the sound output data. Specifically, when the sound output data (L2, R2) is generated by the above (7) or the like, the smartphone 2 transmits the sound indicated by the sound output data (L2, R2) to the speaker 2HS1 and the speaker 2HS2 (FIG. Output to 5).

  In the figure, the process for generating and outputting the display image and the process related to the sound are described separately, but the information processing system 10 may perform these processes in parallel.

<Functional configuration example>
FIG. 19 is a functional block diagram illustrating a functional configuration example of an information processing system according to an embodiment of the present invention. The information processing system 10 includes an audio input unit 10F1, a first conversion unit 10F2, a setting unit 10F3, a photographing unit 10F4, a second conversion unit 10F5, an audio output unit 10F6, and an image output unit 10F7.

  The voice input unit 10F1 performs a voice input procedure in which the input voice VIN is input at a plurality of locations from the range captured by the shooting unit 10F4 to generate voice input data DIN. For example, the voice input unit 10F1 is realized by the microphones 1HM1, 1HM2, 1HM3, and 1HM4 (FIG. 4).

  The first conversion unit 10F2 is an example of a first conversion unit, and converts the audio input data DIN based on the predetermined area T (FIG. 9) to generate virtual speaker data DVS. I do. For example, the first conversion unit 10F2 is realized by an arithmetic device such as the CPU 2H5 (FIG. 5).

  The setting unit 10F3 performs a setting procedure for setting the viewpoint position (X, Y, Z) that identifies the predetermined region T and the viewing angle α of the viewpoint. For example, the setting unit 10F3 is realized by the input / output device 2H3 (FIG. 5) or the like.

  The photographing unit 10F4 performs a photographing procedure for photographing a plurality of images. For example, the photographing unit 10F4 is realized by the photographing apparatus 1 shown in FIG.

  The second conversion unit 10F5 performs a second conversion procedure for converting the virtual speaker data DVS to generate the audio output data DOUT. For example, the second conversion unit 10F5 is realized by an arithmetic device such as the CPU 2H5 (FIG. 5).

  The audio output unit 10F6 performs an audio output procedure for outputting the output audio VOUT from a plurality of locations based on the audio output data DOUT. For example, the audio output unit 10F6 is realized by the speaker 2HS1, the speaker 2HS2 (FIG. 5), and the like.

  The image output unit 10F7 performs an image output procedure for outputting the display image IMGOUT based on a plurality of images captured by the imaging unit 10F4. For example, the image output unit 10F7 is realized by the input / output device 2H3 (FIG. 5) or the like.

  With the functional configuration of FIG. 19, the information processing system 10 first generates an omnidirectional image from a plurality of hemispherical images and the like photographed by the photographing unit 10F4 (FIG. 3). Then, the information processing system 10 inputs the input voice VIN at a plurality of locations by the voice input unit 10F1. When the input voice VIN is input at a plurality of locations, the information processing system 10 generates voice input data DIN such as an Ambisonics B format.

  The setting unit 10F3 sets the viewpoint position (X, Y, Z) and the viewing angle α of the viewpoint, which are parameters for specifying the predetermined region T. Based on this setting, as shown in FIG. 9, the range output to the display image IMGOUT is determined from the entire range indicated by the omnidirectional image.

  The first conversion unit 10F2 generates audio input data DIN that matches the display image IMGOUT determined by the predetermined region T, that is, the position (X, Y, Z) of the viewpoint and the viewing angle α of the viewpoint. Specifically, the user UR inputs to the information processing system 10 an operation for zooming, parallel moving, or rotating the predetermined area T. For example, when a zoom operation is input, the predetermined area T and the display image IMGOUT are changed as shown in FIG.

Therefore, the information processing system 10 adjusts the viewpoint position (X, Y, Z) and the viewing angle α of the viewpoint, and virtual speakers such as virtual speakers VSF, VSR, VSL, and VSB as shown in FIGS. Is arranged around the user UR.
The first conversion unit 10F2 calculates the outputs from the virtual speakers VSF, VSR, VSL, and VSB using the above equation (6) and the like. Based on this calculation, virtual speaker data (L, R, F, B) is generated.

  In the case of virtual speaker data (L, R, F, B), the information processing system 10 can output the output sound VOUT in accordance with the display image IMGOUT, and therefore can output sound with a sense of presence. For example, the information processing system 10 can output the output sound VOUT that is a three-dimensional sound from the direction matching the display image IMGOUT.

  The second conversion unit 10F5 converts the virtual speaker data (L, R, F, B) to generate audio output data DOUT. Based on this conversion, audio output data DOUT indicating output audio VOUT suitable for hardware or the like that implements audio output unit 10F6 is generated.

  Moreover, when there exists state sensor 2H4 (FIG. 5) etc., the information processing system 10 can detect the attitude | position of the smart phone 2. FIG. For example, the arrangement of hardware that implements the audio output unit 10F6 may change depending on whether the smartphone 2 is in the portrait orientation or the landscape orientation. In the case of the portrait orientation, the speakers 2HS1 and 2HS2 are also arranged in the portrait orientation. On the other hand, in the landscape orientation, the speakers 2HS1 and 2HS2 are also arranged in the landscape orientation. Therefore, the information processing system 10 detects the arrangement of hardware using the state sensor 2H4 and the like. Then, the information processing system 10 may generate the audio output data DOUT in accordance with the hardware arrangement.

  Therefore, since the sound transfer function such as HRTF or RTF can be converted into a sound suitable for hardware based on the conversion of the second conversion unit 10F5, a sound with a sense of reality can be output.

  In addition, based on the conversion of the second conversion unit 10F5, even if the hardware that realizes the audio output unit 10F6 is 5.1 ch or the like, it is possible to output realistic sound.

  Note that the voice input unit 10F1 preferably inputs the input voice VIN at more places than the voice output unit 10F6. For example, the number of microphones is preferably larger than the number of speakers. In this case, the information processing system 10 can improve the sound quality of the output sound VOUT.

  When the ambisonics method is used, a phase difference can be expressed. Further, the information processing system 10 can express the difference in the volume level of the sound having a phase difference by the functional configuration of FIG. Specifically, a sound having a left-right phase difference ILD (interaural level difference) and an interaural time difference ITD (interaural time difference) or phase difference IPD (interaural phase difference) A feeling can be heightened.

  In addition, when the input voice VIN is input at a plurality of locations, and the voice level difference between the plurality of input voices VIN is different from a predetermined value or more, the information processing system 10 inputs the input voice input with the microphone having the largest dynamic range. Select VIN. Therefore, the information processing system 10 can acquire a monaural signal and can detect an abnormality in the input voice VIN.

  In addition, the abnormality may be obtained from a correlation coefficient or the like. In particular, when the microphone is omnidirectional, there is little difference between the input voices VIN, so the correlation coefficient is difficult to make a difference. Therefore, the information processing system 10 can detect an abnormality based on the correlation coefficient.

<Other embodiments>
The embodiment according to the present invention may be realized by a program written in a programming language or the like. That is, the embodiment according to the present invention may be realized by a program for causing a computer such as an information processing apparatus to execute an information processing method. The program can be stored and distributed in a recording medium such as a flash memory, an SD (registered trademark) card, or an optical disk. The program can be distributed through a telecommunication line such as the Internet.

  In the embodiment according to the present invention, part or all of the processing may be processed and realized by a programmable device (PD) such as a field programmable gate array (FPGA). Further, in the embodiment according to the present invention, a part or all of the processing may be realized by being processed by ASIC (Application Specific Integrated Circuit).

  Further, the information processing apparatus is not limited to one information processing apparatus, and may be configured by a plurality of information processing apparatuses. That is, the embodiment according to the present invention may be realized by an information processing system having one or more information processing apparatuses.

  As mentioned above, although the preferable Example of this invention was explained in full detail, this invention is not limited to the specific embodiment which concerns. That is, various modifications and changes can be made within the scope of the gist of the present invention described in the claims.

10 Information processing system 1 Shooting device 2 Smartphone VIN Input audio VOUT Output audio VSF, VSR, VSL, VSB Virtual speaker DIN Audio input data DOUT Audio output data

Japanese Patent No. 6044328

Claims (8)

An information processing system having an information processing apparatus connected to an imaging apparatus that captures a plurality of images,
A voice input unit for inputting a plurality of voices;
An image output unit that outputs a display image based on the plurality of images;
A setting unit for setting a predetermined area to be output to the display image;
A first conversion unit that converts a plurality of voices input to the voice input unit based on the predetermined area;
An information processing system comprising: an audio output unit that outputs a plurality of sounds converted by the first conversion unit.   The information processing system according to claim 1, wherein the setting unit sets a position determined by three-dimensional coordinates and a viewing angle that is an angle of view of the virtual camera as the predetermined area.   The information processing system according to claim 1, wherein a direction in which the sound is generated is associated with the sound input to the sound input unit. The voice input unit inputs the voice at four or more locations,
The information processing system according to claim 1, wherein the sound output unit outputs the converted sound at two or more locations.   The information processing system according to claim 4, wherein the voice input unit inputs the plurality of voices at more places than the voice output unit.   3. The information processing system according to claim 2, wherein the first conversion unit changes an arrangement of a virtual speaker or an output from the virtual speaker when the position, the viewing angle, or a combination thereof is changed. An information processing method performed by an information processing system having an information processing apparatus connected to an imaging apparatus that captures a plurality of images,
A voice input procedure in which the information processing system inputs a plurality of voices;
An image output procedure in which the information processing system outputs a display image based on the plurality of images;
A setting procedure in which the information processing system sets a predetermined area to be output to the display image;
A first conversion procedure in which the information processing system converts a plurality of voices input by the voice input procedure based on the predetermined area;
An information processing method, wherein the information processing system includes a sound output procedure for outputting a plurality of sounds converted by the first conversion procedure. A program for causing a computer having an information processing apparatus connected to an imaging apparatus for capturing a plurality of images to execute an information processing method,
A voice input procedure in which the computer inputs a plurality of voices;
An image output procedure in which the computer outputs a display image based on the plurality of images;
A setting procedure in which the computer sets a predetermined area to be output to the display image;
A first conversion procedure in which the computer converts a plurality of voices input by the voice input procedure based on the predetermined area;
A program for causing the computer to execute a sound output procedure for outputting a plurality of sounds converted by the first conversion procedure.