JP2018182528A - Omnidirectional camera and audio processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent or reduce misalignment between an omnidirectional image and a three-dimensional audio field.SOLUTION: The omnidirectional camera includes an optical system, an element for outputting an image as image data, an acceleration sensor, an image processing circuit for processing the image data, a plurality of microphones, and an audio processing circuit for processing audio data. The image processing circuit corrects an inclination of an image caused by an inclination of a camera on the basis of acceleration, and the audio processing circuit corrects an inclination of an audio field caused by an inclination of the camera on the basis of the acceleration.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to an omnidirectional camera and an audio processing method, and more particularly to sound field correction of a digital camera.

  As in Patent Document 1, there is known a technology for correcting a omnidirectional image in accordance with the tilt of a digital camera in a digital camera.

  As in Patent Document 2, there is known a technology for correcting the directivity of sound in accordance with the tilt of a digital camera in a digital camera.

JP, 2016-149733, A JP, 2016-163181, A

  In the omnidirectional camera that acquires the omnidirectional image, the vertical direction that is the reference is detected, and the acquired image is corrected according to the tilt of the camera from the vertical direction, and the omnidirectional image without discomfort A technique for making it possible to visually recognize is known in Patent Document 1 and the like. In order to enhance the realism of the omnidirectional image, it is conceivable to provide a microphone in the camera to pick up a three-dimensional sound field. However, when the sound is simply picked up, since the tilt of the camera is not taken into consideration, the omnidirectional image and the three-dimensional sound field shift. Therefore, the viewer may feel discomfort due to the deviation.

  For example, consider a case where a photographer wears an omnidirectional camera on his head and shoots while moving. By correcting the image as the photographer changes the tilt and orientation of the head, it is possible to obtain an image similar to that facing the front. In this case, if the sound field is not corrected, the sound field changes according to the tilt and the direction of the head, so the image and the sound field do not match, and the viewer feels uncomfortable.

  An optical system, an element for outputting an image incident through the optical system as image data, an acceleration sensor for detecting a signal representing acceleration in three axial directions, an image processing circuit for processing the image data, and a sound field A omnidirectional camera comprising: a plurality of microphones that receive voice and outputting voice data representing the received voice; and a voice processing circuit that processes the voice data, wherein the image processing circuit is configured to Based on the inclination of the image caused by the inclination of the camera, the sound processing circuit corrects the inclination of the sound field caused by the inclination of the camera based on the acceleration.

  An optical system, an element for outputting an image incident through the optical system as image data, an angular velocity sensor for detecting a signal representing an angular velocity in three axial directions, an image processing circuit for processing the image data, and a sound field A omnidirectional camera comprising: a plurality of microphones that receive voice and outputting voice data representing the received voice; and a voice processing circuit that processes the voice data, wherein the image processing circuit The tilt or rotation of the image due to the rotation of the camera is corrected based on that, and the sound processing circuit corrects the tilt or the rotation of the sound field due to the rotation of the camera based on the angular velocity.

  An optical system, an element that outputs an image incident through the optical system as image data, an acceleration sensor that outputs a signal representing acceleration in three axial directions, an audio that constitutes a sound field, and an audio that represents received audio In a omnidirectional camera comprising a plurality of microphones outputting data, the method for processing speech with respect to the audio data, the method including correcting the inclination of the sound field caused by the inclination of the camera based on the acceleration. .

  An optical system, an element for outputting an image incident through the optical system as image data, an angular velocity sensor for outputting a signal representing an angular velocity in three axial directions, and an audio forming a sound field In a omnidirectional camera comprising a plurality of microphones outputting data, the method for processing sound with respect to the sound data, comprising: correcting inclination or rotation of the sound field due to rotation of the camera based on the angular velocity including.

  By performing correction of the omnidirectional image and correction of the three-dimensional sound field based on the information of the same acceleration sensor, it is possible to prevent or reduce the difference between the omnidirectional image and the three-dimensional sound field .

FIG. 7 is a schematic view of an omnidirectional camera according to an exemplary embodiment of the present disclosure. It is sectional drawing when the omnidirectional camera is seen from the top. It is a block diagram which shows the structure of the hardware of a omnidirectional camera. It is a functional block diagram of an omnidirectional camera. It is a functional block diagram of a posture calculation unit and a posture correction amount calculation unit. It is a flowchart which shows the process of an attitude | position calculation part and an attitude | position correction amount calculation part. It is a figure which shows the relationship between the angular velocity around the x, y, z axis which an angular velocity sensor outputs, the acceleration of the x, y, z axis direction which an acceleration sensor outputs, and gravity acceleration. It is a figure which shows the algorithm of a speech processing circuit.

  In the following description, the same reference numerals indicate the same components.

Overall Configuration of System FIG. 1 is a schematic view of an omnidirectional camera 100 according to an exemplary embodiment of the present disclosure. The omnidirectional camera 100, for example, performs imaging of the omnidirectional light substantially. The omnidirectional camera 100 typically includes a housing 110 having a shape and a size that can be photographed by a photographer by hand. An optical system 120 is provided on each of two parallel main planes of the housing 110. Although only one optical system 120 is visible in FIG. 1, another optical system 122 corresponding to the optical system 120 on the opposite side of the housing 110 (not shown in FIG. 1, which will be described later with reference to FIG. ) Exists. The case 110 is typically provided with a name plate 130. In the present specification, the surface on which the nameplate 130 is provided is referred to as a front surface.

  Four microphones 151 to 154 are provided on the housing 110. The number of microphones is arbitrary as long as it is plural, but is preferably four. The number of microphones may be five or more. The microphones 151 to 154 are directional microphones of the open rear type. The highest sensitivity direction (directional direction) of the microphones 151, 153, 154 is illustrated by an arrow. The direction of directivity of the microphone 152 is a direction from the back to the front of the drawing. The microphones 151 and 152 are arranged in an XY method in which the directivity direction intersects the optical axis of the optical system 120. The microphones 153 and 154 are also arranged in the XY method in which the direction of directivity intersects the optical axis of the optical system 122 provided on the back side of the optical system 120. The directional orientations and positions of the microphones 151 to 154 are not limited to those shown in FIG. For example, the microphones 151 to 154 may be an AB method opposite to the XY method.

  FIG. 2 is a cross-sectional view of the omnidirectional camera 100 as viewed from above. The optical systems 120 and 122 typically form images of regions substantially half of the whole sky and not substantially overlapping one another on the imaging elements 220 and 222, respectively. That is, the angle of view of the image obtained by the combination of the optical system 120 and the imaging device 220 and the combination of the optical system 122 and the imaging device 222 is substantially 180 °. For example, the imaging device 220 images a hemisphere with an angle θ of 0 ° to 180 ° when viewed from above the housing 110, and the imaging device 222 has an angle θ of 180 ° to 360 ° when viewed from above the housing 110. Capture the hemispheres of By combining the images captured by the imaging elements 220 and 222, an image of an omnidirectional sphere is obtained. The area of the image obtained by the combination of the optical system 120 and the imaging device 220 and the combination of the optical system 122 and the imaging device 222 may be minute or larger or smaller than half of the total celestial sphere. If the area of the image obtained by the combination of the optical system 120 and the imaging device 220 and the combination of the optical system 122 and the imaging device 222 is smaller than half of the total celestial sphere, the images of the two obtained regions are obtained It may be advantageous when synthesizing (also referred to as stitching). The imaging elements 220 and 222 are, for example, area type storage type photoelectric conversion elements such as a CMOS (complementary metal oxide semiconductor) sensor or a CCD (charge coupled device).

Hardware FIG. 3 is a block diagram showing the hardware structure of the omnidirectional camera 100. As shown in FIG. The omnidirectional camera 100 includes a CPU (central processing unit) 310, a ROM (read-only memory) 312, a RAM (random access memory) 314, an external memory 316, an acceleration sensor 430 described later with reference to FIG. 440, which are operatively connected via a bus 318.

  The signal processing circuit 320 receives the image signal A output by the imaging element 220. The signal processing circuit 320 performs necessary image correction on the received image signal A and transfers it to the evaluation circuit 330. The evaluation circuit 330 generates an evaluation A of the image signal A to perform at least one of automatic exposure and automatic white balance, and transfers the evaluation A to the CPU 310.

  The signal processing circuit 322 receives the image signal B output by the imaging element 222. The signal processing circuit 322 performs necessary image correction on the received image signal B, and transfers it to the evaluation circuit 332. The evaluation circuit 332 generates an evaluation B of the image signal B to perform at least one of automatic exposure and automatic white balance, and transfers the evaluation B to the CPU 310.

  The CPU 310 generates a parameter A for image correction of the image signal A based on the evaluation A, and generates a parameter B for image correction of the image signal B based on the evaluation B.

  The signal processing circuit 320 receives the parameter A and corrects the level based on the parameter A. The signal processing circuit 322 receives the parameter B and corrects the level based on the parameter B.

  The combining processing circuit 350 receives the corrected image signals A and B, combines them into one image, and outputs the combined image to, for example, the external memory 316. That is, the levels of the image signals A and B are adjusted as necessary and then combined.

  The evaluation circuit 330 divides the image into evaluation areas, and evaluates its brightness (in the case of automatic exposure) or color tone (in the case of automatic white balance). The evaluation circuit divides the image into evaluation areas and evaluates its brightness (for automatic exposure) or color tone (for automatic white balance).

  The evaluation of the brightness includes a forward light and a back light. For example, when the brightness of the image central area and the brightness of the image upper area in the area are substantially equal, it can be determined that the light is forward. Conversely, for example, when the brightness of the image upper region is significantly larger than the brightness of the image central region in the region, for example, it can be determined that the sun is in the region corresponding to the sky and the backlight is . As the evaluation of the color tone, the photographing mode can be changed based on which kind of light among sunlight, incandescent bulb light, fluorescent light and the like. The evaluation circuits 330 and 332 can similarly evaluate these brightness or color tones.

  The ratings A, B and the parameters A, B are not limited to the examples described above and may be any suitable ratings and parameters.

  Microphones 151-154 are coupled to bus 318 via analog-to-digital converters (A / D) 351-354, respectively. The microphones 151 to 154 output analog signals each representing the received voice. The A / Ds 351 to 354 convert analog signals output from the microphones 151 to 154 into digital signals, and output the digital signals to the bus 318. The CPU 310 corrects the sound field formed by the sound received by the microphones 151 to 154 in accordance with the tilt of the omnidirectional camera 100. Specifically, the tilt of the sound field caused by the tilt of the camera is corrected by audio processing described later with reference to FIG.

  FIG. 4 is a functional block diagram of the omnidirectional camera 100. As shown in FIG. The image sensor 420 and the image sensor 422 in FIG. 4 correspond to the imaging elements 220 and 222 in FIG. 2 respectively. In the following description, “camera” more specifically refers to an omnidirectional camera, and is an example of the omnidirectional camera 100. The image combining unit 426 combines the outputs from the image sensor 420 and the image sensor 422 to output an image of the omnidirectional sphere. The image combining unit 426 may be realized by the combining processing circuit 350.

  The sound processing circuit 480 corrects a sound field formed by the sound signals output from the microphones 151 to 154. More specifically, the audio processing circuit 480 generates the tilt of the omnidirectional camera 100 by correcting the audio signal according to the output of the camera posture calculation unit 450 (representing the tilt of the omnidirectional camera 100). The inclination of the sound field is reduced and is output as a stereo signal 482. Thus, the omnidirectional image is corrected according to the tilt of the omnidirectional camera 100, and the sound field collected by the microphones 151 to 154 is also corrected.

  The omnidirectional camera 100 includes an attitude calculation unit 450, an attitude correction amount calculation unit 460, and an attitude correction amount recording unit 470. The posture calculation unit 450, the posture correction amount calculation unit 460, the posture correction amount recording unit 470, and the audio processing circuit 480 can be typically realized by a combination of the CPU 310 and software, but not limited thereto. It may be realized by only the wear.

The angular velocity sensor 430 outputs angular velocities g _x , g _y and g _z [rad / sec] around the x, y and z axes. The acceleration sensor 440, x, y, acceleration a _x in the z-axis direction, a _y, and outputs the _{a z [G] (1 [} G] ≒ 9.8 [m / s 2]). The angular velocity sensor 430 and the acceleration sensor 440 are provided in the housing 110.

  In the state where the omnidirectional camera 100 is at rest, the acceleration sensor 440 outputs a component in which the gravitational acceleration is decomposed in each axial direction. Thus, the attitude of the omnidirectional camera 100 can be accurately estimated.

  On the other hand, when the omnidirectional camera 100 is rotated about the direction of gravity, the output of the acceleration sensor does not change. As a result, even if only the output of the acceleration sensor 440 is used, the attitude of the omnidirectional camera 100 can not be accurately estimated. In order to compensate for this, posture detection when rotating about the direction of gravity is performed based on the output of the angular velocity sensor 430.

  FIG. 5 is a functional block diagram of the posture calculation unit 450. The posture calculation unit 450 includes a gravity direction error calculation unit 550, an addition element 560, and a posture quaternion calculation unit 570. In the case described above, the attitude calculation unit 450 estimates the attitude of the omnidirectional camera 100 by using the output of the angular velocity sensor 430 and the output of the acceleration sensor 440 in combination.

  Specifically, global attitude estimation is performed based on the output of the acceleration sensor 440. A change in attitude of the omnidirectional camera 100 due to rotation about the direction of gravity is estimated by integrating the output of the angular velocity sensor 430. In order to use each sensor output together, the integration error of posture estimation by the angular velocity sensor 430 is corrected by comparing the output of the acceleration sensor 440 and the camera posture calculation result by the gravity direction error calculation unit 550.

Algorithm FIG. 6 is a flowchart showing the process 600 of the posture calculation unit 450.

At 630, the direction of gravity is calculated from the quaternion representing the previous camera pose. Expressing quaternion with _{q = [q 0, q 1} , q 2, q 3] T, the gravity direction vector _{v = [v x, v y} , v z] T can be calculated by the following equation.

In 640, calculates the gravity direction vector v calculated from quaternion, vector a = the acceleration sensor output _{[a x, a y, a} z] the error of ^T. The error vector e = [e _x , e _y , e _z ] ^T is determined by e = a × v using the outer product of the vectors. Here, the component of the error vector e represents an angle component formed by the vectors a and v.

  At 650, the output of the angular velocity sensor 430 is corrected by Equation 2 based on the error component.

  Here, Δt [sec] is a sampling cycle of the angular velocity sensor 430, and k is a correction coefficient. The correction coefficient k indicates how much the error is corrected. For example, a value such as a correction coefficient k = 0.001 may be used.

  At 660, the camera posture quaternion at time (t + Δt) is calculated by using the quaternion at time t as q (t) and the sampling period of the angular velocity sensor as Δt [sec] by Eq. It is output to the correction amount calculation unit 460.

  The posture correction amount calculation unit 460 calculates an amount for actually correcting the tilt of the image based on the tilt of the omnidirectional camera 100 from the calculated posture amount.

  The posture correction amount recording unit 470 records the posture correction amount on an external device coupled to the omnidirectional camera 100 by wire or wirelessly, for example, together with the omnidirectional image. As a result, the user can view the omnidirectional image to which the amount of posture correction has been applied using viewer software or the like.

Image Correction FIG. 7 shows angular velocity g _x , g _y , g _z [rad / sec] around the x, y, z axes output by the angular velocity sensor 430 and x, y, z axis directions output by the acceleration sensor 440. acceleration a _x, a _y, and a _z [G], is a diagram showing the relationship between the acceleration of gravity.

  The posture calculation unit 450 can correct the inclination of the image based on the acceleration, and correct the rotation angle around the gravity direction based on the angular velocity around the x, y, and z axes.

  As described above, according to an embodiment, image correction in a short period such as camera shake is performed based on angular velocity, and image correction in a long period where camera shake correction error is accumulated is performed based on acceleration.

  Specifically, the posture correction amount calculation unit 460 corrects the tilt of the image caused by the tilt of the camera based on the acceleration output from the acceleration sensor 440. Alternatively, the posture correction amount calculation unit 460 corrects the tilt or rotation of the image caused by the rotation of the camera based on the angular velocity output by the angular velocity sensor 430. Preferably, the posture correction amount calculation unit 460 corrects the tilt or rotation of the image caused by the rotation of the camera based on both the acceleration output from the acceleration sensor 440 and the angular velocity output from the angular velocity sensor 430.

Sound Field Correction The correction of the sound field picked up by the microphones 151 to 154 at the time of shooting the omnidirectional image will be described below.

  FIG. 8 shows an algorithm of the speech processing circuit 480. At 810, the camera attitude calculation unit 450 obtains a camera attitude quaternion represented by Equation 3. At 820, the sound field in the three-dimensional space defined by the microphones 151 to 154 is corrected by the camera posture amount. Specifically, when there is tilt of the camera, rotation in a three-dimensional space is applied to the four audio data to cancel it.

  In camera pose quaternion based correction at 820, a first order ambisonic basis is typically used. At 830, the sound field in three-dimensional space is converted to stereo and output. The audio processing circuit 480 can be realized by, for example, software executed by the CPU 310 or a dedicated chip such as a DSP (digital signal processing device).

  As described above, the tilt of the omnidirectional image caused by the tilt of the camera is corrected based on the signal indicating the acceleration from the acceleration sensor 440, and the tilt of the omnidirectional image caused by the tilt or rotation of the camera is the angular velocity The correction is made based on the signal indicating the angular velocity from the sensor 430. Similar to the inclination correction of the image, the inclination correction of the sound image is also performed. That is, the tilt correction of the sound field reduces the tilt or rotation of the sound field as well as the tilt of the omnidirectional image. As a result, a sound image localized at a position deviated from the original sound image because the camera is tilted is localized so as to properly reflect the position in the real space.

  For example, in a situation where a camera is fixed on the head and watching a sport, a situation is assumed where the photographer turns his head down to look at his hand or chases the player and shakes his head from side to side. Both the image and the sound field are corrected by the angular velocity output from the angular velocity sensor 430 and the acceleration output from the acceleration sensor 440. As a result, the viewer can view the image and the sound field while the image and the sound field match.

  Specifically, the inclination or rotation of the sound field can be corrected by the angular velocity obtained by the angular velocity sensor 430, and the inclination of the sound field can be corrected by the acceleration obtained by the acceleration sensor 440. For sound field correction, an effect can be obtained by correcting tilt or rotation due to camera rotation based on any one of acceleration and angular velocity. Further effects can be obtained by correcting the tilt or rotation of the sound field, preferably based on both acceleration and angular velocity.

  According to one exemplary embodiment, image correction and sound field correction are performed based on the output from the same acceleration sensor and the camera attitude quaternion. As a result, an inclination correction of the sound image is obtained, which coincides with the inclination correction of the omnidirectional image. As a result, it is possible to provide a sense of incongruent sound field inclination correction that is consistent with the omnidirectional image for the viewer.

  According to another exemplary embodiment, image correction and sound field correction are performed based on the output from the same angular velocity sensor and the camera attitude quaternion. Thereby a correction of the tilt or rotation of the sound image is obtained which corresponds to the correction of tilt or rotation of the omnidirectional image. As a result, for the viewer, correction of the tilt or rotation of the sound field without discomfort can be provided, which is consistent with the omnidirectional image.

  Preferably, with regard to image correction and sound field correction, the correction by the acceleration and the correction by the angular velocity are used in combination.

  The invention (or any part (s) or function (s) thereof) may be realized using hardware, software, or a combination thereof, and may be realized in one or more computer systems or other processing systems obtain.

  What has been described above includes various examples of the present invention. It is of course not possible to describe every conceivable combination of elements or procedures for the purpose of describing the invention, but one of ordinary skill in the art will appreciate that many additional combinations and permutations of the invention are possible. right. Accordingly, the present invention is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.

151 to 154 microphone 420 image sensor 1
422 Image sensor 2
430 angular velocity sensor 432 acceleration sensor 450 attitude calculation unit 460 attitude correction amount calculation unit 470 attitude correction amount recording unit 480 audio processing circuit 482 stereo output

Claims (6)

Optical system,
An element that outputs an image incident through an optical system as image data;
An acceleration sensor for detecting a signal representing acceleration in three axial directions;
An image processing circuit that processes the image data;
A plurality of microphones that receive voices constituting a sound field and output voice data representing the received voices;
An audio processing circuit that processes the audio data;
A omnidirectional camera with
The image processing circuit
Correcting the tilt of the image due to the tilt of the camera based on the acceleration;
The voice processing circuit
An omnidirectional camera that corrects the tilt of the sound field caused by the tilt of the camera based on the acceleration. It further comprises an angular velocity sensor for detecting a signal representing the angular velocity in three axial directions,
The image processing circuit
Correcting tilt or rotation of the image due to camera rotation based on the angular velocity;
The voice processing circuit
The omnidirectional camera according to claim 1, wherein tilt or rotation of the sound field caused by rotation of the camera is corrected based on the angular velocity. Optical system,
An element that outputs an image incident through an optical system as image data;
An angular velocity sensor that detects a signal representing an angular velocity in three axial directions;
An image processing circuit that processes the image data;
A plurality of microphones that receive voices constituting a sound field and output voice data representing the received voices;
An audio processing circuit that processes the audio data;
A omnidirectional camera with
The image processing circuit
Correcting tilt or rotation of the image due to camera rotation based on the angular velocity;
The voice processing circuit
An omnidirectional camera that corrects the tilt or rotation of the sound field caused by the rotation of the camera based on the angular velocity. Optical system,
An element that outputs an image incident through an optical system as image data;
An acceleration sensor that outputs a signal representing acceleration in three axial directions;
A plurality of microphones that receive voices constituting a sound field and output voice data representing the received voices;
An omnidirectional camera comprising: an audio processing method for the audio data;
A voice processing method comprising correcting an inclination of the sound field caused by an inclination of a camera based on the acceleration. The omnidirectional camera further includes an angular velocity sensor that detects a signal representing an angular velocity in three axial directions,
The image processing method is
Correcting tilt or rotation of the image due to camera rotation based on the angular velocity;
The voice processing method is
The sound processing method according to claim 4, further comprising: correcting inclination or rotation of the sound field caused by rotation of a camera based on the angular velocity. Optical system,
An element that outputs an image incident through an optical system as image data;
An angular velocity sensor that outputs a signal representing an angular velocity in three axial directions;
A plurality of microphones that receive voices constituting a sound field and output voice data representing the received voices;
An omnidirectional camera comprising: an audio processing method for the audio data;
An audio processing method comprising correcting tilt or rotation of the sound field caused by rotation of a camera based on the angular velocity.

Images