JP7434846B2 - Image processing device, image processing method, program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

2. Description of the Related Art VR (virtual reality) goggles that provide a 360-degree surrounding image depending on the direction in which the user is viewing are known. For this reason, an image captured in a wide-angle range of 360 degrees is required. Furthermore, by displaying images for the right eye and left eye, VR goggles can provide a virtual reality world with a sense of depth.

2. Description of the Related Art An imaging device that obtains a 360° surrounding image (hereinafter referred to as a spherical image) with a single imaging operation is known (see, for example, Patent Document 1). Patent Document 1 discloses an imaging device that generates one spherical image by combining two hemispherical images captured by optical imaging systems arranged back to back.

However, conventionally, there has been a problem that it is not easy to prepare omnidirectional images for stereoscopic viewing.

First, an imaging device for spherical images has only one lens in each direction (two in total), so even though it can image the surroundings as a 360-degree plane, it is not suitable for stereoscopic viewing. When viewed with VR goggles, etc., the plane is emphasized and lacks impact.

There are broadly two methods for preparing spherical images for stereoscopic viewing.
1. A spherical image capturing device with one lens in each direction captures images two or more times from different positions, and a computer or user edits and combines the images.
2. A special camera that can capture 3D images is used.

Regarding method 1, a time lag occurs in imaging because imaging is performed two or more times. When a time lag occurs, things like the movement of people, the movement of trees due to the wind, and changes in clouds occur during that time, making it difficult to align the two spherical images. At tourist spots, etc., there are many people, and it is virtually impossible to match the movements of all the people between two photos with a time lag.

Regarding method 2, the special camera is expensive and the housing has a bulky shape such as a sphere, making it difficult to use for general purposes.

In addition, with either method 1 or 2, the user cannot stereoscopically view an already existing spherical image taken at a tourist spot. It becomes necessary to take the image again.

SUMMARY OF THE INVENTION In view of the above-mentioned problems, an object of the present invention is to provide an image processing device that can provide a stereoscopic spherical image.

In view of the above problems, the present invention provides an image processing device that applies an estimation algorithm to one spherical image to generate a spherical image for stereoscopic viewing, and the estimation algorithm uses a neural network. The modeling data created with 3D modeling software is used as an input spherical image taken by the center camera of three virtual cameras arranged horizontally. Of the three virtual cameras, the spherical image taken by the right camera and the spherical image taken by the left camera, which includes the part corresponding to the back of the right camera. Swap the back image and the back image of the spherical image showing the portion corresponding to the back of the left camera to generate a spherical image for the right eye and a spherical image for the left eye, which serve as training data. It is characterized by having a learning data creation section .
The present invention also provides an image in which a distance image is generated by applying an estimation algorithm to one spherical image, and a spherical image for stereoscopic viewing is generated from the distance image and the spherical image. The processing device is a processing device, and the estimation algorithm is constructed by learning using a neural network, and the estimation algorithm is constructed by combining modeling data created with 3D modeling software with a spherical image captured by a virtual camera and a spherical image of the spherical image. Among the distance images, the estimation algorithm is constructed which inputs the omnidirectional image, uses the distance image as training data, and outputs the distance image for the omnidirectional image. The spherical image is converted into a three-dimensional point group in an orthogonal coordinate system using the distance of pixels of the distance image, and a straight line that rotates around a predetermined point on a horizontal plane in the orthogonal coordinate system, The present invention is characterized in that it includes a parallax calculation unit that converts a three-dimensional point near the straight line into a cylindrical image when the elevation angle of the straight line is changed at each position rotated by an angle.

It is possible to provide an image processing device that can provide a stereoscopic spherical image.

FIG. 3 is a diagram illustrating an outline of a method for creating a stereoscopically viewable omnidirectional image in this embodiment. 1 is a diagram illustrating an example of a hardware configuration of an image processing device. 1 is an example of a hardware configuration diagram of an imaging device. FIG. 2 is a diagram illustrating the format of a spherical image. This is an example of an image format for stereoscopic viewing. 1 is an example of a functional block diagram showing functions of an image processing device in a block form. FIG. 2 is a flowchart illustrating the flow of a learning data creation method performed by the image processing device. FIG. 2 is a diagram illustrating a method of creating learning data using 3D modeling software. It is an example of the figure which shows the imaging result of one time of three virtual cameras of the left side, the center, and the right side. FIG. 3 is a diagram illustrating the relationship between left and right images and eyes. FIG. 3 is a diagram illustrating processing performed in left and right image swapping. FIG. 6 is a diagram illustrating swapping of left and right images when the omnidirectional image is a three-dimensional sphere. 1 is a diagram showing an example of the configuration of a neural network of CNN (Convolutional Neural Network). FIG. 2 is a diagram schematically explaining convolution and deconvolution. FIG. 2 is an example of a flowchart illustrating a process in which the image processing device outputs a 3D stereoscopic spherical image; FIG. FIG. 2 is a diagram illustrating an outline of a process in which the image processing device creates a 3D stereoscopic spherical image. 1 is an example of a functional block diagram showing functions of an image processing device in a block form. FIG. 2 is a flowchart illustrating the flow of a learning data creation method performed by the image processing device. It is a figure which shows an example of a rendering result. FIG. 2 is a diagram showing a configuration example of a neural network. It is an example of a flowchart figure explaining the process which a parallax calculation part performs. This is an example of a diagram illustrating the direction in which the eyes are facing and the position of the eyes. FIG. 2 is a diagram schematically showing the relationship between an equirectangular image and a three-dimensional point group in a certain eye direction. FIG. 2 is a diagram illustrating a flow of converting a moving image into a three-dimensional stereoscopic moving image.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing apparatus and an image processing method performed by the image processing apparatus will be described below as an example of a mode for carrying out the present invention with reference to the drawings.

<Summary>
FIG. 1 is a diagram illustrating an outline of a method for creating a stereoscopic spherical image in this embodiment.
(1) The omnidirectional image imaging device 9 performs imaging processing and generates one omnidirectional image. The image may have already been captured.
(2) The image processing device 10 executes the estimation algorithm (program) and outputs two omnidirectional images from one omnidirectional image. These two spherical images are spherical images that can be viewed stereoscopically. The estimation algorithm is an algorithm that uses a neural network to estimate a left eye spherical image and a right eye spherical image from one spherical image. However, it is not limited to algorithms using neural networks.
(3) For example, users can view a 360-degree space with stereoscopic vision using VR goggles.

Note that the imaging device 9 only needs to be able to generate a spherical image in the format of equirectangular projection (described later). Alternatively, a spherical image may be rendered using 3D modeling software. The projection is not limited to the equirectangular projection, but may be a Mercator projection, a Miller projection, a centripetal projection, or the like.

In this way, the image processing device 10 of this embodiment can generate two omnidirectional images that can be viewed stereoscopically from one omnidirectional image. There is no need for timed imaging or special imaging equipment. Furthermore, a spherical image that can be viewed stereoscopically can be generated from a spherical image that has already been captured.

<About terms>
A spherical image is image data that captures 360 degrees of the surrounding area. It is not necessary that all 360 degrees be captured, and a portion may be omitted to improve image quality. A celestial sphere image may take a state where it is converted into a flat image (equirectangular image) or a state where it is a three-dimensional sphere.

The estimation algorithm is a program that generates two omnidirectional images for stereoscopic viewing from one omnidirectional image. Alternatively, it is a program that generates a distance image in which distance information is included in one or more pixels from one spherical image.

<Hardware configuration example>
<<Image processing device>>
FIG. 2 shows an example of the hardware configuration of the image processing device 10. As shown in FIG. 2, the image processing device 10 is constructed by a computer, and includes a CPU 501, ROM 502, RAM 503, HD 504, HDD (Hard Disk Drive) controller 505, It includes a display 506, an external device connection I/F (Interface) 508, a network I/F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I/F 516. .

Among these, the CPU 501 controls the overall operation of the image processing apparatus 10 . The ROM 502 stores programs used to drive the CPU 501 such as IPL. RAM 503 is used as a work area for CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading and writing of various data to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images. External device connection I/F 508 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. The network I/F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 2.

Further, the keyboard 511 is a type of input means that includes a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 514 controls reading and writing of various data on a DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may be DVD-R or the like. The media I/F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

Note that although it is omitted in FIG. 2, it is preferable to include a GPU (Graphics Processing Unit). GPUs are excellent at parallel processing of numerical calculations and can perform calculations generated by neural networks at high speed.

<<Imaging device>>
The hardware configuration of the imaging device 9 will be explained using FIG. 3. FIG. 3 is a hardware configuration diagram of the imaging device 9. As shown in FIG. In the following, the imaging device 9 is assumed to be a spherical (omnidirectional) imaging device using two imaging devices, but the number of imaging devices may be any number greater than or equal to two. Furthermore, it does not necessarily have to be a device exclusively for omnidirectional imaging; it may be possible to have substantially the same functions as the imaging device 9 by attaching an aftermarket omnidirectional imaging unit to a normal digital camera, smartphone, etc. good.

As shown in FIG. 3, the imaging device 9 includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, and a ROM (Read Only Memory). ) 612, SRAM (Static Random Access Memory) 613, DRAM (Dynamic Random Access Memory) 614, operation unit 615, external device connection I/F 616, communication unit 617, antenna 617a, and acceleration/direction sensor 618. There is.

Among these, the imaging unit 601 includes wide-angle lenses (so-called fisheye lenses) 602a and 602b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It includes elements 603a and 603b. The image sensors 603a and 603b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images formed by fisheye lenses 602a and 602b into electrical signal image data and output the image data. It has a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The imaging elements 603a and 603b of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I/F bus. On the other hand, the imaging elements 603a and 603b of the imaging unit 601 are connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation section 615, an external device connection I/F (Interface) 616, a communication section 617, an acceleration/direction sensor 618, and the like are also connected to the bus 610.

The image processing unit 604 takes in image data output from the image sensors 603a and 603b through the parallel I/F bus, performs predetermined processing on each image data, and then synthesizes these image data. , create equirectangular image data.

Generally, the imaging control unit 605 uses the I2C bus to set commands and the like in register groups of the imaging devices 603a and 603b, with the imaging control unit 605 as a master device and the imaging devices 603a and 603b as slave devices. Necessary commands and the like are received from the CPU 611. Further, the imaging control unit 605 also uses the I2C bus to take in status data and the like of the register groups of the imaging elements 603a and 603b, and sends it to the CPU 611.

Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data at the timing when the shutter button of the operation unit 615 is pressed. Depending on the imaging device 9, it may have a preview display function or a function corresponding to video display on a display (for example, a smartphone display). In this case, image data is output continuously from the image sensors 603a and 603b at a predetermined frame rate (frames/minute).

The imaging control unit 605 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 603a and 603b in cooperation with the CPU 611, as will be described later. Note that in this embodiment, the imaging device 9 is not provided with a display, but may be provided with a display section.

Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in sound data output from the microphone 608 through the I/F bus, and performs predetermined processing on the sound data.

The CPU 611 controls the overall operation of the imaging device 9 and executes necessary processing. ROM612 stores various programs for CPU611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data being processed, and the like. In particular, the DRAM 614 stores image data that is currently being processed by the image processing unit 604 and data of a processed equirectangular image.

The operation unit 615 is a general term for operation buttons such as the shutter button 615a. By operating the operation unit 615, the user inputs various imaging modes, imaging conditions, and the like.

The external device connection I/F 616 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a PC (Personal Computer), or the like. The equirectangular image data stored in the DRAM 614 can be recorded on an external media via this external device connection I/F 616, or transferred to an external terminal such as a smartphone via the external device connection I/F 616 as necessary. (device).

The communication unit 617 communicates with an external terminal (device) such as a smartphone via an antenna 617a provided on the imaging device 9 using short-range wireless communication technology such as Wi-Fi, NFC (Near Field Communication), or Bluetooth (registered trademark). ). The communication unit 617 can also transmit equirectangular image data to an external terminal (device) such as a smartphone.

The acceleration/azimuth sensor 618 calculates the azimuth of the imaging device 9 from the earth's magnetism and outputs azimuth information. This orientation information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. Note that the related information also includes data such as the date and time when the image was captured and the data capacity of the image data. Further, the acceleration/azimuth sensor 618 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) accompanying movement of the imaging device 9. A change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image. Furthermore, the acceleration/direction sensor 618 is a sensor that detects acceleration in three axial directions. The imaging device 9 calculates the attitude (angle with respect to the direction of gravity) of its own device (imaging device 9) based on the acceleration detected by the acceleration/azimuth sensor 618. By providing the acceleration/direction sensor 618 in the imaging device 9, the accuracy of image correction is improved.

<Format of spherical image>
FIG. 4 is a diagram illustrating the format of a spherical image. FIG. 4(a) is an equirectangular cylinder image, and FIG. 4(b) is a three-dimensional sphere. An equirectangular image is a two-dimensional transfer of the three-dimensional coordinate system of the real world. FIG. 4 shows a method of converting a coordinate system in transfer.

A vector A passing through the center of the three-dimensional sphere can be expressed by θ and φ.
θ: Angle in the horizontal direction in 3D space φ: Angle in the vertical direction in 3D space The two-dimensional transfer of the pixels specified by θ and φ is the normal angle generally used for spherical images. This is a rectangular cylinder image. In this embodiment, the spherical image in a planar state will be described as an equirectangular image. As shown in FIG. 4(a), the equirectangular cylindrical image has an angle of view of 360 degrees in the horizontal direction and 180 degrees in the vertical direction.

<Format of image for stereoscopic viewing>
FIG. 5 is an example of the format of a stereoscopic image. In addition to spherical images, there are several formats for stereoscopic images. The one described here is one of these formats, called the top and bottom format. The top image corresponds to the image displayed to the left eye, and the bottom image corresponds to the image displayed to the right eye. The image processing apparatus 10 of this embodiment outputs a top-and-bottom format image in which two equirectangular images are vertically arranged as the final output image format.

Note that there are several other formats for stereoscopic images. The output image may be any of the following.
・Side-by-side format: A format in which images for the left eye and images for the right eye are arranged side by side. ・Frame sequential format: A format for videos in which images for the left eye and images for the right eye are arranged alternately as video frames. Since the equirectangular image has an aspect ratio of horizontally long (1 (vertical): 2 (horizontal)), the top and bottom format is often adopted for spherical images for stereoscopic viewing. This is because if the top and bottom format is used, the image size will be exactly square based on the above aspect ratio.

<About the functions of the image processing device>
FIG. 6 is an example of a functional block diagram showing the functions of the image processing device 10 in a block form. First, FIG. 6(a) shows the image processing apparatus 10 in a learning phase. The image processing device 10 includes a storage section 41, a learning section 42, an image output section 43, and a learning data creation section 44. Among these, the image output unit 43 is constructed by learning and is therefore shown with a dotted line. Each of these functions that the image processing device 10 has is a function or means that is realized by the CPU 501 of the image processing device 10 executing a program loaded from the HD 504 to the RAM 503. Further, the storage unit 41 is formed in at least one of the HD 504 and the RAM 503 included in the image processing apparatus 10.

The learning data creation unit 44 creates learning data. In this embodiment, for example, 3D modeling software is used to create a spherical image for input and two spherical images for the right eye and for the left eye, which serve as teacher data.

A learning data storage section 49 is constructed in the storage section 41 . The learning data storage unit 49 stores learning data. The learning data of this embodiment includes a spherical image for input and a prepared spherical image for stereoscopic viewing (two spherical images for the left eye and for the right eye). A pair of spherical images for the left eye and for the right eye is the training data. The method for creating learning data will be described later.

Note that the learning data may be downloaded from a server via a network. The server can be in the cloud or on-premises. Further, the storage unit 41 may be provided outside the image processing device 10.

The learning unit 42 uses various machine learning approaches such as neural networks to learn the correspondence between the input spherical image and the teacher data spherical image. Machine learning is a technology that allows computers to acquire human-like learning abilities, in which computers autonomously generate algorithms necessary for data identification etc. from pre-loaded learning data, and then apply these algorithms to new data. This refers to the technology that is applied to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or may be a learning method that combines these learning methods. It doesn't matter what learning method you use.

Since the teacher data is a spherical image for stereoscopic viewing, an image output unit 43 that estimates (outputs) a spherical image for stereoscopic viewing from an input spherical image is obtained as a result of learning.

Note that the image processing device 10 during learning may exist on a network. In this case, a client terminal such as a PC (Personal Computer) transmits learning data to the image processing device 10, and returns the constructed image output section 43 to the client terminal.

FIG. 6(b) is an example of a functional block diagram of the image processing device 10 in the image estimation phase. The image processing device 10 has the above-mentioned image output section 43. The image output unit 43 acquires the spherical image from the spherical image capturing device 9 and outputs the spherical image for stereoscopic viewing (the spherical image for the right eye and the spherical image for the left eye). do. The spherical image for input is not limited to one captured by the imaging device 9, but may be one stored in the storage unit 41, a USB memory, or a network.

Further, the image processing device 10 in the image estimation phase may exist on a network. In this case, the client terminal transmits the input spherical image to the image processing device 10, and the image output unit 43 returns the stereoscopic spherical image output to the client terminal.

<How to create learning data>
Next, a method for creating learning data will be explained using FIGS. 7 to 12. First, FIG. 7 is a flowchart illustrating the flow of a learning data creation method performed by the image processing apparatus 10. As shown in FIG. 7, the learning data creation unit 44 installs a camera, performs rendering, and swaps left and right images.

(S1 Camera installation)
FIG. 8 is a diagram illustrating a method for creating learning data using 3D modeling software. In this embodiment, the teacher data is created using 3D modeling software. 3D modeling software is an application that visualizes 3D CAD and 3DCG data on a computer. 3D CAD mainly represents a three-dimensional shape using mathematical formulas, and 3DCG represents a three-dimensional shape using a combination of polygons.

A developer or the like reproduces appropriate modeling data of a city using 3D modeling software. The learning data creation unit 44 installs a virtual camera 410 within the city. There is no limit to where it can be installed, as long as it can capture images of the surrounding area. This camera is a camera capable of capturing a spherical image, but this camera does not need to have two optical systems. This is because since it is a virtual space, it is only necessary to set the angle of view to 360 degrees as a camera parameter. However, for example, it may be a camera that has a set of two optical imaging systems placed back to back, and the lenses of each optical imaging system have internal parameters of a fisheye lens.

A total of three cameras will be installed. For example, they are arranged in parallel on the left, center, and right. Since the left and right cameras correspond to the left and right human eyes, the left and right cameras are installed at a distance equivalent to the parallax between the left and right human eyes. Note that if you want to emphasize the parallax, the distance between the left and right cameras may be made wider than the actual human parallax. The center camera captures a spherical image for input, and the left and right cameras capture spherical images for teacher data.

The virtual camera needs to take enough images to provide learning data, and each of the three cameras takes about 300 images, for example. A fixed number of images are repeatedly captured at regular intervals or at equal intervals while moving the camera position. Note that the required number of sheets is the number necessary for the weights (parameters) of the neural network to converge, so 300 sheets is an example.

(S2 rendering)
FIG. 9 shows the results of one time of imaging by three virtual cameras on the left, center, and right. 9(a) is the spherical image of the left camera, FIG. 9(b) is the spherical image of the center camera, and FIG. 9(c) is the spherical image of the right camera. Although the images are almost the same, strictly speaking, there is a parallax.

Note that the camera does not actually capture images, but obtains a spherical image by two-dimensionally projecting 3D CAD or 3DCG data within the viewing angle. This process is sometimes called rendering. In general, developers set external parameters related to the camera position and internal parameters (in a matrix) related to the projection method (fisheye lens), and then multiply them by 3D CAD or 3DCG data to create a spherical image. The pixel value is determined.

(S3 Swap left and right images)
Next, swapping of left and right images will be explained based on FIG. 10. FIG. 10 is a diagram illustrating the relationship between the left and right images and the eyes. The spherical image for the right eye and the spherical image for the left eye shown in FIG. 9 do not serve as training data. This is because the positional relationship between the left and right cameras and the position of the human eye change depending on the orientation of the human body.

As shown in FIG. 10, when a person is facing forward, the camera installed on the left side corresponds to the left eye, and the camera installed on the right side corresponds to the right eye. However, when a person faces in the opposite direction, the camera installed on the left side corresponds to the right eye, and the camera installed on the right side corresponds to the left eye.

Therefore, as shown in FIG. 11, the learning data creation unit 44 swaps the left and right images. FIG. 11 is a diagram illustrating the processing performed in left and right image swapping. Swapping the left and right images is a process of converting the spherical image of the left camera and the spherical image of the right camera into a left-eye image and a right-eye image, respectively.

In the case of an imaging device 9 that has two optical imaging systems (lens and image sensor, etc.) placed back to back, the front lens captures the image from 1/4 to 3/4 from the left, and the rear lens captures the rest. Take an image. Therefore, the image of the back of the right camera and the image of the back of the left camera may be exchanged.

FIG. 12 is a diagram illustrating swapping of left and right images when the omnidirectional image is a three-dimensional sphere. As shown in Figure 12, if the partial image of the front lens of the spherical image of the left camera is combined with the partial image of the rear lens of the spherical image of the right camera, it becomes the image of the left eye. . Similarly, if a partial image of the front lens of the spherical image of the right camera is combined with a partial image of the rear lens of the spherical image of the left camera, the resulting image becomes the image of the right eye. In other words, creating an image by swapping the rear lens side of the spherical images of the left and right cameras creates a spherical image of the left and right eyes.

(S4 Completion Judgment)
The learning data creation unit 44 performs S1 to S3 until a sufficient number of learning data (more than a threshold value) is generated.

As described above, the learning data creation unit 44 was able to generate learning data including the input spherical image and the teacher data (the spherical image for the left eye and the spherical image for the right eye). The image processing device 10 performs similar processing on all spherical images captured by the three virtual cameras.

<Structure of neural network>
FIG. 13 shows an example of the configuration of a CNN (Convolutional Neural Network) neural network. Among neural networks and deep learning, a network that uses convolutional operations is called a CNN. The above estimation algorithm (image output unit) is constructed by CNN. Although CNN is mainly used in this embodiment, any algorithm may be used as long as it can generate an image.

FIG. 14 is a diagram schematically explaining convolution and deconvolution. FIG. 14(a) is an image of convolution processing, and FIG. 14(b) is an image of deconvolution. The convolution units 51 and 53 multiply the input image 301 by each element of the filter (kernel) 302 to obtain a value for one pixel of the feature map 303. The convolution units 51 and 53 perform the same calculation with a shift amount called stride. In FIG. 14A, the input image is 5×5, the filter is 3×3, and the stride is 1, so a 3×3 feature map 303 is obtained. Since as many filters as there are channels (for example, RGB) are prepared, feature maps corresponding to the number of channels can be obtained. Note that it is common to perform nonlinear transformation using a RELU function or the like for each convolution.

The deconvolution units 52 and 54 use the feature map obtained by convolution (3×3 as an example) as an input image 305, add spaces around the pixels to enlarge the size, and then apply a filter 306 (3×3) to the input image 305. 3). The deconvolution units 52 and 54 perform the same calculations with a shift amount called a stride. As a result, a 5×5 feature map 307 is obtained in FIG. 14(b). The values embedded around the feature map 303 do not have to be blank (zero). For example, there is a method that uses feature maps of the same size obtained by convolution operations.

The explanation will be returned to FIG. 13. In FIG. 13, the learning unit 42 copies the input spherical image to create two spherical images. The upper row is for the left eye, and the lower row is for the right eye. The convolution units 51 and 53 perform several convolutions and extract features. A filter is prepared in the process of feature extraction. The coefficients of this filter are automatically learned to extract features. The original image becomes smaller depending on the filter size and stride. In FIG. 13, features are extracted up to a size of "2×1", which is just an example. Note that 8, 16, . . . 2048 is the number of channels in convolution.

In the deconvolution units 52 and 54, the learning unit 42 performs deconvolution on the feature map. In FIG. 13, images of the same size in the convolution layer are incorporated in order to bring the feature map obtained by deconvolution closer to the input spherical image. However, this process may not be necessary.

During learning, the learning unit 42 calculates the difference between the prepared teacher data for the right eye and the left eye and the images output by the deconvolution units 52 and 54. This difference is back-propagated in the order of deconvolution units 52 and 54 and convolution units 51 and 53. The coefficients of the filter are updated by backpropagation.

The learning unit 42 performs the process shown in FIG. 13 until the change in the filter coefficients converges or for all the learning data, so that the filter coefficients are learned and the left-eye omnidirectional image is applied to the input omnidirectional image. An image output unit 43 is constructed that outputs a spherical image and a spherical image for the right eye.

Note that the configuration in FIG. 13 is an example, and a pooling layer may be provided after the convolutional layer, and a fully connected layer may be provided instead of the deconvolutional layer.

<Output processing of spherical images for stereoscopic viewing>
FIG. 15 is an example of a flowchart illustrating a process in which the image processing device 10 outputs a 3D spherical image.

First, the image output unit 43 acquires a spherical image for input (S11). The spherical image may be captured by the imaging device 9 in real time, or may be stored in the storage unit 41.

Next, the image output unit 43 copies the input spherical image to generate spherical images for the right eye and for the left eye (S12).

The image output unit 43 applies the estimation algorithm to each of the right-eye and left-eye spherical images (S13). That is, the convolution and deconvolution shown in FIG. 13 are performed to output spherical images for the right eye and for the left eye.

The image output unit 43 generates a spherical image for stereoscopic vision from the spherical images for the right eye and the left eye (S14). In other words, it is converted into a format called top and bottom format.

<Main effects>
As described above, the image processing device 10 of this embodiment can generate a stereoscopically viewable omnidirectional image from one omnidirectional image. There is no need for timed imaging or special imaging equipment. Furthermore, a spherical image that can be viewed stereoscopically can be generated from a spherical image that has already been captured.

In this embodiment, an image processing device 10 that generates a distance image and converts it into left and right eye images will be described.

FIG. 16 is a diagram illustrating an outline of a process in which the image processing apparatus 10 of this embodiment creates a 3D image for stereoscopic viewing. In this embodiment as well, only one spherical image 400 is input, and the final output is a spherical image for the left eye and a spherical image for the right eye.

(1) In this embodiment, the neural network 401 does not directly output the omnidirectional images for the left eye and the right eye, but instead outputs the distance image 402 from the neural network 401. The distance image 402 is an image in which distance information to an object reflected in this pixel is arranged for each pixel.

(2) The image processing device 10 uses the input spherical image 400 and the distance image 402 to create a three-dimensional point group 404, each of which has a pixel value.

(3) The image processing device 10 uses the pixel values of the three-dimensional point group 404 to generate equirectangular cylinder images 405 for the left eye and the right eye.

This method has an advantage over the method of the first embodiment in that it is easier to increase the resolution of the final output spherical images for the left eye and for the right eye. In the method of Example 1, the spherical images for the left and right eyes were directly generated by the neural network, so the quality of the spherical images for the left and right eyes depends on the amount of memory consumed by the neural network. do.

Neural networks for image generation consume a lot of memory, so unless you have a large-scale GPU, there are physical constraints on the size. Since the imaging device 9 has a high resolution, in the first embodiment, the input spherical image must be reduced in size and applied to the neural network, resulting in a decrease in resolution.

In contrast, in this embodiment, a distance image is generated. The resolution of the distance image itself is the same as in Example 1 in that it is created using a neural network, so it is approximately 2048 x 1024. However, since the distance image has a simple structure, the image quality is less likely to deteriorate even if it is enlarged. Rather, the advantage is that it should be expanded. In other words, it is possible to make full use of the original resolution, increase the resolution, and output spherical images for the left and right eyes.

<About functions>
FIG. 17 is an example of a functional block diagram showing the functions of the image processing device 10 in a block form. FIG. 17A is an example of a functional block diagram showing in block form the functions of the image processing device 10 in the learning phase. In this embodiment, the function of the learning data creation section 44 is different from that in the first embodiment. The learning data creation unit 44 of this embodiment only needs to create one distance image from the input spherical image, and there is no need to create spherical images for the right eye and for the left eye (teacher data).

Further, the point that the learning section 42 constructs the image output section 43 is the same as in the first embodiment, but the image output section 43 creates one distance image.

FIG. 17(b) is an example of a functional block diagram showing in block form the functions of the image processing device 10 in the image estimation phase. This embodiment additionally includes a parallax calculation section 45. The parallax calculation unit 45 creates a spherical image for the left eye and a spherical image for the right eye from the distance image created by the image output unit 43 and the input spherical image.

<How to create learning data>
FIG. 18 is a flowchart illustrating the flow of the learning data creation method performed by the image processing device 10. In the explanation of FIG. 18, differences from FIG. 7 will be mainly explained. As shown in FIG. 18, the learning data creation unit 44 installs a camera and performs rendering, but there is no need to swap left and right images.

(S1 Camera installation)
In this embodiment, an equirectangular cylinder image and a corresponding distance image serve as learning data. The training data is a distance image. 3D modeling software is also used in this embodiment. In this embodiment, the learning data creation unit 44 only needs to arrange one virtual camera.

(S2 rendering)
As in the first embodiment, the learning data creation unit 44 renders a spherical image to a virtual camera. Furthermore, in this embodiment, since the learning data creation unit 44 creates a distance image, RGB luminance values are not necessary, and depth information called a z-buffer is rendered. 3D modeling software can use depth information for the camera called z-buffer. Depth information is information representing the distance between the camera and a specific point on an object. In a 3D model, when the camera is fixed, objects in space may overlap when viewed from the camera, and in this case, in order to proceed with the drawing process efficiently, it is unnecessary to draw objects that are far from the camera. Become. The data used here is the z-buffer, which is used to speed up the drawing speed by rendering only the objects closest to the camera.

The learning data creation unit 44 uses external and internal parameters of the camera to project not RGB values but distance information (z buffer) onto an equirectangular image. Therefore, with the virtual camera installed in the 3D model, distance information can be measured for all objects that constitute the 3D model space using the camera as a reference.

FIG. 19 is a diagram showing an example of a rendering result. FIG. 19(a) is a spherical image for input, and FIG. 19(b) is a distance image. Although it cannot be read due to drawing limitations, in FIG. 19(b), colors are assigned according to the distance.

Generally, distance images are expressed as grayscale images. However, in grayscale, the color width is only 256 gradations, so the distance resolution is low. Therefore, in FIG. 19(b), the distance is expressed in higher resolution rather than in 256 steps (for convenience of drawing, there are fewer types of shading than in reality). Thereby, the learning data creation unit 44 can render a high-density distance image.

<Structure of neural network>
FIG. 20 shows an example of the configuration of the neural network of this embodiment. Since FIG. 20 outputs one distance image, there is only one processing flow, but the configurations of the convolution unit 56 and deconvolution unit 57 are similar to those in FIG. 13. However, the neural networks used may be the same or different. Since the training data is changed, even if the same neural network is used, characteristics such as filter coefficients will change after learning, so it is possible to construct the image output unit 43 according to the distance data. An image generation neural network may be used.

During learning, the learning unit 42 calculates the difference between the prepared distance image (teacher data) and the output image output by the deconvolution unit 57. This difference is back-propagated to the deconvolution unit 57 and then to the convolution unit 56 in this order. The coefficients of the filter are updated by backpropagation.

The learning unit 42 performs the process shown in FIG. 20 until the change in the filter coefficients converges or for all learning data, thereby learning the filter coefficients and outputting a distance image for the input spherical image. An image output section 43 is constructed.

<About parallax calculation>
Next, parallax calculation will be explained using FIG. 21. FIG. 21 is a flowchart illustrating the processing performed by the parallax calculation unit 45.

・S101
First, the parallax calculation unit 45 generates a three-dimensional point group using the input omnidirectional image and the corresponding distance image. As explained with reference to FIG. 4, each point on the spherical image corresponds to a point on the spherical surface of the three-dimensional sphere. The horizontal angle θ (radians) and vertical angle φ when transferred to a three-dimensional sphere with respect to the point (u,v) of the spherical image are calculated as follows. Note that w and h are the width and height of the spherical image, respectively, and are introduced to normalize u and v to 0 to 1. An example is w=5376 and h=2688.
θ = −π + 2π(u/w)
φ = −π/2 + π(v/h)
Although the coordinates (θ, φ) of the three-dimensional sphere are polar coordinates, conversion between the polar coordinates and the orthogonal coordinate system can be realized by the following equation (1).

なお、(u,v)の点における距離画像の距離dは立体球の半径に対応するので、距離dが右辺に乗じられている。

Note that the distance d of the distance image at the point (u,v) corresponds to the radius of the three-dimensional sphere, so the right side is multiplied by the distance d.

By applying this calculation to all the points (u,v) of the spherical image, the parallax calculation unit 45 calculates all the pixels of the spherical image and the orthogonal coordinate system having the distance corresponding to each pixel. A three-dimensional point cloud can be generated.

・S102
The parallax calculation unit 45 performs the following processing at each three-dimensional point.

・S103
What the parallax calculation unit 45 creates are equirectangular cylinder images corresponding to the left and right eyes, respectively. Since the three-dimensional space has been restored in step S101, a camera (viewpoint) is placed at an arbitrary coordinate, and how the image appears there is determined by calculation. In order to create left and right eye images, the parallax calculation unit 45 creates an equirectangular cylindrical image from a group of three-dimensional points by shifting the camera by the distance between the eyes (for example, 5 cm on each side, 10 cm in total).

When rendering an image of a three-dimensional point cloud, the position of the right eye (or left eye) is fixed at one place at a certain point in time, but it should be noted that the position of the eye changes depending on the direction the eye is facing.

FIG. 22 is a diagram illustrating the direction in which the eyes are facing and the position of the eyes. First, FIG. 22(a) shows how the eye direction rotates 360 degrees around the xz plane around the y axis. For example, assuming that a human being is at the origin, the direction of the human eye can be rotated 360 degrees around the xz plane. FIG. 22(b) is a top view of a human being viewed from the xz plane from the negative direction of the y axis. In FIG. 22(b), a person faces four directions. The four directions in FIG. 22(b) are designated as A to D. In this case, if we take the xz axis as shown in Figure 22(c), the position of the right eye in orientation A is 5 cm from the origin in the positive direction of the x axis, and the position of the right eye in direction B is 5 cm from the origin in the positive direction of the z axis. 5 cm, the position of the right eye in orientation C is 5 cm from the origin in the negative direction of the x-axis, and the position of the right eye in direction D is 5 cm from the origin in the negative direction of the z-axis.

Therefore, when rendering a 360-degree three-dimensional point group, the parallax calculation unit 45 draws a circle the width of the eyes on the xz plane (horizontal plane) with respect to the centers of the left and right eyes (predetermined points). The orientation is rotated 360 degrees, and rendering is performed at each position (each angle in the radial direction) rotated by a predetermined angle until the camera position returns to its original position. Each position can be calculated from the number of pixels in the horizontal direction of the equirectangular image. If the width of the equirectangular image is w, then the amount of change in the eye direction is set to 2π/w in order to correspond to the amount of rotation of 2π radians in w.

・S104
Next, since the distance image with the least distortion can be created in the direction straight to the eyes, as shown in FIG. 22(d), the parallax calculation unit 45 Rendering is performed using a three-dimensional point group in the direction of a vertical straight line 62.

・S105
FIG. 23 is a diagram schematically showing the relationship between an equirectangular image and a three-dimensional point group in a certain eye direction. The position of the right eye is the intersection, and a straight line 62 representing the direction of the eye is shown. In this case, all pixels in the vertical direction (on the dotted line 61) are to be rendered. A dotted line 61 is a trajectory obtained when the elevation angle of the straight line 62 is changed while the direction of the eye on the xz plane remains the same (the direction of the straight line 62 remains the same). The elevation angle is the angle when the straight line 62 is changed in the y-axis direction (vertical direction). Therefore, the parallax calculation unit 45 renders the pixels on the dotted line 61 into an equirectangular image. However, since there are not always pixels on the dotted line, the point closest to the straight line 62, which is changed by 180 degrees in the y-axis direction, is selected and rendered while the direction of the eye on the xz plane remains the same.

・S106
As a result, a three-dimensional point is determined, and the parallax calculation unit 45 converts the selected three-dimensional point (x, y, z) into a horizontal angle θ and a vertical angle φ using equation (2), and calculates the three-dimensional The pixel value of the point (x, y, z) is set to the pixel value of the point in the equirectangular image.

以上では、主に右目用の正距円筒画像を生成したので、同様に左目用の正距円筒画像も作成する。左目用の場合は回転が逆方向になる。以上の手順により、入力された正距円筒画像から左右の目の視差を考慮した２つの全天球画像を取得できる。

In the above, since an equirectangular image for the right eye was mainly generated, an equirectangular image for the left eye is also created in the same way. If it is for the left eye, the rotation will be in the opposite direction. Through the above procedure, two spherical images can be obtained from the input equirectangular image in consideration of the parallax between the left and right eyes.

<Main effects>
According to this embodiment, in addition to the effects of the first embodiment, a high-density, high-definition equirectangular image can be obtained.

In this embodiment, additional information will be given regarding moving images. In this embodiment, the hardware configuration diagrams shown in FIGS. 2 and 3 and the functional block diagrams shown in FIG. 6 or 17 described in the above embodiments can be used.

FIG. 24 is a diagram illustrating the flow of converting a moving image into a three-dimensional stereoscopic moving image. As shown in Figure 24,
(1) The image processing device 10 regards moving images as continuous images and converts them into continuous still images.
(2) For each, the image processing device 10 generates an image for 360-degree stereoscopic viewing using the estimation algorithm of the first or second embodiment.
(3) Recombine those images as a video.

If audio is present in the video, the audio can be divided during image segmentation, an estimation algorithm can be applied to the audio, and the audio from the original video can be added when the videos are recombined. Since the number of frames etc. have not changed, no audio deviation occurs.

<Main effects>
According to this embodiment, in addition to the effects of embodiments 1 and 2, it is also possible to deal with moving images.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications can be made without departing from the gist of the present invention. and substitutions can be added.

Furthermore, the configuration examples shown in FIGS. 6 and 17 are divided according to main functions in order to facilitate understanding of the processing performed by the image processing apparatus 10. The present invention is not limited by the method of dividing the processing units or the names thereof. The processing of the image processing device 10 can also be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

9 Imaging device 10 Image processing device

Publication of JP-A-2015-019344

Claims (7)

An image processing device that applies an estimation algorithm to one spherical image to generate a spherical image for stereoscopic viewing,
The estimation algorithm is constructed by learning using a neural network ,
Modeling data created with 3D modeling software is input by using a spherical image captured by the center camera of three virtual cameras arranged horizontally as an input spherical image.
A spherical image captured by the right camera of the three virtual cameras, and a rear image of the spherical image that shows a portion corresponding to the back of the right camera among the spherical image captured by the left camera. Learning data that generates a spherical image for the right eye and a spherical image for the left eye as training data by swapping the spherical image and the back image of the spherical image that shows the part corresponding to the back of the left camera. An image processing device comprising a creation section . An image processing device that generates a distance image by applying an estimation algorithm to one omnidirectional image, and generates an omnidirectional image for stereoscopic viewing from the distance image and the omnidirectional image,
The estimation algorithm is constructed by learning using a neural network ,
Among the spherical image obtained by capturing modeling data created with 3D modeling software with a virtual camera and the distance image of the spherical image,
The estimation algorithm is constructed to input the spherical image, use the distance image as training data, and output the distance image for the spherical image,
converting the omnidirectional image into a three-dimensional point group in an orthogonal coordinate system using the pixel distance of the distance image estimated by the estimation algorithm;
A straight line that rotates around a predetermined point on the horizontal plane of the orthogonal coordinate system, and when the elevation angle of the straight line is changed at each position rotated by a predetermined angle, a three-dimensional point near the straight line is transformed into a cylinder. An image processing device characterized by having a parallax calculation unit that converts into an image . 3. The image processing apparatus according to claim 1, wherein the spherical image of a moving image is converted into a still image, a spherical image for stereoscopic viewing is generated, and then the image is combined with the moving image. An image processing method that applies an estimation algorithm to one spherical image to generate a spherical image for stereoscopic viewing, the method comprising:
The estimation algorithm is constructed by learning using a neural network ,
Modeling data created with 3D modeling software is input by using a spherical image captured by the center camera of three virtual cameras arranged horizontally as an input spherical image.
A spherical image captured by the right camera of the three virtual cameras, and a rear image of the spherical image that shows a portion corresponding to the back of the right camera among the spherical image captured by the left camera. Image processing that swaps the back image of the spherical image showing the portion corresponding to the back of the left camera to generate a spherical image for the right eye and a spherical image for the left eye as training data. Method. An image processing method that applies an estimation algorithm to one spherical image to generate a distance image, and generates a spherical image for stereoscopic viewing from the distance image and the spherical image, the method comprising:
The estimation algorithm is constructed by learning using a neural network ,
Among the spherical image obtained by capturing modeling data created with 3D modeling software with a virtual camera and the distance image of the spherical image,
The estimation algorithm is constructed to input the spherical image, use the distance image as training data, and output the distance image for the spherical image,
converting the omnidirectional image into a three-dimensional point group in an orthogonal coordinate system using the pixel distance of the distance image estimated by the estimation algorithm;
A straight line that rotates around a predetermined point on the horizontal plane of the orthogonal coordinate system, and when the elevation angle of the straight line is changed at each position rotated by a predetermined angle, a three-dimensional point near the straight line is transformed into a cylinder. An image processing method characterized by converting into an image . In the image processing device,
A program that applies an estimation algorithm to one spherical image to generate a spherical image for stereoscopic viewing,
The estimation algorithm is constructed by learning using a neural network ,
The image processing device,
Modeling data created with 3D modeling software is input by using a spherical image captured by the center camera of three virtual cameras arranged horizontally as an input spherical image.
A spherical image captured by the right camera of the three virtual cameras, and a rear image of the spherical image that shows a portion corresponding to the back of the right camera among the spherical image captured by the left camera. Learning data that generates a spherical image for the right eye and a spherical image for the left eye as training data by swapping the spherical image and the back image of the spherical image that shows the part corresponding to the back of the left camera. A program to function as a creation section. In the image processing device,
A program that applies an estimation algorithm to one omnidirectional image to generate a distance image, and generates an omnidirectional image for stereoscopic viewing from the distance image and the omnidirectional image, the program comprising:
The estimation algorithm is constructed by learning using a neural network ,
Among the spherical image obtained by capturing modeling data created with 3D modeling software with a virtual camera and the distance image of the spherical image,
The estimation algorithm is constructed to input the spherical image, use the distance image as training data, and output the distance image for the spherical image,
The image processing device,
converting the omnidirectional image into a three-dimensional point group in an orthogonal coordinate system using the pixel distance of the distance image estimated by the estimation algorithm;
A straight line that rotates around a predetermined point on the horizontal plane of the orthogonal coordinate system, and when the elevation angle of the straight line is changed at each position rotated by a predetermined angle, a three-dimensional point near the straight line is transformed into a cylinder. A program that functions as a parallax calculation unit that converts images.

Images