JP2019220114A - Signal processor, convolutional neural network, signal processing method, and signal processing program - Google Patents

Abstract

To provide filter processing on a spherical signal with small computation complexity and high approximation precision.SOLUTION: When performing filter processing on a spherical surface signal having a value allocated to a point on a spherical surface, an axial decomposition part expands or contracts the spherical surface signal in one axial direction of polar coordinates while reflecting a distance on the spherical surface to decompose the signal into a plurality of partial signals. A filter processing part performs filter processing on the partial signals. Then an axial integration part performs axial integration processing to expand or contract and integrate the partial signals having been subjected to the filter processing while reflecting the distance on the spherical surface. Consequently, the filter processing on the spherical surface signal etc., with small computational complexity and high approximation precision is provided.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a signal processing device, a convolutional neural network, a signal processing method, and a signal processing program.

  At present, there is known an apparatus which acquires, outputs, or records information in all directions, such as up, down, left, right, and oblique directions. If the direction is expressed by a unit vector, such omnidirectional information (signal) can be regarded as a signal (a mapping from a unit sphere to a set of signal values: spherical signal) in which values are assigned on a unit sphere. . Filter processing on a spherical signal has a wide range of uses such as components of a convolutional neural network in addition to uses such as noise removal and feature extraction.

  However, the conventional signal processing device that performs a filtering process on a spherical signal or the like has a problem that the approximation accuracy is low and the amount of calculation is large.

  The present invention has been made in view of the above-described problems, and has been made in consideration of a signal processing apparatus, a convolutional neural network, a signal processing method, and a signal processing program that can perform a filtering process on a spherical signal or the like with a small amount of calculation and high approximation accuracy. For the purpose of providing.

  In order to solve the above-described problems and achieve the object, the present invention is a signal processing device that performs a filtering process on a spherical signal in which a value is assigned to a point on a spherical surface. An axis disassembly unit that performs an axis disassembly process that expands and contracts by reflecting a distance on a spherical surface and decomposes the signal into a plurality of partial signals in one axial direction; a filter processing unit that performs a filter process on the partial signals; And an axis integration unit that performs an axis integration process of integrating the expanded and contracted partial signals by reflecting and expanding the distance on the spherical surface.

  Advantageous Effects of Invention According to the present invention, there is an effect that it is possible to perform filter processing with high approximation accuracy with a small amount of calculation.

FIG. 1 is a diagram for explaining the equirectangular cylinder type. FIG. 2 is a configuration diagram of the signal processing device according to the first embodiment. FIG. 3 is a flowchart illustrating a flow of signal processing of the signal processing device according to the first embodiment. FIG. 4 is a diagram for explaining axis disassembly processing in the signal processing device according to the first embodiment. FIG. 5 is a diagram schematically showing the flow of the axis decomposition processing, the plane filter processing, and the axis integration processing. FIG. 6 is a diagram for explaining how to calculate an integral image in the signal processing device according to the second embodiment. FIG. 7 is a diagram for explaining axis disassembly processing in the signal processing device according to the second embodiment. FIG. 8 is a diagram illustrating a multi-scale filter process of a signal processing device according to a modification of the second embodiment. FIG. 9 is a configuration diagram of a signal processing device according to the third embodiment. FIG. 10 is a flowchart illustrating a flow of signal processing of the signal processing device according to the third embodiment. FIG. 11 is a diagram for describing vertical and horizontal decomposition processing of a filter in the signal processing device according to the third embodiment. FIG. 12 is a schematic diagram illustrating a flow of signal processing using a horizontal filter and a vertical filter in the signal processing device according to the third embodiment. FIG. 13 is a diagram showing an example in which a matrix G of filters is decomposed into a plurality of vertical filters and horizontal filters. FIG. 14 is a schematic diagram illustrating a flow of a series of processes performed on a third-order unit spherical surface in the signal processing device according to the fourth embodiment. FIG. 15 is a configuration diagram of a signal processing device according to the fifth embodiment.

  Hereinafter, although only one example, implementation that can be used for processing and analysis of directional signals such as omnidirectional image processing (omnidirectional camera device), sound field processing, weather analysis, or astrophysics, etc. The signal processing device according to the embodiment will be described.

(First Embodiment)
(Overview)
Information (signals) in all directions, such as up, down, left, right, and diagonal, can be regarded as signals in which values are assigned to unit spheres (mapping from a unit sphere to a set of signal values) when each direction is represented by a unit vector. Hereinafter, the signal in each direction is referred to as a “spherical signal”. Filtering a discrete spherical signal is a convolution (or correlation) operation of the signal and the kernel of the filter. In the case of filter processing of signal values on a square lattice, convolution operation may be performed by shifting the filter by one grid. However, it is very difficult to construct such a grid on a secondary or higher spherical surface.

  That is, a signal point is set at a contact point with a circumscribed regular polyhedron, thereby forming a rotation-invariant grid on the spherical surface. However, for example, a regular polyhedron in a three-dimensional space exists only up to an icosahedron, so that a secondary sphere cannot form a grid of 21 points or more that is invariant to three-dimensional rotation. Therefore, in order to perform a filtering process on a spherical surface, it is necessary to interpolate a signal value or a filter kernel or to perform an approximation process instead of the interpolation.

  Further, the interpolation amount of the signal value or the filter kernel requires a large amount of calculation. For example, when the number of signal points is “N” and the filter size is “M”, at most NM points are formed by interpolation processing. This means that it is not practical unless the number of signal points is small or the filter size is small.

  Here, Patent Literature 1 (Japanese Patent No. 5734327), Patent Literature 2 (Japanese Patent No. 6067934), Non-Patent Literature 1 (TCCohen, et al., “Spherical CNNs”. ArXiv: 1801.10130, 2018.) or Non-Patent Document 2 (C. Esteves, et al., “3D object classification and retrieval with Spherical CNNs.” ArXiv: 1711.06721, 2017) discloses a practical calculation amount approximation method. Patent Literature 1, Patent Literature 2, Non-Patent Literature 1, and Non-Patent Literature 2 disclose a filtering process in a spectral domain using a spherical harmonic function and a Wigner function, which are generalized Fourier transform processes.

  In this filtering process, signal points are arranged so as to have the same density as much as possible on the spherical surface, and then a filter operation (multiplication) is performed by approximating the signal value and the filter kernel in the spectral domain. However, even when the generalized fast Fourier transform is used to convert the signal value to the spectral domain, the calculation amount is O (N log N) for the number N of signal points. Note that “O” indicates Landau's asymptotic notation o (or omicron). When filtering is performed in the spatial domain, since the kernel size is M and O (NM), the calculation efficiency is poor when the kernel size is small. Further, since the data is once converted into the spectral domain, a memory of the same amount as the signal value to be processed is required. For this reason, it is not appropriate to apply the filtering in the spectral domain using the spherical harmonic function and the Wigner function to the filtering of a spherical signal having a large number of signal points.

  On the other hand, Patent Document 3 (Japanese Patent Application Laid-Open No. 2017-207960) and Non-Patent Document 3 (W. Boomsma, et al., "Spherical convolutions and their application in molecular modeling", Advances in Neural Information Processing Systems, 2017. ) Discloses a filtering process in which a spherical signal is projected onto a plane (square lattice) and is alternately performed on the plane.

  In the case of the filter processing performed in place of this plane, the interpolation processing of the signal points is not required after the plane projection, so that when the filter processing is performed in multiple stages as in a convolutional neural network, the calculation efficiency is high. However, if the number of projection planes is increased in order to increase the approximation accuracy of the spherical filtering, the amount of calculation increases. In addition, artifacts occur in the processing result at the overlapping portion of the projection plane. Further, it is difficult to apply a filter larger than the projection plane.

  In addition, Non-Patent Document 4 (YCSu, et al., “Learning spherical convolution for fast features from 360 imagery”, Advances in Neural Information Processing Systems, 2017.) discloses that a spherical signal is represented by polar coordinates as shown in FIG. There is disclosed a method of converting into an equirectangular form (a projection in which the surface of a sphere is expanded into a cylinder) in which sampling is performed at equal intervals, and then performing a filtering process. In the example shown in FIGS. 1A and 1B, the angle formed by the z-axis and the moving radius v passing through the center O of the sphere is represented by a first angle θ (0 ≦ θ ≦ π), z The angle between the x-axis of the plane perpendicular to the axis and the projection of the radial radius v onto this plane is defined as a second angle -φ (0 ≦ φ <2π), and the signal point P on the surface of the sphere is defined as the surface of the sphere This is an example of projecting (signal point P ′) on an equidistant cylinder in which is expanded into a cylinder.

  In the case of the equirectangular format, the image is enlarged in the horizontal direction as the latitude increases, so that the filter kernel needs to be modified according to the enlargement ratio. For this reason, the higher the latitude, the larger the size of the filter processing to be performed on the signals of many points, and the lower the calculation efficiency.

  As described above, any of the above-described filtering processes on the spherical signal requires a large amount of calculation, which makes it difficult to perform the filtering process with a large kernel, and also has a problem that an artifact occurs.

  The signal processing device according to the first embodiment reduces spherical filter processing to planar filter processing in a specific axis direction. As a result, the spherical filter processing is realized with high approximation accuracy with a small amount of calculation and without generating artifacts and without being limited by the filter size.

(Configuration of signal processing device)
FIG. 2 shows a configuration diagram of the signal processing device according to the first embodiment. As shown in FIG. 2, the signal processing device includes a signal processing unit 1 (an example of a signal processing device) and a storage unit 2. In addition, the signal processing unit 1 includes a signal acquisition unit 11, an axis decomposition unit 12, a plane filter processing unit 13 (an example of a filter processing unit), an axis integration unit 14, and an output unit 15. As the storage unit 2, for example, a semiconductor storage device such as a random access memory (RAM) or a read only memory (ROM), a hard disk drive (HDD), or an optical storage medium can be used. The signal acquisition unit 11 to the output unit 15 can be realized by hardware or software. In the case of realization by software, for example, the signal processing unit reads a signal processing program stored in the storage unit 2, and develops and executes the signal acquisition unit 11 to the output unit 15 in a RAM or the like.

  A part of the signal acquisition unit 11 to the output unit 15 may be realized by software, and the other part may be realized by hardware such as an IC (Integrated Circuit). Further, each processing of the signal acquisition unit 11 to the output unit 15 may be executed by a single signal processing program, or a part of processing may be executed by another program. Alternatively, each processing of the signal acquisition unit 11 to the output unit 15 may be executed indirectly through another program.

  In addition, the signal processing program may be provided by being recorded in a recording medium readable by a computer device such as a CD-ROM or a flexible disk (FD) in a file in an installable format or an executable format. Alternatively, the program may be provided by being recorded on a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), and a semiconductor memory. Further, it may be provided by being installed via a network such as the Internet, or may be provided by being incorporated in a ROM or the like in the device in advance.

(Signal processing operation)
FIG. 3 is a flowchart illustrating a flow of signal processing of the signal processing device according to the first embodiment. An example of performing a filtering process on the input spherical signal in the form of an equirectangular cylinder (FIG. 1) will be described with reference to the flowchart of FIG. First, in step S <b> 1, the signal acquisition unit 11 acquires a spherical signal to be processed (FIG. 5A) from the storage unit 2. At this time, it is assumed that the spherical signal is represented by values on a square grid grid in the form of an equirectangular cylinder as shown in FIG. The example of FIG. 1 shows an example in which the spherical signal “P” shown in FIG. 1A is represented by a value “P ′” on a square grid grid in the equirectangular cylinder format shown in FIG. 1B. ing.

In step S2, the axis disassembling unit 12 performs expansion / contraction processing and decomposition processing on the spherical signal acquired in step S1 in the vertical (θ) direction (FIG. 5B). Specifically, the shaft decomposition section 12, first, the horizontal direction of the signal points of the spherical signal as shown in FIG. 4 (a) and N _w, the size of the vertical direction of the kernel of the filter and M _h, the spherical signal Each row extracting the size partial signal of M _h × N _w around the. Then, as shown in FIG. 4B, the axis disassembling unit 12 converts the extracted partial signal into a radial radius v passing through the coordinate system θ (the z axis and the center O of the sphere shown in FIG. 1A). With respect to the first angle θ (range of 0 ≦ θ ≦ π))), the horizontal direction is reduced at the rate of sin θ. For this reduction processing, for example, nearest neighbor interpolation processing, linear interpolation processing, bicubic interpolation processing, or the like can be used.

  Thereby, geometric distortion can be corrected. Strictly speaking, there is a slight distortion in the portion expanded by the kernel width. However, if the kernel width is small, high approximation accuracy can be obtained. When the kernel is out of the signal, zero padding (zero fill) may be used.

In step S3, the plane filter processing unit 13 performs plane filter processing (FIG. 5C) on the axis-decomposed partial signal. This is an approximation of spherical filtering. Note that it is desirable that the left and right ends of the partial signals be subjected to the filter processing assuming that they are cyclically connected. This process, for each row of the spherical signal, the size of the partial signals of the 1 × N _w can be obtained.

  In step S4, the axis integration unit 14 expands / contracts the partial signals subjected to the plane filter processing, integrates them, returns the signals to the original equidistant cylindrical form, expands them by the inverse of the reduction ratio of 1 / sin θ, and performs integration processing ( 5 (d)) and output (FIG. 5 (e)).

(Effects of the First Embodiment)
The signal processing device according to the first embodiment can perform a filtering process on a spherical signal in the equirectangular form with a small amount of calculation. As the input signal format, for the sake of convenience in explanation, an equirectangular cylinder format was shown as an example, but in the axis resolution processing and axis integration processing, it is sufficient to reflect polar coordinates, so any arbitrary spherical signal expression format is used. Is also good.

  Further, when the input signal is subjected to the axis decomposition process in the vertical direction, the distance on the spherical surface can be reflected by resizing according to the polar coordinate θ shown in FIG. For this reason, it is possible to uniformly perform the filter processing without performing the deformation processing on the kernel. Since the kernel deformation processing can be made unnecessary, as in Non-Patent Document 4 (YCSu, et al., “Learning spherical convolution for fast features from 360 imagery”, Advances in Neural Information Processing Systems, 2017.) In addition, it is possible to eliminate the need for filter processing with a size larger than the actual size, and to reduce the amount of calculation.

Note that resizing processing is required instead of kernel deformation processing. However, if linear interpolation processing is performed, it can be realized by addition and division at most N _w M _h sin θ per row, so that the total amount of calculation is small. It can be.

  Patent Document 1 (Japanese Patent No. 5734327), Patent Document 2 (Japanese Patent No. 6067934), Non-Patent Document 1 (TCCohen, et al., “Spherical CNNs.” ArXiv: 1801.10130, 2018.) In a method using a generalized Fourier transform disclosed in Patent Document 2 (C. Esteves, et al., “3D object classification and retrieval with Spherical CNNs.” ArXiv: 1711.06721, 2017.), the number of signal points N, The calculation amount is O (N log N) for the kernel size M.

  On the other hand, in the case of the signal processing device according to the first embodiment, the calculation amount is O (NM) for the number of signal points N and the kernel size M, which is smaller than the method of performing the Fourier transform processing when the kernel is small. The amount of calculation can be reduced.

  Patent Document 3 (JP-A-2017-207960) or Non-Patent Document 3 (W. Boomsma, et al., “Spherical convolutions and their application in molecular modeling”, Advances in Neural Information Processing Systems, 2017.) For such a method of performing plane projection, the degree of calculation depends on the number of plane divisions and the overlap ratio of the divided planes. However, the signal processing device according to the first embodiment can prevent the inconvenience that an artifact occurs at the overlapping portion of the planar projection, and can prevent the inconvenience that it is difficult to perform the filtering process with a size larger than the projected plane. .

  Summarizing such effects, the signal processing device according to the first embodiment can realize the spherical filter processing with a small amount of calculation, no generation of artifacts, no limitation on filter size, and high approximation accuracy. .

(Second embodiment)
Next, a signal processing device according to a second embodiment will be described. The signal processing device according to the second embodiment is an example in which an integral image is included when filtering processing is performed on a spherical signal, and the amount of calculation can be reduced even for a large-sized filter. is there. Note that the second embodiment differs from the above-described first embodiment in the axis disassembling process of step S2 in the flowchart of FIG. 3 in the axis disassembly unit 12 shown in FIG. Therefore, hereinafter, only the difference between the two will be described, and redundant description will be omitted.

  In the case of the signal processing device according to the second embodiment, when step S2 in the flowchart of FIG. 3 is reached, the axis decomposing unit 12 expands and contracts the spherical signal acquired in step S1 in the vertical (θ) direction and decomposes the spherical signal. At this time, first, the axis disassembling unit 12 applies a non-patent document 5 (P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features”) to the spherical signal in the form of an equirectangular cylinder to be processed. , in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, vol.1.). In the method using the integral image, a cumulative sum from the origin (0, 0) to the position (i, j) is stored in a memory with respect to the signal on the square lattice shown in FIG. This is a method for realizing an arbitrary box filter at high speed.

  Specifically, the axis disassembling unit 12 calculates a recurrence formula represented by the following expression (1) with the signal value of the non-negative integer coordinates (i, j) as f (i, j), and calculates the integral image F Ask for.

  Note that when any argument of the integral image F in Expression (1) is negative, the value of the integral image F is set to “0”. The sum of the signal values f (i, j) from the origin (0,0) to the position (i, j) is recorded in the integral image F (i, j) thus calculated. Therefore, to obtain the sum of the signal values f of the rectangular area surrounded by the position (i1, j1) and the position (i0, j0) shown in FIG. 6, the right side of the following equation (2) may be calculated.

  The number of calculations required to calculate the integral image F is 3N times for addition performed on N signal points based on the equation (1) and a specific rectangular shape performed based on the equation (2). The operation of addition and subtraction of the sum of the signal values of the area is three times. Since this calculation does not depend on the area of the rectangular region for which the sum of the signal values is obtained, the calculation efficiency is particularly high when the sum of the signal values of a large region is obtained.

In step S2, the axis disassembling unit 12 creates such an integral image F with respect to the spherical signal in the equirectangular form. Then, the axis disassembling unit 12 uses the integral image F to perform expansion / contraction processing in the vertical (θ) direction and decomposition processing. The row coordinate theta, directed to a vertical size M _h rows of kernel filter centered on the line. Each of the target rows is decomposed with a horizontal width (rounded to an integer value by rounding or the like) proportional to 1 / sin θ, and the sum of signal points is calculated in the decomposed area.

Example shown in FIG. 7 (a) and 7 (b), the horizontal proportional vertical size M _h pieces of target row of the kernel of the filter around the row coordinate .theta.i, each N _{w /} sinθ _i This is an example in which a signal is decomposed by a width (rounded to an integer value by rounding or the like) and the sum of signal points is calculated in the decomposed region.

Similarly, the example shown in FIG. 7 (c) and FIG. 7 (d), the vertical size _{M h} pieces of target row of the kernel filter centered on coordinates theta _{i + 1} row, each _N w / _{sinθ i + 1} This is an example in which the signal is decomposed with a horizontal width (rounded to an integer value by rounding or the like) proportional to, and the sum of signal points is calculated in the decomposed region.

  Since the horizontal decomposition width depends on 1 / sin θ, when the value of θ changes, the decomposition width changes as can be seen by comparing FIGS. 7B and 7D. By using the integral image, the sum can be calculated with a constant calculation amount without depending on the decomposition width, so that the calculation amount can be reduced. After the spherical signal is axially decomposed as described above, the same filter processing as that in the first embodiment is performed.

  In the above example, since the height of the rectangle that takes the sum of the signal values is 1, it is not actually necessary to create an integral image in the vertical direction. A horizontal one-dimensional integral image (line) may be created for each row. That is, assuming that the signal value at the position (i, j) is f (i, j), the following recurrence formula (formula (3)) is calculated to obtain the integral image F.

  Further, in order to obtain the sum of the signal values f in the region surrounded by the position (i, j1) and the position (i, j0), the right side of the following equation (4) may be calculated.

  When filtering is performed at a resolution lower than the vertical resolution of the signal, the horizontal and vertical two-dimensional integral images are used in the form of the above-described formula (1) to separate the axes according to the resolution of the filter. A signal may be generated. By using such a method, spatially wide filter processing can be performed with a small amount of calculation.

(Effect of Second Embodiment)
The signal processing device according to the second embodiment can perform the filtering process on the spherical signal of the equirectangular form with a small amount of calculation. In particular, it is possible to reduce the amount of calculation when performing axis decomposition using an integral image, and it is also possible to perform filter processing with a large kernel size with a small amount of calculation.

(First Modification of Second Embodiment)
Next, a first modification of the second embodiment will be described. As the integral image, an integral image generated by the recurrence formula of the following expression (5) can be used.

  At this time, the convolution with respect to the power sequence of α can be realized by the calculation on the right side of the following equation (6) (see Patent Document 4 (Japanese Patent Application Laid-Open No. 2006-103089)).

  Here, if α = exp (√−1ω), convolution with a signal of angular frequency ω is possible. Note that the phase can be adjusted by further multiplying by α. Alternatively, even when the integral image F is a two-dimensional vector and α is a real companion matrix, the convolution with the kernel of the angular frequency ω can be performed by performing the operations of the following expressions (7) and (8).

  In this method, since the second element of F (i, j) is equal to the first element of F (i, j-1), the memory usage can be reduced by half compared to the former method using a complex number for F. . If such a power-type integral image is used, a triangular function-type filter coefficient can be realized instead of a box filter that simply takes the sum of signal values. For example, if the filter coefficient is designed so that the maximum value of the trigonometric function is located at the center of the rectangle whose sum is obtained by axial decomposition, a weighted smoothing effect such as a Gaussian filter can be produced.

  Since the horizontal enlargement ratio changes in accordance with the vertical coordinate θ, it is desirable to make the above-mentioned ω proportional to 1 / sin θ. Also, as in Patent Document 4 (JP-A-2006-103089), a sink function (sinc function) is approximated by the sum of a plurality of trigonometric functions, and a resynchronization based on the sinc function is performed using a plurality of power-type integral images. Sizing can be realized. By using the above-described axis decomposition based on the power-type integral image, highly accurate resizing can be realized at high speed, and as a result, highly accurate filter processing can be realized with a small amount of calculation.

(Second Modification of Second Embodiment)
The use of the integral image can speed up multi-scale filtering. For example, in an image signal, an object photographed near a camera and an object photographed far away include signals of similar shapes with different scales, such as different sizes but similar shapes. May have been. There is an application that performs multi-scale filtering on such a signal.

  For example, as shown in FIG. 8A, it is considered that a filter having a filter coefficient of 3 × 3 in height and width is applied on a plurality of scales. As shown in FIGS. 8 (b) to 8 (d) and FIGS. 8 (e) to 8 (g), when taking out three rows by axial decomposition at different resolutions on the equirectangular form, one integral If the calculation is performed from the image by the above-described method, the amount of calculation can be reduced. In this example, the filter coefficient is shared between the scales, but the filter coefficient may be changed according to the scale.

(Third embodiment)
Next, a signal processing device according to a third embodiment will be described. The signal processing device according to the third embodiment is an example in which a kernel of a filter is decomposed in a horizontal and a vertical direction with respect to an input spherical signal to perform a filter process (kernel decomposition of a filter).

  FIG. 9 is a configuration diagram of a signal processing device according to the third embodiment. In the case of the signal processing device according to the third embodiment, as can be seen from comparison with FIG. 2, a horizontal filter processing unit 21 is provided in front of the axis integration unit 14 instead of the plane filter processing unit 13, and the axis integration unit A vertical filter processing unit 22 is provided at a stage subsequent to 14.

  FIG. 10 is a flowchart illustrating a flow of signal processing of the signal processing device according to the third embodiment. In the case of the signal processing device according to the third embodiment, as can be seen from comparison with FIG. 3, instead of the plane filter processing in step S3, the horizontal filter processing in step S11 is performed as preprocessing of the axis integration processing in step S4. Have. Further, instead of the plane filter processing in step S3, the post-axis integration processing in step S4 includes a vertical filter processing in step S12.

  The first embodiment and the third embodiment described above differ only in this point. Therefore, hereinafter, only the difference between the two will be described, and redundant description will be omitted.

In the flowchart of FIG. 10, in step S2, the axis decomposing unit 12 decomposes the input signal in the horizontal direction. However, the axis decomposing unit 12 of the third embodiment converts the input signal into a kernel size in the vertical direction. Decompose as M _h = 1.

Next, in step S11, the horizontal filter processing unit 21 applies a filter in the horizontal direction (kernel size is 1 × M _w ) to the partial signal decomposed as described above. When the filter kernel is given by, for example, a two-dimensional M _h × M _w matrix G, a vertical filter h (M _h) obtained to minimize the square of the Frobenius norm shown in the following equation (9) Dimensional vector) and a horizontal filter w (M _w- dimensional vector).

(9) vertical filter h and the horizontal direction filter w minimizes the square of the Frobenius norm in the expression, the eigenvectors e _h largest eigenvalues α respectively GG ^T and G ^{T _G,} is obtained as a constant multiple of e _w. Proportionality constant is the maximum intrinsic λ of the square root of the GG ^T and G ^{T G,} there is one of uncertainty to the vertical filters h and the horizontal direction filter w many times each. There is an example, but can be a _{^{"h = λ 1/4 e h, w}} = λ 1/4 e w ".

FIGS. 11A and 11B show two decomposition examples of the 3 × 3 filter. In the example shown in FIG. 11A, three filter coefficients “−1, −2, −1”, “0, 0, 0” and “1, 2, 1” arranged three by one in the horizontal direction are vertically set. This is an example in which a filter of a 3 × 3 matrix G formed by laminating in the direction is decomposed into a vertical filter h and a horizontal filter w based on the above equation (9). In the case of this example, the vertical direction filter h is calculated as “−1.3161, 0, 1.3161”, and the horizontal direction filter w ^T is calculated as “0.7598, 1.5197, 0.7598”. This is an example. Note that the resolution error in this case is “0.0”.

In the example shown in FIG. 11B, three filter coefficients “1, 2, 0”, “−1, 4, 1”, and “1, 3, 1” arranged three by one in the horizontal direction are arranged in the vertical direction. This is an example in which a filter of a 3 × 3 matrix G formed in the form of lamination is decomposed into a vertical filter h and a horizontal filter w based on the above equation (9). In this example, the vertical filter h is calculated as “0.8396, 1.7351, 1.3524”, and the horizontal filter w ^T is calculated as “0.0824, 2.2864, 0.5569”. This is an example. The decomposition error in this case is “3.2591”.

  In step S11 in the flowchart of FIG. 9, the horizontal filter processing unit 21 applies the horizontal filter w obtained in this manner to the partial signal.

  Next, in step S12 of the flowchart of FIG. 9, the vertical filter processing unit 22 applies a vertical filter h to the signal subjected to the axis integration processing. Since there is no geometric distortion in the vertical direction of the equirectangular cylinder format, no error occurs even if the vertical filter h is applied as it is after the axis integration processing of the partial signals.

  FIG. 12 is a diagram schematically illustrating a flow of signal processing of the signal processing device according to the third embodiment. As shown in FIG. 12, first, the axis disassembling unit 12 expands and contracts the input signal (spherical signal: FIG. 12A) in the equirectangular format in the vertical (θ) direction (FIG. 12). (B)). The horizontal filter processing unit 21 performs a horizontal filter process by applying a horizontal filter w to the partial signal subjected to the decomposition process.

  The axis integrating unit 14 expands and contracts the partial signals subjected to the horizontal filter processing, integrates the partial signals, and returns the signals to the original equidistant cylindrical signal, thereby forming an integrated signal. The vertical filter processing unit 22 applies a vertical filter h to the integrated signal, performs a vertical filter process (FIG. 12H), and outputs the resultant signal (FIG. 12G).

(Effect of Third Embodiment)
The signal processing apparatus according to the third embodiment uses the filter kernel one-dimensionally decomposed in the horizontal and vertical directions. Thereby, the axis disassembly processing in step S2 of the flowchart of FIG. 10 can be made into one-dimensional signal disassembly processing, and the amount of calculation can be reduced. Further, by using the filter kernel one-dimensionally decomposed in the horizontal direction and the vertical direction, it is possible to prevent the inconvenience of generating geometric distortion in the partial signal.

  Further, as described using the equation (9), the filter matrix G is decomposed into a vertical filter h and a horizontal filter w so as to minimize the square of the Frobenius norm, thereby approximating the two-dimensional filter. Errors can be reduced as much as possible. The criterion of the kernel decomposition may be a formula other than the formula (9), or a general error function such as robustness using the L1 norm (sum of the absolute values of the values of each dimension) may be used.

  Also, as for the filtering process for a signal having a plurality of channels, such as RGB (red, green, blue) in an image signal, "one axis in the horizontal and channel directions", "one axis in the vertical direction", "one axis in the horizontal direction" Or "one axis in which the channel direction and the vertical direction are combined" to form a matrix G of filter kernels and perform axis decomposition in the same manner as described above.

As shown in FIG. 13, the filter matrix G may be decomposed into a plurality of vertical filters and horizontal filters. In this case, the predetermined number obtaining the eigenvalues of GG ^T and G ^{T G} in descending order, may be the eigenvectors with the filter coefficients. In the example of FIG. 13, the matrix G of the filters is set to correspond to each of the RGB (red, green, blue) image signals, and the vertical filters h ₁ , h ₂ , h ₃ and the horizontal filters w ₁ ^T , w ₂ ^{T are used.} , W ₃ ^T. In this _example, the decomposition error of _{^{h 1 w 1 T + h 2}} w 2 T is _0.2723, With _{^{h 1 w 1 T + h 2}} w 2 T + h 3 w 3 T, the decomposition error 0 .0.

  By decomposing the filter matrix G into a plurality of vertical filters and a plurality of horizontal filters in this manner, it is possible to perform filter processing on signals of a plurality of channels, and to reduce filter decomposition errors. Can be.

(Effect of Third Embodiment)
As described above, the signal processing device according to the third embodiment can convert a partial signal into a one-dimensional signal by performing one-dimensional decomposition of the filter. In addition, since generation of geometric distortion can be prevented and the number of calculations can be reduced, high-speed spherical filter processing with high approximation accuracy can be performed. In particular, such an effect becomes more conspicuous as the filter size is larger.

(Fourth embodiment)
Next, a signal processing device according to a fourth embodiment will be described. The signal processing devices of the first to third embodiments are examples in which signal processing of a spherical signal on a secondary spherical surface is performed. On the other hand, the signal processing device according to the fourth embodiment is an example in which the input p-order spherical signal is subjected to filter processing.

The p-order unit sphere S ^p, is the unit sphere in the following (10) p + 1-dimensional Euclidean space of Formula.

The p-th spherical signal f is a mapping from the ^p-th unit spherical surface ^Sp to a set Y of signal values. The set Y is often a set of real numbers or complex numbers. The signal processing device according to the fourth embodiment performs signal processing based on signal values assigned to a finite number of points on a ^p-th unit spherical surface Sp, which is a discrete spherical signal. The representation of the spherical signal, as an example, using a format that extends Equirectangular format S ² to S ^p. Therefore, first, ^S the point x on the ^p p pieces of angle variable θi {i = 1,2, ···, p} by using the following (11) the polar coordinates of points on ^{S p} type Express as:

Here, θ _i ∈ [0 2π] (i <p) and θ _p ∈ [0 π]. The angle variable θ _i is decomposed at equal intervals to obtain a signal on a p-dimensional square lattice. It will be referred to as form of the spherical signal Equirectangular form of order p the unit sphere S ^p.

Here, when the expression (11) is partially differentiated with respect to θ _i and the L2 norm (the sum of squares of the values of each dimension) is obtained, the following expression (12) is obtained.

This is because, in equirectangular format, theta _i direction, shows that the length change in the ratio of the above equation.

  An example of processing on a p-order spherical signal in the signal processing device according to the third embodiment will be described. The configuration of the signal processing device according to the third embodiment is the same as that of the signal processing device according to the first embodiment shown in FIG. 2, except that the axis decomposing unit 12, the plane filter processing unit 13, and the axis integrating unit 14 indicates a different operation.

That is, the shaft decomposition unit 12 decomposes the p-th order spherical signal along the p number of axes of equirectangular format p-th order unit sphere S ^p. First, the shaft decomposition unit 12 decomposes the p-th order spherical signal in the direction of theta _p. Then, the decomposed p-order spherical signal is reduced at a ratio of | sin θ _p |. Next, the axis decomposing unit 12 decomposes the p-th spherical signal in the direction of θ _p −1, and reduces the decomposed signal at a ratio of | sin θ _p−1 |. Shaft decomposition section 12, thus to decompose sequentially selects axis, repeating this until decomposition of theta ₂ axes. By this decomposition, it is possible to correct the ratio of the length on the spherical surface and the equirectangular form (= geometric distortion) in the above equation (12).

If the signal is the θ _i axis, the signal is decomposed by giving a filter kernel width in the θ _i axis direction. If the kernel size is not 1, an overlapping area exists and a slight distortion occurs in that area. However, if the kernel width is small, high approximation accuracy can be obtained.

Next, the plane filter processing unit 13 applies a p-dimensional kernel filter to the partial signal decomposed as described above. The axis integrating unit 14 repeats the processing of integrating the signals with respect to the θ _i axis and enlarging the signal to | 1 / sin θ _i | times to the original p dimension in a process opposite to that of the axis disassembling unit 12.

(Example of processing of cubic unit spherical surface)
FIG. 14 is a schematic diagram showing a flow of a series of processes for a third-order (p = 3) unit spherical surface. Shaft decomposition unit 12 decomposes the third-order spherical signal along the three axes theta ₁ through? ₃ of equirectangular format cubic unit sphere S ³ shown in FIG. 14 (a). First, the shaft decomposition unit 12 decomposes the third-order spherical signal in the direction of the theta _3. Then, the axis disassembling unit 12 reduces the decomposed tertiary sphere signal at a ratio of | sin θ ₃ | as shown in FIG.

Next, the shaft decomposition unit 12 decomposes the third order spherical signal theta ₂ direction (= theta ₃ -1 direction) as shown in FIG. 14 (c), the decomposed signals, | sin [theta ₂ | in a ratio to shrink. Shaft decomposition section 12, thus to decompose sequentially selects axis, repeating this until decomposition of theta ₂ axes. As described above, by this decomposition processing, the ratio of the length on the spherical surface and the equirectangular form (= geometric distortion) in equation (12) can be corrected.

Next, as shown in FIG. 14D, the plane filter processing unit 13 performs a filter process by applying a three-dimensional kernel filter to the partial signal decomposed as described above. Axis Integrated unit 14 is a reverse process to the shaft decomposition section 12, as shown in FIG. 14 (e), integrates the signal to theta ₂ axes | a processing for enlarging doubled _| 1 / sin [theta ₂ To the original three dimensions (enlargement and integration at the ratio of | 1 / sin θ ₃ | in the θ ₃ direction).

(Effect of Fourth Embodiment)
The signal processing device according to the fourth embodiment can realize the filtering of the p-order spherical signal with a small amount of calculation and high approximation accuracy. Incidentally, based on the first embodiment described above, were subjected to this fourth embodiment is applicable integral image processing in the unit sphere S ^p described in the second embodiment. When applying the integral image processing described in the second embodiment in the unit sphere S ^p, extension to p-dimensional integral image is required. This can be achieved by creating an integral image based on the following recurrence formula (13) instead of the formula (1).

It is also applicable to the unit sphere S ^p filtering decomposition treatment also described in the third embodiment. When applied to a filter decomposition treatment also the unit sphere S ^p described in the third embodiment decomposed, for example, by using the CP decomposition algorithm p floor tensor like, the p-dimensional filter p number of one-dimensional filter . Then, on the final degraded signal, performs theta _i direction filtering, then applying the one-dimensional filter in the theta _i direction every time the integrated theta _i axis.

(Fifth embodiment)
Next, a signal processing device according to a fifth embodiment will be described. The signal processing device according to the fifth embodiment is an example in which a multi-stage spherical filter process on an input equirectangular spherical signal is used as a convolutional layer of a convolutional neural network.

(Configuration of Fifth Embodiment)
FIG. 15 shows a configuration diagram of a signal processing unit 1 according to the fifth embodiment. As shown in FIG. 15, the signal processing unit 1 according to the fifth embodiment includes a spherical filter unit 31, an activation function application unit 32, an output unit 33, and a learning unit 34, in addition to the signal acquisition unit 11 described above. .

  The signal acquisition unit 11 acquires a spherical signal to be processed from the storage unit 2 or the like. The spherical filter unit 31 performs any one of the spherical filter processes described in the first to fourth embodiments. The activation function application unit 32 applies an activation function to the processing result of the spherical filter unit 31. As the activation function, for example, an activation function generally used in a neural network, such as a sigmoid function, a hyperbolic function, and a ReLU (Rectified Linear Unit), can be used.

  The spherical filtering and the application of the activation function are repeated a specific number of times. This makes it possible to perform a non-linear conversion process that is difficult to achieve with a single spherical filter process. Note that a layer generally used in a neural network, such as a pooling layer or a normalization layer, may be added.

  The output unit 33 outputs a signal processed by the neural network. The learning unit 34 adjusts the parameters of the neural network so as to minimize a specific loss function for a plurality of pairs of the input signal and the output signal. These parameters include the filter coefficients of the spherical filter, and the values can be obtained by using a general neural network learning method such as a stochastic gradient descent method.

(Effect of Fifth Embodiment)
As described above, the signal processing device according to the fifth embodiment can realize a neural network for spherical signals using spherical filter processing.

(Summary of effects of the embodiment)
The effects of each of the above embodiments will be described below.

  First, in one axial direction of the polar coordinates, expansion and contraction are performed by reflecting the distance on the spherical surface, and the axis is decomposed into a plurality of partial signals, and filtering is performed. As a result, the filtering process on the spherical signal can be realized with a small amount of calculation and high approximation accuracy. In addition, the calculation can be performed without being limited by the filter size, and the occurrence of artifacts can be suppressed.

  In addition, by performing the decomposition while giving the width of the kernel of the filter corresponding to the decomposition direction, it is possible to realize the filter processing on the spherical surface with high approximation accuracy while maintaining the simplicity of the processing.

  In addition, by performing the expansion and contraction reflecting the distance on the spherical surface using the integral image, the speed of the expansion and contraction processing can be increased.

  In addition, the filter kernel is one-dimensionally decomposed along each axis direction of the polar coordinates, and after the axis integration processing, the integrated one-dimensional filter in the axial direction is applied. Thereby, the memory usage w of the partial signal can be reduced. In addition, since the filtering process is not affected by the local geometric deformation of the signal, the spherical filtering process with high approximation accuracy can be performed with a small amount of calculation.

  Further, by sequentially repeating the axis resolution step and the axis integration step p-1 times for the p-dimensional spherical signal, it is possible to realize the filtering process for the p-dimensional spherical signal with a small amount of calculation and high approximation accuracy. .

  Further, by using the spherical signal filter processing as a convolution layer of a convolutional neural network, a convolutional neural network that receives a spherical signal as an input can be realized with a small amount of calculation and high approximation accuracy.

  Finally, the above-described embodiments have been presented by way of example, and are not intended to limit the scope of the present invention. These new embodiments can be implemented in other various forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. Further, each embodiment and modifications of each embodiment are included in the scope and spirit of the invention, and are also included in the invention described in the claims and equivalents thereof.

  For example, the present invention can be implemented by an ASIC (Application Specific Integrated Circuits) or an apparatus configured by connecting conventional circuit modules to a technician having ordinary knowledge in the information processing technology field. is there.

  Further, each function described in each of the above-described embodiments can be realized by one or a plurality of processing circuits (Circuit). The “processing circuit” includes a processor programmed to execute each function by software, an ASIC designed to execute each function, and hardware such as a circuit module.

DESCRIPTION OF SYMBOLS 1 Signal processing apparatus 2 Storage part 11 Signal acquisition part 12 Axis decomposition part 13 Planar filter processing part 14 Axis integration part 15 Output part 21 Horizontal filter processing part 22 Vertical filter processing part 31 Spherical filter part 32 Activation function application part 33 Output part 34 Learning Department

Japanese Patent No. 5734327 Japanese Patent No. 6067934 JP 2017-207960 A JP-A-2006-103089

T.C.Cohen, et al., `` Spherical CNNs. '' ArXiv: 1801.10130, 2018. C. Esteves, et al., `` 3D object classification and retrieval with Spherical CNNs. '' ArXiv: 1711.06721,2017. W. Boomsma, et al., `` Spherical convolutions and their application in molecular modeling '', Advances in Neural Information Processing Systems, 2017. Y.C.Su, et al., `` Learning spherical convolution for fast features from 360 imagery '', Advances in Neural Information Processing Systems, 2017. P. Viola and M. Jones, `` Rapid object detection using a boosted cascade of simple features '', in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, vol.1.

Claims (8)

A signal processing device that performs a filtering process on a spherical signal whose value is assigned to a point on a spherical surface,
An axis disassembly unit that performs an axis disassembly process that expands and contracts the spherical signal in one axial direction of polar coordinates by reflecting a distance on the spherical surface and decomposes the signal into a plurality of partial signals;
A filter processing unit that performs filter processing on the partial signal,
An axis integration unit that performs axis integration processing of integrating the filtered partial signals by expanding and contracting to reflect the distance on the spherical surface,
A signal processing device comprising: The signal processing device according to claim 1, wherein the axis decomposing unit decomposes the image by giving a kernel width of a filter corresponding to a decomposing direction. The signal processing device according to claim 1, wherein the axis disassembling unit performs expansion and contraction reflecting a distance on a spherical surface using an integral image. The method according to claim 1, further comprising a one-dimensional filter processing unit that performs one-dimensional decomposition of the filter kernel along each axis direction of the polar coordinate, and applies a one-dimensional filter in the integrated axial direction after the axis integration unit. Item 2. The signal processing device according to item 1. The axis decomposing unit, the filter processing unit, and the axis integrating unit perform the axis decomposing process and the filtering on a spherical signal in which a value is assigned to a point on a p-dimensional (p is an arbitrary natural number) hypersphere. The signal processing device according to any one of claims 1 to 4, wherein the processing and the axis integration processing are sequentially repeated p-1 times. A convolutional neural network using the signal processing device according to any one of claims 1 to 5 as a convolutional layer. A signal processing method of a signal processing device that performs a filtering process on a spherical signal whose value is assigned to a point on a spherical surface,
An axis disassembling step of performing an axis disassembly process of expanding and contracting the spherical signal in one axial direction of polar coordinates by reflecting a distance on the spherical surface and decomposing the spherical signal into a plurality of partial signals;
A filtering step of performing a filtering process on the partial signal,
An axis integration step of performing an axis integration process of integrating the filter-processed partial signal by expanding and contracting by reflecting a distance on a spherical surface,
A signal processing method comprising: A signal processing program for causing a computer to perform a filtering process on a spherical signal whose value is assigned to a point on a spherical surface,
Said computer,
An axis disassembly unit that performs an axis disassembly process that expands and contracts the spherical signal in one axial direction of polar coordinates by reflecting a distance on the spherical surface and decomposes the signal into a plurality of partial signals;
A filter processing unit that performs filter processing on the partial signal,
The filtered partial signal, functioning as an axis integration unit that performs axis integration processing for integrating by expanding and contracting to reflect the distance on the spherical surface,
A signal processing program characterized by the following.

Images