JP2023013821A - Image processing device, image reproducing device, and program - Google Patents

Abstract

To provide an image processing device, an image reproducing device, and a program capable of detecting attractive regions and performing processing corresponding to wide-angle image data.SOLUTION: An image processing device includes a control unit, a storage unit, an operation unit, a display unit, and an interface unit. The control unit includes: an acquisition unit 22 that acquires image data to be processed; a candidate region extraction unit 23 that extracts a plurality of candidate regions, which are candidates for attractive regions that satisfy a predetermined condition from the acquired image data; a saliency map estimation unit 25 that estimates, based on the acquired image data, saliency map information related to the image data; and a region determination unit 26 that determines attractive regions in the acquired image data, based on the saliency map and information on the candidate regions.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device, an image reproducing device, and a program.

2. Description of the Related Art In recent years, opportunities to view various image data (regardless of whether they are still images or moving images) using computers are increasing. Under such circumstances, it is desired to guide the user to the area to be noticed from each of various image data. Such a demand is conspicuous in wide-angle image data such as omnidirectional images in which a part of the entire image is displayed, and moving image data.

Daniel Martin, Ana Serrano, and Belen Masia. 2020. Panoramic convolutions for 360° single-image saliency prediction. In CVPR Workshop on Computer Vision for Augmented and Virtual Reality

Non-Patent Document 1 discloses a technique for generating a saliency map by detecting characteristic portions in input image data. By using such a technique, it is possible to find a portion of interest from the image data.

However, when using the conventional saliency map, there are the following problems. First of all, since the saliency map shows the distribution of saliency in the image data, it cannot always extract regions that are meaningful to the viewer. Secondly, when generating a saliency map for wide-angle image data such as an omnidirectional image, processing is performed after conversion into a rectangular image such as an equirectangular image. However, when converting to an equirectangular image, the significant part is divided horizontally or vertically, and the intended saliency map may not be obtained, making it difficult to detect the region of interest. .

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and aims to provide an image processing apparatus, an image reproducing apparatus, and a program capable of detecting an attention area and performing processing corresponding to wide-angle image data. , as one of its purposes.

One aspect of the present invention that solves the problems of the conventional example is an image processing apparatus comprising an acquisition unit that acquires image data to be processed, and an image processing apparatus that satisfies a predetermined condition from the acquired image data. Candidate area extraction means for extracting a plurality of candidate areas that are candidates for the attention area; Saliency map estimation means for estimating saliency map information related to the acquired image data based on the acquired image data; determining means for determining a region of interest in the acquired image data based on the map and information of the candidate region.

According to this image processing apparatus, a region of interest can be detected, and processing corresponding to wide-angle image data can be performed.

1 is a block diagram showing a configuration example of an image processing device according to an embodiment of the present invention; FIG. 1 is a functional block diagram showing an example of an image processing device according to an embodiment of the present invention; FIG. 1 is an explanatory diagram showing an example of a virtual celestial sphere and its coordinate system used by an image processing apparatus according to an embodiment of the present invention; FIG. It is a flowchart figure showing the operation example of the image processing apparatus which concerns on embodiment of this invention. FIG. 5 is another flowchart showing an operation example of the image processing device according to the embodiment of the present invention; FIG. 9 is yet another flowchart showing an operation example of the image processing device according to the embodiment of the present invention; FIG. 4 is an explanatory diagram showing an output example of information generated by the image processing apparatus according to the embodiment of the present invention;

An embodiment of the present invention will be described with reference to the drawings. An image processing apparatus 1 according to an embodiment of the present invention includes a control section 11, a storage section 12, an operation section 13, a display section 14, and an interface section 15, as illustrated in FIG.

Here, the control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12 . In this embodiment, the control unit 11 acquires image data to be processed, and extracts a plurality of candidate regions that are candidates for attention regions that satisfy a predetermined condition from the acquired image data. In addition, the control unit 11 receives the acquired image data as an input, performs machine learning processing for performing machine learning on a neural network for estimating saliency map information related to the image data, and and an estimation process of estimating saliency map information related to the acquired image data using a neural network that has undergone machine learning to estimate saliency map information corresponding to the image data.

Here, the saliency map information is image data having the same shape as the image data to be processed, and the pixel value (here, luminance) of each pixel corresponds to the image data to be processed. is a value (scalar value) that represents the salience of the image portion represented by the pixel (or group of pixels) of .

Then, the control unit 11 determines a region of interest in the acquired image data based on the saliency map obtained by the estimation process and information on the candidate region. The control unit 11 stores the determined attention area information in the storage unit 12 in association with the image data. These operations of the control unit 11 will be described later in detail.

The storage unit 12 is a memory device or disk device, and holds programs executed by the control unit 11 . This program may be provided by being stored in a computer-readable, non-temporary recording medium and stored in the storage section 12 . The storage section 12 also operates as a work memory for the control section 11 .

The operation unit 13 is a keyboard, a mouse, or the like, and receives an operation of the viewer of the image data and outputs the content of the operation to the control unit 11 . The display unit 14 is a display or the like, and displays information according to instructions input from the control unit 11 .

The interface unit 15 includes a network interface, a USB (Universal Serial Bus), etc., receives various information such as image data from an external computer device, and outputs the information to the control unit 11 . The interface unit 15 also outputs information to an indicated output destination (computer equipment, storage device, etc.) in accordance with an instruction input from the control unit 11 .

Next, the operation of the control section 11 of this embodiment will be described. By executing the program stored in the storage unit 12, the control unit 11 functionally functions as an acceptance unit 21, an acquisition unit 22, a candidate region extraction unit 23, a saliency It includes a map learning processing unit 24 , a saliency map estimation unit 25 , an area determination unit 26 and an output unit 27 .

Here, the receiving unit 21 receives data to be subjected to machine learning processing or estimation processing. Here, the object of the machine learning process is the saliency map estimation unit 25, and the data to be the object of the machine learning process are image data and correct saliency map information (hereinafter referred to as , called teacher map data). Data to be subjected to estimation processing is image data. If the image data included in the data targeted for machine learning processing is moving image data, the data targeted for machine learning processing includes teacher map data corresponding to each frame of the moving image data. It is assumed that there is

In addition, the image data included in the data targeted for machine learning processing and the data targeted for estimation processing can be video data, still image data of omnidirectional images, video data of omnidirectional images, etc. Among them, it shall be data of one kind.

The acquiring unit 22 acquires at least one piece of image data or the like to be processed from the data received by the receiving unit 21 . The processing of this acquisition unit 22 may differ depending on the type of data received by the receiving unit 21 and the content of processing (whether it is machine learning processing or estimation processing).

[For wide-angle image data]
For example, if the image data received by the receiving unit 21 is wide-angle image data projected to a range exceeding at least a hemisphere of the celestial sphere, such as still image data of an omnidirectional image (including omnidirectional images projected on the entire celestial sphere). For example, the acquisition unit 22 projects the image projected on the celestial sphere (in the case of wide-angle image data that does not cover the entire celestial sphere, the wide-angle image data is projected onto the virtual celestial sphere. In this case, the part without the image is can be set to a predetermined pixel value), the following processing is performed.

That is, in any process, the acquisition unit 22 takes the center of the virtual celestial sphere on which the wide-angle image data is projected as shown in FIG. For example, the central meridian C is the semicircle where the YZ plane intersects the celestial sphere in the positive direction of the X axis, and the predetermined standard latitude line D (a plane parallel to the XY plane where Z = z (where z is It is a predetermined value in the range of 0 ≤ z ≤ r, where r is the radius of the celestial sphere. Wide-angle image data projected onto the celestial sphere is converted into a rectangular equirectangular image. For example, when performing estimation processing, the acquisition unit 22 may determine the parameters of the standard parallel and the central meridian according to the viewer's operation. As an example, the acquisition unit 22 sets the parameters of the standard parallel and the central meridian so that the line-of-sight direction specified by the viewer is the center of the converted equirectangular image, and performs conversion to the equirectangular image. .

In addition, at least when performing machine learning processing, the acquisition unit 22 rotates the virtual celestial sphere around the Z-axis (from the center of the celestial sphere to the zenith direction) at angles θ1, θ2, . The direction is rotated, and the meridian of the direction rotated from the positive direction of the X axis at each angle is set as the central meridian, and the wide-angle image data is converted into a rectangular shape using each of these central meridians and a predetermined standard latitude as parameters. A plurality of equirectangular images may be obtained by converting to an equirectangular image of the shape.

This process corresponds to a process of acquiring a plurality of image data to be processed while rotating the celestial sphere on which the wide-angle image data is projected.

Furthermore, the acquisition unit 22 not only rotates around the Z-axis (rotation in the horizontal direction) as described above, but also rotates around each coordinate axis with another axis as the rotation axis, as processing performed at least when performing machine learning processing. (Because the center of the celestial sphere is the origin, each coordinate axis must intersect at this origin.) Then, the meridian in the positive direction of the X-axis after rotation by each angle pair is defined as the central meridian, and the wide-angle image data is converted into a rectangular shape using a predetermined standard latitude after rotation by each angle pair as parameters. , to obtain a plurality of equirectangular images (each centered on a different line-of-sight direction).

At least when performing machine learning processing, the acquisition unit 22 outputs a plurality of equirectangular images obtained by the above method as image data to be processed. Also, when performing the estimation process, the acquisition unit 22 may output only one rectangular equirectangular image obtained based on the wide-angle image data. The coordinates of each pixel of the equirectangular image obtained here and the coordinates on the celestial sphere onto which the wide-angle image data is projected can be mutually converted.

Furthermore, when performing machine learning processing, the acquisition unit 22 also sets the teacher map data associated with the received image data to the same range as the projected image data associated with the virtual celestial sphere, similarly to the image data. Project to range.

Then, the acquisition unit 22 obtains the rotation angles θ1, θ2, . …) is used to rotate the celestial sphere onto which the teacher map data is projected around each coordinate axis at each angle, and the meridian in the positive direction of the X axis after rotation at each angle (or each angle set) is taken as the central meridian. , a predetermined standard parallel (as already mentioned, a plane parallel to the XY plane where Z=z (where z is a predetermined value in the range of 0≦z≦r, where r is the radius of the celestial sphere ) is a circle formed by cutting the celestial sphere, and if it is further rotated around the X axis as φ1, φ2, …, and ψ1, ψ2, … around the Y axis, then Z = z to define the standard parallel corresponding to each angle), the teacher map data is converted into a rectangular equirectangular image (hereinafter referred to as an equirectangular image of the image data). , called teacher images) to obtain a plurality of teacher images.

As a result, the acquiring unit 22 acquires a plurality of teacher images corresponding to each of the plurality of equirectangular images (each centered on a different line-of-sight direction) obtained based on the wide-angle image data.

[For moving image data]
Further, when the image data included in the data received by the receiving unit 21 is moving image data, the acquiring unit 22 performs the moving image data regardless of whether the machine learning process or the estimation process is performed. generates a series of still image data based on The still image data may be obtained by extracting key frame data (for example, I frame) included in the moving image data, or may be still image data of each frame obtained by reproducing the moving image data. may The obtaining unit 22 outputs the still image data (frames) at a plurality of playback points obtained by the above method as image data to be processed.

Further, when performing machine learning processing, the acquiring unit 22 extracts teacher map data corresponding to the frame at each reproduction time point, which is the image data to be processed, from the data received by the receiving unit 21 .

[For wide-angle video image data]
Furthermore, when the data received by the receiving unit 21 is wide-angle image data projected to a range exceeding at least a hemisphere of the celestial sphere, such as still image data of an omnidirectional image, and is moving image data, the acquiring unit 22 may be processed as follows:

When performing machine learning processing, the acquiring unit 22 in this example preliminarily sets the rotation angles θ1, θ2, . , . . . are randomly determined by uniform random numbers.

Then, the acquisition unit 22 acquires the wide-angle image data of the still image by extracting the key frame data (for example, I frame) of the wide-angle image included in the moving image data, or obtains the wide-angle image data by reproducing the moving image data. Acquire wide-angle image data (of a still image) for each frame at multiple playback time points.

Then, when performing the estimation process, the acquisition unit 22 projects each of the obtained wide-angle image data of the still image onto a virtual celestial sphere, and projects the wide-angle image data onto the virtual celestial sphere in a predetermined direction ( For example, the direction set by the viewer) is set to the center of the equirectangular image after conversion. to obtain equirectangular images corresponding to each keyframe, and output them. Note that if the estimation process is to be executed during playback of moving image data, the acquisition unit 22 will, when generating the image based on the keyframes, set the direction set by the viewer at that time (for example, forward direction of the viewer). ) is the center of the equirectangular image after transformation, and the wide-angle image data projected on the celestial sphere is converted into a rectangular equirectangular image by setting the parameters of the central meridian and standard parallels. Transform and output the transformed equirectangular image.

When performing machine learning processing, the acquiring unit 22 projects each of the wide-angle image data of the obtained still image onto a virtual celestial sphere, and determines the virtual celestial sphere onto which the wide-angle image data is projected. Rotate around the corresponding axis by the rotation angle. The acquisition unit 22 sets the meridian in the positive direction of the X axis after rotation by each angle (or each angle set) as the central meridian, and the predetermined standard parallel (as already described, on the XY plane where Z = z) It is a circle formed by cutting the celestial sphere with parallel planes (where z is a predetermined value in the range of 0≤z≤r, where r is the radius of the celestial sphere). . . , ψ1, ψ2, . Using the central meridian as a parameter, the wide-angle image data projected onto the celestial sphere is converted into rectangular equirectangular images to obtain a plurality of equirectangular images. This process corresponds to a process of rotating the celestial sphere onto which the wide-angle image data is projected and converting it into rectangular image data to obtain a plurality of image data to be processed.

The acquiring unit 22 outputs, as image data to be processed, a plurality of equirectangular images for each frame at a plurality of reproduction time points obtained by the above method. Also in this example, the coordinates of each pixel of the obtained equirectangular image and the original coordinates on the celestial sphere can be mutually converted.

Furthermore, when performing the machine learning process, the acquiring unit 22 also obtains the associated moving image data of the virtual celestial sphere for the teacher map data associated with the received moving image data as well as the moving image data described above. Project to the same range as the projected range.

Then, the acquisition unit 22 obtains a plurality of equirectangular images for each frame, and calculates the rotation angles θ1, θ2, . . . (or (φ1, ψ1, θ1), (φ2 , ψ2, θ2), …), rotate the celestial sphere onto which the training map data corresponding to the frame is projected around each coordinate axis at each angle, and after rotating at each angle (or each angle set) The meridian in the positive direction of the X axis is defined as the standard meridian, and this standard meridian and a predetermined standard parallel (as already mentioned, a plane parallel to the XY plane where Z = z (where z is in the range of 0 ≤ z ≤ r is a predetermined value, where r is the radius of the celestial sphere). In the case of rotation, the standard latitude corresponding to each angle is determined by Z=z after rotation by each set of angles. A plurality of teacher images corresponding to the image data to be processed are obtained for each frame by converting into an equirectangular image (teacher image) of the shape.

As a result, the acquisition unit 22 obtains, for each of a plurality of frames obtained based on the wide-angle image data of the moving image, a Obtain multiple teacher images.

[When the received image data is non-wide-angle still image data]
If the data received by the receiving unit 21 is still image data that has a relatively small angle of view and is not wide-angle, the acquiring unit 22 outputs the received data as it is as image data to be processed.

In this example, the acquiring unit 22 outputs teacher map data associated with the received image data when performing machine learning processing.

[Process after obtaining image data to be processed]
The candidate area extraction unit 23 operates when the image processing apparatus 1 performs estimation processing, and extracts candidate areas Ik (k=1, 2, . . n) are extracted, and information representing the extracted candidate regions is output. The candidate regions Ik extracted here are extracted with overlapping regions allowed. For example, the candidate area Ii and the candidate area Ij (i≠j) may overlap.

As an example, the candidate region extraction unit 23 performs a so-called selective search (Selective Search: J.R. Uijlings et al., "Selective Search for object recognition", International journal of computer vision, Vol.104, No.2, pp.154- 171 (2013)) to extract a plurality of candidate regions. The information representing the candidate area is, for example, the coordinates (x, y) of the upper left vertex of the candidate area on the image data to be processed, and the height and width (h, w) of the candidate area. should be included.

However, this method is only an example, and the candidate area extraction unit 23 may extract the candidate area by another process. An example of this alternative processing will be described later.

The saliency map learning processing unit 24 operates during learning processing, and uses the image data output by the acquisition unit 22 and the corresponding teacher map data to perform, for example, D. Martin, et al., “Panoramic convolutions for 360 single -image saliency prediction”, machine learning neural networks such as convolutional networks in CVPR Workshop on Computer Vision for Augmented and Virtual Reality, 2020. Then, by this machine learning, the neural network receives input of image data, estimates and outputs a remarkable area of the image data, and performs machine learning.

A widely known method can be adopted for this machine learning method, so a detailed explanation is omitted here. By converting the corresponding teacher map data into a plurality of equirectangular images with different center directions by rotating the celestial sphere onto which the wide-angle image data and teacher map data are projected, enrichment of learning data is achieved. That's what it was.

The saliency map estimation unit 25 operates during estimation processing, and inputs each image data to be processed output by the acquisition unit 22 to the neural network machine-learned by the saliency map learning processing unit 24 . The saliency map estimation unit 25 then estimates saliency map information corresponding to each image data output by the neural network.

The region determination unit 26 operates during estimation processing, and selects an image to be processed based on the saliency map information estimated by the saliency map estimation unit 25 and the candidate region information generated by the candidate region extraction unit 23. Determine the region of interest in the data. As an example, the region determination unit 26 evaluates a set of candidate regions based on the degree of overlap between the candidate regions and saliency information within the candidate regions obtained from the saliency map information, while determining the candidate regions. The set is corrected, such as by reduction, and the region of interest is determined.

Specifically, on the saliency map information, the sum of values representing the saliency corresponding to the inside of the candidate region Ik is expressed as g(Ik), and the overlapping rate (degree of overlap) between the candidate regions Ii and Ij is When expressed as IoU (Ii, Ij), the evaluation value SIoU of the set S of area candidates Ik is defined as follows.

The region determination unit 26 determines the sum g(Ii) of the values representing the salience corresponding to the inside of each of the candidate regions Ii included in the candidate region set R extracted by the region candidate extraction unit 23. keep asking

The region determination unit 26 extracts a predetermined number n of candidate regions I1, I2, . . . Here, the conditions are, for example, conditions for extracting n candidate regions in descending order of g(Ii). The region determination unit 26 obtains the evaluation value SIoU(S) of the set S of the extracted n candidate regions I1, I2, . . .

Thereafter, the area determination unit 26 sequentially extracts the candidate areas Ij that are not extracted from the set S from the candidate area set R, and executes the following processing. In other words, the area determining unit 26 replaces any of the candidate areas I1, I2, . Among the candidate areas included in the set S, the candidate areas to be replaced may be randomly determined by uniform random numbers, for example.

The area determining unit 26 obtains the evaluation value SIoU(S') from (1) for the set S' obtained by the replacement. Area determination unit 26 then compares the evaluation value SIoU(S) of set S with the evaluation value SIoU(S') of set S', and if SIoU(S')<SIoU(S), Set S′ is replaced with set S, and set S is updated. If SIoU(S')<SIoU(S), the area determining unit 26 leaves the set S as it is.

The area determination unit 26 extracts the next candidate area Ij from the candidate area set R, generates another set S', and repeats the above process. After performing the above process on all of the candidate areas included in the set R (other than the candidate areas included in the initial set S), the area determination unit 26 determines n , and outputs information specifying the n regions as the regions of interest.

The output unit 27 records information specifying the area output by the area determining unit 26 in association with the image data to be processed or the original data thereof. Specifically, the output section 27 performs the following processing according to the type of data received by the receiving section 21 .

[For wide-angle image data]
For example, if the data received by the receiving unit 21 is wide-angle image data, the output unit 27 uses the information for specifying the area output by the area determining unit 26 for each image data to be processed as follows. process. In this example, the information specifying the area output by the area determination unit 26 is specified by coordinates on the image data to be processed. Therefore, the output unit 27 converts the information specifying the area output by the area determining unit 26 into information on the area on the virtual celestial sphere onto which the wide-angle image data is projected. This transformation may be performed in the reverse manner to the transformation of the image of the celestial sphere into the equirectangular image.

Then, the output unit 27 records the information of the converted area in association with the wide-angle image data received by the receiving unit 21 . The output unit 27 executes the above processing for each of the image data to be processed, and records information of each attention area on the virtual celestial sphere when the wide-angle image data is projected onto the virtual celestial sphere. .

[For moving image data]
Further, when the data received by the receiving unit 21 is moving image data, the output unit 27 outputs the information specifying the area output by the area determining unit 26 for each image data to be processed, Information relating frame numbers of image data (a number indicating the order of reproduction with the first frame at the time of reproduction being "1" and information indicating the reproduction time) is generated. The output unit 27 then records the generated information in association with the moving image data received by the receiving unit 21 .

[For wide-angle video image data]
Furthermore, when the data received by the receiving unit 21 is wide-angle image data projected to a range exceeding at least a hemisphere of the celestial sphere, such as still image data of an omnidirectional image, and is moving image data, this output unit 27 processes as follows.

In this example, the output unit 27 outputs the information specifying the area output by the area determination unit 26 for each image data to be processed, as in the case of wide-angle image data of a still image, on a virtual celestial sphere. Convert to area information. Then, the output unit 27 generates information that associates the information representing the frame number of the image data to be processed with the information on the virtual celestial sphere obtained by the conversion. The output unit 27 records the information generated here in association with the data received by the receiving unit 21 .

[For non-wide-angle still image data]
If the data received by the receiving unit 21 is still image data that has a relatively small angle of view and is not wide-angle, the output unit 27 causes the area determination unit 26 to output based on the image data to be processed. The information specifying the area is recorded as it is in association with the data of the still image received by the receiving unit 21 .

[motion]
An example of the operation of the image processing apparatus 1 according to the present embodiment will be described below using a case in which input data is moving image data of a wide-angle image as an example. I will explain separately.

[During learning process]
During the learning process, the image processing apparatus 1 uses wide-angle image data, which is image data, as data to be machine-learned (the image data does not necessarily have to be moving images for machine learning; (assumed to be video data of images) and saliency map information (hereinafter referred to as teacher map data) that is the correct answer when this image data is input.

The image processing apparatus 1 randomly determines a plurality of rotation angles of the celestial sphere on which a wide-angle image is projected using uniform random numbers. The rotation angles of the celestial sphere are defined as (φ1, ψ1, θ1), (φ2, ψ2, θ2), .

The image processing apparatus 1 extracts key frame data (for example, I frame) included in the moving image data from the received data, and acquires a plurality of wide-angle image data of still images. The image processing apparatus 1 obtains celestial image data projected onto a virtual celestial sphere for each of the acquired wide-angle image data of still images.

The image processing apparatus 1 also acquires teacher map data corresponding to the wide-angle still images of the key frames extracted from the moving image data, among the received data. The image processing apparatus 1 obtains data projected onto a virtual celestial sphere for each acquired teacher map data, similarly to the wide-angle image data.

The image processing apparatus 1 rotates the virtual celestial sphere onto which the image data and the teacher map data are projected, at each of the previously determined rotation angles, around the corresponding axis, and rotates the corresponding angle (or set of angles). Let the meridian in the positive direction of the X axis after rotation be the central meridian, and this central meridian and a predetermined standard parallel (the celestial sphere after being rotated by each angle or each angle set is placed on the XY plane where Z = z A wide-angle projected onto the celestial sphere, using a parallel plane (where z is a predetermined value in the range of 0 ≤ z ≤ r, where r is the radius of the celestial sphere) as a parameter. Image data and teacher map data are converted into rectangular equirectangular images. It is assumed that the coordinates of each pixel of the equirectangular image and the original coordinates on the celestial sphere can be mutually converted.

The image processing apparatus 1 executes the process of obtaining this equirectangular image for each of the previously determined rotation angles, and obtains a plurality of equirectangular images for each of the wide-angle image and the corresponding teacher map data.

The image processing apparatus 1 repeatedly executes this process for each of the wide-angle image of the key frame and the corresponding teacher map data. As a result, the wide-angle image data for each keyframe is projected onto the celestial sphere, and the wide-angle image data for each keyframe projected onto the celestial sphere is transformed with mutually different parameters to obtain a plurality of equirectangular images and the corresponding teacher map. Obtain an equirectangular image of the data. Here, the celestial sphere onto which each keyframe is projected is rotated using the same set of rotation angles, but it may be rotated using a different set of rotation angles for each keyframe.

Then, the image processing device 1 uses the set of the image data obtained here and the corresponding teacher map data to perform, for example, D. Martin, et al., “Panoramic convolutions for 360 single-image saliency prediction”, CVPR Workshop Machine learning neural networks such as convolutional networks in on Computer Vision for Augmented and Virtual Reality, 2020. Through this machine learning, the neural network receives input of wide-angle image data and outputs saliency map information that estimates the salient area of the neural network.

After this machine learning process, the image processing apparatus 1 can execute the estimation process. When performing the estimation process, the image processing apparatus 1 receives moving image data of, for example, omnidirectional images as data of moving images of wide-angle images. The image processing apparatus 1 acquires image data to be processed from the received data as follows.

As illustrated in FIG. 4, the image processing apparatus 1 extracts wide-angle image key frame data (for example, I frame) included in the moving image data to acquire wide-angle image data of a still image (S11), Each of the obtained wide-angle image data (here, omnidirectional image) of the still image is projected onto a virtual celestial sphere (S12).

The image processing apparatus 1 uses the central meridian as the meridian in the positive direction of the X axis of the virtual celestial sphere on which the omnidirectional image is projected, and uses this central meridian and a predetermined standard parallel as parameters to create a wide-angle image projected onto the celestial sphere. The data is converted into a rectangular equirectangular image (S13). It is assumed that the coordinates of each pixel of the equirectangular image and the original coordinates on the celestial sphere can be mutually converted.

The image processing apparatus 1 outputs the equirectangular image obtained in step S13 as image data to be processed together with information (for example, a frame number) specifying the key frame used as the celestial image data in step S11 (S14). .

The image processing apparatus 1 repeatedly executes the processes S13 and S14 for each of the key frame data extracted in the process S11. Thus, an equirectangular image generated for each key frame is obtained as image data to be processed.

Further, as illustrated in FIG. 5, the image processing apparatus 1 sequentially selects each of the equirectangular images corresponding to the plurality of frames to be processed (S21), and uses the selective search method described above. Then, a plurality of candidate areas Ik (k=1, 2, . . . n) are extracted from the selected equirectangular image (S22). The image processing apparatus 1 also estimates saliency map information related to the selected equirectangular image using a machine-learned neural network or the like (S23).

Then, the image processing apparatus 1 determines a region of interest from each of the image data to be processed, based on the candidate region information and the saliency map obtained in steps S22 and S23 (S24).

In this process S24, the image processing apparatus 1, as exemplified in FIG. , the sum g(Ii) of the values representing the saliency obtained in step S23 is obtained (S241).

により、当該集合Ｓの評価値ＳＩｏＵ（Ｓ）を求め（Ｓ２４３）、現在の集合Ｓの評価値ＳＩｏＵ（Ｓ）とする。 Also, the image processing apparatus 1 extracts a predetermined number n of candidate areas I1, I2, . . . The image processing apparatus 1 calculates the set S of the extracted n candidate regions I1, I2, . . . In by formula (1):

Then, the evaluation value SIoU(S) of the set S is obtained (S243) and set as the current evaluation value SIoU(S) of the set S.

Next, the image processing apparatus 1 sequentially selects the candidate areas Ij that do not belong to the set S (original set S) from among the candidate areas belonging to the set R of the candidate areas (S244). is replaced with any of the candidate areas I1, I2, . . . In included in the set S to generate the set S' (S245). Among the candidate areas included in the set S, the candidate areas to be replaced are randomly determined by uniform random numbers, for example.

The image processing apparatus 1 obtains the evaluation value SIoU(S') from the equation (1) for the set S' obtained by the replacement. Then, the image processing apparatus 1 compares the current evaluation value SIoU(S) of the set S with the evaluation value SIoU(S') of the set S' obtained in the process S245, and obtains the evaluation value SIoU(S') of the set S'. S') is smaller (S246). If SIoU(S')<SIoU(S) (S246: Yes), the set S' is replaced with the set S and the set S is updated (S247). Also, the evaluation value SIoU(S') of the set S' obtained in the process S245 is newly set as the current evaluation value SIoU(S) of the set S (S248: Update evaluation value).

On the other hand, if SIoU(S')<SIoU(S) is not satisfied in step S246 (S246: No), the image processing apparatus 1 leaves the set S as it is.

The image processing apparatus 1 repeatedly executes the processes from S244 to S248 for each of the candidate areas Ij that do not belong to the initial set S among the candidate areas belonging to the set R of the candidate areas. The n regions represented by the candidate regions included in the set S obtained after the above are set as the regions of interest, and information specifying the n regions is obtained.

The image processing apparatus 1 returns to the process of FIG. 5 and records the information of the attention area determined for the image data to be processed (S25). At this time, the image processing apparatus 1 records the information specifying the key frame that is the source of the image data to be processed in association with the information of the attention area.

The image processing apparatus 1 repeatedly executes the processing from S21 to S25, and when the processing for all image data to be processed is completed, each piece of information specifying the recorded key frame is recorded in association with the information. Extract the information of the region of interest. The information processing apparatus 1 converts the extracted information of the attention area into information of the coordinates on the celestial sphere when the key frame is projected onto the celestial sphere, and converts the information of the attention area represented by the coordinates on the celestial sphere to the key. It is stored in the storage unit 12 as a region-of-interest database in association with information specifying a frame (for example, the frame number of a keyframe, which is information related to the reproduction time of the keyframe) (S26).

Then, the image processing apparatus 1 associates this attention area database with the input moving image data of the omnidirectional image and outputs them (S27).

[Playback process]
An information processing apparatus (such as a personal computer, which corresponds to the image reproducing apparatus of the present invention) that reproduces image data uses the attention area information (attention area database) generated by the image processing apparatus 1 in this way. and perform the following processing. The information processing device reproduces a series of still images (frames) to be sequentially displayed based on moving image data of omnidirectional images.

Here, since each frame is an omnidirectional image, the information processing device sequentially projects the omnidirectional image of each frame onto a virtual celestial sphere and creates a virtual image centered on the celestial sphere. Render and display the image seen from the camera. Since this process can employ a method widely known as a process for displaying an omnidirectional image, a detailed description thereof will be omitted here.

Further, when rendering the projected image of each frame, if the attention area database has information on the attention area associated with the frame number of the frame, the information processing apparatus renders the area on the celestial sphere represented by the information. The graphic image of the enclosing frame (FIG. 7(X)) is superimposed on the rendered image of the frame and displayed. FIG. 7 shows an example in which a region of interest (X) is set near the entrance of one of the buildings lined up on the left side in celestial image data of a moving image taken from a vehicle traveling on the road.

Note that the information processing device that displays wide-angle image data may change the angle of view, such as the line-of-sight direction of a camera placed at the center of the virtual celestial sphere, in response to an instruction from the viewer. This allows the viewer to refer to different points on the celestial sphere.

In the above example, only the keyframes are associated with the attention area information. Based on the information on the attention area associated with the , each frame (playback time ti+1, ti +2, etc.) may be overlaid on the image and drawn.

In this way, the display based on the information of the attention area associated with the information specifying the keyframe Kj is displayed from the time when a certain keyframe Kj is played until the time when the next keyframe Kj+1 is played. It will be done.

Also, the method of displaying the information of the attention area is not limited to this example. An information processing device for displaying wide-angle image data (corresponding to the image reproducing device of the present invention) receives attention area information associated with information specifying each key frame (attention area information at the time of reproduction of each key frame). See For each of the attention areas specified by the referenced information, the information processing apparatus determines the time at which the frame associated with the attention area should be reproduced (here, the reproduction time of the key frame specified by the information associated with the attention area). ), when reproducing a frame (not limited to a key frame) to be reproduced at a point in time t-Δt (where Δt>0) determined by a predetermined method before t, specified by the information referred to above. information about the region of interest may be displayed. In this example, the position of the attention area is indicated at time Δt before the display of the key frame in which the image to be the attention area is captured (or at the time of reproducing the frame whose frame number is a predetermined number before). A graphic image of the frame and the like are drawn. This allows the viewer to know the position of the attention area before the attention area appears.

Although an example of reproducing a moving image of a wide-angle image is shown here, when still image data associated with information representing an attention area is displayed, an information processing device that displays the still image data A still image represented by is displayed and output, and a graphic image of a frame surrounding the area represented by the information is superimposed and drawn on the displayed image according to the information of the associated attention area.

When displaying still image data of a wide-angle image, the information processing device projects the wide-angle image onto a predetermined virtual celestial sphere, and an image viewed from a virtual camera placed at the center of the celestial sphere. is rendered and displayed. Then, the information processing apparatus draws and displays a graphic image of a frame surrounding the area on the celestial sphere represented by the information of the associated attention area so as to be superimposed on the rendered image. In this case as well, the information processing apparatus may change the angle of view, such as the line-of-sight direction of the camera placed at the center of the virtual celestial sphere, in response to the viewer's instruction. This allows the viewer to refer to different points on the celestial sphere.

In the case of moving image data that is not a wide-angle image, as already explained, the information representing the attention area is recorded in association with the information of the corresponding frame number. Therefore, an information processing apparatus that reproduces the image data reads the record, starts reproducing the moving image data based on the image data, and records each frame in association with the frame number information of the frame. If there is information representing a focused area, a graphic image of a frame surrounding the focused area represented by the information is superimposed on the displayed image (image of each frame) and drawn. Also, if there is no information representing the attention area recorded in association with the information of the frame number of the frame, drawing of the graphic image of the frame enclosing the attention area currently superimposed and drawn may be continued.

Also, when reproducing moving image data, it is not always necessary to present the viewer with all the attention areas specified by the associated information. For example, the information processing apparatus defines an integer P0 of 1 or more and an integer P of 2 or more, and defines a frame surrounding a region of interest specified by information associated with frame numbers whose values are P0, P0+P, P0+2P, . A graphic image may be displayed. Also, while the frame number f is P0+(k-1)P<f<P0+kP (where k=1, 2, . . . ), it is specified by information associated with the frame number P0+(k-1)P or P0+kP display a graphical image of a frame surrounding the region of interest. According to this example, the information of the attention area is updated once every P frames are reproduced.

[Example of determining attention area during playback]
If the image data is moving image data or moving image data of a wide-angle image, the region of interest may be determined during reproduction of the image data (that is, during estimation processing). In this example, when reproducing moving image data of a wide-angle image, the image processing apparatus 1 determines whether the image to be rendered for reproduction is a key frame, instead of step S11 in the processing of FIG. If it is not a key frame, the image of the frame is projected onto a virtual celestial sphere, and the image viewed from the virtual camera placed at the center of the celestial sphere is rendered in the direction specified by the viewer. indicate.

On the other hand, when rendering the wide-angle image of the key frame for reproduction, the image processing apparatus 1 executes step S12, and then performs the following processing in the processing of step S13. In this example, the image processing device 1 selects a point on the virtual celestial sphere onto which the wide-angle image of the key frame is projected, in the direction specified by the user during playback (for example, in front of the viewer standing at the center of the celestial sphere). The wide-angle image data is converted into a rectangular equirectangular image so that the desired line-of-sight direction, etc., becomes the center of the converted equirectangular image. The image processing apparatus 1 then uses this equirectangular image to execute processing for estimating a region of interest.

[Another example of candidate region extraction]
In the description of the present embodiment so far, the image processing apparatus 1 executes selective search when extracting candidate regions, but the present embodiment is not limited to this example.

For example, the image processing apparatus 1 receives image data in advance, detects image portions such as store signboards and store entrances from the image data, and performs machine learning to output information representing the range. A certain neural network or the like may be used. When using such a neural network, an image to be processed is input, and the range of the output image portion is used as a candidate area. In this example, a store signboard, an entrance, or the like is specified as the attention area.

[Modified Example of Attention Area Determination Processing]
Furthermore, in at least one of machine learning processing and estimation processing, the control unit 11 of the image processing apparatus 1 performs the following when processing an equirectangular image obtained by converting image data projected onto the celestial sphere. You may process as follows. For example, when performing processing as the region determination unit 26 during estimation processing, the control unit 11 selects a relatively high latitude range (from the upper side or the lower A candidate area whose overlapping range with the predetermined range) occupies a predetermined ratio or more of its area may be excluded. For example, the region determination unit 26 defines a high latitude range that is 5% of the height of the equirectangular image from each of the upper and lower sides of the equirectangular image, and identifies the portion of each candidate region that overlaps with this range. do. Then, if the area of the identified portion is equal to or greater than a predetermined percentage (for example, 70%) of the area of the candidate area, the area determination unit 26 removes the candidate area from the set of candidate areas.

The region determining unit 26 uses the set of candidate regions obtained in this way as an initial set S, the degree of overlap between the candidate regions included in this set S, and each candidate included in this set S obtained from the saliency map. While evaluating a set of candidate regions based on saliency information in the region, the set is corrected by, for example, reducing candidate regions, and a region of interest is determined.

According to this example, the information of the high latitude portion where the degree of deformation is relatively large in the conversion to the equirectangular image is excluded. This prevents erroneous detection as candidate regions due to relatively large deformations.

[Frame associated with attention area]
Furthermore, in the description so far, when reproducing moving image data (regardless of whether it is a wide-angle image), for information on a certain attention area, the frame associated with the information on the attention area is reproduced. An example of displaying a graphic image representing the range of the attention area was shown at the previous point. However, the processing for obtaining this result may be performed at the time of generating the information of the attention area, not at the time of reproduction.

When the attention area is determined, the image processing apparatus 1 of this example selects a frame number of a predetermined reproduction time earlier than the reproduction time (frame number) of the frame to be processed when the attention area is determined. , information on the determined attention area may be associated and recorded.

For example, when the image processing apparatus 1 determines a region of interest for a frame of frame number F as a processing target, it records information of the region of interest in association with the frame number of frame number min(F−ΔF, 1). Here, min(a, b) means taking the smaller value of a and b. Both F and ΔF are positive integers, and ΔF is predetermined.

1 image processing device, 11 control unit, 12 storage unit, 13 operation unit, 14 display unit, 15 interface unit, 21 reception unit, 22 acquisition unit, 23 candidate region extraction unit, 24 saliency map learning processing unit, 25 saliency map estimation unit, 26 area determination unit, 27 output unit.

Claims (7)

acquisition means for acquiring image data to be processed;
Candidate area extracting means for extracting a plurality of candidate areas serving as attention area candidates that satisfy a predetermined condition from the acquired image data;
saliency map estimation means for estimating saliency map information related to the acquired image data based on the acquired image data;
determining means for determining a region of interest in the acquired image data based on the saliency map and the information of the candidate region;
An image processing device including The image processing device according to claim 1,
The determining means is
An image processing device that determines a region of interest based on the saliency map and information on overlap of the candidate regions. The image processing device according to claim 1 or 2,
The saliency map estimation means performs the estimation using a neural network that has undergone machine learning so as to receive input of image data and estimate and output saliency map information related to the image data,
The acquisition means receives an input of wide-angle image data projected to a range exceeding at least a hemisphere of the celestial sphere, and projects the wide-angle image data onto the celestial sphere at angles about at least one axis that are different from each other, or obtaining a plurality of pieces of image data to be processed by converting the wide-angle image data projected onto the celestial sphere after being rotated by a set of angles into rectangular image data;
An image processing apparatus for subjecting the image data to be subjected to the plurality of processes to machine learning processing of the saliency map estimation means. The image processing device according to any one of claims 1 to 3,
The acquisition means receives input of moving image data, extracts still images at a plurality of playback times from the moving image data, and acquires image data of the extracted still images. The image processing device according to claim 4,
The determining means, when determining the attention area, records information of the determined attention area in association with image data at a predetermined reproduction time before the reproduction time of the image data related to the attention area. Moving image data representing still image data reproduced in chronological order, and including information specifying a predetermined attention area in the still image data reproduced at any point in time t (where t>0) means for accepting input of
When the moving image data is reproduced, the information specifying the attention area is referred to, and a time point t- determined by a predetermined method before the time point t at which the still image data related to the attention area specified by the information is to be reproduced. An image reproducing apparatus for displaying information about an attention area specified by the information when reproducing still image data reproduced at Δt (where Δt>0). the computer,
acquisition means for acquiring image data to be processed;
Candidate area extracting means for extracting a plurality of candidate areas serving as attention area candidates that satisfy a predetermined condition from the acquired image data;
saliency map estimation means for estimating saliency map information related to the acquired image data based on the acquired image data;
determining means for determining a region of interest in the acquired image data based on the saliency map and the information of the candidate region;
A program that acts as

Images