JP2019012516A - Image processor and image processing method - Google Patents

Abstract

To appropriately convert an entire celestial sphere image into a plane image.SOLUTION: Provided are: acquisition means for acquiring at least one or more pieces of input image data for expressing a spherical image; determination means for determining a focus region in the input image data; and conversion means for converting, based on the focus region, the input image data into output image data expressing at least part of the spherical image by a positive pyramid projection technique.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing technique for holding plane image data representing a spherical image.

  A technique for displaying an image in accordance with the movement of the viewer's line of sight for a viewer wearing a head-mounted display or the like is known. By displaying a part of the omnidirectional image representing all 360-degree images at a certain position in accordance with the direction of the viewer's line of sight, the viewer can feel as if they are present. You can experience it. Since the omnidirectional image represents an image in all directions, the amount of data is enormous. Therefore, in recent years, methods for appropriately storing omnidirectional images have been studied. In the method disclosed in Patent Document 1, the omnidirectional image is divided into a plurality of fragment images, and the upper and lower pole portions such as the sky and the ground with little change are projected on a plane circle and expressed in polar coordinates, and other than the pole portions. This part is interpolated into a rectangle and a compression process based on an orthogonal coordinate system is applied. Thereby, the omnidirectional image is efficiently compressed.

Japanese Patent Laid-Open No. 2001-298652

  However, in the method disclosed in Patent Document 1, since the omnidirectional image is expressed in different coordinate systems depending on the region, different processing is required for each region in order to generate a display image from the omnidirectional image. turn into. Therefore, an object of the present invention is to divide the omnidirectional image and appropriately convert it into a planar image without requiring different processing for each region.

  In order to solve the above problems, the present invention provides an acquisition unit that acquires at least one input image data for representing a spherical image, a determination unit that determines a region of interest in the input image data, And converting means for converting the input image data into output image data representing at least a part of a spherical image by equirectangular projection.

  According to the present invention, an omnidirectional image can be appropriately converted into a planar image.

1 is a block diagram illustrating a configuration of an image processing apparatus. FIG. 2 is a block diagram showing a detailed logical configuration of the image processing apparatus. 3 is a flowchart showing a flow of processing in the image processing apparatus. The flowchart which shows the flow of rotation amount calculation processing. The figure which shows the example of the omnidirectional image of equirectangular projection. The graph which shows the change of the resolution with respect to the elevation angle of equirectangular projection. The figure which shows the example of a display image. The figure which shows input image data. The figure which shows the image after a rotation process in case there is one attention point. The figure which shows the image after a rotation process in case the angle difference of two attention points is less than 90 degree | times. The figure which shows the image after a rotation process in case the angle difference of two attention points is 90 degree | times or more. 1 is a block diagram showing a logical configuration of an image processing apparatus. 3 is a flowchart showing a flow of processing in the image processing apparatus. FIG. 2 is a block diagram showing a detailed logical configuration of the image processing apparatus. 3 is a flowchart showing a flow of processing in the image processing apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not necessarily limit the present invention, and not all combinations of features described in the present embodiment are essential for the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

<First Embodiment>
In the first embodiment, an attention area to be stored at a high resolution is determined in the omnidirectional image, the axis of the omnidirectional image is changed so that the attention area corresponds to a position near the pole, and then the equirectangular cylinder method is used. A method for converting to a flat image will be described. The omnidirectional image is a spherical image and is also called an omnidirectional image, an omnidirectional image, or the like, and means an image representing a 360-degree omnidirectional scene from a certain position. In addition, the image processing apparatus according to the present embodiment stores an omnidirectional image used in a head mounted display (hereinafter, HMD) that can be worn on the head and viewed. In the HMD, a partial image in the line-of-sight direction in the omnidirectional image appropriately stored according to the present embodiment is displayed on the display according to the line of sight of the viewer who wears it. The image displayed on the HMD is assumed to be performed by the image processing apparatus that executes image output to the HMD or HMD. The image displayed on the HMD is an omnidirectional image stored by image processing in the present embodiment, or an image obtained after various image processing is performed on a part thereof.

  A hardware configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. The image processing apparatus according to the present embodiment will be described using a personal computer that reads image data and executes various image processes as an example. The CPU 101 executes programs stored in the ROM 103 and the hard disk drive (HDD) 105 using the RAM 102 as a work memory, and controls each component to be described later via the system bus 110. Thereby, various processes described later are executed. The HDD interface (I / F) 104 connects a secondary storage device such as the HDD 105 or an optical disk drive. For example, an interface such as serial ATA (SATA). The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD I / F 104. Further, the CPU 101 can expand the data stored in the HDD 105 in the RAM 102 and similarly store the data expanded in the RAM 102 in the HDD 105. The CPU 101 can execute the data expanded in the RAM 102 as a program. An input interface (I / F) 106 is a serial bus interface such as USB or IEEE 1394 for connecting an input device 107 such as a keyboard, a mouse, a digital camera, and a scanner. The CPU 101 can read data from the input device 107 via the input I / F 106. An output interface (I / F) 108 connects an output device 109 of an image display device such as a display. The output interface 108 is a video output interface such as DVI or HDMI (registered trademark). The CPU 101 can send data to the output device 109 via the output I / F 108 to execute display. As described above, in the present embodiment, an HMD is assumed as the output device 109.

  FIG. 2 is a block diagram illustrating a logical configuration of the image processing apparatus according to the present embodiment. The CPU 101 plays a role as each functional block shown in FIG. 2 by reading a program stored in the ROM 103 or the HDD 105 and executing the RAM 102 as a work area. Note that the CPU 101 does not have to play the role of all the functional blocks, and a dedicated processing circuit corresponding to each functional block may be provided.

  The image acquisition unit 201 acquires input image data representing the omnidirectional sphere. Input image data representing the omnidirectional sphere is image data in which the colors of incident light rays in all directions at a certain point are stored. The input image data acquired here is image data developed on a plane by equirectangular projection. FIG. 5 shows an omnidirectional image represented by equirectangular projection. The equirectangular projection is known as one of map projection methods, and is a method of developing on a plane by assigning the latitude and longitude of the earth to the vertical and horizontal planes. It is known that equirectangular projection is used as a method for storing image data of omnidirectional images. In equirectangular projection, the horizontal axis of the image data is the azimuth angle θ. The range of the azimuth angle θ is [−ππ]. The vertical axis is the elevation angle φ, and the range of the elevation angle φ is [−π / 2 π / 2]. The image data consists of pixels that store color information as pixel values at positions where the azimuth angle θ and elevation angle φ are discretized. In the image data, the width is w pixels and the height is h pixels. FIG. 6 is a graph showing a change in resolution with respect to a change in elevation angle φ in equirectangular projection. The resolution represents the angle of light incident on one pixel of the omnidirectional image. The graph shown in FIG. 6 represents the ratio of the resolution at each elevation angle to the resolution of the image at an elevation angle of 0 degrees. According to the graph shown in FIG. 6, in the region where the absolute value of the elevation angle exceeds 70 degrees, the resolution is three times or more compared to the region where the elevation angle is 0 degree. It can be seen that the resolution increases as the absolute value of the elevation angle increases.

  The positions corresponding to the north pole (zenith) and the south pole (nadir) when the omnidirectional image developed on a plane by equirectangular projection is mapped to a spherical surface are called poles. In an image represented by equirectangular projection, the poles are not single points but have a spread, and the uppermost row and the lowermost row of the image correspond to the positions of the poles. On the other hand, the position of the equator on the spherical surface corresponds to the central horizontal line in the omnidirectional image. As shown in FIG. 6, in the omnidirectional image developed on a plane by equirectangular projection, the position of the pole has the highest resolution, and the resolution is higher in the region closer to the pole than in the region corresponding to the equator. . Therefore, a subject whose color information is stored in the polar region can be stored with a higher resolution using more pixels than a subject whose color information is stored in other regions.

  When displaying a partial area of the omnidirectional image (hereinafter referred to as a partial image) on the display of the HMD in accordance with the viewing direction of the viewer wearing the HMD, various image processing is performed on the partial image. It is assumed that it will be applied. As a result of performing image processing such as reduction on the omnidirectional image or the partial image, the image quality of the partial image may deteriorate. Therefore, in the present embodiment, it is used that the resolution closer to the pole is higher in the planar image developed by the equirectangular projection. Identify the region of the omnidirectional image that should be stored at a higher resolution as the region of interest, convert the omnidirectional image so that the position of the region of interest corresponds to the position of the polar region, and save it. Keep it.

  The attention area determination unit 202 specifies an area to be stored at high resolution in the input image data. In the present embodiment, the omnidirectional image to be processed is displayed on the display that is the output device 109 via the output I / F 108, and the user is allowed to specify the subject to be stored at high resolution. The designation is performed via the input device 107 such as a mouse or a touch. In the present embodiment, it is assumed that the user clicks on a subject to be set as a region of interest using a mouse.

  The conversion unit 206 converts the input image data according to the position of the attention area. The conversion unit 206 includes a rotation amount calculation unit 203 and an image rotation unit 204. The rotation amount calculation unit 203 calculates a rotation amount for associating the position of the designated attention point with the polar region of the omnidirectional image. Note that the omnidirectional image is developed in a plan view by equirectangular projection, but corresponds to a spherical image. Here, the amount of rotation means the angle of rotation necessary for rotating the coordinate axis and changing the position of the pole when the omnidirectional image is mapped onto the spherical surface. The image rotation unit 204 substantially rotates the omnidirectional image acquired by the image acquisition unit 206 according to the rotation amount calculated by the rotation amount calculation unit 203.

  The output unit 205 outputs output image data obtained by converting the input image data so that the position of the target point designated by the image rotation unit 204 is associated with the polar region of the omnidirectional image.

  FIG. 7 shows an example of a display image created from the omnidirectional image. As described above, the omnidirectional image held by the image processing apparatus according to the present embodiment is read by another image processing apparatus or HMD, and a display image as shown in FIG. 7 is generated. A display image can be created by shooting a sphere with a celestial sphere image on the inside with a virtual camera. As can be seen from FIG. 7, the display image is created with reference to a partial region of the omnidirectional image. Therefore, in this embodiment, a spherical omnidirectional image is input, but only a part of the omnisphere such as a hemispherical image necessary for the display image has pixel data, and pixel data of a part of the region has pixel data. It does n’t matter.

  FIG. 3 is a flowchart showing the flow of image processing in the first embodiment. The CPU 101 is realized by reading and executing a program capable of executing the flowchart shown in FIG. In the following description of the flowchart, the symbol for each step is denoted as S. First, in S301, the image acquisition unit 201 acquires input image data in which the omnidirectional sphere is developed into a planar image.

In step S <b> 302, the attention area determination unit 202 determines an attention point capable of defining an area of interest according to an input by the user. A point of interest is a point that represents a region of interest. As described above, the user clicks the position of the subject of interest in the omnidirectional image displayed on the display. The attention area determination unit 202 sets the clicked position as the attention point (x _i , y _i ). Here, x and y indicate the coordinates in the image, and i indicates the index of the determined point of interest. In addition, when the area including the subject is designated as the attention area instead of the click, the center of gravity of the area is set as the attention point (x _i , y _i ).

  In S303, the rotation amount calculation unit 203 calculates the rotation amount of the input image data based on the position of the target point. In the present embodiment, image rotation is represented using a rotation matrix R. For the rotation, another expression format such as Euler angle or quaternion may be used. Details of this rotation amount calculation processing will be described later.

  In S304, the image rotation unit 204 performs a rotation process on the input image data I that is an omnidirectional image, and output image data I that is an omnidirectional image converted so that the subject of interest corresponds to a polar region in the image. 'Is output. Here, the omnidirectional image is rotated not in the image coordinate system but in the spherical coordinate system. In other words, instead of rotating on a two-dimensional plane, the omnidirectional image is mapped to a spherical surface, the mapped sphere is rotated, rotated, and then converted again to an image on a plane developed by equirectangular projection. . Each pixel in the output image data I ′ after rotation is calculated by sampling pixel values from the input image data I in consideration of rotation. First, for each pixel (x ′, y ′) in the output image data I ′ after rotation, the coordinates (x, y) of the acquired input image data I corresponding to the pixel are calculated. Next, the pixel value of the coordinates (x, y) is sampled to obtain the pixel value of the pixel (x ′, y ′) in the output image data I ′. Hereinafter, a method for obtaining the coordinates (x, y) of the input image data I corresponding to the coordinates (x ′, y ′) of the output image data I ′ after the rotation processing will be described. First, (x ′, y ′) in the image coordinate system is converted into an azimuth angle and an elevation angle. When the width of the input image data I is w and the height is h, the azimuth angle θ ′ and the elevation angle φ ′ corresponding to (x ′, y ′) are calculated by the equation (1).

Next, the azimuth angle θ ′ and the elevation angle φ ′ are represented as rotation matrices, and the matrix M integrated with the inverse matrix R ⁻¹ of the rotation matrix R of the omnidirectional image calculated in S303 is converted into representations of azimuth angles and elevation angles. Thus, the azimuth angle θ and the elevation angle φ in the acquired omnidirectional image are calculated. The azimuth angle θ and the elevation angle φ are calculated by equation (2).

  Here, atan represents an arc tangent and asin represents an arc sine function.

  Then, the azimuth angle θ and the elevation angle φ are converted into coordinates (x, y) in the acquired omnidirectional image. This conversion can be calculated by equation (3).

  Finally, the pixel value at the coordinates (x, y) of the acquired input image data I is calculated and set as the pixel value at the coordinates (x ′, y ′) of the output image data I ′ after rotation. Here, a pixel value is calculated by look-ahead interpolation from four pixels in the vicinity of the coordinates (x, y), and set as the pixel value of the coordinates (x ′, y ′) of the output image data I ′ after rotation. However, not only linear interpolation but also an interpolation method such as bicubic may be used.

  In step S <b> 305, the output unit 205 outputs the output image data after the rotation process and the rotation matrix R used for the rotation process to the HDD 105 or the RAM 102. The above is the flow of processing performed by the image processing apparatus of this embodiment.

  Here, details of the rotation amount calculation processing executed by the rotation amount calculation unit 203 in S303 will be described. In the rotation amount calculation process, the rotation amount of the input image data is calculated so that the subject of interest corresponds to an area with high resolution. When the omnidirectional image is an equirectangular projection, the resolution increases as it approaches the top and bottom of the sphere pole, and the resolution is low at the center height (near the equator) of the omnidirectional image. Therefore, a rotation amount is calculated so that the subject of interest moves to a region above or below the image, that is, a position where the absolute value of the elevation angle is large. When there is only one subject of interest, the amount of rotation is calculated so that the point moves to one of the poles. When there are a plurality of subjects of interest, the amount of rotation is calculated so as to be distributed to one or two poles. FIG. 4 is a flowchart of the rotation amount calculation process.

  In step S401, a process to be performed next is selected according to the number of attention points. When the number of attention points output by the attention area determination unit 202 is one, S402 is executed, when it is two, S403 is executed, and when it is three or more, S409 is executed.

In S402, a rotation matrix for moving the attention point to the pole position is calculated. There are two poles at the top and bottom in the image represented by equirectangular projection, either of which may be selected. FIG. 8 shows an example of the acquired omnidirectional image. Assume that the point 801 is a given point of interest. FIG. 9 shows an example of a case where the point 801 is rotated so as to move to a point 901 representing one point of the zenith among the poles in the omnidirectional image developed by the equirectangular projection. In S402, the amount of rotation by which the point of interest 801 moves to the position of the point 901 is calculated. When the omnidirectional image is considered as a sphere, the pole corresponds to the top row of the omnidirectional image. Therefore, rotation that moves a point at an arbitrary position to the pole can be expressed by rotation only in the pitch direction. . Rotation in this case, since the elevation angle phi _a of the point of interest 801 to elevation [pi / 2 of the point 901 is rotated to move, the rotation amount is π / 2-φ _a. As described above, the rotation amount as to move the poles of one target point is calculated by the pitch direction π / 2-φ _a.

  In S403, the process is further branched according to the angle difference between the two specified points of interest. If the angle difference between the two points is less than 90 degrees, the process proceeds to S404, and if the angle difference is 90 degrees or more, the process proceeds to S409. In S404, a rotation amount is calculated such that the two attention points approach the same pole. On the other hand, in the processing from S405 to S407, the rotation amount is calculated such that the two attention points approach different poles.

In S404, the amount of rotation is calculated so that the middle point of the two points of interest (θ _a , φ _a ) (θ _b , φ _b ) moves to one pole. In this case, either the upper or lower pole may be selected. In order to consider the rotation of the sphere, the midpoint of the two points of interest must be calculated on the spherical coordinates, not on the image coordinates. For example, spherical linear interpolation can be used to calculate the midpoint. The amount of rotation to move one of the midpoints (θ _c , φ _c ) to the pole is calculated as π / 2−φ _a only in the pitch direction so that the elevation angle is π / 2, as in S402. To do. FIG. 10 is a diagram illustrating a rotation result when two attention points 802 and 803 are designated. The amount of rotation by which the midpoint of the two points of interest moves to the zenith pole, that is, the top row, as indicated by point 1001, is calculated. On the other hand, in S405, the amount of rotation for moving one of the two points of interest (θ _a , φ _a ) (θ _b , φ _b ) to the pole is calculated. The attention point to be selected may be either, but here, the attention point (θ _a , φ _a ) designated first is selected. When the azimuth angle and elevation angle of the point of interest are θ _a and φ _a respectively, the rotation is such that the azimuth angle is θ _a and the elevation angle is π / 2 and the pitch direction is rotated by π / 2−φ _a. Matrix R _A is calculated as in equation (4).

In S406, after applying the rotation matrix _RA calculated in S405 to the input image data, the azimuth of the attention point (θ _b , φ _b ) not selected in S405 in the omnidirectional image becomes 0 degree. The amount of rotation in the yaw direction is calculated. This is to calculate a rotation matrix R _B that rotates by −θ _b when the azimuth angle of the target point is θ _b as shown in Equation (5).

In step S407, a rotation matrix R _c that performs rotation in the pitch direction so that the midpoints (θ _c , φ _c ) of the two attention points after applying the rotation matrices R _{A and} R _B to the input image data is the front surface. calculate. By applying the rotation matrices R _{A and} R _B , one point of interest has moved to the pole, and the other point of interest has moved to the position (0, φ _B2 ) at the azimuth angle 0. Therefore, the midpoint between the two attention points is expressed by Equation (6).
(Θ, φ) = (0, (π / 2−φ _b2 ) × 1/2) (6)

In order to obtain rotation for moving the midpoint to the front (θ, φ) = (0, 0), a rotation matrix R _C that rotates by − (π / 2−φ _b2 ) × 1/2 in the pitch direction. Is calculated as in equation (7).

In S408, the rotation is calculated such that the two points of interest (θ _a , φ _a ) (θ _b , φ _b ) approach different poles. This can be calculated by accumulating the rotation matrices R _A, R _B , and R _C calculated in S405 to S407.

  In S409, the specified three or more attention points are classified into two groups. In this embodiment, classification is performed using the k-means method. In this case, the distance between the points of interest uses the distance on the spherical coordinates. In the processing after S409, the centroids of the two classified groups are treated as two attention points, respectively. As a result, the same processing as in the case where there are two attention points can be performed. For example, when the points 802, 803, and 804 in FIG. 8 are input as points of interest, the first group is the point 804 and the second group is the point 802 and the point 803. In the first group, the point 804 is an attention point, and in the second group, the center of gravity positions of the points 802 and 803 are the attention points. The above completes the rotation amount calculation process performed by the rotation amount calculation unit 203 of the present embodiment.

  As described above, in the first embodiment, a subject (region) that is desired to be stored in a high-resolution region in a spherical image such as a spherical image is specified, and the specified subject is a polar region in the spherical image. Is converted into a omnidirectional image rotated so as to correspond to and saved. Thereby, when performing image processing such as reduction on the omnidirectional image or a partial image of the omnidirectional image, the subject of interest can be maintained at a high resolution.

Second Embodiment
In the first embodiment, the method of storing a omnidirectional image after a desired rotation process using one omnidirectional image as input image data has been described. In the second embodiment, a method for storing a subject of interest with high resolution when a plurality of captured images that are not omnidirectional images are combined and the omnidirectional image is stored will be described. In this example, not the synthesized image but the omnidirectional image is generated by rotating the coordinate system before the synthesis. In addition, about the structure and process similar to 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  FIG. 12 is a block diagram illustrating a logical configuration of the image processing apparatus according to the second embodiment. FIG. 13 is a flowchart of processing executed by each configuration shown in FIG. The CPU 101 plays a role as each component shown in FIG. 12 by reading a program that can realize the flowchart of FIG. 13 stored in the ROM 103 or the HDD 105 and executing the RAM 102 as a work area.

  In step S <b> 1301, the image acquisition unit 1201 outputs a plurality of pieces of image data acquired from the HDD 105 or the RAM 102 to the posture acquisition unit 1202. Note that a plurality of pieces of image data are taken from the same viewpoint. In this embodiment as well, as in the first embodiment, the omnidirectional image is treated as a flat image developed by equirectangular projection. In addition, for the sake of simplicity, the image data is taken from the same viewpoint with the intersection of the optical axis of the lens and the image plane being the center of the image without distortion, but these are taken into consideration in the following steps. You may make it perform the process which carried out.

  In step S <b> 1302, the posture acquisition unit 1202 acquires posture information indicating the posture of the camera when the image input from the image acquisition unit 1201 is captured from the HDD 105 or the RAM 102. The posture of the camera is information indicating which direction the camera is facing, and is represented by a rotation matrix. Here, the posture of the camera is calculated in advance, but may be calculated from an image.

In step S1303, the attention area determination unit 1203 determines one or more attention areas as attention points. The point of interest (x _i , y _i ) is converted into a polar coordinate system, and the azimuth angle θ _i and elevation angle φ _{i are} calculated. This is based on the rotation matrix R _j representing the posture of the camera that captured the image including the attention point, and the angles θ _i and φ _i of the light rays passing through the attention point (x _i , y _i ) in the image from the camera viewpoint. Obtain by calculating. Here, j represents an image index. Assuming that the focal length of the camera is f, the coordinate X of the intersection of the light beam and the image plane can be calculated by equation (8).

The azimuth and elevation angles of the three-dimensional point X when viewed from the origin are θ _i and φ _i , and the attention area determination unit 1203 outputs the attention point (θi, φi) to the rotation amount calculation unit 1204.

  In step S <b> 1304, the rotation amount calculation unit 1204 calculates the rotation amount of the image based on the position of the target point (θi, φi), and outputs it to the posture update unit 1205. In the present embodiment, the rotation amount of the omnidirectional image is calculated in the same manner as in the first embodiment.

Attitude updating unit 1205 in S1305, on the basis of the position R _j of the cameras taking the rotation matrix R and the image representing the amount of rotation of the celestial sphere image, and updates the posture information of the camera. Utilized as the posture of the camera rotation matrix R the integrated posture R _{j 'next} processing indicating the amount of rotation of the posture R _j to the image of the camera. By this update, since the coordinate system of the input image can be synthesized with the rotated coordinate system, the omnidirectional image obtained by the synthesis can be rotated.

In step S _< b> 1306, the image composition unit 1206 composes an image by using the plurality of image data I _j and the updated posture R _j as inputs. The image composition unit 1206 projects an image captured by each camera based on the input posture of each camera onto an omnidirectional image in a spherical coordinate system. Then, the pixel values of the overlapping images are blended in the overlapping region when each image is projected. The image composition unit 1206 outputs the synthesized omnidirectional image data to the output unit 1207.

  In step S <b> 1307, the output unit 1207 outputs the rotated spherical image and its rotation matrix R to the HDD 105 or RAM 102. Thus, the image processing in the second embodiment is completed. According to the present embodiment, even when synthesizing an omnidirectional image from a plurality of input images, by synthesizing in a coordinate system such that the region of interest is stored at a position where it can be stored at high resolution, It can be saved in high resolution.

<Third Embodiment>
In the above-described embodiment, a method has been described in which a single omnidirectional image is stored after being rotated based on a manually set attention area. In the third embodiment, a method for automatically detecting and storing a subject of interest will be described. In this example, the subject of interest is detected based on the set detection mode. In addition, about the structure and process similar to 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  FIG. 14 is a block diagram illustrating a logical configuration of the image processing apparatus according to the third embodiment. FIG. 15 is a flowchart of processing executed by each configuration shown in FIG. The CPU 101 plays a role as each component shown in FIG. 14 by reading a program that can realize the flowchart of FIG. 14 stored in the ROM 103 or the HDD 105 and executing the RAM 102 as a work area.

  In S1401, the detection mode selection unit 1401 selects a mode for determining a method for detecting a region of interest from an image. The detection mode is a mode for specifying a feature to be set as a region of interest in the input omnidirectional image. Here, as examples of the detection mode, there are a person mode for detecting a human face and a pet mode for detecting an animal. The user selects either the portrait mode or the pet mode. The detection mode selection unit 1401 outputs the designated mode to the attention area determination unit 1402. As the detection mode, another mode such as a statistical mode that uses statistical information for detection, a focus mode at the time of shooting that uses information at the time of shooting, or a landscape mode that detects a high-frequency region may be selected. In the embodiment, a default detection mode used when the attention area is not detected in the designated detection mode is also set.

  In step S <b> 1402, the attention area determination unit 1402 automatically determines a point of interest from the input image data based on the detection mode output from the detection mode selection unit 1401. When the detection mode is the person mode, face detection processing is performed on the input image data, and the detected face area is set as the attention area. In the case of the pet mode, an animal detection process is performed from the input image data, and the detected animal region is set as a region of interest. If no region of interest is detected, detection is performed using a predetermined detection mode. Here, the landscape mode is set as the default detection mode. In the landscape mode, the input omnidirectional image is divided into a plurality of regions, the frequency is calculated in each region, and a region having a high frequency component is set as a region of interest. The attention area determination unit 1402 sets the position of the center of gravity of the area detected in the omnidirectional image as the attention point (xi, yi).

  Thus, the image processing in the third embodiment is completed. According to the present embodiment, the attention area can be automatically specified based on the detection mode, and the subject of interest can be stored at a high resolution according to the scene.

<Other embodiments>
In the above-described embodiment, an attention area that is desired to be stored at a high resolution is specified, and an omnidirectional image is generated so that the attention area corresponds to a position near the pole. As described above, in the plan view developed by the equirectangular projection, the resolution near the equator is the lowest. Therefore, among the subjects in the omnidirectional image, a non-focused area such as the ground, floor, or sky may be specified, and the omnidirectional image may be generated so that the non-focused area preferentially corresponds to a position near the equator. . This is because an omnidirectional image is generated so that a non-attention area does not correspond to a position with a high resolution, thereby indirectly identifying an attention area that is desired to be stored at a position with a high resolution. It is none other than that.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 201 Image acquisition part 202 Attention area determination part 203 Rotation amount calculation part 204 Image rotation part 205 Output part

Claims (20)

Obtaining means for obtaining at least one or more input image data for representing an image;
Determining means for determining a region of interest in the input image data;
An image processing apparatus comprising: conversion means for converting the input image data into output image data representing at least a part of the image by equirectangular projection based on the attention area.   The converting means converts the input image data into the output image data so that the region of interest corresponds to a position near the pole of the sphere when the output image data is mapped to a sphere. The image processing apparatus according to claim 1. The input image data and the output image data represent a spherical image,
The image processing apparatus according to claim 1, wherein the omnidirectional image represented by the output image data is an image obtained by changing a position of a subject in the omnidirectional image represented by the input image data.   The image processing apparatus according to claim 1, wherein the conversion unit changes a position of the region of interest in an image represented by the input image data.   The said determining means converts the said input image data into the said output image data according to the angle difference of the said 2 attention area, when the said determination means designates the said 2 attention area. The image processing apparatus according to any one of 1 to 4.   The conversion means converts the input image data so that the angle difference between the two regions of interest is 90 degrees or more, and the position of the different poles approaches the position of the same pole when the angle difference is less than 90 degrees. The image processing apparatus according to claim 5, wherein the image processing apparatus converts the output image data.   7. The image processing according to claim 5, wherein when the determining unit specifies three or more regions of interest, the converting unit classifies the regions into two groups according to the positions of the regions of interest. apparatus. The acquisition means acquires one input image data representing a spherical image,
The image processing according to claim 1, wherein the conversion unit obtains the output image data by rotating the input image data acquired by the acquisition unit in a spherical coordinate system. apparatus. The conversion means calculates a rotation amount of the attention point in a spherical coordinate system so that the coordinates of the attention point representing the attention area approach the upper part or the lower part of the output image data;
The image processing apparatus according to claim 8, further comprising a rotation processing unit that rotates the input image data according to the rotation amount. Furthermore, it has a selection means for selecting a mode for determining the region of interest,
The image processing apparatus according to claim 1, wherein the determination unit automatically detects a plurality of regions of interest based on the detection mode.   The said determination means detects an attention area | region using the preset detection mode, when the said attention area cannot be detected in the detection mode selected by the said selection means. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the determination unit determines the region of interest in accordance with a designation from a user.   In the planar image represented by the output image data, the resolution of the pixels located near the top and bottom is higher than the resolution of the pixels located near the central horizontal line. The image processing apparatus according to claim 1, wherein the image processing apparatus performs conversion so as to be positioned in a vicinity of an uppermost part and a lowermost part of the planar image.   The image processing apparatus according to claim 1, wherein the input image data is a plurality of captured images having different viewpoints, and the conversion unit generates the output image data from the plurality of captured images. . Furthermore, posture acquisition means for acquiring posture information indicating the posture of the camera when each of the plurality of imaging devices is photographed,
Based on the posture information, comprising a synthesizing unit for synthesizing an omnidirectional image from the plurality of imaging devices;
The image processing apparatus according to claim 14, wherein the conversion unit converts the omnidirectional image synthesized by the synthesis unit into the output image data.   The image processing apparatus according to claim 10, wherein the selection unit includes a person mode that detects a person and uses at least one of the detected persons as the attention area.   12. The image according to claim 10, wherein the selection unit divides the input image data into a plurality of regions, calculates a frequency in each region, and sets a region having a high frequency component as the region of interest. Processing equipment.   The image processing apparatus according to claim 1, wherein the input image data and the output image data are omnidirectional images that hold light information in all directions from a certain viewpoint.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 9. An acquisition step of acquiring at least one or more input image data for representing a spherical image; a determination step of determining a region of interest in the input image data;
Converting the input image data into output image data representing a spherical image by equirectangular projection based on the region of interest;
The converting step converts the input image data into the output image data so that the region of interest corresponds to a position near a pole of a spherical surface when the output image data is mapped onto a spherical surface. Image processing method.

Images