JP6705477B2 - Image processing system, image processing method and program - Google Patents

Description

The present invention relates to an image processing technique, and more particularly, to an image processing system, an image processing method, and a program for projecting an image configured with a wide angle of view based on a three-dimensional model.

2. Description of the Related Art Conventionally, a panoramic image viewer is known as a system for displaying an image captured by a camera or the like on a flat display. The panoramic image viewer is a system that joins a plurality of images captured in different shooting directions and overlapping some subjects, and displays a stitched composite image on a display.

Conventional panoramic image viewers have functions that enable various display range changing operations such as pan (moving the field of view left and right), tilt (moving the field of view up and down), and zoom (enlargement/reduction) for the combined panoramic images. There is. The panoramic image viewer often uses a method of projecting and displaying an image pasted on the side surface or the spherical surface of the cylinder on the plane when the image is viewed from the center of gravity of the cylinder or sphere. In this case, on the flat display, an image on a three-dimensional surface in which a focus is formed along the side surface of the cylinder or the spherical surface corresponding to the pan, tilt, and zoom set values set by the user is displayed. It is displayed by projecting it onto a plane image.

However, in the conventional panoramic image viewer, when the view is widened to a certain extent or more due to the zoom function of the display range changing operation, the image may be distorted or distorted at the edge of the view. There was a problem that it might end up.

Non-Patent Document 1 is known as a technique for displaying an image with a wide angle of view such as a panoramic image without a sense of discomfort. Non-Patent Document 1 discloses a viewer that displays a panoramic image in a required field of view. The viewer of Non-Patent Document 1 continuously adjusts the projection method to perform perspective projection in a narrow field of view and cylindrical projection or spherical projection in a wide field of view.

However, in the conventional technique of Non-Patent Document 1, since the projection method is switched according to the zoom, the processing becomes complicated, and high computing performance is required to realize the real-time processing. On the other hand, in recent years, the panoramic image viewer is often executed not only on a personal computer but also on an information terminal such as a smartphone terminal or a tablet terminal having a relatively low CPU (Central Processing Unit) computing capability. .. With such an information terminal having a low computing capability, it becomes difficult to perform real-time display at, for example, about 30 fps (Frame per Second) with the complicated processing as in Non-Patent Document 1.

In recent years, many information terminals are equipped with a GPU (Graphics Processing Unit) that is in charge of graphics calculation in addition to the CPU. The GPU typically has a calculation function corresponding to an API (Application Programming Interface) such as OpenGL and can realize high-speed image processing calculation. However, a GPU included in a smartphone or the like corresponds to a subset version of OpenGL, and is forced to perform a calculation with a relatively simple model.

From the background described above, even in the case of an information terminal having a limited computing capacity, when displaying a panoramic image, in a display in a wide angle of view area, it is possible to reduce the discomfort caused by stretching the subjects at the top, bottom, left, and right edges. The development of technology capable of high-speed display has been desired.

The present invention has been made in view of the inadequacies of the above-described prior art, and the present invention is such that, when displaying an image, the subject at the upper, lower, left, and right ends is stretched in the display in a wide visual field region. It is an object of the present invention to provide an image processing system, an image processing method, and a program that enable high-speed display while reducing discomfort due to the above, and have relaxed requirements for computing power.

In order to solve the above problems, the present invention provides an image processing system having the following features. The image processing system includes an acceptance unit that accepts an input value that defines an output range, a generation unit that generates a three-dimensional model in which a target image is attached to a three-dimensional shape, and a position and a visual field of a viewpoint based on the input value. It includes a determining means for determining an angle and a projecting means for projecting the three-dimensional model from a viewpoint. When the input value described above is in the first range, the determining unit changes the range included in the field of view of the target image by preferentially changing the viewing angle. On the other hand, when the input value is in the second range, which is on the wider visual field side than the first range, the range of the visual field is changed by preferentially changing the position of the visual point.

According to the above configuration, when displaying an image, in the display in the area on the wide field of view side, the requirement for the calculation capability when performing high-speed display while reducing the discomfort caused by the subject at the upper, lower, left, and right ends being stretched Can be relaxed.

The schematic diagram showing the omnidirectional image display system by this embodiment. FIG. 3 is a functional block diagram related to spherical image output processing in the spherical image display system according to the present embodiment. The image data flow diagram in the spherical image output processing. The figure which illustrates the projection system employ|adopted with a fisheye lens. The figure which shows the data structure of the image data of the spherical image format used by this embodiment. The figure explaining the perspective projection (Perspective Projection) performed by three-dimensional graphics display. The flowchart which shows the omnidirectional image display process which the image processing apparatus of this embodiment performs. The figure which illustrates the image viewer screen which displays a spherical image in a predetermined range. 3 is a functional block diagram of a plane image generation unit on the image processing apparatus according to the present embodiment. FIG. The figure explaining a model coordinate system, the position (d) of a camera, a viewing angle ((theta)), and the relationship with an angle of view ((phi)). FIG. 6 is a diagram (1/2) illustrating how to determine an image generation parameter according to a zoom designated value. FIG. 3B is a diagram (2/2) explaining how to determine the image generation parameter according to the designated zoom value. The hardware block diagram of the image processing apparatus in this embodiment.

Hereinafter, embodiments of the present invention will be described, but the embodiments of the present invention are not limited to the embodiments described below. In the embodiment to be described, as the image processing system, an omnidirectional imaging device, and an image processing device that receives an image captured by the omnidirectional imaging device and generates an output image to be output to a display device or the like. A spherical image display system including and will be described as an example.

FIG. 1 is a diagram illustrating a schematic configuration of a spherical image display system 100 according to the present embodiment. The celestial sphere image display system 100 shown in FIG. 1 includes a celestial sphere imaging device 110 that captures an image of the celestial sphere, a smartphone 120, a tablet terminal 122, and a personal computer 124. The smartphone 120, the tablet terminal 122, and the personal computer 124 each constitute an image processing device according to the present embodiment, which has an image viewer function of displaying an image captured by the omnidirectional imaging device 110 on a display or the like.

In the embodiment shown in FIG. 1, the omnidirectional imaging device 110 and the image processing devices 120 to 124 are not particularly limited, but a wireless LAN (Local Area Network), a wireless USB (Universal Serial Bus), or It is connected by a wireless connection such as Bluetooth (registered trademark). The image in the predetermined format captured by the omnidirectional imaging device 110 is transmitted to the image processing devices 120 to 124 via wireless communication, subjected to predetermined image processing, and provided in the image processing devices 120 to 124. It is displayed on the display device. Note that the above connection form is an example, and may be connected by wire via a wired LAN, a wired USB, or the like.

In the embodiment to be described, the omnidirectional imaging device 110 includes two imaging optical systems each including an imaging optical system and a solid-state imaging device, and captures images from each direction for each imaging optical system to generate a captured image. .. Each of the image forming optical systems can be configured as a fisheye lens with, for example, 7 elements in 6 groups. The fisheye lens has a total angle of view larger than 180 degrees (=360 degrees/n; n=2), preferably a total angle of view of 185 degrees or more, and more preferably 190 degrees or more. Has full angle of view. In the embodiment described, the fisheye lens includes a wide-angle lens and a so-called ultra-wide-angle lens.

The celestial sphere image pickup device 110 joins the picked-up images picked up by the plurality of solid-state image pickup devices, and synthesizes the images to generate an image having a solid angle of 4π radians (hereinafter, referred to as “all-sky image”). The celestial sphere image is a photograph of all directions that can be viewed from the photographing point. As described above, since the fish-eye lens has a total angle of view of more than 180 degrees, in the captured image captured by each imaging optical system, the capturing range overlaps at the portion exceeding 180 degrees. When the images are stitched together, this overlapping area is referred to as reference data representing the same image, and a spherical image is generated.

Here, in the embodiment to be described, a celestial sphere image with a solid angle of 4π radians is generated, but in another embodiment, a so-called panoramic image in which only a horizontal plane is photographed at 360 degrees may be used. Moreover, in the embodiment to be described, the configuration is described as including two imaging optical systems, but the number of imaging optical systems is not particularly limited. In another embodiment, the celestial sphere imaging device 110 includes an imaging body that includes three or more fisheye lenses in its optical system, and a celestial sphere image based on a plurality of captured images captured by the three or more fisheye lenses. May be provided. In still another embodiment, the omnidirectional imaging device 110 includes an imaging body that includes a single fish-eye lens in its optical system, and the omnidirectional sphere is based on a plurality of picked-up images picked up by the single fish-eye lens in different directions. It may have a function of generating an image.

The generated spherical image is transmitted in a predetermined format to the external image processing devices 120 to 124 by communication, or is output to an external storage medium such as an SD (registered trademark) card or a compact flash (registered trademark). To be done.

The image processing devices 120 to 124 receive the omnidirectional image through the connection, or acquire the omnidirectional image via an external recording medium in which the omnidirectional image is recorded, and once record their own. Save the spherical image on the device. The image processing apparatuses 120 to 124 generate an output image for display output on a flat display device such as a display included in itself or a projector connected to itself from the obtained omnidirectional image, and output from the flat display device. The output image can be displayed. The image processing apparatuses 120 to 124 can also print out the generated output image from an image forming apparatus connected to itself to a paper medium. Details of the process of generating the output image from the spherical image will be described later.

In the embodiment shown in FIG. 1, the omnidirectional imaging device 110 and the image processing devices 120 to 124 are further connected to the Internet 102 via a communication device 104 such as an access point, a mobile router or a broadband router. .. An image display server 130 is provided on the Internet 102.

The image display server 130 shown in FIG. 1 receives the omnidirectional images sent from the omnidirectional imaging device 110 or the image processing devices 120 to 124, accumulates and manages the received omnidirectional images. The image display server 130 also generates an output image based on the spherical image in response to a display request for the spherical image from the image processing devices 120 to 124 and other information processing terminals, and the generated output image. To the request source device. Thereby, the output image can be displayed on the flat display device of the request source device.

The image display server 130 may be configured as a web server in a specific embodiment, receives a request for an image registration request including a omnidirectional image according to HTTP (HyperText Transfer Protocol), and displays the omnidirectional image. Can be accumulated. The image display server 130 further receives a request for an image display request in which the omnidirectional image to be output is designated, reads the omnidirectional image to be processed, performs image processing, and generates an output image, It is possible to respond with a response including the output image. In the request source device that has received the response, the web browser displays the received output image on the flat display device. Then, the web browser appropriately prints out the output image. The image display server 130 is also configured as an image processing device that generates an output image in this embodiment.

Hereinafter, the omnidirectional image output processing of generating an output image from the omnidirectional image in the present embodiment will be described in more detail with reference to FIGS. 2 to 12. FIG. 2 is a diagram showing a functional block 200 related to the spherical image output processing in the spherical image display system in the present embodiment. The functional block 200 shown in FIG. 2 is configured to include a functional block 210 on the omnidirectional imaging device 110 and a functional block 250 on the image processing devices 120 to 124 and 130.

The functional block 210 of the celestial sphere image pickup device 110 receives two image pickup optical systems 212A and 212B for picking up images in respective directions and the respective picked-up images picked up by the image pickup optical systems 212, and receives a omnidirectional image. Is generated and output.

The functional block 250 of the image processing apparatus includes an input unit 252, an output unit 254, a celestial sphere image storage unit 256, a user input reception unit 258, a plane image generation unit 260, and an image output unit 262. To be done. The input unit 252 is an input device such as a touch panel, a mouse, and a keyboard. The output unit 254 is an output device such as a flat display device that displays an image processing result according to an input of a user operation performed on the input unit 252, an image forming apparatus that prints out the image processing result, and the like. The input unit 252 and the output unit 254 may be provided in the image processing apparatus itself, or may be provided in an external device to which the image processing apparatus is connected.

The celestial sphere image storage unit 256 is a unit that stores the celestial sphere image captured by the celestial sphere imaging device 110 and input to the image processing devices 120 to 124 via the above-mentioned connection or an external recording medium. The user input accepting unit 258 accepts an input value defining the output range of the omnidirectional image according to the operation based on the output range changing operation performed via the input unit 252, and outputs the input value to the planar image generating unit 260. Pass to.

Examples of the output range changing operation include a pan operation for moving the visual field left and right, a tilt operation for moving the visual field up and down, and a zoom operation for enlarging or reducing the range of the image to be output. As a result of the change by the output range changing operation or as a result of direct input, the pan specified value, the tilt specified value, and the zoom specified value are acquired as input values that define the output range of the omnidirectional image.

The plane image generation unit 260 determines a parameter for image generation (hereinafter referred to as an image generation parameter) based on the received input value, and the spherical image based on the determined image generation parameter. Generate an output image from. The image output unit 262 causes the output unit 254 to output the generated output image. The output image is a plane image for proper display on the plane display device.

When the image processing apparatus operates as a web server such as the image display server 130, the configurations of the input unit 252 and the output unit 254 are as follows. That is, the input unit 252 is configured as an HTTP receiving unit that receives an HTTP request for image registration. The output unit 254 is configured as an HTTP transmission unit that returns the generated output image to the request source as a response in response to the HTTP request for image display.

FIG. 3 is a diagram illustrating the data structure of each image and the data flow of the image in the spherical image output processing. The imaging optical system 212 according to the present embodiment generates two captured images by the imaging process. In the present embodiment, the light incident on the lens optical system is imaged on the light receiving area of the corresponding solid-state image sensor according to a predetermined projection method. The picked-up image is picked up by a two-dimensional solid-state image pickup device in which the light receiving area forms a plane area, and is image data expressed in a plane coordinate system. Further, in the present embodiment, a so-called circumferential fisheye lens having a smaller image circle diameter than the diagonal line of the image is adopted. Therefore, the obtained captured image is configured as a plane image including the entire image circle onto which each shooting range is projected, as shown by "captured image A" and "captured image B" in FIG.

FIG. 4 is a diagram illustrating a projection method that can be adopted in a fisheye lens. Although various designs are conceivable for the fisheye lens, the projection methods thereof include the orthogonal projection method (FIG. 4A), the equidistant projection method (FIG. 4B), and the stereoscopic projection as shown in FIG. The method (FIG. 4(C)) and the equal solid angle projection method (FIG. 4(D)) can be mentioned. Further, in the embodiment to be described, it is assumed that the captured image captured by one fish-eye lens is captured in a direction of approximately a hemisphere (a part of the total angle of view exceeds 180 degrees extends from the hemisphere) from the shooting point. Become. Then, as shown in FIG. 4, the image is generated with an image height r corresponding to the incident angle β with respect to the optical axis. The pixel position (image height: distance in the radial direction from the lens focus) r on the light receiving area that receives light from the incident angle β is calculated by using the following projection function according to a predetermined projection model, where f is the focal length. You can decide.

The azimuth (incident angle and rotation angle around the optical axis) is associated with the coordinates of the pixel position on the planar image by the above formula corresponding to the projection method adopted by the fisheye lens. In a preferred embodiment, the fisheye lens can use the stereoscopic projection method shown in FIG.

The synthesizing processing block 214 utilizes information from a three-axis acceleration sensor (not shown) on the two captured images obtained from the two image-capturing optical systems 212A and 212B, performs distortion correction as well as top-bottom correction, and synthesizes the images. In the image synthesizing process, first, each celestial sphere image including each complementary semi-hollow portion is generated from each captured image configured as a plane image. Then, the two spherical images including the respective hemispherical parts are aligned based on the matching of the overlapping regions and the images are combined to generate the spherical image including the entire spherical surface.

FIG. 5 is a diagram illustrating the data structure of image data in the spherical image format used in this embodiment. As shown in FIG. 5(A), the image data in the spherical image format has coordinates of a vertical angle φ with respect to a predetermined axis and a horizontal angle θ corresponding to the rotation angle around the predetermined axis. Is expressed as an array of pixel values. Here, as shown in FIG. 3, the omnidirectional image has a vertical angle (latitude in latitude/longitude coordinates) about the zenith direction of the captured scene and a horizontal angle around the zenith direction (latitude/longitude coordinates). It is assumed to be represented by a coordinate system composed of (longitude in). The vertical angle φ is in the range of −90 degrees to +90 degrees, and the horizontal angle θ is in the range of −180 degrees to +180 degrees.

Each coordinate value (θ, φ) in the omnidirectional format is associated with each point on the spherical surface that represents the azimuth around the shooting point, as shown in FIG. It is associated with the spherical image. The plane coordinates of the captured image captured by the fisheye lens and the coordinates on the spherical surface of the spherical image format are associated with each other by performing the above-mentioned projection function and appropriate coordinate conversion.

In the following description, the two-dimensional coordinates in the spherical image format shown in FIG. 5A are the coordinate system of the lower left origin, as indicated by the gray axis in FIG. 5A. Then, in the following description, the two-dimensional coordinates are converted into a horizontal angle value x in the range corresponding to the number of pixels in the 0 to horizontal direction and a vertical angle value y in the range corresponding to the number of pixels in the 0 to vertical direction. Shall be handled. For example, when pixels are formed in 1/10 degree increments, the horizontal angle value x is in the range of 0 to 3799 and the vertical angle value y is in the range of 0 to 1800. The relationship between the horizontal angle value x and the horizontal angle θ and the relationship between the vertical angle value y and the vertical angle φ are represented by the following (1) and (2). In equations (1) and (2) below, w and h correspond to the image width (for example, 3600 pixels) and the image height (for example, 1801 pixels) of the spherical image format, respectively.

Then, between the two-dimensional coordinates (x, y) of the spherical image shown in FIG. 5(A) and the three-dimensional coordinates (xs, ys, zs) of the surface of the spherical image shown in FIG. 5(B). The relationship of is calculated by the following equations (3) and (4). The three-dimensional coordinate shown in FIG. 5B is a right-handed system, where the center of the sphere is the origin and r represents the radius.

The relationship between the two images (captured image A and captured image B) captured by the two capturing optical systems 212A and 212B and the image area on the spherical image is shown in FIG. The correspondence is represented by "celestial sphere image". The image-combined spherical image is stored in a built-in memory or an external recording medium in a predetermined format. The file format in which the spherical image is stored may be an uncompressed still image such as a bitmap, JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), PNG (Portable Network Graphics). It may be a still image in a compressed format such as. In still another embodiment, the spherical image may be included as a frame image in a moving image such as MPEG (Moving Picture Experts Group) or AVI (Audio Video Interleave). In the embodiments described below, the spherical image is described as a still image.

In the functional block 250 of the image processing apparatus, the omnidirectional image storage unit 256 stores the omnidirectional image, and the plane image generation unit 260 continuously inputs the omnidirectional image and outputs an image from the omnidirectional image. Image processing is performed to convert to. It should be noted that in the embodiment to be described, the input omnidirectional image is described as being imaged by the omnidirectional imaging device 110 in the preferred embodiment, but the origin of the omnidirectional image is not necessarily limited. Not something. For example, the captured image may be subjected to predetermined image processing, or an image generated by computer graphics may be included.

The planar image generation unit 260 receives, from the user input reception unit 258, an input value including the determined pan designation value, tilt designation value, and zoom designation value as a result of the change or the like by the output range changing operation. The plane image generation unit 260 determines an image generation parameter as described later based on the received input value, and executes the image generation process of the output image based on the determined image generation parameter.

As described above, the spherical image can be associated with the three-dimensional coordinates by the above equations (3) and (4). In the image generation process, a three-dimensional model in which the input spherical image is pasted on the inner surface of a predetermined three-dimensional shape is constructed, and under a predetermined condition, a virtual camera (hereinafter, simply referred to as a camera). .) to obtain an output image S.

FIG. 6 is a diagram for explaining perspective projection (Perspective Projection) performed in three-dimensional graphics display. As shown in FIG. 6, the plane image generation unit 260 generates an output image of a plane image by performing perspective projection processing on the three-dimensional model in which the spherical image is attached to the inner surface of the sphere. The image generation parameter for perspective projection is determined according to the input value. The image generation parameters may include the position (d) of the viewpoint of the camera, the direction (v), the viewing angle (Θ) and the projection range (zNear, zFar) in certain embodiments.

The output image S is an image observed within a specific viewing angle (Θ) when the three-dimensional model is viewed from the center of the sphere in a specific latitude/longitude direction (v) according to the shape of the display area. The image is cut out. The parameters of the projection range (zNear, zFar) specify the range of perspective projection, but an appropriate value is determined. The details of the method of determining the image generation parameter and the projection process will be described later with reference to FIGS.

In the embodiment shown in FIG. 2, the image pickup optical system 212 and the combining processing block 214 are components on the omnidirectional image pickup device. The input unit 252, the output unit 254, the celestial sphere image storage unit 256, the user input reception unit 258, the plane image generation unit 260, and the image output unit 262 are distributed and mounted as components on the image processing apparatus. However, the mode of implementation is not particularly limited.

In other embodiments, all components may be placed on a single device to form a spherical image display system. In still another embodiment, an arbitrary part of the components is arranged on any one of the plurality of devices forming the spherical image display system, and the spherical image is obtained as a combination of the plurality of devices. The display system may be configured. For example, in a specific embodiment, the image processing device may include a composition processing block, and may receive two captured images from the omnidirectional image capturing device to prepare the omnidirectional image.

Hereinafter, the flow of the omnidirectional image output processing in this embodiment will be described in more detail with reference to FIGS. 7 and 8. Note that, as the output process, a display process in an image viewer that displays a spherical image will be described below. FIG. 7 is a flowchart showing the spherical image display processing executed by the image processing apparatus of this embodiment. FIG. 8: is a figure which illustrates the image viewer screen which displays a spherical image in a predetermined range.

The process shown in FIG. 7 is started from step S100 in response to, for example, an operator of the image processing device 250 issuing a display instruction specifying a predetermined spherical image. In step S101, the image processing apparatus 250 determines the initial image processing parameters based on the preset default pan designation value, tilt designation value, and zoom designation value by the planar image generation unit 260. In step S102, the image processing apparatus 250 causes the planar image generation unit 260 to generate a planar image from the spherical image based on the determined image processing parameters.

In step S103, the image processing device 250 causes the image output unit 262 to display the generated planar image at a predetermined position on the image viewer screen. The image viewer screen 300 shown in FIG. 8 includes an image display area 310 and GUI (Graphical User Interface) components 322 and 324 for changing the display range of the image displayed in the image display area 310. In the image display area 310, the output image generated by the plane image generation unit 260 in the range corresponding to the input value is displayed by the image output unit 262.

In step S104, the image processing apparatus 250 determines whether the display range changing operation has been received by the user input receiving unit 258. Here, the display range changing operation is detected by the occurrence of an operation event such as click or flick performed on the GUI components 322 and 324 corresponding to each operation. The image viewer screen 300 illustrated in FIG. 8 includes a GUI component 322I that waits for a zoom-in instruction and a GUI component 322O that waits for a zoom-out instruction for changing the designated zoom value. The image viewer screen 300 further waits for a left button 324L and a right button 324R for waiting for a pan instruction to the left and right directions and for waiting for a tilt instruction for up and down, in order to change the pan specified value and the tilt specified value. An upper button 324U and a lower button 324D are included.

The display range changing operation can be detected by the occurrence of an operation event such as a shortcut key associated with each display range changing operation, a gesture, or a multi-touch operation, in addition to an operation on the GUI component. For example, the shortcut keys may include "+" and "-" buttons for instructing zoom-in and zoom-out on the keyboard. Further, key operations for the left and right arrow buttons and the up and down arrow buttons for instructing left and right pan and up and down tilt may be used as the shortcut keys. Multi-touch operations can include pinch in and pinch out associated with zoom operations.

In step S104, until the display range changing operation is received (while NO in step S104), the inside of step S104 is looped to wait for the display range changing operation. If it is determined in step S104 that the display change operation has been accepted (YES), the process branches to step S105.

In step S105, the changed image processing parameter is determined based on the pan designation value, tilt designation value, and zoom designation value determined as a result of the display change operation, and the process branches to step S102. In this case, in subsequent step S102, the plane image generation unit 260 performs the plane image generation process based on the changed image processing parameter. In step S103, the image output unit 262 updates the image display area 310 of the image viewer screen 300 with the plane image newly generated according to the user operation.

Hereinafter, the omnidirectional image output processing according to the present embodiment will be described in more detail with reference to FIGS. 9 to 12. FIG. 9 is a diagram showing detailed functional blocks of the plane image generation unit 260 on the image processing apparatus in this embodiment. The plane image generation unit 260 shown in FIG. 9 includes a parameter determination unit 264, a texture mapping unit 266, and a projection unit 268.

The parameter determination unit 264 determines the viewpoint position (d) and the viewing angle (Θ) of the camera according to various input values (pan designated value, tilt designated value, and zoom designated value) passed from the user input reception unit 258. It is a determining means for determining the image generation parameter to be included.

The texture mapping unit 266 is a generation unit that generates a three-dimensional model in which a spherical image that is a display target image is attached to a predetermined three-dimensional shape. The three-dimensional model can be generated by a so-called texture mapping method. Texture mapping is a general graphics process such as OpenGL that is compatible with GPUs installed in information processing terminals such as smartphones and tablets that have limited computing power. Say. The texture mapping unit 266 reads out the specified spherical image, transfers it to the texture buffer that stores the texture, and allocates it to the three-dimensional model.

In the described embodiment, the three-dimensional shape may be a sphere, a cylinder, or any other three-dimensional shape capable of projecting an output image that does not make the viewer feel uncomfortable. From the viewpoint of not making the viewer feel uncomfortable and simplifying the arithmetic processing, a sphere can be preferably used. The three-dimensional shape has at least one inner surface to which the spherical image is attached, and when a sphere is adopted, includes a spherical surface to which the spherical image is attached.

The projection unit 268, in accordance with the image generation parameter determined by the parameter determination unit 264, uses a predetermined projection method to project the three-dimensional model to which the spherical image is attached from the camera whose viewpoint is set at a predetermined position. Projection means for projecting and generating an output image. An output image can be generated by rendering an image of a texture-mapped three-dimensional model observed from an arbitrary camera viewpoint under a predetermined condition.

According to Non-Patent Document 1, it is known that it is effective to change the projection method between the wide-field display and the narrow-field display. However, if a plurality of projection methods are continuously applied, the performance requirements for the image processing apparatus become strict.

Therefore, in the present embodiment, in order to relax the hardware requirements of the image processing apparatus, the plane image generation unit 260 changes the image generation parameter of the display model by a single projection method, so that the viewer can appreciate the image. Adopt a configuration that obtains a suitable display effect. Hereinafter, the process of determining the image generation parameter according to the input value will be described with reference to FIGS. Note that, in the embodiment described, the perspective projection method (perspective projection) is used as the projection method, but in other embodiments, another projection method such as an orthogonal projection method may be adopted.

As described above, the image generation parameters are, in a specific embodiment that employs perspective projection as a projection method, the position (d), the direction (v), the viewing angle (Θ), and the projection range (zNear, zFar) of the viewpoint of the camera. including. Three-dimensional computer graphics typically defines three coordinate systems: a world coordinate system, a model coordinate system, and a camera coordinate system. The world coordinate system defines an absolute three-dimensional space, and cameras and objects are arranged in the three-dimensional space defined by the world coordinate system. The model coordinate system is a coordinate system centered on a predetermined object. In the described embodiment, a sphere model is constructed and this sphere model is placed at the origin of the world coordinate system. Therefore, the world coordinate system and the model coordinate system of the spherical model may have the same origin and different axes. The camera coordinate system represents the direction (v) of the visual field centered on the viewpoint of the camera.

Then, the projecting unit 268 projects the spherical model having the omnidirectional image pasted on the inner surface thereof from the viewpoint of the camera onto a two-dimensional screen, and sets the obtained projected image as a display image. The screen is arranged on a plane passing through the origin of camera coordinates, and a spherical image is projected on the screen by perspective projection.

FIG. 10 is a diagram illustrating the relationship between the model coordinate system, the position (d) and the viewing angle (Θ) of the camera, and the angle of view (Φ) that represents the range of the image captured in the field of view. When the viewpoint of the camera is located at the center of the spherical model, the angle of view (Φ) representing the range of the image reflected in the field of view and the angle of view (Θ) of the camera match. However, as indicated by a double circle in FIG. 10, when the position (d) of the viewpoint of the camera departs from the center of the three-dimensional model, the angle of view (Φ) and the viewing angle (Θ) of the camera are Will take different values. Zooming in and zooming out corresponds to an operation of changing the angle of view (Φ). In the described embodiment, a change in the angle of view (Φ) is generated by changing either the position (d) of the viewpoint of the camera or the viewing angle (Θ) according to the range of the designated zoom value.

The change of the image generation parameter according to the operation of the pan, tilt and zoom display ranges in the present embodiment is summarized in Table 1 below.

In Table 1, the movement of the display position of the image due to the tilt and pan is realized by fixing the direction (v) of the visual field and rotationally converting the spherical model in the world coordinate system. However, in another embodiment, the movement of the image display position may be realized by fixing the spherical model with respect to the world coordinate system and changing the direction (v) of the visual field of the camera.

Hereinafter, with reference to FIGS. 11 and 12, the process of determining the image generation parameter according to the zoom designated value in the present embodiment will be described in more detail. 11 and 12 are diagrams for explaining how to determine the image generation parameter according to the designated zoom value, and also show the display range of the output image and the spherical model at that time. 11 and 12 show a method of determining the image generation parameter when the specific zoom designation values (z) shown by A to E are given.

Further, Table 2 below collectively shows the image generation parameters determined according to the designated zoom value (z), the display magnification and the angle of view (Φ) at that time. In Table 2 below, viewWH represents the width or height of the display area of the output image. When the display area is horizontally long, it is the width of the display area, and when the display area is vertically long, it is the height of the display area. In Table 2 below, imgWH represents the width or height of the image. When the display area is horizontally long, it is the image width, and when the display area is vertically long, it is the image height. imgDeg represents the angle of the display range of the image, and is 360 degrees for the image width and 180 degrees for the image height.

In the present embodiment, the range of the designated zoom value (z) is, as shown in Table 2, four ranges including the range of A to B, the range of B to C, the range of C to D, and the range of D to E. It is divided into ranges. The zoom designation value (z) is a value corresponding to the angle of view to be displayed, and in the embodiment to be described, the zoom designation value (z) is designated by the user in the range from the minimum angle of view 60 degrees to the maximum angle of view 300 degrees.

Considering zoom-out, when the specified zoom value (z) is in the range A to B, the viewing angle (Θ) of the camera is fixed at 60 degrees and the camera position (d) moves away from the center. Then, as shown in A of FIG. 11 and B of FIG. 11, the angle of view (Φ) is widened. The camera position (d) in the range of A to B is expressed by the formula in Table 2 from the origin 0 shown in A of FIG. 11 to 1 corresponding to the outer edge of the spherical model shown in B of FIG. As described above, the value is determined according to the angle of view (Φ) and the designated zoom value (z).

When the designated zoom value (z) is in the range of B to C, which is on the wider visual field side than A to B, the camera position (d) is the spherical model as shown in B of FIG. 11 and C of FIG. The angle of view (Φ) is widened by fixing it to the outer edge (d=1) and widening the viewing angle (Θ) of the camera. The viewing angle (Θ) of the camera increases from 60 degrees, which was fixed in the range A to B, to 120 degrees, according to the calculation formula Θ=Φ/2. The angle of view (Φ), which represents the range of the image reflected in the field of view, coincides with the designated zoom value (z) in the range A to B and the range B to C, and monotonically increases.

When the designated zoom value (z) is in the range of C to D and the range of D to E, which are on the side of the wider field of view than the range of B to C, C in FIG. 11, D in FIG. As shown. That is, the angle of view (Φ) is expanded by moving the position (d) of the viewpoint of the camera again away from the center while the viewing angle (θ) of the camera is fixed at 120 degrees. The position (d) of the camera viewpoint is determined according to the zoom designated value (z) according to the calculation formula described in Table 2. Further, in the range of C to D and the range of D to E, the angle of view (Φ) does not match the zoom specified value (z).

Dmax1 in Table 2 corresponding to the position indicated by D in FIG. 12 is the distance at which the spherical model is displayed at the maximum angle of view in the full rectangle of the display area as shown in FIG. Correspondingly, in a particular embodiment, it can be calculated by equation (5) below. Dmax2 in Table 2 corresponding to the position (D) indicated by E in FIG. 12 is the maximum angle of view at which the spherical model is inscribed in the rectangle of the display area as shown in FIG. Corresponding to the distance at which the sphere model is displayed, in a particular embodiment it is calculated by equation (6) below.

In the above formulas (5) and (6), viewW and viewH represent the width and height of the display area. Therefore, dmax1 and dmax2 are values that depend on the size (width and height, diagonal length) of the display screen. dmax2 corresponds to the farthest position that the viewpoint of the camera can take, and the designated zoom value (z) is limited according to the size of the display area. By limiting the zoom specified value (z) so that the position of the camera viewpoint falls within the range (to dmax2) shown in Table 2, the image fits on the display screen or the spherical image becomes a spherical image at a predetermined magnification. You can end the zoom out with the shape displayed. As a result, the viewer can visually recognize that the displayed image is a celestial sphere image, and the zoom-out can be ended without any discomfort.

Further, as is apparent from Table 2 and FIG. 11, the angle of view (Φ) is continuous between the ranges described above, but the angle of view (Φ) remains constant due to zooming out to the wide angle side. Not increasing. That is, in the range of C to D in the range of C to D to E in which the viewpoint position of the camera is changed, the view angle (angle of view) increases as the position (d) of the viewpoint of the camera becomes farther from the center of the spherical model. Φ) increases, but in the range of D to E, the angle of view (Φ) decreases as the distance increases. This is because the area outside the spherical model is reflected in the visual field. Then, by moving the viewpoint position (d) of the camera in the wide field of view with the specified zoom value of 240 degrees or more, it is possible to change the angle of view (Φ) while displaying with less discomfort.

Therefore, considering the change of the designated zoom value to the wide angle direction, basically, the angle of view (Φ) becomes wider. At this time, the increase in the viewing angle (Θ) of the camera is suppressed, and the camera is moved away from the model coordinate system, so that a feeling of openness at the time of wide-angle display is expressed, and thus image distortion is reduced. Further, since the movement of the camera away is similar to the behavior when a person actually tries to confirm a wide range, it is considered that the zoom-out is performed with less discomfort. In the range of D to E, the angle of view (Φ) decreases as the designated zoom value goes to the wide field of view side, but this causes the viewer to feel as if he or she is moving away from the sphere, which makes the viewer feel uncomfortable. There will be less zoom out.

In the above description, in each range, only one of the camera viewpoint position (d) and the viewing angle (Θ) is changed and the other is fixed. However, in another embodiment, a mode in which one of the camera viewpoint position (d) and the viewing angle (Θ) is preferentially changed and the other is changed with a relatively small change amount is not hindered. Further, in the above description, the method of determining the image generation parameter has been described in the context of changing to the zoom-out direction, but the image generation parameter can be similarly determined when zooming in.

Hereinafter, the hardware configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 13 is a diagram showing the hardware configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment is configured as a mobile information terminal such as a tablet terminal 122. A tablet terminal 122 shown in FIG. 13 includes a mobile processor 10 in which a single-core or multi-core CPU (Central Processing Unit), a GPU, a baseband processor, a memory controller, and the like are integrated as a SoC (System on Chip), and the mobile processor 10. A memory 12 such as an LPDDR (Low-Power Double Data Rate) SDRAM that is connected and provides a work area such as a CPU, a flash memory 14, and an external recording medium slot 16 such as an SD card are included.

The flash memory 14 stores an OS for controlling the tablet terminal 122, a control program for realizing the above-described functional units, various system information and various setting information, and user data including the above-mentioned spherical image. A recording medium in which user data such as a spherical image is stored is attached to the external recording medium slot 16.

The mobile processor 10 is further connected via a touch screen controller 18 and a display interface 20 to a display 22 equipped with a touch screen sensor. The display 22 displays various setting screens and application screens, and in the present embodiment, can display an image viewer screen including an output image generated from a omnidirectional image. The tablet terminal 122 further includes a video output interface 24 such as HDMI (registered trademark, High-Definition Multimedia Interface) connected to the mobile processor 10, and can be connected to an external display or projector.

The tablet terminal 122 further includes a camera 28 including an image sensor such as CMOS (Complementary Metal Oxide Semiconductor). The camera 28 is connected to the mobile processor 10 via the camera interface 26. The tablet terminal 122 further includes an audio codec 30 that performs audio encoding and decoding processing, and an audio switch 32 that switches audio of headphones and speakers.

The mobile processor 10 is further connected to a wireless LAN port 34 and a short-range wireless port 36 such as Bluetooth (registered trademark) so that it can be connected to an external device by wireless communication. In the embodiment described, the tablet terminal 122 is connected to the external omnidirectional imaging device 110 via the wireless LAN port 34 or the short-range wireless port 36. The tablet terminal 122 includes a power management unit 38, and the power management unit 38 manages the external power of the tablet terminal 122 and the power of the battery.

The tablet terminal 122 according to the present embodiment reads the control program from the flash memory 14 and expands it in the work space provided by the memory 12 to control each of the above-described functional units under the control of the CPU integrated in the mobile processor 10. And realize each processing. At that time, the arithmetic function of the GPU integrated in the mobile processor 10 is called via the graphic processing API such as OpenGL, and the image calculation such as the texture mapping processing and the projection processing described above is executed.

According to the embodiment described above, when displaying an image, in a display in a wide field of view, high-speed display is possible while reducing discomfort caused by stretching of subjects at the top, bottom, left, and right edges, and It is possible to provide an image processing system, an image processing method, and a program whose requirements are relaxed.

According to the above-described embodiment, since the display model is constructed as a single projection method, even an image processing apparatus having a limited image calculation function can perform smooth zoom display in real time. Become. Then, since the zoom-out is expressed by moving the viewpoint position of the camera away from the three-dimensional model, it is possible to suppress the increase of the viewing angle and to provide the feeling of expanse, while suppressing the distortion of the image.

The functional unit can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark), or an object-oriented programming language, and can be ROM, EEPROM, EPROM. , Flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, or other device-readable recording medium, or through an electric communication line. It can be distributed. In addition, a part or all of the above-mentioned functional units can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA), or as an ASIC (application specific integration). Circuit configuration data (bit stream data) to be downloaded to the PD in order to implement the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, and VHDL (VHSIC (Very High Speed). Integrated Circuits) Hardware Description Language), Verilog-HDL, and the like can be distributed by a recording medium as data.

Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art can conceive of other embodiments, additions, changes, deletions, and the like. The present invention can be modified within the scope of the present invention, and is included in the scope of the present invention as long as the effects and advantages of the present invention are exhibited in any of the aspects.

Reference numeral 100... Global spherical image display system, 102... Internet, 104... Communication equipment, 110... Global spherical imaging device, 120-124... Image processing device, 130... Image display server, 200... Functional block, 210... Functional block, Reference numeral 212... Imaging optical system, 214... Compositing processing block, 250... Functional block, 252... Input section, 254... Output section, 256... Spherical image storage section, 258... User input receiving section, 260... Planar image generating section, 262... Image output unit, 264... Parameter determination unit, 264... Texture mapping unit, 266... Projection unit, 300... Image viewer screen, 310... Image display area, 312, 314... GUI component, 10... Mobile processor, 12... Memory , 14... Flash memory, 16... External recording medium slot, 18... Touch screen controller, 20... Display interface, 22... Display, 24... External video interface, 26... Camera interface, 28... Camera, 30... Audio codec , 32... Voice switch, 34... Wireless LAN port, 36... Short-range wireless port, 38... Power management unit

J. Kopf, et.al, "Capturing and Viewing Gigapixel Images", ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2007, Volume 26, Issue 3, July 2007, Article No. 93, ACM New York, NY, USA

Claims (14)

A processor for generating an image in which a spherical image is pasted into a predetermined three-dimensional shape and inputting it to the display as a display image , wherein the processor is
A viewpoint capable of continuously moving inside and outside of a predetermined three-dimensional shape to which the spherical image is pasted is provided,
A process of changing the viewing angle of the display image ,
A process of changing the position of the viewpoint with respect to the display image ,
Is to
When enlarging or reducing the display image,
Performing a process of generating the display image by changing the viewing angle in a range of a predetermined magnification,
A process of generating the display image by changing the position of the viewpoint outside the range of the predetermined magnification is performed.
A device characterized by the above. The processor is
When enlarging or reducing the display image,
2. The apparatus according to claim 1, wherein one of the view angle and the position of the viewpoint is changed preferentially. The processor is
The apparatus according to claim 1 , wherein the angle of view is continuously changed in the range of all magnifications. The predetermined three-dimensional shape is a sphere,
The processor is
The apparatus according to claim 1 , wherein an image of a sphere is generated in a range where the magnification is smaller than a predetermined magnification. A processor, the processor comprising:
A process of pasting the photographed spherical image on a predetermined three-dimensional shape, generating planar image data obtained by projecting on a plane at a specified viewpoint, and outputting to the output unit ,
A process of moving continuously between the inside and the outside of the predetermined three-dimensional shape of the viewpoint the celestial sphere image pasted,
A process of changing the viewing angle at the viewpoint,
A process of changing the position of the viewpoint,
Is intended to run,
When enlarging or reducing the planar image,
Performing a process of generating the planar image data by changing the viewing angle at the viewpoint in a range of a predetermined magnification,
Performing processing of changing the position of the viewpoint outside the range of the predetermined magnification to generate the planar image data
A device characterized by the above. The processor is
The spherical image is pasted on the inner surface of the predetermined three-dimensional shape,
The device according to claim 1 or claim 5 . Corresponding to enlargement or reduction of the display image or the planar image,
Position of the viewpoint, and wherein the moving between the center and the outside of the three-dimensional shape, according to claim 1 or claim 5. A processor, the processor comprising:
A process of pasting the captured omnidirectional image on a predetermined three-dimensional shape, generating planar image data obtained by projecting on a plane at a specified viewpoint, and outputting to the output unit,
A process of continuously moving the viewpoint inside and outside the predetermined three-dimensional shape to which the spherical image is pasted,
A process of changing the viewing angle at the viewpoint,
A process of changing the position of the viewpoint,
Is to
An apparatus for continuously changing the angle of view in the range of all magnifications when enlarging or reducing the planar image . The processor is
Performing a process of generating the planar image data by changing the viewing angle at the viewpoint in a range of a predetermined angle of view,
9. The apparatus according to claim 8 , wherein a process of changing the position of the viewpoint to generate the planar image data outside the range of the predetermined angle of view is performed. On the computer,
A process of pasting the captured omnidirectional image on a predetermined three-dimensional shape, generating planar image data obtained by projecting it on a plane at a specified viewpoint, and outputting it to an output unit ,
Process for continuously moving the inside and outside of the predetermined three-dimensional shape of the viewpoint the celestial sphere image pasted,
Be shall be executed changes make processing the viewing angle, and the change causes processing position of the viewpoint in the viewpoint
When enlarging or reducing the planar image,
A process of generating the planar image data by changing a viewing angle at the viewpoint within a range of a predetermined magnification, and
Processing for changing the position of the viewpoint outside the range of the predetermined magnification to generate the planar image data
Because of the program to the execution. On the computer,
The program according to claim 10 , which executes a process of pasting the spherical image on the inner surface of the predetermined three-dimensional shape. Corresponding to enlargement or reduction of the plane image,
The program according to claim 10 , wherein the position of the viewpoint is moved between the center of the three-dimensional shape and the outside. On the computer,
A process of pasting the captured omnidirectional image on a predetermined three-dimensional shape, generating planar image data obtained by projecting it on a plane at a specified viewpoint, and outputting it to an output unit,
A process of continuously moving the viewpoint inside and outside the predetermined three-dimensional shape to which the spherical image is pasted,
Processing for changing the viewing angle at the viewpoint, and
Processing for changing the position of the viewpoint
Is executed,
A program for continuously changing the angle of view in the range of all magnifications when enlarging or reducing the planar image . On the computer,
When enlarging or reducing the planar image,
The program according to claim 10 , which executes a process of preferentially changing one of the view angle and the position of the viewpoint.