JP2022043132A - Image processing system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system, an image processing method, and a program for projecting an image formed at a wide angle of view based on a three-dimensional model.

SOLUTION: The present image forming system 100 comprises: receiving means 258 that receives an input value defining an output range; creation means (260, 266) that create a three-dimensional model in which an object image is attached in a three-dimensional shape; determination means (260, 264) that determine the position of a viewpoint and a viewing angle based on the input value; and projection means (260, 268) that project the three-dimensional model from the viewpoint. When the input value is within a first range, the determination means (260, 264) preferentially change the viewing angle to change a range in which the object image enters a visual field, and when the input value is within a second range closer to a wide visual field than the first range, preferentially change the position of the viewpoint to change the range in which the object image enters the visual field.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an image processing technique, and more particularly to an image processing system, an image processing method, and a program for projecting an image composed of a wide angle of view based on a three-dimensional model.

Conventionally, a panoramic image viewer is known as a system for displaying an image taken by a camera or the like on a flat display. The panoramic image viewer is a system that stitches together a plurality of images shot in different shooting directions and overlaps some subjects, and displays the stitched composite image on the display.

The conventional panoramic image viewer has a function that enables various display range change operations such as pan (horizontal movement of the field of view), tilt (up / down movement of the field of view), and zoom (enlargement / reduction) for the stitched panoramic images. There is. The panoramic image viewer often uses a method of projecting an image pasted on a side surface of a cylinder or a spherical surface onto a plane when viewed from the center of gravity of the cylinder or sphere. In this case, on the planar display, an image on a three-dimensional surface having a focal point formed on the side surface of the cylinder or along the spherical surface corresponding to each setting value of pan, tilt, and zoom set by the user is displayed. It is displayed by projecting onto a flat image.

However, in the panoramic image viewer of the prior art, when the field of view is widened beyond a certain level by the zoom function of the display range changing operations, the image may be discomforted or distorted at the edge of the field of view. There was a problem that it could end up.

Non-Patent Document 1 is known as a technique for displaying a wide angle of view image such as a panoramic image without discomfort. Non-Patent Document 1 discloses a viewer that displays a panoramic image in a required field of view. The viewer of Non-Patent Document 1 continuously adjusts the projection method so as to perform perspective projection in a narrow field of view and cylindrical projection or spherical projection in a wide field of view.

However, in the prior art of Non-Patent Document 1, since the projection method is switched according to the zoom, the processing becomes complicated, and high calculation performance is required to realize real-time processing. On the other hand, in recent years, panoramic image viewers are often executed not only on personal computers but also on information terminals such as smartphone terminals and tablet terminals, which have relatively low CPU (Central Processing Unit) computing power. .. With such an information terminal having low computing power, it becomes difficult to perform real-time display of, for example, about 30 fps (Frame per Second) in a complicated process as in Non-Patent Document 1.

In recent years, many information terminals are equipped with a GPU (Graphics Processing Unit) in charge of graphics calculation in addition to the CPU. The GPU typically has a calculation function corresponding to an API (Application Programming Interface) such as OpenGL, and makes it possible to realize high-speed image processing calculation. However, the GPU provided in smartphones and the like corresponds to a subset version of OpenGL, and is obliged to perform operations with a relatively simple model.

From the above background, even in an information terminal with limited computing power, when displaying a panoramic image, while reducing the discomfort caused by stretching the subject at the top, bottom, left, and right edges in the display in the wide angle of view area. It has been desired to develop a technology capable of high-speed display.

The present invention has been made in view of the deficiencies of the above-mentioned prior art, and the present invention is to stretch the subject at the upper, lower, left and right edges in the display in a wide field of view when displaying an image. It is an object of the present invention to provide an image processing system, an image processing method, and a program in which the requirements for computing power are relaxed, which enables high-speed display while reducing the discomfort caused by.

The present invention provides an image processing system having the following features in order to solve the above problems. This image processing system has a reception means that accepts an input value that defines an output range, a generation means that generates a three-dimensional model in which a target image is pasted into a three-dimensional shape, and a viewpoint position and a field of view based on the above input values. It includes a determination means for determining an angle and a projection means for projecting the above-mentioned three-dimensional model from a viewpoint. When the above-mentioned input value is in the first range, the determination means preferentially changes the viewing angle to change the range within the field of view of the target image. On the other hand, when the input value is in the second range on the wide field of view side of the first range, the range within the visual field is changed by preferentially changing the position of the viewpoint.

According to the above configuration, when displaying an image, in the display in the area on the wide field of view side, a requirement for computing power when performing high-speed display while reducing discomfort due to stretching of the subject at the top, bottom, left, and right edges. Can be alleviated.

The schematic diagram which shows the spherical image display system by this embodiment. The functional block diagram about the spherical image output processing in the spherical image display system in this embodiment. Image data flow diagram in spherical image output processing. The figure which illustrates the projection method adopted in the fisheye lens. The figure which shows the data structure of the image data of the spherical image format used in this embodiment. A diagram illustrating a perspective projection performed in a three-dimensional graphics display. The flowchart which shows the spherical image display processing which performs the image processing apparatus of this embodiment. The figure which illustrates the image viewer screen which displays the whole celestial sphere image in a predetermined range. The functional block diagram of the plane image generation part on the image processing apparatus in this embodiment. The figure explaining the relationship between the model coordinate system, the position (d) of a camera, the viewing angle (Θ), and the angle of view (Φ). The figure (1/2) explaining how the image generation parameter is determined according to the zoom specified value. It is a figure (2/2) explaining how the image generation parameter is determined according to the zoom specified value. The hardware configuration diagram of the image processing apparatus in this embodiment.

Hereinafter, embodiments of the present invention will be described, but the embodiments of the present invention are not limited to the embodiments described below. In the embodiment described, as the image processing system, an omnidirectional image pickup device and an image processing device that receives an image captured by the omnidirectional image pickup device and generates an output image for output to a display device or the like. An omnidirectional image display system including and will be described as an example.

FIG. 1 is a diagram illustrating a schematic configuration of the spherical image display system 100 according to the present embodiment. The spherical image display system 100 shown in FIG. 1 includes a spherical imager 110 for imaging spherical images, a smartphone 120, a tablet terminal 122, and a personal computer 124. The smartphone 120, the tablet terminal 122, and the personal computer 124 each constitute an image processing device according to the present embodiment, which has an image viewer function for displaying an image captured by the celestial sphere image pickup device 110 on a display or the like.

In the embodiment shown in FIG. 1, the celestial sphere image pickup device 110 and the image processing devices 120 to 124 are not particularly limited, but are a wireless LAN (Local Area Network), a wireless USB (Universal Serial Bus), or the like. It is connected by a wireless connection such as Bluetooth®. Images in a predetermined format captured by the celestial sphere image pickup device 110 are transmitted to the image processing devices 120 to 124 via wireless communication, subjected to predetermined image processing, and included in the image processing devices 120 to 124. It is displayed on the display device. The above connection form is an example, and may be wiredly connected by a wired LAN, a wired USB, or the like.

In the embodiment described, the celestial sphere image pickup device 110 includes two image pickup optical systems each composed of an image pickup optical system and a solid-state image pickup element, and each image pickup optical system is photographed from each direction to generate an image pickup image. .. The imaging optical system can be configured as a fisheye lens, for example, with 7 elements in 6 groups. The fisheye lens has a total angle of view larger than 180 degrees (= 360 degrees / n; n = 2), preferably a total angle of view of 185 degrees or more, and more preferably 190 degrees or more. Has all angles of view. In the embodiment to be described, the fisheye lens includes a wide-angle lens and a so-called ultra-wide-angle lens.

The spherical image sensor 110 joins and synthesizes images captured by a plurality of solid-state image sensors to generate an image having a solid angle of 4π radians (hereinafter referred to as “spherical image”). The spherical image is taken in all directions that can be seen from the shooting point. As described above, since the fisheye lens has a total angle of view exceeding 180 degrees, in the captured images captured by each imaging optical system, the imaging range overlaps in the portion exceeding 180 degrees. When joining the images, this overlapping region is referred to as reference data representing the same image, and a spherical image is generated.

Here, in the embodiment described, a spherical image having a solid angle of 4π radians is generated, but in other embodiments, a so-called panoramic image in which only the horizontal plane is photographed at 360 degrees may be used. Further, in the embodiment described, the description is made assuming that the two imaging optical systems are included, but the number of the imaging optical systems is not particularly limited. In another embodiment, the spherical imager 110 includes an imager including three or more fisheye lenses in an optical system, and a spherical image based on a plurality of captured images captured by the three or more fisheye lenses. It may have a function to generate. In yet another embodiment, the spherical imager 110 comprises an imager including a single fisheye lens in its optical system and is based on a plurality of captured images captured in different orientations by the single fisheye lens. It may have a function to generate an image.

The generated spherical image is transmitted to an external image processing device 120 to 124 by communication in a predetermined format, or output to an external storage medium such as an SD (registered trademark) card or a compact flash (registered trademark). Will be done.

The image processing devices 120 to 124 receive the spherical image via the connection, or acquire the spherical image via an external recording medium on which the spherical image is recorded, and once record themselves. Save the spherical image on the device. The image processing devices 120 to 124 generate an output image to be displayed and output from the acquired whole celestial sphere image to a plane display device such as a display provided by the image processing device or a projector connected to the image processing device 120 to 124, and the image processing device 120 to 124 generate an output image to be displayed and output from the plane display device. The output image can be displayed. The image processing devices 120 to 124 can also print out the generated output image from the image forming device connected to the image processing device 120 to 124 on a paper medium. The details of the process of generating the output image from the spherical image will be described later.

In the embodiment shown in FIG. 1, the spherical image pickup device 110 and the image processing devices 120 to 124 are further connected to the Internet 102 via a communication device 104 such as an access point, a mobile router, and a broadband router. .. An image display server 130 is provided on the Internet 102.

The image display server 130 shown in FIG. 1 receives the spherical images sent from the spherical image pickup device 110 or the image processing devices 120 to 124, and accumulates and manages the received spherical images. The image display server 130 also generates an output image based on the spherical image in response to a display request for the spherical image from the image processing devices 120 to 124 and other information processing terminals, and the generated output image. To the requesting device. As a result, the output image can be displayed on the flat display device of the requesting device.

In a specific embodiment, the image display server 130 can be configured as a web server, receives a request for an image registration request including an all-sky image according to HTTP (HyperText Transfer Protocol), and displays the all-sky image. Can be accumulated. The image display server 130 further receives a request for an image display request for which the spherical image to be output is specified, reads the spherical image to be output, performs image processing, and generates an output image. You can respond with a response that includes an output image. In the requesting device that received the response, the received output image is displayed on the flat display device by the web browser. Then, the web browser appropriately prints and outputs the output image. The image display server 130 is also configured as an image processing device that generates an output image in the present embodiment.

Hereinafter, the spherical image output processing for generating an output image from the spherical image in the present embodiment will be described in more detail with reference to FIGS. 2 to 12. FIG. 2 is a diagram showing a functional block 200 related to spherical image output processing in the spherical image display system according to the present embodiment. The functional block 200 shown in FIG. 2 includes a functional block 210 on the spherical image pickup device 110 and a functional block 250 on the image processing devices 120 to 124, 130.

The functional block 210 of the celestial sphere image pickup device 110 receives inputs of two image pickup optical systems 212A and 212B for photographing each direction and each image pickup image taken by each image pickup optical system 212, and receives an all celestial sphere image. Is included with the synthesis processing block 214 to generate and output.

The functional block 250 of the image processing device includes an input unit 252, an output unit 254, a spherical image storage unit 256, a user input reception unit 258, a plane image generation unit 260, and an image output unit 262. Will be done. The input unit 252 is an input device such as a touch panel, a mouse, and a keyboard. The output unit 254 is an output device such as a plane display device that displays an image processing result according to an input of a user operation performed on the input unit 252, and an image forming device that prints out the image processing result. The input unit 252 and the output unit 254 may be provided in the image processing device itself, or may be provided in an external device to which the image processing device is connected.

The spherical image storage unit 256 is a means for accumulating spherical images captured by the spherical image pickup device 110 and input to the image processing devices 120 to 124 via the connection or an external recording medium. The user input reception unit 258 receives an input value that defines the output range of the spherical image according to the operation based on the output range change operation performed via the input unit 252, and the input value is the plane image generation unit 260. Pass to.

Examples of the output range changing operation include a pan operation for moving the field of view left and right, a tilt operation for moving the field of view up and down, and a zoom operation for expanding or contracting the range of the output image. As a result of the change by the output range change operation or as a result of being directly input, the pan specified value, the tilt specified value and the zoom specified value are acquired as input values defining the output range of the spherical image.

The plane image generation unit 260 determines a parameter for image generation (hereinafter referred to as an image generation parameter) based on the received input value, and based on the determined image generation parameter, an all-sky image. Generate an output image from. The image output unit 262 outputs the generated output image to the output unit 254. The output image is a flat image for proper display on the flat display device.

When the image processing device operates as a web server such as an image display server 130, the configuration of the input unit 252 and the output unit 254 is as follows. That is, the input unit 252 is configured as an HTTP receiving unit that receives an HTTP request for image registration. The output unit 254 is configured as an HTTP transmission unit that returns the generated output image as a response to the request source in response to the HTTP request for displaying the image.

FIG. 3 is a diagram illustrating a data structure of each image and a data flow of the image in the spherical image output processing. The image pickup optical system 212 according to the present embodiment generates two captured images by an image pickup process. In the present embodiment, the light incident on the lens optical system is imaged in the light receiving region of the corresponding solid-state image pickup device according to a predetermined projection method. The captured image is captured by a two-dimensional solid-state image sensor whose light receiving region forms a plane area, and is image data expressed in a plane coordinate system. Further, in the present embodiment, a configuration of a so-called circumferential fisheye lens having an image circle diameter smaller than that of the diagonal line of the image is adopted. Therefore, the obtained captured image is configured as a plane image including the entire image circle on which each shooting range is projected, as shown by "captured image A" and "captured image B" in FIG.

FIG. 4 is a diagram illustrating a projection method that can be adopted in a fisheye lens. Various designs can be considered for the fisheye lens, and the projection methods include the orthographic projection method (FIG. 4 (A)), the equidistant projection method (FIG. 4 (B)), and the stereographic projection as shown in FIG. Methods (FIG. 4 (C)) and equidistant angle projection (FIG. 4 (D)) can be mentioned. Further, in the embodiment to be described, the captured image taken by one fisheye lens is taken in the direction of a hemisphere (the part where the total angle of view exceeds 180 degrees is out of the hemisphere) from the shooting point. Become. Then, as shown in FIG. 4, the image is generated at the image height r corresponding to the incident angle β with respect to the optical axis. The pixel position (image height: distance in the radiation direction from the lens focal length) r on the light receiving region that receives light from the incident angle β is determined by using the following projection function according to a predetermined projection model, with the focal length as f. Can be decided.

According to the above equation corresponding to the projection method adopted by the fisheye lens, the orientation (angle of incidence and the angle of rotation around the optical axis) is associated with the coordinates of the pixel position on the plane image. In a preferred embodiment, the fisheye lens can use the stereographic projection method shown in FIG. 4 (C).

The compositing processing block 214 uses information from a 3-axis accelerometer (not shown) to perform top-bottom correction as well as distortion correction on the two captured images obtained from the two imaging optical systems 212A and 212B, and synthesizes the images. In the image composition process, first, each spherical image including each complementary half-rest portion is generated from each captured image configured as a plane image. Then, the two spherical images including each hemisphere portion are aligned based on the matching of the overlapping regions, and the images are combined to generate the spherical image including the entire spherical surface.

FIG. 5 is a diagram illustrating a data structure of image data in the spherical image format used in the present embodiment. As shown in FIG. 5A, the image data in the all-sky image format coordinates the vertical angle φ made with respect to a predetermined axis and the horizontal angle θ corresponding to the rotation angle around the predetermined axis. It is expressed as an array of pixel values. Here, as shown in FIG. 3, the whole celestial sphere image has a vertical angle (latitude in latitude / longitude coordinates) about the zenith direction of the captured scene and a horizontal angle (latitude / longitude coordinates) around the axis in the zenith direction. It shall be expressed in a coordinate system composed of (longitude in). The vertical angle φ is in the range of −90 degrees to +90 degrees, and the horizontal angle θ is in the range of −180 degrees to +180 degrees.

As shown in FIG. 5 (B), each coordinate value (θ, φ) in the spherical format is associated with each point on a spherical surface representing all directions centered on the shooting point, and all directions are It is associated on the spherical image. The plane coordinates of the captured image taken by the fisheye lens and the coordinates on the spherical surface of the spherical image format are associated with each other by performing the above-mentioned projection function and appropriate coordinate conversion.

In the following description, the two-dimensional coordinates shown in FIG. 5A of the spherical image format are the coordinate system of the lower left origin as shown by the gray axis in FIG. 5A. Then, in the following description, the two-dimensional coordinates are converted into a horizontal angle value x in the range corresponding to the number of pixels in the horizontal direction from 0 and a vertical angle value y in the range corresponding to the number of pixels in the 0 to the vertical direction. It shall be handled. For example, when the pixels are configured in 1/10 degree increments, the horizontal angle value x is in the range of 0 to 3799, and the vertical angle value y is in the range of 0 to 1800. The relationship between the horizontal angle value x and the horizontal angle θ and the relationship between the vertical angle value y and the vertical angle φ are represented by the following (1) and (2). In the following equations (1) and (2), w and h correspond to the image width (for example, 3600 pixels) and the image height (for example, 1801 pixels) of the spherical image format, respectively.

Then, between the two-dimensional coordinates (x, y) of the spherical image shown in FIG. 5 (A) and the three-dimensional coordinates (xs, ys, zs) of the surface of the spherical surface shown in FIG. 5 (B). The relationship between is calculated by the following equations (3) and (4). The three-dimensional coordinates shown in FIG. 5B are a right-handed system, with the center of the sphere as the origin and r representing the radius.

The relationship between the two images (captured image A and captured image B) taken by the two imaging optical systems 212A and 212B and the image region on the whole celestial sphere image is shown in FIG. The correspondence is shown in "Tenkyu image". The image-combined spherical image is stored in a built-in memory or an external recording medium in a predetermined format. The file format in which the whole celestial sphere image is saved may be an uncompressed still image such as a bitmap, JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), PNG (Portable Network Graphics). It may be a still image in a compressed format such as. In still another embodiment, the spherical image may be included as a frame image in a moving image such as MPEG (Moving Picture Experts Group) or AVI (Audio Video Interleave). In the embodiment described below, the spherical image will be described as a still image.

In the functional block 250 of the image processing device, the all-sky image is saved by the all-sky image storage unit 256, and the plane image generation unit 260 continues to input the all-heaven sphere image and output the image from the all-heaven sphere image. Image processing is performed to convert to. In the embodiment described, the input spherical image is described as being captured by the spherical imager 110 in a preferred embodiment, but the origin of the spherical image is not necessarily limited. It's not something. For example, the captured image may be subjected to predetermined image processing, or an image generated by computer graphics may be included.

The plane image generation unit 260 receives from the user input reception unit 258 an input value including the determined pan designation value, tilt designation value, and zoom designation value as a result of the change due to the output range change operation or the like. The plane image generation unit 260 determines an image generation parameter as described later based on the received input value, and executes an image generation process of the output image based on the determined image generation parameter.

As described above, the spherical image can be associated with the three-dimensional coordinates by the above equations (3) and (4). In the above image generation process, a three-dimensional model in which the input spherical image is pasted on the inner surface of a predetermined three-dimensional shape is constructed, and under predetermined conditions, a virtual camera (hereinafter, simply referred to as a camera) is used. The output image S is obtained by displaying the projection from.).

FIG. 6 is a diagram illustrating a perspective projection performed in a three-dimensional graphics display. As shown in FIG. 6, the plane image generation unit 260 generates an output image of a plane image by performing a perspective projection process on a three-dimensional model in which a whole celestial sphere image is attached to the inner surface of a sphere. The image generation parameters for perspective projection are determined according to the above input values. In certain embodiments, the image generation parameters include the position (d), direction (v), viewing angle (Θ), and projection range (zNear, zFar) of the viewpoint of the camera.

The output image S is an image observed within a specific viewing angle (Θ) when the three-dimensional model is viewed from the center of the sphere in a specific latitude / longitude direction (v), depending on the shape of the display area. The image will be cut out. The parameters of the projection range (zNear, zFar) specify the range to be perspectively projected, but an appropriate value shall be determined. The details of the image generation parameter determination method and the projection processing will be described later with reference to FIGS. 7 to 12.

In the embodiment shown in FIG. 2, the image pickup optical system 212 and the synthesis processing block 214 are components on the spherical image pickup device. Then, the input unit 252, the output unit 254, the spherical image storage unit 256, the user input reception unit 258, the plane image generation unit 260, and the image output unit 262 are distributed and mounted as components on the image processing device. However, the mounting mode is not particularly limited.

In other embodiments, all components may be placed on a single device to configure a spherical image display system. In yet another embodiment, any part of the component is placed on any of the plurality of devices constituting the spherical image display system, respectively, and the spherical image is formed as a combination of the plurality of devices. A display system may be configured. For example, in a specific embodiment, the image processing device may include a synthesis processing block, receive two captured images from the spherical image pickup device, and prepare the spherical image.

Hereinafter, the flow of the spherical image output processing in the present embodiment will be described in more detail with reference to FIGS. 7 and 8. Hereinafter, as the output processing, the display processing in the image viewer for displaying the spherical image will be described. FIG. 7 is a flowchart showing the spherical image display processing executed by the image processing apparatus of the present embodiment. FIG. 8 is a diagram illustrating an image viewer screen that displays a spherical image in a predetermined range.

The process shown in FIG. 7 is started from step S100 in response to, for example, an operator of the image processing device 250 giving a display instruction specifying a predetermined spherical image. In step S101, the image processing apparatus 250 determines the initial image processing parameters by the plane image generation unit 260 based on the preset default pan designation value, tilt designation value, and zoom designation value. In step S102, the image processing device 250 generates a plane image from the spherical image based on the image processing parameters determined by the plane image generation unit 260.

In step S103, the image processing device 250 causes the image output unit 262 to display the generated planar image at a predetermined position on the image viewer screen. The image viewer screen 300 shown in FIG. 8 includes an image display area 310 and GUI (Graphical User Interface) components 322 and 324 for changing the display range of the image to be displayed in the image display area 310. In the image display area 310, the output image in the range corresponding to the input value generated by the plane image generation unit 260 is displayed by the image output unit 262.

In step S104, the image processing device 250 determines whether or not the display range change operation has been accepted by the user input reception unit 258. Here, the display range change operation is detected by the occurrence of an operation event such as a click or a flick performed on the GUI parts 322 and 324 corresponding to each operation. The image viewer screen 300 illustrated in FIG. 8 includes a GUI component 322I that awaits a zoom-in instruction and a GUI component 322O that awaits a zoom-out instruction to change the zoom specified value. The image viewer screen 300 further waits for the left button 324L and the right button 324R, which await a pan instruction in the left and right directions, and an up and down tilt instruction, as a means of changing the pan specified value and the tilt specified value. Includes up button 324U and down button 324D.

The display range change operation can be detected by the occurrence of an operation event such as a shortcut key, a gesture, or a multi-touch operation associated with each display range change operation, in addition to the operation for the GUI component. For example, shortcut keys may include "+" and "-" buttons that direct zoom-in and zoom-out on the keyboard. Further, key operations for the left and right arrow buttons and the up and down arrow buttons for instructing left and right pan and up and down tilt may be used as shortcut keys. Multi-touch operations include pinch-in and pinch-out associated with zoom operations.

In step S104, the display range change operation is awaited by looping in step S104 until the display range change operation is accepted (between NO in step S104). If it is determined in step S104 that the display change operation has been accepted (YES), the process is branched to step S105.

In step S105, the changed image processing parameters are determined based on the pan specified value, the tilt specified value, and the zoom specified value determined as a result of the display change operation, and the processing is branched to step S102. In this case, in the following step S102, the plane image generation unit 260 performs the plane image generation processing based on the changed image processing parameters. In step S103, the image output unit 262 updates the image display area 310 of the image viewer screen 300 with a plane image newly generated in response to a user operation.

Hereinafter, the spherical image output processing according to the present embodiment will be described in more detail with reference to FIGS. 9 to 12. FIG. 9 is a diagram showing a detailed functional block of the plane image generation unit 260 on the image processing apparatus according to the present embodiment. The plane image generation unit 260 shown in FIG. 9 includes a parameter determination unit 264, a texture mapping unit 266, and a projection unit 268.

The parameter determination unit 264 determines the position (d) and viewing angle (Θ) of the viewpoint of the camera according to various input values (pan specified value, tilt specified value and zoom specified value) passed from the user input receiving unit 258. It is a determination means for determining the image generation parameters to be included.

The texture mapping unit 266 is a generation means for generating a three-dimensional model in which a spherical image, which is a display target image, is attached to a predetermined three-dimensional shape. The generation of the three-dimensional model can be performed by the so-called texture mapping method. Texture mapping is a general graphics process even in OpenGL, which is supported by GPUs installed in information processing terminals with limited computing power such as smartphones and tablets, and is a process of pasting a texture image on the surface of a three-dimensional shape. say. The texture mapping unit 266 reads the designated spherical image, transfers it to the texture buffer that stores the texture, and allocates it to the three-dimensional model.

In the embodiments described, the three-dimensional shape can be a sphere, a cylinder, or any other three-dimensional shape capable of projecting an output image that does not give the viewer a sense of discomfort. From the viewpoint of not making the viewer feel uncomfortable and simplifying the arithmetic processing, a sphere can be preferably used. The three-dimensional shape has at least one inner surface to which the spherical image is attached, and when a sphere is adopted, includes a spherical surface to which the spherical image is attached.

The projection unit 268 uses a predetermined projection method to project a three-dimensional model to which a spherical image is pasted from a camera whose viewpoint is set at a predetermined position according to an image generation parameter determined by the parameter determination unit 264. It is a projection means that projects and produces an output image. An output image can be generated by rendering an image when a three-dimensional model with texture mapping is observed from an arbitrary camera viewpoint under predetermined conditions.

According to Non-Patent Document 1, it is known that it is effective to make the projection method different between the wide-field display and the narrow-field display. However, when a plurality of projection methods are continuously applied, the performance requirements for the image processing device become strict.

Therefore, in the present embodiment, in order to relax the hardware requirements of the image processing device, the plane image generation unit 260 changes the image generation parameters of the display model by a single projection method, so that the viewer can view the images. Adopt a configuration that obtains a suitable display effect. Hereinafter, the process of determining the image generation parameter according to the input value will be described with reference to FIGS. 10 to 12. In the embodiment described, a perspective projection method (perspective projection) is used as the projection method, but in other embodiments, another projection method such as an orthographic projection method may be adopted.

As described above, the image generation parameters are the position (d), direction (v), viewing angle (Θ) and projection range (zNear, zFar) of the viewpoint of the camera in a specific embodiment in which perspective projection is adopted as the projection method. including. In 3D computer graphics, three types of coordinate systems are typically defined: a world coordinate system, a model coordinate system, and a camera coordinate system. The world coordinate system defines an absolute three-dimensional space, and cameras and objects are placed in the three-dimensional space defined by the world coordinate system. The model coordinate system is a coordinate system centered on a predetermined object. In the embodiment described, a sphere model is constructed and the sphere model is placed at the origin of the world coordinate system. Therefore, the origin of the world coordinate system and the model coordinate system of the sphere model may be the same, and only the axes may be different. The camera coordinate system represents the direction (v) of the field of view centered on the viewpoint of the camera.

Then, the projection unit 268 projects a spherical model on which the spherical image is pasted on the inner surface onto a two-dimensional screen from the viewpoint of the camera, and uses the obtained projected image as a display image. The screen is arranged in a plane passing through the origin of the camera coordinates, and the spherical image is projected on the screen by perspective projection.

FIG. 10 is a diagram illustrating the relationship between the model coordinate system, the camera position (d) and the viewing angle (Θ), and the angle of view (Φ) representing the range of the image captured in the field of view. When the viewpoint of the camera is located at the center of the spherical model, the angle of view (Φ) representing the range of the image captured in the field of view coincides with the viewing angle (Θ) of the camera. However, as shown by the double circle in FIG. 10, when the position (d) of the viewpoint of the camera is separated from the center of the three-dimensional model, the angle of view (Φ) and the viewing angle (Θ) of the camera Will take different values. Zoom-in and zoom-out correspond to the operation of changing the angle of view (Φ). In the embodiment to be described, the angle of view (Φ) is changed by changing either the position (d) of the viewpoint of the camera or the viewing angle (Θ) according to the range of the specified zoom value.

The changes in the image generation parameters according to the operation of the pan, tilt, and zoom display ranges in this embodiment are summarized in Table 1 below.

In Table 1, the movement of the display position of the image by the tilt and pan is realized by fixing the direction (v) of the field of view and rotating the sphere model in the world coordinate system. However, in another embodiment, the sphere model may be fixed with respect to the world coordinate system and the direction (v) of the field of view of the camera may be changed to realize the movement of the image display position.

Hereinafter, the process of determining the image generation parameter according to the specified zoom value in the present embodiment will be described in more detail with reference to FIGS. 11 and 12. 11 and 12 are diagrams illustrating how the image generation parameters are determined according to the zoom specified value, and also show the output image and the displayed range of the sphere model at that time. 11 and 12 show a method of determining an image generation parameter when a specific zoom designation value (z) indicated by A to E is given.

Further, the image generation parameters determined according to the zoom specified value (z), the display magnification and the angle of view (Φ) at that time are collectively shown in Table 2 below. In Table 2 below, viewWH represents the width or height of the display area of the output image. When the display area is horizontally long, it is the width of the display area, and when the display area is vertically long, it is the height of the display area. In Table 2 below, imgWH represents the width or height of the image, and when the display area is horizontally long, it is the image width, and when the display area is vertically long, it is the image height. The imgDeg represents the angle of the display range of the image, which is 360 degrees in the case of the image width and 180 degrees in the case of the image height.

In the present embodiment, as shown in Table 2, the range of the specified zoom value (z) is four, including the range of A to B, the range of B to C, the range of C to D, and the range of D to E. It is divided into ranges. The zoom specified value (z) is a value corresponding to the angle of view to be displayed, and is designated by the user in the range from the minimum angle of view of 60 degrees to the maximum angle of view of 300 degrees in the embodiment described.

Considering zoom-out, when the specified zoom value (z) is in the range of A to B, the viewing angle (Θ) of the camera is fixed at 60 degrees, and the camera position (d) moves away from the center. Then, as shown by A in FIG. 11 and B in FIG. 11, the angle of view (Φ) is widened. The camera position (d) in the range of A to B is shown by the formula in Table 2 from the origin 0 shown by A in FIG. 11 to 1 corresponding to the outer edge of the spherical model shown by B in FIG. The value is determined according to the angle of view (Φ) and the specified zoom value (z).

When the specified zoom value (z) is in the range of B to C on the wide field of view side of A to B, the camera position (d) is the spherical model as shown by B in FIG. 11 and C in FIG. It is fixed to the outer edge (d = 1), and the angle of view (Φ) is widened by widening the viewing angle (Θ) of the camera. The viewing angle (Θ) of the camera increases from 60 degrees to 120 degrees, which is fixed in the range of A to B, according to the formula Θ = Φ / 2. The angle of view (Φ) representing the range of the image captured in the field of view matches the zoom specified value (z) in the range of A to B and the range of B to C, and increases monotonically.

When the specified zoom value (z) is in the range of C to D and the range of D to E on the wider field of view side than the range of B to C, C in FIG. 11, D in FIG. 12 and E in FIG. Will be shown. That is, the angle of view (Φ) is widened by moving the position (d) of the viewpoint of the camera away from the center again while the viewing angle (Θ) of the camera is fixed at 120 degrees. The position (d) of the camera viewpoint is determined according to the zoom specified value (z) according to the calculation formula shown in Table 2. Further, in the range of C to D and the range of D to E, the angle of view (Φ) does not match the zoom specified value (z).

The dmax1 in Table 2 corresponding to the position indicated by D in FIG. 12 is the distance at which the sphere model is displayed at the maximum angle of view in the entire rectangle of the display area as shown together with D in FIG. Correspondingly, in a specific embodiment, it can be calculated by the following equation (5). Dmax2 in Table 2 corresponding to the position (D) indicated by E in FIG. 12 is the maximum angle of view in which the sphere model is inscribed in the rectangle of the display area as shown together with E in FIG. Corresponding to the distance at which the sphere model is displayed, in a particular embodiment, it is calculated by the following equation (6).

In the above equations (5) and (6), viewW and viewH represent the width and height of the display area. Therefore, dmax1 and dmax2 are values that depend on the size (width and height, diagonal length) of the display screen. dmax2 corresponds to the farthest position that the viewpoint of the camera can take, and the zoom specified value (z) is limited according to the size of the display area. By limiting the zoom specified value (z) so that the position of the viewpoint of the camera falls within the range (to dmax2) shown in Table 2, the spherical image fits on the display screen or the spherical image is spherical at a predetermined magnification. You can end the zoom out while it is displayed on the shape. As a result, the viewer can visually recognize that the displayed image is a spherical image, and the zoom-out can be completed without any discomfort.

Further, as is clear from Table 2 and FIG. 11, the angle of view (Φ) is continuous between the above-mentioned ranges, but the angle of view (Φ) is one due to the zoom-out to the wide-angle side. It has not increased so much. That is, in the range of C to D in the range of C to D to E where the viewpoint position of the camera is changed, the angle of view ( Φ) increases, but in the range of D to E, the angle of view (Φ) decreases as the distance increases. This is due to the fact that the outer region of the sphere model is reflected in the field of view. Then, by moving the viewpoint position (d) of the camera in a wide field of view with a zoom specified value of 240 degrees or more, it is possible to change the angle of view (Φ) while displaying with less discomfort.

Therefore, considering the change of the zoom specified value in the wide-angle direction, the angle of view (Φ) is basically widened. At this time, the increase in the viewing angle (Θ) of the camera is suppressed, and the camera moves away from the model coordinate system to express a feeling of openness during wide-angle display, so that distortion of the image is reduced. Further, since the movement of the camera moving away is similar to the behavior when a human actually tries to confirm a wide range, it is considered that the zoom-out is less uncomfortable. The angle of view (Φ) decreases in the range of D to E as the specified zoom value goes toward the wide field of view, but this makes the viewer feel as if he / she is moving away from the sphere, which makes him feel uncomfortable. Zoom out with less.

In the above description, it is assumed that only one of the camera viewpoint position (d) and the viewing angle (Θ) is changed and the other is fixed in each range. However, in other embodiments, it is not hindered to preferentially change either the position (d) or the viewing angle (Θ) of the camera viewpoint and change the other with a relatively small amount of change. Further, in the above description, the method of determining the image generation parameter has been described in the context of being changed in the zoom-out direction, but the image generation parameter can be similarly determined when zooming in.

Hereinafter, the hardware configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. 13. FIG. 13 is a diagram showing a hardware configuration of an image processing device according to the present embodiment. The image processing device according to the present embodiment is configured as a mobile information terminal such as a tablet terminal 122. The tablet terminal 122 shown in FIG. 13 includes a mobile processor 10 in which a single-core or multi-core CPU (Central Processing Unit), a GPU, a baseband processor, a memory controller, and the like are integrated as a SoC (System on Chip), and the mobile processor 10. It includes a memory 12 such as an LPDDR (Low-Power Double DataRate) SDRAM that is connected and provides a work area such as a CPU, a flash memory 14, and an external recording medium slot 16 such as an SD card.

The flash memory 14 stores user data including an OS for controlling the tablet terminal 122, a control program for realizing the above-mentioned functional unit, various system information and various setting information, and the above-mentioned spherical image. A recording medium in which user data such as a spherical image is stored is installed in the external recording medium slot 16.

The mobile processor 10 is further connected to a display 22 equipped with a touch screen sensor via a touch screen controller 18 and a display interface 20. The display 22 displays various setting screens and application screens, and in the present embodiment, the screen of the image viewer including the output image generated from the spherical image can be displayed. The tablet terminal 122 further includes a video output interface 24 such as HDMI (registered trademark, High-Definition Multimedia Interface) connected to the mobile processor 10, and can be connected to an external display or projector.

The tablet terminal 122 further includes a camera 28 including an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor). The camera 28 is connected to the mobile processor 10 via the camera interface 26. The tablet terminal 122 further includes an audio codec 30 that encodes and decodes audio, and an audio switch 32 that switches audio from headphones and speakers.

Further, a wireless LAN port 34 and a short-range wireless port 36 such as Bluetooth (registered trademark) are connected to the mobile processor 10, and it is possible to connect to an external device by wireless communication. In the embodiment described, the tablet terminal 122 is connected to an external spherical image pickup device 110 via a wireless LAN port 34 or a short-range wireless port 36. The tablet terminal 122 includes a power management unit 38, and the power management unit 38 manages the external power supply of the tablet terminal 122 and the power supply of the battery.

The tablet terminal 122 according to the present embodiment reads a control program from the flash memory 14 and expands it in the work space provided by the memory 12, so that each functional unit described above is controlled by a CPU integrated in the mobile processor 10. And realize each process. At that time, the calculation function of the GPU integrated in the mobile processor 10 is called via the API for graphic processing such as OpenGL, and the image calculation such as the texture mapping processing and the projection processing described above is executed.

According to the embodiment described above, when displaying an image, it is possible to perform high-speed display while reducing a sense of discomfort due to stretching of the subject at the top, bottom, left, and right edges in the display in a wide field of view. It is possible to provide an image processing system, an image processing method and a program with relaxed requirements.

According to the above-described embodiment, since the display model is constructed as a single projection method, it is possible to perform real-time and smooth zoom display even with an image processing device having a limited image calculation function. Become. Since the zoom-out is expressed by moving the viewpoint position of the camera away from the three-dimensional model, it is possible to suppress the increase in the viewing angle and suppress the distortion of the image while giving a feeling of spaciousness.

The above functional unit can be realized by a computer-executable program described in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark), an object-oriented programming language, or the like, and is ROM, EEPROM, EPROM. , Flash memory, flexible disc, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, etc. Stored in a device-readable recording medium, or through a telecommunications line Can be distributed. In addition, some or all of the functional parts can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA), or as an ASIC (application specific integrated circuit). Circuit configuration data (bit stream data) to be downloaded to PD to realize the above functional unit on PD, HDL (Hardware Description Language) to generate circuit configuration data, VHDL (VHSIC (Very High Speed)) Integrated Circuits) Hardware Description Language), can be distributed by recording medium as data described by Verilog-HDL and the like.

Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art may think of other embodiments, additions, changes, deletions, and the like. It can be changed within the range possible, and is included in the scope of the present invention as long as the action / effect of the present invention is exhibited in any of the embodiments.

100 ... All-sky image display system, 102 ... Internet, 104 ... Communication equipment, 110 ... All-sky imager, 120-124 ... Image processing device, 130 ... Image display server, 200 ... Functional block, 210 ... Functional block, 212 ... Imaging optical system, 214 ... Composite processing block, 250 ... Functional block, 252 ... Input unit, 254 ... Output unit, 256 ... All-sky image storage unit, 258 ... User input reception unit, 260 ... Planar image generation unit, 262 ... Image output unit, 264 ... Parameter determination unit, 264 ... Texture mapping unit, 266 ... Projection unit, 300 ... Image viewer screen, 310 ... Image display area, 312, 314 ... GUI parts, 10 ... Mobile processor, 12 ... Memory , 14 ... Flash memory, 16 ... External recording medium slot, 18 ... Touch screen controller, 20 ... Display interface, 22 ... Display, 24 ... External video interface, 26 ... Camera interface, 28 ... Camera, 30 ... Voice codec , 32 ... voice switch, 34 ... wireless LAN port, 36 ... short-range wireless port, 38 ... power management unit

J. Kopf, et. Al, "Capturing and Viewing Gigapixel Images", ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2007, Volume 26, Issue 3, July 2007, Article No. 93, ACM New York, NY, USA

Claims (9)

A reception means that accepts input values that specify the output range,
A generation means for generating a 3D model in which a target image is pasted into a 3D shape,
A determination means for determining the position and viewing angle of the viewpoint based on the input value, and
The determination means includes a projection means for projecting the three-dimensional model from the viewpoint, and the determination means preferentially changes the viewing angle of the target image when the input value is in the first range. The range within the visual field is changed, and when the input value is in the second range on the wider visual field side than the first range, the position of the viewpoint is preferentially changed into the visual field. An image processing system that changes the range of entry. The second range includes a range in which the range within the visual field increases as the position of the viewpoint moves away from the center of the three-dimensional shape, and a range in which the range within the visual field decreases as the position moves away. The image processing system according to claim 1, which includes. The image processing system according to claim 2, wherein the position of the viewpoint is limited according to the size of the output area. The image processing according to any one of claims 1 to 3, wherein the determination means determines the viewpoint at a position away from the center of the three-dimensional shape when the input value is in the first range. system. When the input value is in a third range on the narrower visual field side than the first range, the determining means changes the range within the visual field by preferentially changing the position of the viewpoint. The image processing system according to any one of claims 1 to 4. The input value is a zoom specified value that specifies the degree of zoom, and the receiving means further includes a tilt specified value that swings the output range up and down, a pan specified value that swings the output range left and right, or at least one of these. Accept one,
Any of claims 1-5, further comprising means for rotating the three-dimensional model or changing the viewing direction with respect to the world coordinate system according to the tilt and pan designations or at least one of them. The image processing system according to item 1. The three-dimensional shape is a solid body having at least one inner surface to which the target image is attached, the target image is an image represented by a coordinate system including at least one angular coordinate, and the projection means is a projection means. The image processing system according to any one of claims 1 to 6, wherein the image is projected by a single projection method. An image processing method that produces an output image, which is a computer
A step that accepts an input value that defines the output range,
Steps to determine the position and viewing angle of the viewpoint based on the input values,
Steps to generate a 3D model with the target image pasted into a 3D shape,
In the step of projecting the three-dimensional model from the viewpoint and the step of determining the determination, the computer performs the step of projecting the three-dimensional model.
When the input value is in the first range, a sub-step that changes the range within the field of view of the target image by preferentially changing the viewing angle, or
When the input value is in the second range on the wide field of view side of the first range, the sub-step of changing the range within the field of view by preferentially changing the position of the viewpoint is executed. Image processing method. Computer,
Reception means that accepts input values that specify the output range,
A generation means for generating a 3D model in which a target image is pasted into a 3D shape,
It is a program for functioning as a determining means for determining a viewpoint position and a viewing angle based on the input value, and a projection means for projecting the three-dimensional model from the viewpoint.
When the input value is in the first range, the determination means preferentially changes the viewing angle to change the range within the field of view of the target image, and the input value is the first range. A program that changes the range within the visual field by preferentially changing the position of the viewpoint when it is in the second range on the wide visual field side of the range 1.

Images